Generate a code coverage report (#1353)

* Generate a code coverage report

* make linters happy

* adding action and badge

* no report for me

* Upload the test report

* adding folders

* spaces not tabs

* makefiles love tabs

* spacing is important

* rawr

* Fix makefile

* Add diagnostic info

* Set UID and GID

* Make info

* Add docker images in diagnostic info

* Move info before testing the local action

* Add missing -C to git diff-tree

* Reset ownership

* Add missing quotes

* Sudo

* Attempt to fix ARM test

* Attempt to fix go test

Co-authored-by: Admiral Awkbar <admiralawkbar@github.com>

.automation/test/arm/reports/expected-ARM.tap

@@ -2,6 +2,6 @@ TAP version 13
 1..2
 not ok 1 - arm_bad_1.json
   ---
-  message: \nValidating arm\arm_bad_1.json\n deploymentTemplate\n [+] adminUsername Should Not Be A Literal\n [+] apiVersions Should Be Recent\n [+] artifacts parameter\n [+] CommandToExecute Must Use ProtectedSettings For Secrets\n [+] DependsOn Best Practices\n [+] Deployment Resources Must Not Be Debug\n [+] DeploymentTemplate Must Not Contain Hardcoded Uri\n [+] DeploymentTemplate Schema Is Correct\n [+] Dynamic Variable References Should Not Use Concat\n [+] IDs Should Be Derived From ResourceIDs\n [+] Location Should Not Be Hardcoded\n [+] ManagedIdentityExtension must not be used\n [+] Min And Max Value Are Numbers\n [+] Outputs Must Not Contain Secrets\n [+] Parameters Must Be Referenced\n [+] Parameters Property Must Exist\n [+] providers apiVersions Is Not Permitted\n [+] ResourceIds should not contain\n [+] Resources Should Have Location\n [+] Resources Should Not Be Ambiguous\n [+] Secure String Parameters Cannot Have Default\n [-] Template Should Not Contain Blanks\n Empty property [] found on line 28 Index 1331\n Empty property "" found on line 3 Index 121\n\n [-] Variables Must Be Referenced\n Unreferenced variable Network.Location\n Unreferenced variable Test.Blank.Variable\n\n [+] Virtual Machines Should Not Be Preview\n [+] VM Images Should Use Latest Version\n [+] VM Size Should Be A Parameter\n \n
+  message: \nValidating arm\arm_bad_1.json\n deploymentTemplate\n [+] adminUsername Should Not Be A Literal\n [+] apiVersions Should Be Recent\n [+] artifacts parameter\n [+] CommandToExecute Must Use ProtectedSettings For Secrets\n [+] DependsOn Best Practices\n [+] Deployment Resources Must Not Be Debug\n [+] DeploymentTemplate Must Not Contain Hardcoded Uri\n [+] DeploymentTemplate Schema Is Correct\n [+] Dynamic Variable References Should Not Use Concat\n [+] IDs Should Be Derived From ResourceIDs\n [+] Location Should Not Be Hardcoded\n [+] ManagedIdentityExtension must not be used\n [+] Min And Max Value Are Numbers\n [+] Outputs Must Not Contain Secrets\n [+] Parameters Must Be Referenced\n [+] providers apiVersions Is Not Permitted\n [+] ResourceIds should not contain\n [+] Resources Should Have Location\n [+] Resources Should Not Be Ambiguous\n [+] Secure String Parameters Cannot Have Default\n [-] Template Should Not Contain Blanks\n Empty property [] found on line 28 Index 1331\n Empty property "" found on line 3 Index 121\n\n [-] Variables Must Be Referenced\n Unreferenced variable Network.Location\n Unreferenced variable Test.Blank.Variable\n\n [+] Virtual Machines Should Not Be Preview\n [+] VM Images Should Use Latest Version\n [+] VM Size Should Be A Parameter\n \n
   ...
 ok 2 - arm_good_1.json

.automation/test/go/reports/expected-GO-4.tap

@@ -2,6 +2,6 @@ TAP version 13
 1..2
 not ok 1 - golang_bad_01.go
   ---
-  message: level=warningg="[runner] The linter 'maligned' is deprecated due to The repository of the linter has been archived by the owner. Use govet 'fieldalignment' instead."\nlevel=warningg="[runner] Can't run linter goanalysis_metalinter failed prerequisites [buildir@command-line-arguments analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']]"\nlevel=warningg="[runner] Can't run linter unused buildir analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']"\nlevel=errorg="Running error buildir analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']"\n
+  message: level=warningg="[runner] The linter 'maligned' is deprecated (since v1.38.0) due to The repository of the linter has been archived by the owner. Replaced by govet 'fieldalignment'."\ngo/golang_bad_01.go 1 1 expected 'package', found 'if' (typecheck)\nif len(in) == 0 {\n^\n
   ...
 ok 2 - golang_good_01.go

.github/workflows/deploy-DEV.yml

@@ -20,7 +20,7 @@ on:
   push:
     branches-ignore: [master]
   pull_request:
-    branches-ignore: []
+    branches-ignore: [ ]
 
 ###############
 # Set the Job #
@@ -32,7 +32,8 @@ jobs:
     # Set the agent to run on
     runs-on: ubuntu-latest
     # Prevent duplicate run from happening when a forked push is committed
-    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    if: github.event_name == 'push' ||
+        github.event.pull_request.head.repo.full_name != github.repository
     ##################
     # Load all steps #
     ##################
@@ -43,7 +44,8 @@ jobs:
       - name: Checkout Code
         uses: actions/checkout@v2.3.4
         with:
-          # Full git history is needed to get a proper list of changed files within `super-linter`
+          # Full git history is needed to get a proper list
+          # of changed files within `super-linter`
           fetch-depth: 0
 
       ########################
@@ -66,12 +68,6 @@ jobs:
           --build-arg BUILD_VERSION=${{ github.sha }} \
           -t ghcr.io/github/super-linter:${{ github.sha }} -f Dockerfile .
 
-      ##############
-      # Print info #
-      ##############
-      - name: Get Docker info
-        run: docker images
-
       ########################################
       # Validates the metadata docker labels #
       ########################################
@@ -86,6 +82,14 @@ jobs:
         run: |
           sed -i "s/super-linter:.*/super-linter:${GITHUB_SHA}'/g" action.yml
 
+      ######################
+      # Gather information #
+      ######################
+      - name: Gather information about the runtime environment
+        shell: bash
+        run: |
+          make info
+
       ##########################
       # Test in action context #
       ##########################
@@ -99,12 +103,47 @@ jobs:
           VALIDATE_ALL_CODEBASE: false
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
+      ###########################################################################################
+      # Fix file and dir ownership. Workaround for https://github.com/actions/runner/issues/434 #
+      ###########################################################################################
+      - name: Fix file and directory ownership
+        shell: bash
+        run: |
+          sudo chown -R "$(id -u)":"$(id -g)" "$(pwd)"
+
+      ##################
+      # Run test cases #
+      ##################
+      - name: Run the test suite
+        shell: bash
+        run: |
+          make test
+
+      ##########################
+      # Codacy Coverage Report #
+      ##########################
+      - name: Upload the code coverage report
+        uses: codacy/codacy-coverage-reporter-action@0.2.0
+        with:
+          project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
+          coverage-reports: test/reports/cobertura/runTests.sh/cobertura.xml
+
       #####################################
       # Run Linter against Test code base #
       #####################################
       - name: Run Test Cases
         shell: bash
-        run: docker run -e RUN_LOCAL=true -e TEST_CASE_RUN=true -e ANSIBLE_DIRECTORY=.automation/test/ansible -e OUTPUT_FORMAT=tap -e OUTPUT_FOLDER=${GITHUB_SHA} -e OUTPUT_DETAILS=detailed -e ACTIONS_RUNNER_DEBUG=true -e ERROR_ON_MISSING_EXEC_BIT=true -v ${GITHUB_WORKSPACE}:/tmp/lint ghcr.io/github/super-linter:${GITHUB_SHA}
+        run: |
+          docker run \
+          -e RUN_LOCAL=true \
+          -e TEST_CASE_RUN=true \
+          -e ANSIBLE_DIRECTORY=.automation/test/ansible \
+          -e OUTPUT_FORMAT=tap -e OUTPUT_FOLDER=${GITHUB_SHA} \
+          -e OUTPUT_DETAILS=detailed \
+          -e ACTIONS_RUNNER_DEBUG=true \
+          -e ERROR_ON_MISSING_EXEC_BIT=true \
+          -v ${GITHUB_WORKSPACE}:/tmp/lint \
+          ghcr.io/github/super-linter:${GITHUB_SHA}
 
       #########################################
       # Clean code base to run against it all #
@@ -118,4 +157,11 @@ jobs:
       ############################################
       - name: Run against all code base
         shell: bash
-        run: docker run -e RUN_LOCAL=true -e OUTPUT_DETAILS=detailed -e ACTIONS_RUNNER_DEBUG=true -e ERROR_ON_MISSING_EXEC_BIT=true -v ${GITHUB_WORKSPACE}:/tmp/lint ghcr.io/github/super-linter:${GITHUB_SHA}
+        run: |
+          docker run \
+          -e RUN_LOCAL=true \
+          -e OUTPUT_DETAILS=detailed \
+          -e ACTIONS_RUNNER_DEBUG=true \
+          -e ERROR_ON_MISSING_EXEC_BIT=true \
+          -v ${GITHUB_WORKSPACE}:/tmp/lint \
+          ghcr.io/github/super-linter:${GITHUB_SHA}

.gitignore

@@ -69,3 +69,9 @@ super-linter.report
 
 # Git directory (useful for .dockerignore)
 .git
+
+# Code coverage data for tests
+.coverage
+
+# Test reports
+test/reports

Makefile

@@ -0,0 +1,60 @@
+# Inspired by https://github.com/jessfraz/dotfiles
+
+.PHONY: all
+all: info test ## Run all targets.
+
+.PHONY: test
+test: info clean kcov prepare-test-reports ## Run tests
+
+# if this session isn't interactive, then we don't want to allocate a
+# TTY, which would fail, but if it is interactive, we do want to attach
+# so that the user can send e.g. ^C through.
+INTERACTIVE := $(shell [ -t 0 ] && echo 1 || echo 0)
+ifeq ($(INTERACTIVE), 1)
+	DOCKER_FLAGS += -t
+endif
+
+.PHONY: info
+info: ## Gather information about the runtime environment
+	echo "whoami: $$(whoami)"; \
+	echo "pwd: $$(pwd)"; \
+	echo "ls -ahl: $$(ls -ahl)"; \
+	docker images
+
+.PHONY: kcov
+kcov: ## Run kcov
+	docker run --rm $(DOCKER_FLAGS) \
+		--user "$$(id -u)":"$$(id -g)" \
+		-v "$(CURDIR)":/workspace \
+		-w="/workspace" \
+		kcov/kcov \
+		kcov \
+		--bash-parse-files-in-dir=/workspace \
+		--clean \
+		--exclude-pattern=.coverage,.git \
+		--include-pattern=.sh \
+		/workspace/test/.coverage \
+		/workspace/test/runTests.sh
+
+COBERTURA_REPORTS_DESTINATION_DIRECTORY := "$(CURDIR)/test/reports/cobertura"
+
+.PHONY: prepare-test-reports
+prepare-test-reports: ## Prepare the test reports for consumption
+	mkdir -p $(COBERTURA_REPORTS_DESTINATION_DIRECTORY); \
+	COBERTURA_REPORTS="$$(find "$$(pwd)" -name 'cobertura.xml')"; \
+	for COBERTURA_REPORT_FILE_PATH in $$COBERTURA_REPORTS ; do \
+		COBERTURA_REPORT_DIRECTORY_PATH="$$(dirname "$$COBERTURA_REPORT_FILE_PATH")"; \
+		COBERTURA_REPORT_DIRECTORY_NAME="$$(basename "$$COBERTURA_REPORT_DIRECTORY_PATH")"; \
+		COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX="$${COBERTURA_REPORT_DIRECTORY_NAME%.*}"; \
+		mkdir -p "$(COBERTURA_REPORTS_DESTINATION_DIRECTORY)"/"$$COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX"; \
+		cp "$$COBERTURA_REPORT_FILE_PATH" "$(COBERTURA_REPORTS_DESTINATION_DIRECTORY)"/"$$COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX"/cobertura.xml; \
+	done
+
+.PHONY: clean
+clean: ## Clean the workspace
+	rm -rf $(CURDIR)/test/.coverage; \
+	rm -rf $(CURDIR)/test/reports
+
+.PHONY: help
+help: ## Show help
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

README.md

@@ -3,6 +3,8 @@
 This repository is for the **GitHub Action** to run a **Super-Linter**.
 It is a simple combination of various linters, written in `bash`, to help validate your source code.
 
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/51071879604e4f319859d4daf91c68f5)](https://www.codacy.com/gh/github/super-linter/dashboard?utm_source=github.com&utm_medium=referral&utm_content=github/super-linter&utm_campaign=Badge_Grade)
+
 **The end goal of this tool:**
 
 - Prevent broken code from being uploaded to the default branch (_Usually_ `master` or `main`)

lib/functions/buildFileList.sh

@@ -100,7 +100,7 @@ function BuildFileList() {
       ################
       # push event   #
       ################
-      DIFF_TREE_CMD="git diff-tree --no-commit-id --name-only -r ${GITHUB_SHA}"
+      DIFF_TREE_CMD="git -C ${GITHUB_WORKSPACE} diff-tree --no-commit-id --name-only -r ${GITHUB_SHA}"
       GenerateFileDiff "$DIFF_TREE_CMD"
 
       ###############################################################

test/runTests.sh

@@ -0,0 +1 @@
+#!/usr/bin/env sh