Generate a code coverage report (#1353)

* Generate a code coverage report * make linters happy * adding action and badge * no report for me * Upload the test report * adding folders * spaces not tabs * makefiles love tabs * spacing is important * rawr * Fix makefile * Add diagnostic info * Set UID and GID * Make info * Add docker images in diagnostic info * Move info before testing the local action * Add missing -C to git diff-tree * Reset ownership * Add missing quotes * Sudo * Attempt to fix ARM test * Attempt to fix go test Co-authored-by: Admiral Awkbar <admiralawkbar@github.com>
2024-09-13 07:57:10 -04:00 · 2021-04-22 20:56:18 +02:00 · 2021-04-22 20:56:18 +02:00 · 527c5a3986
commit 527c5a3986
parent 786e7056de
8 changed files with 129 additions and 14 deletions
--- a/.automation/test/arm/reports/expected-ARM.tap
+++ b/.automation/test/arm/reports/expected-ARM.tap
@ -2,6 +2,6 @@ TAP version 13
 1..2
 not ok 1 - arm_bad_1.json
  ---
-  message: \nValidating arm\arm_bad_1.json\n deploymentTemplate\n [+] adminUsername Should Not Be A Literal\n [+] apiVersions Should Be Recent\n [+] artifacts parameter\n [+] CommandToExecute Must Use ProtectedSettings For Secrets\n [+] DependsOn Best Practices\n [+] Deployment Resources Must Not Be Debug\n [+] DeploymentTemplate Must Not Contain Hardcoded Uri\n [+] DeploymentTemplate Schema Is Correct\n [+] Dynamic Variable References Should Not Use Concat\n [+] IDs Should Be Derived From ResourceIDs\n [+] Location Should Not Be Hardcoded\n [+] ManagedIdentityExtension must not be used\n [+] Min And Max Value Are Numbers\n [+] Outputs Must Not Contain Secrets\n [+] Parameters Must Be Referenced\n [+] Parameters Property Must Exist\n [+] providers apiVersions Is Not Permitted\n [+] ResourceIds should not contain\n [+] Resources Should Have Location\n [+] Resources Should Not Be Ambiguous\n [+] Secure String Parameters Cannot Have Default\n [-] Template Should Not Contain Blanks\n Empty property [] found on line 28 Index 1331\n Empty property "" found on line 3 Index 121\n\n [-] Variables Must Be Referenced\n Unreferenced variable Network.Location\n Unreferenced variable Test.Blank.Variable\n\n [+] Virtual Machines Should Not Be Preview\n [+] VM Images Should Use Latest Version\n [+] VM Size Should Be A Parameter\n \n
+  message: \nValidating arm\arm_bad_1.json\n deploymentTemplate\n [+] adminUsername Should Not Be A Literal\n [+] apiVersions Should Be Recent\n [+] artifacts parameter\n [+] CommandToExecute Must Use ProtectedSettings For Secrets\n [+] DependsOn Best Practices\n [+] Deployment Resources Must Not Be Debug\n [+] DeploymentTemplate Must Not Contain Hardcoded Uri\n [+] DeploymentTemplate Schema Is Correct\n [+] Dynamic Variable References Should Not Use Concat\n [+] IDs Should Be Derived From ResourceIDs\n [+] Location Should Not Be Hardcoded\n [+] ManagedIdentityExtension must not be used\n [+] Min And Max Value Are Numbers\n [+] Outputs Must Not Contain Secrets\n [+] Parameters Must Be Referenced\n [+] providers apiVersions Is Not Permitted\n [+] ResourceIds should not contain\n [+] Resources Should Have Location\n [+] Resources Should Not Be Ambiguous\n [+] Secure String Parameters Cannot Have Default\n [-] Template Should Not Contain Blanks\n Empty property [] found on line 28 Index 1331\n Empty property "" found on line 3 Index 121\n\n [-] Variables Must Be Referenced\n Unreferenced variable Network.Location\n Unreferenced variable Test.Blank.Variable\n\n [+] Virtual Machines Should Not Be Preview\n [+] VM Images Should Use Latest Version\n [+] VM Size Should Be A Parameter\n \n
  ...
 ok 2 - arm_good_1.json
--- a/.automation/test/go/reports/expected-GO-4.tap
+++ b/.automation/test/go/reports/expected-GO-4.tap
@ -2,6 +2,6 @@ TAP version 13
 1..2
 not ok 1 - golang_bad_01.go
  ---
-  message: level=warningg="[runner] The linter 'maligned' is deprecated due to The repository of the linter has been archived by the owner. Use govet 'fieldalignment' instead."\nlevel=warningg="[runner] Can't run linter goanalysis_metalinter failed prerequisites [buildir@command-line-arguments analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']]"\nlevel=warningg="[runner] Can't run linter unused buildir analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']"\nlevel=errorg="Running error buildir analysis skipped errors in package [go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if' /tmp/lint/.automation/test/go/golang_bad_01.go 1 1 expected 'package', found 'if']"\n
+  message: level=warningg="[runner] The linter 'maligned' is deprecated (since v1.38.0) due to The repository of the linter has been archived by the owner. Replaced by govet 'fieldalignment'."\ngo/golang_bad_01.go 1 1 expected 'package', found 'if' (typecheck)\nif len(in) == 0 {\n^\n
  ...
 ok 2 - golang_good_01.go
--- a/.github/workflows/deploy-DEV.yml
+++ b/.github/workflows/deploy-DEV.yml
@ -20,7 +20,7 @@ on:
  push:
    branches-ignore: [master]
  pull_request:
-    branches-ignore: []
+    branches-ignore: [ ]

 ###############
 # Set the Job #
@ -32,7 +32,8 @@ jobs:
    # Set the agent to run on
    runs-on: ubuntu-latest
    # Prevent duplicate run from happening when a forked push is committed
-    if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository
+    if: github.event_name == 'push' ||
+        github.event.pull_request.head.repo.full_name != github.repository
    ##################
    # Load all steps #
    ##################
@ -43,7 +44,8 @@ jobs:
      - name: Checkout Code
        uses: actions/checkout@v2.3.4
        with:
-          # Full git history is needed to get a proper list of changed files within `super-linter`
+          # Full git history is needed to get a proper list
+          # of changed files within `super-linter`
          fetch-depth: 0

      ########################
@ -66,12 +68,6 @@ jobs:
          --build-arg BUILD_VERSION=${{ github.sha }} \
          -t ghcr.io/github/super-linter:${{ github.sha }} -f Dockerfile .

-      ##############
-      # Print info #
-      ##############
-      - name: Get Docker info
-        run: docker images
-
      ########################################
      # Validates the metadata docker labels #
      ########################################
@ -86,6 +82,14 @@ jobs:
        run: |
          sed -i "s/super-linter:.*/super-linter:${GITHUB_SHA}'/g" action.yml

+      ######################
+      # Gather information #
+      ######################
+      - name: Gather information about the runtime environment
+        shell: bash
+        run: |
+          make info
+
      ##########################
      # Test in action context #
      ##########################
@ -99,12 +103,47 @@ jobs:
          VALIDATE_ALL_CODEBASE: false
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

+      ###########################################################################################
+      # Fix file and dir ownership. Workaround for https://github.com/actions/runner/issues/434 #
+      ###########################################################################################
+      - name: Fix file and directory ownership
+        shell: bash
+        run: |
+          sudo chown -R "$(id -u)":"$(id -g)" "$(pwd)"
+
+      ##################
+      # Run test cases #
+      ##################
+      - name: Run the test suite
+        shell: bash
+        run: |
+          make test
+
+      ##########################
+      # Codacy Coverage Report #
+      ##########################
+      - name: Upload the code coverage report
+        uses: codacy/codacy-coverage-reporter-action@0.2.0
+        with:
+          project-token: ${{ secrets.CODACY_PROJECT_TOKEN }}
+          coverage-reports: test/reports/cobertura/runTests.sh/cobertura.xml
+
      #####################################
      # Run Linter against Test code base #
      #####################################
      - name: Run Test Cases
        shell: bash
-        run: docker run -e RUN_LOCAL=true -e TEST_CASE_RUN=true -e ANSIBLE_DIRECTORY=.automation/test/ansible -e OUTPUT_FORMAT=tap -e OUTPUT_FOLDER=${GITHUB_SHA} -e OUTPUT_DETAILS=detailed -e ACTIONS_RUNNER_DEBUG=true -e ERROR_ON_MISSING_EXEC_BIT=true -v ${GITHUB_WORKSPACE}:/tmp/lint ghcr.io/github/super-linter:${GITHUB_SHA}
+        run: |
+          docker run \
+          -e RUN_LOCAL=true \
+          -e TEST_CASE_RUN=true \
+          -e ANSIBLE_DIRECTORY=.automation/test/ansible \
+          -e OUTPUT_FORMAT=tap -e OUTPUT_FOLDER=${GITHUB_SHA} \
+          -e OUTPUT_DETAILS=detailed \
+          -e ACTIONS_RUNNER_DEBUG=true \
+          -e ERROR_ON_MISSING_EXEC_BIT=true \
+          -v ${GITHUB_WORKSPACE}:/tmp/lint \
+          ghcr.io/github/super-linter:${GITHUB_SHA}

      #########################################
      # Clean code base to run against it all #
@ -118,4 +157,11 @@ jobs:
      ############################################
      - name: Run against all code base
        shell: bash
-        run: docker run -e RUN_LOCAL=true -e OUTPUT_DETAILS=detailed -e ACTIONS_RUNNER_DEBUG=true -e ERROR_ON_MISSING_EXEC_BIT=true -v ${GITHUB_WORKSPACE}:/tmp/lint ghcr.io/github/super-linter:${GITHUB_SHA}
+        run: |
+          docker run \
+          -e RUN_LOCAL=true \
+          -e OUTPUT_DETAILS=detailed \
+          -e ACTIONS_RUNNER_DEBUG=true \
+          -e ERROR_ON_MISSING_EXEC_BIT=true \
+          -v ${GITHUB_WORKSPACE}:/tmp/lint \
+          ghcr.io/github/super-linter:${GITHUB_SHA}
--- a/.gitignore
+++ b/.gitignore
@ -69,3 +69,9 @@ super-linter.report

 # Git directory (useful for .dockerignore)
 .git
+
+# Code coverage data for tests
+.coverage
+
+# Test reports
+test/reports
--- a/60
+++ b/60
@ -0,0 +1,60 @@
+# Inspired by https://github.com/jessfraz/dotfiles
+
+.PHONY: all
+all: info test ## Run all targets.
+
+.PHONY: test
+test: info clean kcov prepare-test-reports ## Run tests
+
+# if this session isn't interactive, then we don't want to allocate a
+# TTY, which would fail, but if it is interactive, we do want to attach
+# so that the user can send e.g. ^C through.
+INTERACTIVE := $(shell [ -t 0 ] && echo 1 || echo 0)
+ifeq ($(INTERACTIVE), 1)
+	DOCKER_FLAGS += -t
+endif
+
+.PHONY: info
+info: ## Gather information about the runtime environment
+	echo "whoami: $$(whoami)"; \
+	echo "pwd: $$(pwd)"; \
+	echo "ls -ahl: $$(ls -ahl)"; \
+	docker images
+
+.PHONY: kcov
+kcov: ## Run kcov
+	docker run --rm $(DOCKER_FLAGS) \
+		--user "$$(id -u)":"$$(id -g)" \
+		-v "$(CURDIR)":/workspace \
+		-w="/workspace" \
+		kcov/kcov \
+		kcov \
+		--bash-parse-files-in-dir=/workspace \
+		--clean \
+		--exclude-pattern=.coverage,.git \
+		--include-pattern=.sh \
+		/workspace/test/.coverage \
+		/workspace/test/runTests.sh
+
+COBERTURA_REPORTS_DESTINATION_DIRECTORY := "$(CURDIR)/test/reports/cobertura"
+
+.PHONY: prepare-test-reports
+prepare-test-reports: ## Prepare the test reports for consumption
+	mkdir -p $(COBERTURA_REPORTS_DESTINATION_DIRECTORY); \
+	COBERTURA_REPORTS="$$(find "$$(pwd)" -name 'cobertura.xml')"; \
+	for COBERTURA_REPORT_FILE_PATH in $$COBERTURA_REPORTS ; do \
+		COBERTURA_REPORT_DIRECTORY_PATH="$$(dirname "$$COBERTURA_REPORT_FILE_PATH")"; \
+		COBERTURA_REPORT_DIRECTORY_NAME="$$(basename "$$COBERTURA_REPORT_DIRECTORY_PATH")"; \
+		COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX="$${COBERTURA_REPORT_DIRECTORY_NAME%.*}"; \
+		mkdir -p "$(COBERTURA_REPORTS_DESTINATION_DIRECTORY)"/"$$COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX"; \
+		cp "$$COBERTURA_REPORT_FILE_PATH" "$(COBERTURA_REPORTS_DESTINATION_DIRECTORY)"/"$$COBERTURA_REPORT_DIRECTORY_NAME_NO_SUFFIX"/cobertura.xml; \
+	done
+
+.PHONY: clean
+clean: ## Clean the workspace
+	rm -rf $(CURDIR)/test/.coverage; \
+	rm -rf $(CURDIR)/test/reports
+
+.PHONY: help
+help: ## Show help
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
--- a/README.md
+++ b/README.md
@ -3,6 +3,8 @@
 This repository is for the **GitHub Action** to run a **Super-Linter**.
 It is a simple combination of various linters, written in `bash`, to help validate your source code.

+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/51071879604e4f319859d4daf91c68f5)](https://www.codacy.com/gh/github/super-linter/dashboard?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=github/super-linter&amp;utm_campaign=Badge_Grade)
+
 **The end goal of this tool:**

 - Prevent broken code from being uploaded to the default branch (_Usually_ `master` or `main`)
--- a/lib/functions/buildFileList.sh
+++ b/lib/functions/buildFileList.sh
@ -100,7 +100,7 @@ function BuildFileList() {
      ################
      # push event   #
      ################
-      DIFF_TREE_CMD="git diff-tree --no-commit-id --name-only -r ${GITHUB_SHA}"
+      DIFF_TREE_CMD="git -C ${GITHUB_WORKSPACE} diff-tree --no-commit-id --name-only -r ${GITHUB_SHA}"
      GenerateFileDiff "$DIFF_TREE_CMD"

      ###############################################################
--- a/test/runTests.sh
+++ b/test/runTests.sh
@ -0,0 +1 @@
+#!/usr/bin/env sh