Ignore files marked with @generated marker (#1689)

* Ignore files marked with @generated marker `@generated` marker is used by certain tools to understand that the file is generated, so it should be treated differently than a file written by a human: * these files do not need to be reformatted, * diffs in these files are less important, * and linters should not be invoked on these files. This PR proposes builtin support for `@generated` marker (and `@not-generated` marker to mark file as not generated when it contains `@generated` marker, like `README.md`). I have not found a standard for a generated file marker, but: * Facebook [uses `@generated` marker](https://tinyurl.com/fb-generated) * Phabricator tool which was spawned from Facebook internal tool [also understands `@generated` marker](https://git.io/JnVHa) * Cargo inserts `@generated` marker into [generated Cargo.lock files](https://git.io/JnVHP) Super-linter supports regex includes and excludes, but they are harder to maintain (each repository needs to be configured) than patching the tools which generate the files. My personal story is that I maintain rust-protobuf crate, which started emitting `@generated` markers [six years ago](https://git.io/JnV5h) after a request of a Phabricator user. Test Plan: Create a test file `test.sh`: ``` echo $a ``` Run: ``` docker run -e RUN_LOCAL=true -v $HOME/tmp/g:/tmp/lint super-linter-test ``` Result is: ``` In /tmp/lint/test.sh line 1: echo $a ^-- SC2148: Tips depend on target shell and yours is unknown. Add a shebang or a 'shell' directive. ^-- SC2154: a is referenced but not assigned. ^-- SC2086: Double quote to prevent globbing and word splitting. ... 2021-06-22 23:46:16 [ERROR] ERRORS FOUND in BASH:[1] ``` Now add `@generated` to the file and run again: ``` 2021-06-22 23:47:13 [NOTICE] All file(s) linted successfully with no errors detected ``` Additionally, add `@not-generated` in addition to `@generated`, and linter error pops up again. * cleanup * remove space * fix non utf return * fix non utf return Co-authored-by: Lukas Gravley <admiralawkbar@github.com>
2024-11-21 16:21:00 -05:00 · 2021-06-28 13:59:11 +01:00 · 2021-06-28 13:59:11 +01:00 · e059100991
commit e059100991
parent de0700eb5f
5 changed files with 82 additions and 5 deletions
--- a/README.md
+++ b/README.md
@ -282,6 +282,7 @@ But if you wish to select or exclude specific linters, we give you full control
 | **FILTER_REGEX_INCLUDE**           | `all`                           | Regular expression defining which files will be processed by linters (ex: `.*src/.*`)                                                                                                                                |
 | **GITHUB_DOMAIN**                  | `github.com`                    | Specify a custom Github domain in case Github Enterprise is used: e.g. `github.myenterprise.com`                                                                                                                     |
 | **GITHUB_CUSTOM_API_URL**          | `api.github.com`                | Specify a custom Github API URL in case Github Enterprise is used: e.g. `https://github.myenterprise.com/api/v3/`                                                                                                    |
+| **IGNORE_GENERATED_FILES**         | `false`                          | If set to `true`, super-linter will ignore all the files with `@generated` marker but without `@not-generated` marker.                                                                                               |
 | **IGNORE_GITIGNORED_FILES**        | `false`                         | If set to `true`, super-linter will ignore all the files that are ignored by Git.                                                                                                                                    |
 | **JAVASCRIPT_ES_CONFIG_FILE**      | `.eslintrc.yml`                 | Filename for [eslint configuration](https://eslint.org/docs/user-guide/configuring#configuration-file-formats) (ex: `.eslintrc.yml`, `.eslintrc.json`)                                                               |
 | **JAVASCRIPT_DEFAULT_STYLE**       | `standard`                      | Flag to set the default style of javascript. Available options: **standard**/**prettier**                                                                                                                            |
@ -399,6 +400,26 @@ Examples:
 - Do not lint files inside test folder: `FILTER_REGEX_EXCLUDE: .*test/.*`
 - Do not lint javascript files inside test folder: `FILTER_REGEX_EXCLUDE: .*test/.*.js`

+<!-- This `README.md` has both markers in the text, so it is considered not generated. -->
+Additionally when `IGNORE_GENERATED_FILES=true`, super-linter
+ignores any file with `@generated` marker in it unless the file
+also has `@not-generated` marker. `@generated` marker is
+[used by Facebook](https://engineering.fb.com/2015/08/20/open-source/writing-code-that-writes-code-with-hack-codegen/)
+and some other projects to mark generated files. For example, this
+file is considered generated:
+
+```bash
+#!/bin/sh
+echo "@generated"
+```
+
+And this file is considered not generated:
+
+```bash
+#!/bin/sh
+echo "@generated" # @not-generated
+```
+
 ## Docker Hub

 The **Docker** container that is built from this repository is located at [github/super-linter](https://hub.docker.com/r/github/super-linter)
--- a/lib/functions/buildFileList.sh
+++ b/lib/functions/buildFileList.sh
@ -64,6 +64,8 @@ function BuildFileList() {

  debug "IGNORE_GITIGNORED_FILES: ${IGNORE_GITIGNORED_FILES}..."

+  debug "IGNORE_GENERATED_FILES: ${IGNORE_GENERATED_FILES}..."
+
  debug "USE_FIND_ALGORITHM: ${USE_FIND_ALGORITHM}..."

  if [ "${VALIDATE_ALL_CODEBASE}" == "false" ] && [ "${TEST_CASE_RUN}" != "true" ]; then
@ -278,27 +280,38 @@ function BuildFileList() {
      debug "TEST_CASE_RUN (${TEST_CASE_RUN}) is true. Skipping ${FILE}..."
    fi

-    #################################################
+    ###############################################
    # Filter files if FILTER_REGEX_INCLUDE is set #
-    #################################################
+    ###############################################
    if [[ -n "$FILTER_REGEX_INCLUDE" ]] && [[ ! (${FILE} =~ $FILTER_REGEX_INCLUDE) ]]; then
      debug "FILTER_REGEX_INCLUDE didn't match. Skipping ${FILE}"
      continue
    fi

-    #################################################
+    ###############################################
    # Filter files if FILTER_REGEX_EXCLUDE is set #
-    #################################################
+    ###############################################
    if [[ -n "$FILTER_REGEX_EXCLUDE" ]] && [[ ${FILE} =~ $FILTER_REGEX_EXCLUDE ]]; then
      debug "FILTER_REGEX_EXCLUDE match. Skipping ${FILE}"
      continue
    fi

+    ###################################################
+    # Filter files if FILTER_REGEX_EXCLUDE is not set #
+    ###################################################
    if [ "${GIT_IGNORED_FILES_INDEX[$FILE]}" ] && [ "${IGNORE_GITIGNORED_FILES}" == "true" ]; then
      debug "${FILE} is ignored by Git. Skipping ${FILE}"
      continue
    fi

+    #########################################
+    # Filter files with at-generated marker #
+    #########################################
+    if [ "${IGNORE_GENERATED_FILES}" == "true" ] && IsGenerated "$FILE"; then
+      debug "${FILE} is generated. Skipping ${FILE}"
+      continue
+    fi
+
    # Editorconfig-checker should check every file
    FILE_ARRAY_EDITORCONFIG+=("${FILE}")
    # jscpd also runs an all files
--- a/lib/functions/detectFiles.sh
+++ b/lib/functions/detectFiles.sh
@ -334,3 +334,43 @@ function IsValidShellScript() {
  trace "$FILE is NOT a supported shell script. Skipping"
  return 1
 }
+################################################################################
+#### Function IsGenerated ######################################################
+function IsGenerated() {
+  # Pull in Vars #
+  ################
+  FILE="$1"
+
+  ##############################
+  # Check the file for keyword #
+  ##############################
+  grep -q "@generated" "$FILE"
+
+  #######################
+  # Load the error code #
+  #######################
+  ERROR_CODE=$?
+
+  if [ ${ERROR_CODE} -ne 0 ]; then
+    trace "File:[${FILE}] is not generated, because it doesn't have @generated marker"
+    return 1
+  fi
+
+  ##############################
+  # Check the file for keyword #
+  ##############################
+  grep -q "@not-generated" "$FILE"
+
+  #######################
+  # Load the error code #
+  #######################
+  ERROR_CODE=$?
+
+  if [ ${ERROR_CODE} -eq 0 ]; then
+    trace "File:[${FILE}] is not-generated because it has @not-generated marker"
+    return 1
+  else
+    trace "File:[${FILE}] is generated because it has @generated marker"
+    return 0
+  fi
+}
--- a/lib/linter.sh
+++ b/lib/linter.sh
@ -294,6 +294,9 @@ DEFAULT_BRANCH="${DEFAULT_BRANCH:-master}" # Default Git Branch to use (master b

 IGNORE_GITIGNORED_FILES="${IGNORE_GITIGNORED_FILES:-false}"

+# Do not ignore generated files by default for backwards compatibility
+IGNORE_GENERATED_FILES="${IGNORE_GENERATED_FILES:-false}"
+
 ################
 # Default Vars #
 ################
--- a/test/inspec/super-linter/controls/super_linter.rb
+++ b/test/inspec/super-linter/controls/super_linter.rb
@ -132,7 +132,7 @@ control "super-linter-installed-commands" do
    { linter_name: "R", version_command: "R --slave -e \"r_ver <- R.Version()\\$version.string; \
            lintr_ver <- packageVersion('lintr'); \
            glue::glue('lintr { lintr_ver } on { r_ver }')\""},
-    { linter_name: "raku"},
+    { linter_name: "raku", version_command: "raku --version | strings -n 8"},
    { linter_name: "rubocop"},
    { linter_name: "rustfmt"},
    { linter_name: "shellcheck"},