Commit graph

35 commits

Author SHA1 Message Date
Marco Ferrari
94920ffcc7
feat: optionally remove color codes from output (#6095)
If REMOVE_ANSI_COLOR_CODES_FROM_OUTPUT is set to true, remove ANSI color
codes from linters stdout and stderr files, and from the Super-linter
log file.

Close #5540
2024-08-31 19:50:00 +02:00
Marco Ferrari
2f0ac20566
fix: avoid concurrent edits to the same file (#6027)
Reduce the number of processes to 1 if at least one FIX_xxxx variable is
set to true. This avoids that Parallel runs multiple processes that
might edit the same file at the same time.
2024-08-14 22:20:56 +02:00
Marco Ferrari
48d5391b7b
fix!: remove sql-lint (#5991)
sql-lint is unmaintained, and its dependencies are impacted by several
vulnerabilities.
2024-08-14 17:30:46 +02:00
Marco Ferrari
91dc6d7234
fix: add missing fix mode options and test cases (#5987)
- Add missing fix mode options for: CLANG_FORMAT, ENV,
  GOOGLE_JAVA_FORMAT, NATURAL_LANGUAGE, PYTHON_ISORT, RUST_CLIPPY.
- Refactor linter tests to make them shorter because there's no need to
  have big test files.
- Refactor 'bad' linter tests for linters that support fix mode so they
  contain only automatically fixable issues. This is needed to avoid
  adding another set of 'bad' linter tests for fix mode.
- Provide configuration files for linters that support fix mode and for
  which the default configuration is not suitable to enable fix mode:
  ansible-lint, ESLint, golangci-lint.
- Add a test case for linter commands options for linters that support
  fix mode, to ensure that fix mode and check-only mode options have
  been defined.
- Refactor the fix mode test to check if linters actually applied
  modifications to files.
- Update documentation about adding test cases for linters that support
  fix mode.
- Don't exit with a fatal error if VALIDATE_xxx is false when testing
  fix mode because not all linters support fix mode. To enable this, set
  the new FIX_MODE_TEST_CASE_RUN variable to true.
2024-08-12 12:31:38 +02:00
Marco Ferrari
6fdc091361
feat: local fix mode (#5978)
Certain linters and formatters support fixing linting and formatting
issues (fix mode). Before this change, Super-linter runs linters and
formatters in a mode that doesn't modify the source code in any way
(check only mode).

With this change, Super-linter supports running linters and formatters
in fix mode if explicitly requested by the configuration. If the
configuration includes a variable named FIX_<language_name>,
Super-linters modifies the command to run the linter or formatter for
<language_name> to enable fix mode.

The modifications to the linter or formatter command that Super-linter
applies depend on what is the default for a particular linter: it either
removes or adds options to the command to run the linter or formatter.
2024-08-07 15:36:16 +02:00
Marco Ferrari
b09ac00c68
docs: add info about standard and prettier (#5948)
Document how to update the configuration when it includes
JAVASCRIPT_DEFAULT_STYLE or TYPESCRIPT_DEFAULT_STYLE.
2024-08-01 12:39:48 +02:00
Marco Ferrari
78ed3ef5fc
fix: store outputs in the main output directory (#5899)
- Store Super-linter outputs in the main output directory.
- Update README with a few fixes and details about outputs.
- Add missing test cases for when the configuration didn't enable any
  output.
2024-07-28 21:34:40 +02:00
Marco Ferrari
2daf461143
feat: allow using both prettier and standardjs (#5679)
- Allow using both Prettier and StandardJS at the same time by removing
  JAVASCRIPT_DEFAULT_STYLE and TYPESCRIPT_DEFAULT_STYLE
  configuration variables. They create a confusing corner case that
  Super-linter already handles with its core logic that's in place to
  enable and disable linters.
- Document VALIDATE_JAVASCRIPT_PRETTIER and VALIDATE_TYPESCRIPT_PRETTIER
  because they were missing from the README.
- Add missing TYPESCRIPT_PRETTIER tests.
2024-07-19 05:04:43 +00:00
Marco Ferrari
d2d73347d3
chore: split validation logic in smaller functions (#5892)
- Move USE_FIND_ALGORITHM and VALIDATE_ALL_CODEBASE validation in a
  dedicated function (ValidateFindMode).
- Move ANSIBLE_DIRECTORY validation to a dedicated function
  (ValidateAnsibleDirectory).
- Move VALIDATE_xxxx variables validation to a dedicated function
  (ValidateValidationVariables).
- Mark ANY_SET, ANY_TRUE, ANY_FALSE as local because we don't need to
  reference them anywhere outside ValidateValidationVariables.
- Add some debug statements in validation functions.
- Merge the loops to initialize VALIDATE_xxx variables and to print
  enable/disable language debug messages.
- Add tests for these validation functions.
- Add test start message for all tests.
2024-07-16 08:22:45 +00:00
Marco Ferrari
57c86588c3
feat: write github actions step summary (#5867)
This also removes an unneeded textlint configuration file because it
matches with the default one.

Close #5650
2024-07-11 08:01:11 +00:00
Marco Ferrari
94bb3f5563
feat: save super-linter output if requested (#5806)
- New SAVE_SUPER_LINTER_OUTPUT variable. When set to true,
  saves super-linter output to ${DEFAULT_WORKSPACE}/${SUPER_LINTER_OUTPUT_DIRECTORY_NAME}
- New SUPER_LINTER_OUTPUT_DIRECTORY_NAME variable to set the output
  directory name instide the default workspace.

Close #5774
2024-07-01 14:50:52 +02:00
Marco Ferrari
cef17760de
feat: configure github server url (#5792)
Allow the configuration of the GitHub server URL, and add some
validation rules that check for common misconfigurations.

Close #5572
2024-06-21 08:54:19 +02:00
Marco Ferrari
c99ec7784a
fix: don't skip processing ansible_directory pwd (#5790)
Don't skip processing the current item (FILE) before we give
BuildFileArrays the chance to process it as an item to eventually add to
the list of directories to lint with ansible-lint.

Fix #5789

Other related changes

- Add a new make target to open a shell in a Super-linter container.
- Use a fixed path for FILE_ARRAYS_DIRECTORY_PATH so we can verify its
  contents in tests
- Remove redundant ValidateBooleanVariable in buildFileList because we
  already check those variables in valudation.
- Move Ansible directory detection to a function so we can reuse it.
- Add missing exports for global configuration variables.
- Remove unused LOG_XXXX variables from tests. These should have been
  deleted when we moved log variables to log.sh
2024-06-19 16:58:11 +00:00
Benjamin Wuethrich
95aabd4cfa
feat(bash-exec): add option to ignore shell library files (#5254)
Introduce a new configuration variable, BASH_EXEC_IGNORE_LIBRARIES. If
set to true, the behaviour of bash-exec is modified: if a shell file has
a file extension and no shebang line, it is ignored, i.e., allowed to be
non-executable. This allows files that are only every sourced from other
shell files, acting as libraries and not executables, to have no
executable bit set without failing the bash-exec linter.
2024-02-27 18:17:22 +00:00
Marco Ferrari
608bd502d8
docs: suggest setting fetch-depth on ref errors (#5316)
Emit a hopefully more helpful error message when the Git ref doesn't
exist. The error message is the same we use when getting changed files
only: it suggests checking that repository clone is not shallow, and
that the full history is available. Additionally, when running on
GitHub Actions, it also suggests checking the fetch-depth option of the
actions/checkout step.

Fix #5315
2024-02-27 13:25:18 +01:00
Marco Ferrari
0f91a56f21
fix: simplify log infrastructure (#5312)
- Move log variables in log.sh to shorten linter.sh
- Source log.sh as soon as possible, so we can use log functions and
  variables as soon as possible.
- Validate 'boolean' log variables: LOG_DEBUG, LOG_VERBOSE, LOG_NOTICE,
  LOG_WARN, LOG_ERROR.
- Move foreground color markers from an associative array (that we
  cannot export), inside the log function. This fixes an issue that we
  discovered while working on #5254 where color markers were not
  available to subprocesses.
- Remove background color markers because we don't use them.
2024-02-21 17:51:47 +00:00
Marco Ferrari
0967cd29d0
feat: enable shell error checks (#5126)
Enable error checks to:

- Exit on errors
- Disallow empty variables
- Fail when a piped command errors
2024-02-20 19:05:39 +00:00
Marco Ferrari
83eca1df43
fix: unset the log_level variable (#5249)
- Super-linter uses the LOG_LEVEL variable to let the user
  configure the desired log level. Checkov and Renovate use a variable
  with the same name for the same purpose, but accept a
  different set of values, and exit with an error if it gets an unknown
  value for that variable.
- Refactor the VERBOSE log level to the more commonly used INFO.
  Configuration validation will warn users if they use VERBOSE and
  instruct them to use INFO instead. This is not a breaking change
  because super-linter falls back on INFO if VERBOSE is set.
- Remove the TRACE log level because we rarely used it. As with VERBOSE,
  configuration validation will warn the user. Fall back to DEBUG if the
  user configured LOG_LEVEL to VERBOSE.

Close #5217
2024-02-09 18:43:58 +00:00
Marco Ferrari
d7790e4f1c
build: move linter verions logic outside runtime (#5197)
Move the logic to build the linter versions file outside the main
runtime. There's no need to include it there because it's used only when
building the image.

Move the list of linters by name in linterVersions.sh because we don't
need it in any other place.
2024-02-05 13:31:36 +01:00
Marco Ferrari
99e41ce451
feat: run linters in parallel (#5177) 2024-01-30 19:24:55 +00:00
Marco Ferrari
4a28fc5e73
feat: validate variables and simplify lowercase (#5128)
- Validate variables representing boolean values.
- Group global variables in the same sections.
- Declare variables as lowercase with the 'declare -l' shell builtin for
  more clarity.
2024-01-11 20:30:00 +00:00
Marco Ferrari
3a5617235c
feat!: deprecate error_on_missing_exec_bit (#5120)
Deprecate the ERROR_ON_MISSING_EXEC_BIT variable to remove a confusing
corner case, and to align the BASH_EXEC check to how super-linter
handles other linters.
2024-01-10 22:54:13 +00:00
Marco Ferrari
cf2038d903
fix: fix GITHUB_BEFORE_SHA initalization for push (#5098) 2024-01-05 23:07:39 +01:00
Marco Ferrari
ededa44d36
fix: fail when validating as expected (#5076)
Fail when USE_FIND_ALGORITHM is true and VALIDATE_ALL_CODEBASE is false
as expected.
2023-12-30 23:02:25 +01:00
Marco Ferrari
69a45e022d
feat!: validate configuration when using find (#5045)
- Fail if the configuration contains USE_FIND_ALGORITHM=true and
  VALIDATE_ALL_CODEBASE=false because this is an unsupported
  configuration. Before this change, super-linter silently ignored
  VALIDATE_ALL_CODEBASE, leading to potentially surprising behavior.
2023-12-25 11:15:42 +01:00
dependabot[bot]
20d12b4c7a
deps(java): bump com.pinterest.ktlint:ktlint-cli in /dependencies/ktlint (#5050)
Remove KOTLIN_ANDROID because ktlint handles that case by reading
a configuration setting in .editorconfig.

Bumps [com.pinterest.ktlint:ktlint-cli](https://github.com/pinterest/ktlint) from 0.47.1 to 1.1.0.
- [Release notes](https://github.com/pinterest/ktlint/releases)
- [Changelog](https://github.com/pinterest/ktlint/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pinterest/ktlint/compare/0.47.1...1.1.0)

---
updated-dependencies:
- dependency-name: com.pinterest.ktlint:ktlint-cli
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-24 19:33:08 +01:00
Marco Ferrari
11b70102c3
feat!: run linters against the workspace (#5041)
- Run jscpd, gitleaks, textlint  against the entire workspace instead of
  running them over single files, one by one.
- Implement a warning function for deprecated variables.
- Deprecate the VALIDATE_JSCPD_ALL_CODEBASE variable.
- Remove duplicate configuration files when they are the same as the
  ones we provide in TEMPLATES.
- Add a missing tests for ansible-lint.
- Move ANSIBLE_DIRECTORY configuration when running tests in
  buildFileList, where similar configs are.
- Simplify ansible-lint test cases to include only what's necessary, and
  not an entire set of roles, playbooks, and inventory.
- Write instructions about major upgrades in the upgrade guide.
2023-12-24 17:56:15 +01:00
Marco Ferrari
9d7268fb99
feat: add support for checkov to lint iac files (#4925)
- Add support to run Checkov against infrastructure as code descriptors
  that are in a given (configurable) directory. Defaults to lint the
  whole workspace.
- Establish a baseline for our own codebase so we don't have to fix
  issues right away with this change.
2023-12-22 13:22:15 +01:00
Marco Ferrari
30317804b1
Lint Go modules (#4984)
* Lint Go projects by directory

* Search for Go modules

* Fix test path

* Add test cases and fix command

* Change workdir

* Add a warning about false positives

* fatal instead of warn

* Move tests to the test directory

* Close group on fatal

* Don't fail in test mode
2023-12-16 08:30:33 +00:00
Marco Ferrari
7a21f934b4
Don't forcefully validate Git repos if not needed (#4953)
* Fix find when linting non-git repos

* Remove uses

* Move safe.directory config after we set GITHUB_WORKSPACE

* Fix Git validation check

* Move 'find' test runner to makefile

* Validate vars before validating Git repo

* Validate Git repo even when VALIDATE_ALL_CODEBASE=false

* Initialize GITHUB_SHA when running locally

* Initialize safe git dirs

* Check git safe dirs error code

* Fix log level color marker

* Fix linting errors

* Fix change dir command

* Fix linting errors

* Set default branch
2023-12-07 19:07:22 +00:00
Marco Ferrari
5a8805dc4f
Exit on errors when running Git (#4889)
* Exit on errors when running Git

* Skip pulling changes entirely

* Enable pipefail when generating diffs

* Cleanup

* Shallow repo check

* Echo GITHUB_SHA update

* Check if GITHUB_SHA exists before using it

* Move GITHUB_SHA validation to validation script

* Rely on cat-file return code

* Check if DEFAULT_BRANCH exists

* Change dir when checking DEFAULT_BRANCH

* Show git branches

* Don't switch branches

* Check GITHUB_SHA only when needed

* Ensure we have permissions before interacting with the repo

* Remove the DIFF_CMD variable

* Move TEST_CASE_RUN and RUN_LOCAL init up

* Validate if Git repo and if SHA exists

* Move validation function

* Change dir when getting branch names

* Move debug messages up to be less verbose

* Move branch validation in a function

* Fix linting errors
2023-12-04 09:47:49 +00:00
Stephen Abbene
35d5a3e9fb
Running superlinter as a user not defined in /etc/passwd causes error messages (#4351)
* no need to run git check-ignore on every file if you are not setting IGNORE_GITIGNORED_FILES

* use id instead of whoami for determining running and discard stderr

* accidently include change from other PR

* remove commented code

---------

Co-authored-by: Zack Koppert <zkoppert@github.com>
2023-06-20 19:57:38 +00:00
Marco Ferrari
b6bbc88e8e
Ignore files that are ignored by Git (#1185)
* Ignore files that are ignored by Git

* Add missing continue statement

* Fix linting errors

* Fix linting errors

* like it

* like it

* Add the IGNORE_GITIGNORED_FILES variable

* Fix setting the Ansible directory when running tests

* Fix var export

* Update linter.sh

Co-authored-by: Admiral Awkbar <admiralawkbar@github.com>
2021-02-18 12:15:50 -06:00
Aaron Sky
328b17bbc8
Improved ANSIBLE_DIRECTORY validation to support the repository root (#1111)
* Improved ANSIBLE_DIRECTORY validation to support the repository root

Fixes #1110, which prevents linting playbooks at the root of the repository from being supported.

* Trim whitespace

* Remove undesired extra validation condition

Addressing review feedback from @ ferrarimarco

* Avoid mutating ANSIBLE_DIRECTORY beyond the extent that was already in place

Co-authored-by: Lukas Gravley <admiralawkbar@github.com>
2021-01-04 13:38:46 -06:00
Lukas Gravley
7943ce1f54
break out: (#990)
* break out:

* fix it

* fix source

* add them exports

* more path

* more path
2020-11-12 11:27:34 -06:00
Renamed from lib/validation.sh (Browse further)