Commit graph

180 commits

Author SHA1 Message Date
Marco Ferrari
c99ec7784a
fix: don't skip processing ansible_directory pwd (#5790)
Don't skip processing the current item (FILE) before we give
BuildFileArrays the chance to process it as an item to eventually add to
the list of directories to lint with ansible-lint.

Fix #5789

Other related changes

- Add a new make target to open a shell in a Super-linter container.
- Use a fixed path for FILE_ARRAYS_DIRECTORY_PATH so we can verify its
  contents in tests
- Remove redundant ValidateBooleanVariable in buildFileList because we
  already check those variables in valudation.
- Move Ansible directory detection to a function so we can reuse it.
- Add missing exports for global configuration variables.
- Remove unused LOG_XXXX variables from tests. These should have been
  deleted when we moved log variables to log.sh
2024-06-19 16:58:11 +00:00
Marco Ferrari
6047e3f732
feat: ignore avif files when building file list (#5604)
Close #5382
2024-05-06 10:47:11 +02:00
Chongyi Zheng
f84508a9c4
feat: find go.mod based on Go files (#5345) 2024-04-30 14:44:42 +02:00
Marco Ferrari
c26430f868
feat: support arbitrary shellcheck config paths (#5571)
- Support passing an arbitrary path to the shellcheck configuration file
  with the BASH_CONFIG_FILE_NAME variable. This brings shellcheck in
  line with other linters that support configuration files.
- Enable shellcheck external sources using a directive in the
  configuration file, so users can eventually override it if not needed.
- Export command configuration variables that subprocesses might need.

Close #5414
2024-04-25 19:47:55 +00:00
Marco Ferrari
49001a2405
fix: respect log level when writing to the log (#5546)
- Write log messages in the log file according to the LOG_LEVEL that the
  user configured (or the default), instead of printing all the messages
  regardless of LOG_LEVEL to the log file.
- Don't emit colors if there is no terminal

Close #5337
2024-04-20 09:18:14 +00:00
Marco Ferrari
8f405c1a9c
fix: handle initial commit (#5534)
Close #5453
2024-04-18 06:48:55 +00:00
Masaya Suzuki
69249882f3
feat: support GoReleaser (#5505) 2024-04-15 12:38:25 +00:00
Ümit Büyükulcay
0ae4572874
feat: add clang-format style configuration (#5424) 2024-04-11 14:59:47 +02:00
Junya Okabe
e71a37d49d
feat: add depndency (#5456)
feat: configure ruff

feat: update the orchestration scripts

feat: update the test suite

docs: update README

feat: add test cases for ruff

fix: CI error

chore: del .github/linters/.ruff.toml

fix: CI error

fix: README

update: LINTER_NAMES_ARRAY

fix: Dockerfile

fix: .github/linters/.jscpd.json

fix: test files

fix: del version_command
2024-04-10 12:02:28 +00:00
Marco Ferrari
cb109f45f1
fix: export test_case_run (#5499)
- Export the TEST_CASE_RUN variable because subprocesses reference it
  when building the file list.
- Remove a duplicate file that we didn't catch because Jscpd was not
  running due to the TEST_CASE_RUN variable not being exported.

Fix #5483
2024-04-08 20:48:30 +02:00
Benjamin Wuethrich
95aabd4cfa
feat(bash-exec): add option to ignore shell library files (#5254)
Introduce a new configuration variable, BASH_EXEC_IGNORE_LIBRARIES. If
set to true, the behaviour of bash-exec is modified: if a shell file has
a file extension and no shebang line, it is ignored, i.e., allowed to be
non-executable. This allows files that are only every sourced from other
shell files, acting as libraries and not executables, to have no
executable bit set without failing the bash-exec linter.
2024-02-27 18:17:22 +00:00
Marco Ferrari
608bd502d8
docs: suggest setting fetch-depth on ref errors (#5316)
Emit a hopefully more helpful error message when the Git ref doesn't
exist. The error message is the same we use when getting changed files
only: it suggests checking that repository clone is not shallow, and
that the full history is available. Additionally, when running on
GitHub Actions, it also suggests checking the fetch-depth option of the
actions/checkout step.

Fix #5315
2024-02-27 13:25:18 +01:00
Marco Ferrari
52b082b1f6
fix: fix log level variable assignment (#5319)
Correctly declare log level variables when initializing the logging
infrastructure.
2024-02-23 21:16:09 +00:00
Marco Ferrari
0f91a56f21
fix: simplify log infrastructure (#5312)
- Move log variables in log.sh to shorten linter.sh
- Source log.sh as soon as possible, so we can use log functions and
  variables as soon as possible.
- Validate 'boolean' log variables: LOG_DEBUG, LOG_VERBOSE, LOG_NOTICE,
  LOG_WARN, LOG_ERROR.
- Move foreground color markers from an associative array (that we
  cannot export), inside the log function. This fixes an issue that we
  discovered while working on #5254 where color markers were not
  available to subprocesses.
- Remove background color markers because we don't use them.
2024-02-21 17:51:47 +00:00
Marco Ferrari
0967cd29d0
feat: enable shell error checks (#5126)
Enable error checks to:

- Exit on errors
- Disallow empty variables
- Fail when a piped command errors
2024-02-20 19:05:39 +00:00
Marco Ferrari
091eaa71e3
feat: show error output when info is disabled (#5251)
In case of linting errors, print stdout and stderr (if present)
at the ERROR level if users set LOG_LEVEL to NOTICE to avoid
failures without any explanation.
2024-02-10 10:16:31 +01:00
Marco Ferrari
5b5e54ad5c
fix: initialize terrascan at runtime (#5246)
Terrascan runs initialization anyway when scanning files, so there's no
point in running it at build time. Also, this works around a Terrascan
bug that caused it to fail its initialization if $HOME/.terrascan
directory is not present. This happens on GitHub Actions because it
configures a $HOME directory that is different from ours.
2024-02-09 22:57:01 +00:00
Marco Ferrari
6fd6830fb4
fix: initialize GitHub domain variable (#5216)
- Initialize GITHUB_DOMAIN as documented in the README
- Configure other URLs according to GITHUB_DOMAIN
- Automatically fetch SSH key fingerprints for GITHUB_DOMAINS
2024-02-09 18:45:44 +00:00
Marco Ferrari
83eca1df43
fix: unset the log_level variable (#5249)
- Super-linter uses the LOG_LEVEL variable to let the user
  configure the desired log level. Checkov and Renovate use a variable
  with the same name for the same purpose, but accept a
  different set of values, and exit with an error if it gets an unknown
  value for that variable.
- Refactor the VERBOSE log level to the more commonly used INFO.
  Configuration validation will warn users if they use VERBOSE and
  instruct them to use INFO instead. This is not a breaking change
  because super-linter falls back on INFO if VERBOSE is set.
- Remove the TRACE log level because we rarely used it. As with VERBOSE,
  configuration validation will warn the user. Fall back to DEBUG if the
  user configured LOG_LEVEL to VERBOSE.

Close #5217
2024-02-09 18:43:58 +00:00
Marco Ferrari
787b63ddb2
fix: write hint about fetch-depth (#5241)
Point users at the fetch-depth option when not running locally.
2024-02-09 17:45:00 +00:00
Marco Ferrari
a26db6d34d
feat: lint xsd files (#5250)
Lint XSD files with xmllint

Close #5248
2024-02-09 17:44:30 +00:00
Marco Ferrari
fe6e29b685
feat: automatically set the default branch (#5242)
Get the default branch from the GitHub Actions event payload when
running on GitHub Actions. Default to 'master', as before, otherwise.
2024-02-08 08:41:07 +00:00
Marco Ferrari
38edbe557a
fix: create mypy cache directory (#5240)
MyPy expects the cache directory to be there. We now remove it after
each super-linter run to avoid leftovers, so we need to create it before
running MyPy.

See https://github.com/python/mypy/issues/10768
and https://github.com/python/mypy/issues/10863
2024-02-06 15:53:30 +01:00
Marco Ferrari
20ded7178b
fix: don't print empty lines with default logging (#5238)
- Check if Stdout and Stderr have elements before printing them.
- Run the super-linter action in a dedicated step using default logging to
  inspect how the output looks during CI.
2024-02-05 18:24:07 +01:00
Marco Ferrari
d7790e4f1c
build: move linter verions logic outside runtime (#5197)
Move the logic to build the linter versions file outside the main
runtime. There's no need to include it there because it's used only when
building the image.

Move the list of linters by name in linterVersions.sh because we don't
need it in any other place.
2024-02-05 13:31:36 +01:00
Marco Ferrari
eded42747b
fix: don't add unnecessary empty lines (#5221) 2024-02-05 11:49:22 +01:00
Marco Ferrari
99e41ce451
feat: run linters in parallel (#5177) 2024-01-30 19:24:55 +00:00
Marco Ferrari
9bab4a90e8
chore: simplify updateSSL (#5130)
Run command directly instead of checking their exit code afterwards.
2024-01-30 09:05:47 +01:00
Marco Ferrari
05009f2816
chore: simplify error code checks (#5131)
Instead of loading the error code and checking it, shorten checks by
running the command directly.
2024-01-30 08:48:24 +01:00
Marco Ferrari
f1873b0374
feat: don't check bake files with terragrunt (#5178)
Exclude the default docker bake files in HCL format from the list of files to
lint with Terragrunt.

Note that Docker bake supports defining arbitrary names for
configuration files, so this commit doesn't cover the scenario that a
user defined an arbitrary name for a Docker bake configuration file in
HCL format. Example: example-bake-config-file.hcl will be included in
the list of files to lint with Terragrunt.

Ref (default bake names):
https://docs.docker.com/build/bake/reference/#file-format
2024-01-29 13:51:28 +01:00
Marco Ferrari
5219feefab
fix: simplify worker and linterVersions (#5123)
- Remove the SKIP_FLAG variable and check for the length of the arrays
  of files to lint directly.
- Remove the LIST_FILES variable, and use the FILE_ARRAY variable
  directly.
- Remove the corner case for RENOVATE because renovate-config-validator
  supports passing the path to the file to lint using an argument as the
  default case does.
- Remove the corner case for ANSIBLE not having 'bad' tests because it
  has them now.
- Set TF_DATA_DIR to avoid any modification to any existing Terraform
  data directory that users might have in their workspace.
- Aggregate GO_MODULES and ANSIBLE corner cases because they are the
  same.
- Remove the corner case for ANSIBLE to add a trailing slash to
  TEST_CASE_FOLDER (similar reason as the previous point about ANSIBLE
  corner case).
- Simplify log messages by removing color markers because they are
  already handled in log.sh.
- Simplify linterVersions by removing redundant checks and functions.
- Avoid printing debug logs in the versions file.
2024-01-15 19:37:45 +01:00
Marco Ferrari
4a28fc5e73
feat: validate variables and simplify lowercase (#5128)
- Validate variables representing boolean values.
- Group global variables in the same sections.
- Declare variables as lowercase with the 'declare -l' shell builtin for
  more clarity.
2024-01-11 20:30:00 +00:00
Marco Ferrari
5a2056d77a
fix: handle log messages in CheckFileType (#5117)
- Honor SUPPRESS_FILE_TYPE_WARN when printing messages in the
  CheckFileType function.
- Reduce duplication when handling log messages in the CheckFileType
  function.
- Don't add files to the array of files to lint with JSCPD because we
  lint the whole codebase with JSCPD anyway.
2024-01-11 20:44:16 +01:00
Marco Ferrari
3a5617235c
feat!: deprecate error_on_missing_exec_bit (#5120)
Deprecate the ERROR_ON_MISSING_EXEC_BIT variable to remove a confusing
corner case, and to align the BASH_EXEC check to how super-linter
handles other linters.
2024-01-10 22:54:13 +00:00
Marco Ferrari
3a784fcfd6
fix: simplify file status checks (#5119)
- Simplify file status checks to assume that a file should be linted
  correctly unless we are running in test mode.
- Avoid the corner case of files having the 'bad' string as part of
  their file name to be wrongly assumed to fail linting.
- Move FILE_STATUS initialization where it's needed, after running the
  linters.
2024-01-10 12:35:05 +01:00
Marco Ferrari
6f70adee89
fix: enable linting changed files with textlint (#5100)
textlint was expensive to run because we added every file in the list of
files to lint to FILE_ARRAY_NATURAL_LANGUAGE. In #5041, we mitigated
this issue but lost the ability to run textlint on changed files only.
Given that textlint ignore files for which it doesn't have a plugin
installed, and that we don't currently install additional plugins
besides the default ones to lint markdown files and text files, we let
textlint run on these files only, so we can have the feature to lint
only changed files with this linter as well, back.
2024-01-10 09:49:04 +01:00
Marco Ferrari
e62b382bf0
feat: don't inspect files if not needed (#5094)
Don't run potentially expensive I/O operations to check file types if
we're not going to analyze them anyway.
2024-01-06 18:39:39 +01:00
Marco Ferrari
cf2038d903
fix: fix GITHUB_BEFORE_SHA initalization for push (#5098) 2024-01-05 23:07:39 +01:00
Marco Ferrari
1d5ed2c386
fix: fix GITHUB_BEFORE_SHA diff on push events (#5096)
- Fix GITHUB_BEFORE_SHA initialization on push events by setting the
  correct key.
- Add an additional check against setting GITHUB_BEFORE_SHA to null.
- Run the CI workflow on push events to trigger required status checks
  when using the merge queue.
2024-01-04 22:54:47 +01:00
Marco Ferrari
9257ba8af3
build: use embedded checkstyle configuration files (#5089)
checkstyle embeds sun_checks.xml and google_checks.xml so we can use
them instead of shipping our own.
2024-01-02 18:47:45 +01:00
Marco Ferrari
ededa44d36
fix: fail when validating as expected (#5076)
Fail when USE_FIND_ALGORITHM is true and VALIDATE_ALL_CODEBASE is false
as expected.
2023-12-30 23:02:25 +01:00
Marco Ferrari
69a45e022d
feat!: validate configuration when using find (#5045)
- Fail if the configuration contains USE_FIND_ALGORITHM=true and
  VALIDATE_ALL_CODEBASE=false because this is an unsupported
  configuration. Before this change, super-linter silently ignored
  VALIDATE_ALL_CODEBASE, leading to potentially surprising behavior.
2023-12-25 11:15:42 +01:00
dependabot[bot]
20d12b4c7a
deps(java): bump com.pinterest.ktlint:ktlint-cli in /dependencies/ktlint (#5050)
Remove KOTLIN_ANDROID because ktlint handles that case by reading
a configuration setting in .editorconfig.

Bumps [com.pinterest.ktlint:ktlint-cli](https://github.com/pinterest/ktlint) from 0.47.1 to 1.1.0.
- [Release notes](https://github.com/pinterest/ktlint/releases)
- [Changelog](https://github.com/pinterest/ktlint/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pinterest/ktlint/compare/0.47.1...1.1.0)

---
updated-dependencies:
- dependency-name: com.pinterest.ktlint:ktlint-cli
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-12-24 19:33:08 +01:00
Marco Ferrari
11b70102c3
feat!: run linters against the workspace (#5041)
- Run jscpd, gitleaks, textlint  against the entire workspace instead of
  running them over single files, one by one.
- Implement a warning function for deprecated variables.
- Deprecate the VALIDATE_JSCPD_ALL_CODEBASE variable.
- Remove duplicate configuration files when they are the same as the
  ones we provide in TEMPLATES.
- Add a missing tests for ansible-lint.
- Move ANSIBLE_DIRECTORY configuration when running tests in
  buildFileList, where similar configs are.
- Simplify ansible-lint test cases to include only what's necessary, and
  not an entire set of roles, playbooks, and inventory.
- Write instructions about major upgrades in the upgrade guide.
2023-12-24 17:56:15 +01:00
Marco Ferrari
b214a59ca7
fix: fix file list when looking for changes (#5044)
- Fix the file diff function on push events.
- Implement a test for the file diff function
2023-12-23 19:33:53 +01:00
Marco Ferrari
9d7268fb99
feat: add support for checkov to lint iac files (#4925)
- Add support to run Checkov against infrastructure as code descriptors
  that are in a given (configurable) directory. Defaults to lint the
  whole workspace.
- Establish a baseline for our own codebase so we don't have to fix
  issues right away with this change.
2023-12-22 13:22:15 +01:00
Marco Ferrari
60983d395f
fix: fail if r package installation fails (#4994)
- Fail if the installation of a R package fails.
- Install the remotes package once during the image build, and not when we scan
  files at runtime.
- Reuse the default R library directory instead of moving it to /home/r-library
2023-12-21 20:52:57 +01:00
Marco Ferrari
30317804b1
Lint Go modules (#4984)
* Lint Go projects by directory

* Search for Go modules

* Fix test path

* Add test cases and fix command

* Change workdir

* Add a warning about false positives

* fatal instead of warn

* Move tests to the test directory

* Close group on fatal

* Don't fail in test mode
2023-12-16 08:30:33 +00:00
Marco Ferrari
22564fb65c
Switch to tflint image because tflint-bundle is deprecated (#4990)
* Switch to tflint image because tflint-bundle is deprecated

* Fix version string

* Don't copy plugins

* Don't copy plugins

* Set terraform log vars globally

* Fix tflint error
2023-12-15 09:29:34 +00:00
Marco Ferrari
e6cf8d3845
Move tests to the test directory (#4985)
* Move tests to the test directory

* Fix linting errors

* Add states back

* Add xml back
2023-12-15 08:50:35 +00:00