Commit graph

60 commits

Author SHA1 Message Date
Marco Ferrari
bde3b9368e
build: set build_version dynamically (#6013)
Set BUILD_VERSION to the content of the version descriptor (version.txt)
if it changed in the last commit, assuming that the last commit was a
release preparation commit that updated the version descriptor.

Close #4928
2024-08-13 12:10:31 +02:00
Marco Ferrari
91dc6d7234
fix: add missing fix mode options and test cases (#5987)
- Add missing fix mode options for: CLANG_FORMAT, ENV,
  GOOGLE_JAVA_FORMAT, NATURAL_LANGUAGE, PYTHON_ISORT, RUST_CLIPPY.
- Refactor linter tests to make them shorter because there's no need to
  have big test files.
- Refactor 'bad' linter tests for linters that support fix mode so they
  contain only automatically fixable issues. This is needed to avoid
  adding another set of 'bad' linter tests for fix mode.
- Provide configuration files for linters that support fix mode and for
  which the default configuration is not suitable to enable fix mode:
  ansible-lint, ESLint, golangci-lint.
- Add a test case for linter commands options for linters that support
  fix mode, to ensure that fix mode and check-only mode options have
  been defined.
- Refactor the fix mode test to check if linters actually applied
  modifications to files.
- Update documentation about adding test cases for linters that support
  fix mode.
- Don't exit with a fatal error if VALIDATE_xxx is false when testing
  fix mode because not all linters support fix mode. To enable this, set
  the new FIX_MODE_TEST_CASE_RUN variable to true.
2024-08-12 12:31:38 +02:00
Marco Ferrari
6fdc091361
feat: local fix mode (#5978)
Certain linters and formatters support fixing linting and formatting
issues (fix mode). Before this change, Super-linter runs linters and
formatters in a mode that doesn't modify the source code in any way
(check only mode).

With this change, Super-linter supports running linters and formatters
in fix mode if explicitly requested by the configuration. If the
configuration includes a variable named FIX_<language_name>,
Super-linters modifies the command to run the linter or formatter for
<language_name> to enable fix mode.

The modifications to the linter or formatter command that Super-linter
applies depend on what is the default for a particular linter: it either
removes or adds options to the command to run the linter or formatter.
2024-08-07 15:36:16 +02:00
Marco Ferrari
16dbf17306
chore: add linter commands tests (#5957) 2024-08-05 09:55:58 +02:00
Marco Ferrari
14a6ee6da6
chore: move rules vars to a dedicated file (#5946) 2024-08-05 08:56:56 +02:00
Marco Ferrari
928972feeb
fix: adjust previous commit id on merge commits (#5950)
- When a commit is a merge commit, adjust the revision so that it
  selects the correct parent.
- Add relevant tests.
- Extract the logic to initialize a Git repository for tests in a
  dedicated function.

Fix #5941
2024-08-02 10:37:27 +02:00
Marco Ferrari
d9592327c6
chore: extract language array definition (#5945)
Move LANGUAGE_ARRAY definition and initialization to a dedicated file
(globals/languages.sh) so we can source it in different places without
having to source the whole linter.sh
2024-07-31 17:00:41 +02:00
shabbir-genetech
55a47ff317
ci: fix output formatting (#5942) 2024-07-31 10:32:56 +00:00
Marco Ferrari
ef57e132e1
fix: avoid duplicated content in summary (#5939)
- Remove the previous summary file is present to avoid showing stale
  contents from old summaries.
- Extract the logic to write summary heading and lines to dedicated
  functions in output.sh to make it easier adding new formats in the
  future.
2024-07-30 08:35:14 +02:00
Marco Ferrari
78ed3ef5fc
fix: store outputs in the main output directory (#5899)
- Store Super-linter outputs in the main output directory.
- Update README with a few fixes and details about outputs.
- Add missing test cases for when the configuration didn't enable any
  output.
2024-07-28 21:34:40 +02:00
Marco Ferrari
57c86588c3
feat: write github actions step summary (#5867)
This also removes an unneeded textlint configuration file because it
matches with the default one.

Close #5650
2024-07-11 08:01:11 +00:00
Marco Ferrari
12da4973c6
ci: enable dev-dependencies docker build checks (#5871)
- Run Docker build checks when building the dev-dependencies container
  image and when running the test target.
- Run Docker build checks when building the Super-linter container, and
  not just when running the test target.
- Use the correct variable when setting the dev-dependencies container
  user.
2024-07-11 08:08:23 +02:00
Marco Ferrari
bcf8ca82ad
ci: print info about the environment, image size (#5869)
- Add information about container image layers size when printing info
  about the environment.
- Print info about the environment during the CI job.
2024-07-10 14:45:32 +02:00
Marco Ferrari
ce59f5c323
ci: run docker build checks (#5864)
Run docker build checks as part of the test suite.

Ref https://docs.docker.com/reference/build-checks/
2024-07-09 11:22:29 +02:00
Marco Ferrari
94bb3f5563
feat: save super-linter output if requested (#5806)
- New SAVE_SUPER_LINTER_OUTPUT variable. When set to true,
  saves super-linter output to ${DEFAULT_WORKSPACE}/${SUPER_LINTER_OUTPUT_DIRECTORY_NAME}
- New SUPER_LINTER_OUTPUT_DIRECTORY_NAME variable to set the output
  directory name instide the default workspace.

Close #5774
2024-07-01 14:50:52 +02:00
Marco Ferrari
c99ec7784a
fix: don't skip processing ansible_directory pwd (#5790)
Don't skip processing the current item (FILE) before we give
BuildFileArrays the chance to process it as an item to eventually add to
the list of directories to lint with ansible-lint.

Fix #5789

Other related changes

- Add a new make target to open a shell in a Super-linter container.
- Use a fixed path for FILE_ARRAYS_DIRECTORY_PATH so we can verify its
  contents in tests
- Remove redundant ValidateBooleanVariable in buildFileList because we
  already check those variables in valudation.
- Move Ansible directory detection to a function so we can reuse it.
- Add missing exports for global configuration variables.
- Remove unused LOG_XXXX variables from tests. These should have been
  deleted when we moved log variables to log.sh
2024-06-19 16:58:11 +00:00
Marco Ferrari
2bb8a0a3e7
fix: no full git validation when ignoring files (#5599)
- Don't require Git SHA and branch validation when
  IGNORE_GITIGNORED_FILES=true because we only need to validate that the
  workspace is a Git repository in this case.

Fix #5383
2024-04-30 17:58:11 +02:00
Marco Ferrari
ff425b9a7d Revert "fix: no full git validation when ignoring files"
This reverts commit 5b0c248f9c.
2024-04-30 10:14:12 +00:00
Marco Ferrari
5b0c248f9c fix: no full git validation when ignoring files
- Don't require Git SHA and branch validation when
  IGNORE_GITIGNORED_FILES=true because we only need to validate that the
  workspace is a Git repository in this case.

Fix #5383
2024-04-30 10:11:14 +00:00
Marco Ferrari
8f405c1a9c
fix: handle initial commit (#5534)
Close #5453
2024-04-18 06:48:55 +00:00
Benjamin Wuethrich
95aabd4cfa
feat(bash-exec): add option to ignore shell library files (#5254)
Introduce a new configuration variable, BASH_EXEC_IGNORE_LIBRARIES. If
set to true, the behaviour of bash-exec is modified: if a shell file has
a file extension and no shebang line, it is ignored, i.e., allowed to be
non-executable. This allows files that are only every sourced from other
shell files, acting as libraries and not executables, to have no
executable bit set without failing the bash-exec linter.
2024-02-27 18:17:22 +00:00
Marco Ferrari
ed27c0146d
fix: github actions debug logging (#5288)
Enable debug logging when one (or more) of the following conditions is
true:

- ACTIONS_RUNNER_DEBUG is set to true
- ACTIONS_STEPS_DEBUG is set to true
- RUNNER_DEBUG is set to 1
2024-02-20 17:27:06 +00:00
Marco Ferrari
091eaa71e3
feat: show error output when info is disabled (#5251)
In case of linting errors, print stdout and stderr (if present)
at the ERROR level if users set LOG_LEVEL to NOTICE to avoid
failures without any explanation.
2024-02-10 10:16:31 +01:00
Marco Ferrari
5b5e54ad5c
fix: initialize terrascan at runtime (#5246)
Terrascan runs initialization anyway when scanning files, so there's no
point in running it at build time. Also, this works around a Terrascan
bug that caused it to fail its initialization if $HOME/.terrascan
directory is not present. This happens on GitHub Actions because it
configures a $HOME directory that is different from ours.
2024-02-09 22:57:01 +00:00
Marco Ferrari
6fd6830fb4
fix: initialize GitHub domain variable (#5216)
- Initialize GITHUB_DOMAIN as documented in the README
- Configure other URLs according to GITHUB_DOMAIN
- Automatically fetch SSH key fingerprints for GITHUB_DOMAINS
2024-02-09 18:45:44 +00:00
Marco Ferrari
83eca1df43
fix: unset the log_level variable (#5249)
- Super-linter uses the LOG_LEVEL variable to let the user
  configure the desired log level. Checkov and Renovate use a variable
  with the same name for the same purpose, but accept a
  different set of values, and exit with an error if it gets an unknown
  value for that variable.
- Refactor the VERBOSE log level to the more commonly used INFO.
  Configuration validation will warn users if they use VERBOSE and
  instruct them to use INFO instead. This is not a breaking change
  because super-linter falls back on INFO if VERBOSE is set.
- Remove the TRACE log level because we rarely used it. As with VERBOSE,
  configuration validation will warn the user. Fall back to DEBUG if the
  user configured LOG_LEVEL to VERBOSE.

Close #5217
2024-02-09 18:43:58 +00:00
Marco Ferrari
ace79ca403
build: ignore changelog when linting codebase (#5205)
CHANGELOG.md is automatically generated by the release workflow.
Ignore it when linting the codebase because it might not pass textlint
validation because of lowercase terms.
2024-01-31 10:31:06 +01:00
Marco Ferrari
99e41ce451
feat: run linters in parallel (#5177) 2024-01-30 19:24:55 +00:00
Marco Ferrari
9bab4a90e8
chore: simplify updateSSL (#5130)
Run command directly instead of checking their exit code afterwards.
2024-01-30 09:05:47 +01:00
Kin Fai Tse
3a21ed5bdf
ci: implement more smoke tests (#4955)
- Non-default workspace
- Git-related flags
2024-01-16 10:23:17 +01:00
Marco Ferrari
3a5617235c
feat!: deprecate error_on_missing_exec_bit (#5120)
Deprecate the ERROR_ON_MISSING_EXEC_BIT variable to remove a confusing
corner case, and to align the BASH_EXEC check to how super-linter
handles other linters.
2024-01-10 22:54:13 +00:00
Marco Ferrari
df911171c4
build: python venvs and npm in dedicated stages (#5078)
Move the following tasks to dedicated stages so we can run these steps
in parallel with other stages:

- Build Python virtual environments
- Install npm packages
- Build clang-format
- Download and install TFlint plugins

Add missing target stage when building the container image
2024-01-10 11:09:53 +01:00
Marco Ferrari
e62b382bf0
feat: don't inspect files if not needed (#5094)
Don't run potentially expensive I/O operations to check file types if
we're not going to analyze them anyway.
2024-01-06 18:39:39 +01:00
Marco Ferrari
cf2038d903
fix: fix GITHUB_BEFORE_SHA initalization for push (#5098) 2024-01-05 23:07:39 +01:00
Marco Ferrari
901a901655
fix: add missing checkov configuration file (#5090)
Add a default (empty) configuration file for Checkov and a smoke test to
ensure that we catch the case where a default configuration file is not
present, and it should be.
2024-01-02 18:03:30 +01:00
Marco Ferrari
11b70102c3
feat!: run linters against the workspace (#5041)
- Run jscpd, gitleaks, textlint  against the entire workspace instead of
  running them over single files, one by one.
- Implement a warning function for deprecated variables.
- Deprecate the VALIDATE_JSCPD_ALL_CODEBASE variable.
- Remove duplicate configuration files when they are the same as the
  ones we provide in TEMPLATES.
- Add a missing tests for ansible-lint.
- Move ANSIBLE_DIRECTORY configuration when running tests in
  buildFileList, where similar configs are.
- Simplify ansible-lint test cases to include only what's necessary, and
  not an entire set of roles, playbooks, and inventory.
- Write instructions about major upgrades in the upgrade guide.
2023-12-24 17:56:15 +01:00
Marco Ferrari
b214a59ca7
fix: fix file list when looking for changes (#5044)
- Fix the file diff function on push events.
- Implement a test for the file diff function
2023-12-23 19:33:53 +01:00
Marco Ferrari
9d7268fb99
feat: add support for checkov to lint iac files (#4925)
- Add support to run Checkov against infrastructure as code descriptors
  that are in a given (configurable) directory. Defaults to lint the
  whole workspace.
- Establish a baseline for our own codebase so we don't have to fix
  issues right away with this change.
2023-12-22 13:22:15 +01:00
Marco Ferrari
641c65a8c4
ci: configure release-please dry-run and changelog (#5039)
- Implement a job to preview the release notes
- Include build, ci, and dependency updates
- Add emoji to section headings to match the existing release notes
- Add documentation about how to run release-please from the CLI
2023-12-21 15:03:14 +01:00
Marco Ferrari
93b5ede1e8
ci: configure release-please (#5016)
- Configure release-please to automatically create pull requests and
  releases.
- Run release-please and tag update in the CD workflow.
- Update Git tags pointing to latest, major.minor, and major versions as
  part of the CD workflow.
- Remove workflows (draft-release, release) that are not necessary
  anymore, and related configuration files.
- Handle automatic updates to README.md, action.yml, and action.yaml
- Mount .github to /tmp/lint/.github so super-linter finds config files,
  and the GitHub Actions to lint.
2023-12-20 14:58:25 +01:00
Marco Ferrari
9db632f0e1
ci: configure commitlint (#5014)
- Check if the PR contains a single commit, and fail otherwise.
- Enable commitlint to check if commits adhere to the
  conventialcommits.org spec.
- Update the the pull request template to point to the conventional
  commit spec.
- Update the dependabot configuration to add the "build(...)" prefix to
  commits.
2023-12-19 19:37:16 +01:00
Marco Ferrari
22b8624f61
Simplify container image build (#4962) 2023-12-15 11:59:36 +01:00
Marco Ferrari
e6cf8d3845
Move tests to the test directory (#4985)
* Move tests to the test directory

* Fix linting errors

* Add states back

* Add xml back
2023-12-15 08:50:35 +00:00
Marco Ferrari
7150e1f8b0
Group log output on GitHub Actions (#4961) 2023-12-12 20:57:15 +01:00
Marco Ferrari
ac4b767bd7
Reduce duplication in CI and CD workflows (#4982)
* Reduce duplication in CI and CD workflows

* Fix indentation in README

* Load token from file

* Fix instructions

* Ignore test leftovers
2023-12-12 18:53:48 +00:00
Marco Ferrari
7a21f934b4
Don't forcefully validate Git repos if not needed (#4953)
* Fix find when linting non-git repos

* Remove uses

* Move safe.directory config after we set GITHUB_WORKSPACE

* Fix Git validation check

* Move 'find' test runner to makefile

* Validate vars before validating Git repo

* Validate Git repo even when VALIDATE_ALL_CODEBASE=false

* Initialize GITHUB_SHA when running locally

* Initialize safe git dirs

* Check git safe dirs error code

* Fix log level color marker

* Fix linting errors

* Fix change dir command

* Fix linting errors

* Set default branch
2023-12-07 19:07:22 +00:00
Marco Ferrari
9869638131
Validate container image labels (#4926)
* Validate labels and avoid busting the cache

* Fix validation

* Validate non-empty labels

* Add build date back

* Don't set build date

* Simplify validation script

* Enable build cache

* Setup buildx

* Dynamically set build revision and version

* Remove leftover

* Disable cache

* Add build date back

* Add build date back

* Fix linting errors

* Add checks

* Get head SHA

* Fix linting errors

* Handle merge_group
2023-12-07 14:18:47 +00:00
Marco Ferrari
bb8ba1ad7b
Run the info target when running the test suite (#4799) 2023-10-31 17:45:19 +00:00
Marco Ferrari
dfedd306f3
Use the same image tags in CI and CD (#4778)
* Use the same image tags in CI and CD

* Define image tag once per workflow

* Fix yq tag

* Don't quote yq to avoid literal interpretation

* yq version

* Fix yq tag

* Reduce duplication

* Fix workflow env var

* Don't remove an already removed container
2023-10-19 21:03:14 +00:00
Zack Koppert
da4dd08cd6
Follow up with several more documentation and automation renames for the org move from github to super-linter (#4543)
* move from github org to super-linter org

* rename to super-linter org

* rename to super-linter org

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter and remove dockerhub reference

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rename org to super-linter

* rm dockerhub section link
2023-08-17 15:12:11 +00:00