actions/superlint
1
0
Fork 0
mirror of https://github.com/super-linter/super-linter.git synced 2024-09-19 02:47:08 -04:00
Code Issues Projects Releases Packages Wiki Activity

Add Gitleaks for secret scanning (#1951)

Browse source 
* Add actionlint

* adding gitleaks

* rm secretlint

* Update .gitleaks.toml

Co-authored-by: Admiral Awkbar <admiralawkbar@github.com>
This commit is contained in:
Masaya Suzuki 2021-09-27 23:32:18 +09:00 committed by GitHub
parent 5d5dba0d99
commit 0e1a1e50d6
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 222 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

20
.automation/test/gitleaks/README.md Normal file
View file

@ -0,0 +1,20 @@
# GitLeaks Test Cases
This folder holds the test cases for **GitLeaks**.
## Additional Docs
No Additional information is needed for this test case.
## Good Test Cases
The test cases denoted: `LANGUAGE_good_FILE.EXTENSION` are all valid, and should pass successfully when linted.
- **Note:** They are linted utilizing the default linter rules.
## Bad Test Cases
The test cases denoted: `LANGUAGE_bad_FILE.EXTENSION` are **NOT** valid, and should trigger errors when linted.
- **Note:** They are linted utilizing the default linter rules.
{"mode":"full","isActive":false}

1
.automation/test/gitleaks/gitleaks_bad_01.txt Normal file
View file

@ -0,0 +1 @@
AWS_SECRET_ACCESS_KEY = wJalrXUtnFEMI/K7MDENG/bPxRfiCYSECRETSKEY

1
.automation/test/gitleaks/gitleaks_good_01.txt Normal file
View file

@ -0,0 +1 @@
AWS_SECRET_ACCESS_KEY =

8
Dockerfile
View file

@ -307,7 +307,13 @@ RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community/" >> /etc/apk/repo
&& find /node_modules/ -type f -name 'LICENSE' -exec rm {} + \
&& find /node_modules/ -type f -name '*.md' -exec rm {} + \
&& find /node_modules/ -type f -name '*.txt' -exec rm {} + \
&& find /usr/ -type f -name '*.md' -exec rm {} +
&& find /usr/ -type f -name '*.md' -exec rm {} + \
####################
# Install GitLeaks #
####################
&& GO111MODULE=on go get github.com/zricethezav/gitleaks/v7 \
&& mv /root/go/bin/gitleaks /usr/bin
################################################################################
# Build the clang-format binary ################################################

7
Dockerfile-slim
View file

@ -245,7 +245,12 @@ RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community/" >> /etc/apk/repo
&& find /node_modules/ -type f -name 'LICENSE' -exec rm {} + \
&& find /node_modules/ -type f -name '*.md' -exec rm {} + \
&& find /node_modules/ -type f -name '*.txt' -exec rm {} + \
&& find /usr/ -type f -name '*.md' -exec rm {} +
&& find /usr/ -type f -name '*.md' -exec rm {} + \
####################
# Install GitLeaks #
####################
&& GO111MODULE=on go get github.com/zricethezav/gitleaks/v7 \
&& mv /root/go/bin/gitleaks /usr/bin
################################################################################
# Build the clang-format binary ################################################

3
README.md
View file

@ -90,6 +90,7 @@ Developers on **GitHub** can call the **GitHub Action** to lint their code base
| **Raku** | [Raku](https://raku.org) |
| **Ruby** | [RuboCop](https://github.com/rubocop-hq/rubocop) |
| **Rust** | [Rustfmt](https://github.com/rust-lang/rustfmt) / [Clippy](https://github.com/rust-lang/rust-clippy) |
| **Secrets** | [GitLeaks](https://github.com/zricethezav/gitleaks) |
| **Shell** | [Shellcheck](https://github.com/koalaman/shellcheck) / [executable bit check] / [shfmt](https://github.com/mvdan/sh) |
| **Snakemake** | [snakefmt](https://github.com/snakemake/snakefmt/) / [snakemake --lint](https://snakemake.readthedocs.io/en/stable/snakefiles/writing_snakefiles.html#best-practices) |
| **SQL** | [sql-lint](https://github.com/joereynolds/sql-lint) / [sqlfluff](https://github.com/sqlfluff/sqlfluff) |
@ -285,6 +286,7 @@ But if you wish to select or exclude specific linters, we give you full control
| **GITHUB_ACTIONS_CONFIG_FILE** | `actionlint.yml` | Filename for [Actionlint configuration](https://github.com/rhysd/actionlint/blob/main/docs/config.md) (ex: `actionlint.yml`) |
| **GITHUB_DOMAIN** | `github.com` | Specify a custom Github domain in case Github Enterprise is used: e.g. `github.myenterprise.com` |
| **GITHUB_CUSTOM_API_URL** | `api.github.com` | Specify a custom Github API URL in case Github Enterprise is used: e.g. `https://github.myenterprise.com/api/v3/` |
| **GITLEAKS_CONFIG_FILE** | `.gitleaks.toml` | Filename for [GitLeaks configuration](https://github.com/zricethezav/gitleaks#configuration) (ex: `.geatleaks.toml`) |
| **IGNORE_GENERATED_FILES** | `false` | If set to `true`, super-linter will ignore all the files with `@generated` marker but without `@not-generated` marker. |
| **IGNORE_GITIGNORED_FILES** | `false` | If set to `true`, super-linter will ignore all the files that are ignored by Git. |
| **JAVA_FILE_NAME** | `sun-checks.xml` | Filename for [Checkstyle configuration](https://checkstyle.sourceforge.io/config.html) (ex: `checkstyle.xml`) |
@ -332,6 +334,7 @@ But if you wish to select or exclude specific linters, we give you full control
| **VALIDATE_EDITORCONFIG** | `true` | Flag to enable or disable the linting process with the editorconfig. |
| **VALIDATE_ENV** | `true` | Flag to enable or disable the linting process of the ENV language. |
| **VALIDATE_GITHUB_ACTIONS** | `true` | Flag to enable or disable the linting process of the GitHub Actions. |
| **VALIDATE_GITLEAKS** | `true` | Flag to enable or disable the linting process of the secrets. |
| **VALIDATE_GHERKIN** | `true` | Flag to enable or disable the linting process of the Gherkin language. |
| **VALIDATE_GO** | `true` | Flag to enable or disable the linting process of the Golang language. |
| **VALIDATE_GROOVY** | `true` | Flag to enable or disable the linting process of the language. |

175
TEMPLATES/.gitleaks.toml Normal file
View file

@ -0,0 +1,175 @@
title = "gitleaks config"
[[rules]]
description = "AWS Access Key"
regex = '''(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}'''
tags = ["key", "AWS"]
[[rules]]
description = "AWS Secret Key"
regex = '''(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"]'''
tags = ["key", "AWS"]
[[rules]]
description = "AWS MWS key"
regex = '''amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
tags = ["key", "AWS", "MWS"]
[[rules]]
description = "Facebook Secret Key"
regex = '''(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]'''
tags = ["key", "Facebook"]
[[rules]]
description = "Facebook Client ID"
regex = '''(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]'''
tags = ["key", "Facebook"]
[[rules]]
description = "Twitter Secret Key"
regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]'''
tags = ["key", "Twitter"]
[[rules]]
description = "Twitter Client ID"
regex = '''(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]'''
tags = ["client", "Twitter"]
[[rules]]
description = "Github Personal Access Token"
regex = '''ghp_[0-9a-zA-Z]{36}'''
tags = ["key", "Github"]
[[rules]]
description = "Github OAuth Access Token"
regex = '''gho_[0-9a-zA-Z]{36}'''
tags = ["key", "Github"]
[[rules]]
description = "Github App Token"
regex = '''(ghu|ghs)_[0-9a-zA-Z]{36}'''
tags = ["key", "Github"]
[[rules]]
description = "Github Refresh Token"
regex = '''ghr_[0-9a-zA-Z]{76}'''
tags = ["key", "Github"]
[[rules]]
description = "LinkedIn Client ID"
regex = '''(?i)linkedin(.{0,20})?(?-i)[0-9a-z]{12}'''
tags = ["client", "LinkedIn"]
[[rules]]
description = "LinkedIn Secret Key"
regex = '''(?i)linkedin(.{0,20})?[0-9a-z]{16}'''
tags = ["secret", "LinkedIn"]
[[rules]]
description = "Slack"
regex = '''xox[baprs]-([0-9a-zA-Z]{10,48})?'''
tags = ["key", "Slack"]
[[rules]]
description = "Asymmetric Private Key"
regex = '''-----BEGIN ((EC|PGP|DSA|RSA|OPENSSH) )?PRIVATE KEY( BLOCK)?-----'''
tags = ["key", "AsymmetricPrivateKey"]
[[rules]]
description = "Google API key"
regex = '''AIza[0-9A-Za-z\\-_]{35}'''
tags = ["key", "Google"]
[[rules]]
description = "Google (GCP) Service Account"
regex = '''"type": "service_account"'''
tags = ["key", "Google"]
[[rules]]
description = "Heroku API key"
regex = '''(?i)heroku(.{0,20})?[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'''
tags = ["key", "Heroku"]
[[rules]]
description = "MailChimp API key"
regex = '''(?i)(mailchimp|mc)(.{0,20})?[0-9a-f]{32}-us[0-9]{1,2}'''
tags = ["key", "Mailchimp"]
[[rules]]
description = "Mailgun API key"
regex = '''((?i)(mailgun|mg)(.{0,20})?)?key-[0-9a-z]{32}'''
tags = ["key", "Mailgun"]
[[rules]]
description = "PayPal Braintree access token"
regex = '''access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}'''
tags = ["key", "Paypal"]
[[rules]]
description = "Picatic API key"
regex = '''sk_live_[0-9a-z]{32}'''
tags = ["key", "Picatic"]
[[rules]]
description = "SendGrid API Key"
regex = '''SG\.[\w_]{16,32}\.[\w_]{16,64}'''
tags = ["key", "SendGrid"]
[[rules]]
description = "Slack Webhook"
regex = '''https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8,12}/[a-zA-Z0-9_]{24}'''
tags = ["key", "slack"]
[[rules]]
description = "Stripe API key"
regex = '''(?i)stripe(.{0,20})?[sr]k_live_[0-9a-zA-Z]{24}'''
tags = ["key", "Stripe"]
[[rules]]
description = "Square access token"
regex = '''sq0atp-[0-9A-Za-z\-_]{22}'''
tags = ["key", "square"]
[[rules]]
description = "Square OAuth secret"
regex = '''sq0csp-[0-9A-Za-z\\-_]{43}'''
tags = ["key", "square"]
[[rules]]
description = "Twilio API key"
regex = '''(?i)twilio(.{0,20})?SK[0-9a-f]{32}'''
tags = ["key", "twilio"]
[[rules]]
description = "Dynatrace ttoken"
regex = '''dt0[a-zA-Z]{1}[0-9]{2}\.[A-Z0-9]{24}\.[A-Z0-9]{64}'''
tags = ["key", "Dynatrace"]
[[rules]]
description = "Shopify shared secret"
regex = '''shpss_[a-fA-F0-9]{32}'''
tags = ["key", "Shopify"]
[[rules]]
description = "Shopify access token"
regex = '''shpat_[a-fA-F0-9]{32}'''
tags = ["key", "Shopify"]
[[rules]]
description = "Shopify custom app access token"
regex = '''shpca_[a-fA-F0-9]{32}'''
tags = ["key", "Shopify"]
[[rules]]
description = "Shopify private app access token"
regex = '''shppa_[a-fA-F0-9]{32}'''
tags = ["key", "Shopify"]
[[rules]]
description = "PyPI upload token"
regex = '''pypi-AgEIcHlwaS5vcmc[A-Za-z0-9-_]{50,1000}'''
tags = ["key", "pypi"]
[allowlist]
description = "Allowlisted files"
files = ['''^\.?gitleaks.toml$''',
'''(.*?)(png|jpg|gif|doc|docx|pdf|bin|xls|pyc|zip)$''',
'''(go.mod|go.sum)$''']

2
lib/functions/buildFileList.sh
View file

@ -316,6 +316,8 @@ function BuildFileList() {
FILE_ARRAY_EDITORCONFIG+=("${FILE}")
# jscpd also runs an all files
FILE_ARRAY_JSCPD+=("${FILE}")
# GitLeaks also runs an all files
FILE_ARRAY_GITLEAKS+=("${FILE}")
#######################
# Get the shell files #

2
lib/functions/linterVersions.sh
View file

@ -68,7 +68,7 @@ BuildLinterVersions() {
if [[ ${LINTER} == "arm-ttk" ]]; then
# Need specific command for ARM
GET_VERSION_CMD="$(grep -iE 'version' "/usr/bin/arm-ttk" | xargs 2>&1)"
elif [[ ${LINTER} == "bash-exec" ]] || [[ ${LINTER} == "gherkin-lint" ]]; then
elif [[ ${LINTER} == "bash-exec" ]] || [[ ${LINTER} == "gherkin-lint" ]] || [[ ${LINTER} == "gitleaks" ]]; then
# Need specific command for Protolint and editorconfig-checker
GET_VERSION_CMD="$(echo "--version not supported")"
elif [[ ${LINTER} == "lintr" ]]; then

8
lib/linter.sh
View file

@ -94,6 +94,8 @@ EDITORCONFIG_FILE_NAME="${EDITORCONFIG_FILE_NAME:-.ecrc}"
# shellcheck disable=SC2034 # Variable is referenced indirectly
GITHUB_ACTIONS_FILE_NAME="${GITHUB_ACTIONS_CONFIG_FILE:-actionlint.yml}"
# shellcheck disable=SC2034 # Variable is referenced indirectly
GITLEAKS_FILE_NAME="${GITLEAKS_CONFIG_FILE:.gitleaks.toml}"
# shellcheck disable=SC2034 # Variable is referenced indirectly
GHERKIN_FILE_NAME=".gherkin-lintrc"
# shellcheck disable=SC2034 # Variable is referenced indirectly
GO_FILE_NAME=".golangci.yml"
@ -200,12 +202,12 @@ fi
LANGUAGE_ARRAY=('ANSIBLE' 'ARM' 'BASH' 'BASH_EXEC' 'CLANG_FORMAT'
'CLOUDFORMATION' 'CLOJURE' 'COFFEESCRIPT' 'CPP' 'CSHARP' 'CSS' 'DART'
'DOCKERFILE' 'DOCKERFILE_HADOLINT' 'EDITORCONFIG' 'ENV' 'GITHUB_ACTIONS'
'GHERKIN' 'GO' 'GROOVY' 'HTML' 'JAVA' 'JAVASCRIPT_ES'
'GITLEAKS' 'GHERKIN' 'GO' 'GROOVY' 'HTML' 'JAVA' 'JAVASCRIPT_ES'
"${JAVASCRIPT_STYLE_NAME}" 'JSCPD' 'JSON' 'JSONC' 'JSX' 'KUBERNETES_KUBEVAL'
'KOTLIN' 'LATEX' 'LUA' 'MARKDOWN' 'OPENAPI' 'PERL' 'PHP_BUILTIN' 'PHP_PHPCS'
'PHP_PHPSTAN' 'PHP_PSALM' 'POWERSHELL' 'PROTOBUF' 'PYTHON_BLACK'
'PYTHON_PYLINT' 'PYTHON_FLAKE8' 'PYTHON_ISORT' 'PYTHON_MYPY' 'R' 'RAKU' 'RUBY'
'RUST_2015' 'RUST_2018' 'RUST_CLIPPY' 'SHELL_SHFMT' 'SNAKEMAKE_LINT'
'RUST_2015' 'RUST_2018' 'RUST_CLIPPY' 'SECRET' 'SHELL_SHFMT' 'SNAKEMAKE_LINT'
'SNAKEMAKE_SNAKEFMT' 'STATES' 'SQL' 'SQLFLUFF' 'TEKTON' 'TERRAFORM_TFLINT'
'TERRAFORM_TERRASCAN' 'TERRAGRUNT' 'TSX' 'TYPESCRIPT_ES' 'TYPESCRIPT_STANDARD'
'XML' 'YAML')
@ -231,6 +233,7 @@ LINTER_NAMES_ARRAY['DOCKERFILE_HADOLINT']="hadolint"
LINTER_NAMES_ARRAY['EDITORCONFIG']="editorconfig-checker"
LINTER_NAMES_ARRAY['ENV']="dotenv-linter"
LINTER_NAMES_ARRAY['GITHUB_ACTIONS']="actionlint"
LINTER_NAMES_ARRAY['GITLEAKS']="gitleaks"
LINTER_NAMES_ARRAY['GHERKIN']="gherkin-lint"
LINTER_NAMES_ARRAY['GO']="golangci-lint"
LINTER_NAMES_ARRAY['GROOVY']="npm-groovy-lint"
@ -831,6 +834,7 @@ LINTER_COMMANDS_ARRAY['DOCKERFILE_HADOLINT']="hadolint -c ${DOCKERFILE_HADOLINT_
LINTER_COMMANDS_ARRAY['EDITORCONFIG']="editorconfig-checker -config ${EDITORCONFIG_LINTER_RULES}"
LINTER_COMMANDS_ARRAY['ENV']="dotenv-linter"
LINTER_COMMANDS_ARRAY['GITHUB_ACTIONS']="actionlint -config-file ${GITHUB_ACTIONS_LINTER_RULES}"
LINTER_COMMANDS_ARRAY['GITLEAKS']="gitleaks -q -c ${GITLEAKS_LINTER_RULES} -p"
LINTER_COMMANDS_ARRAY['GHERKIN']="gherkin-lint -c ${GHERKIN_LINTER_RULES}"
LINTER_COMMANDS_ARRAY['GO']="golangci-lint run -c ${GO_LINTER_RULES}"
LINTER_COMMANDS_ARRAY['GROOVY']="npm-groovy-lint -c ${GROOVY_LINTER_RULES} --failon warning"