[sectoramen]

Closes #769.

Rebased onto main after #786 merged (squash a84b533). This PR was stacked on #786 because both edit internal/validators/secrets/validator.go; the branch is now the single commit 366bc34 on top of main, and its diff is byte-identical to the original a04d1b2 (same git patch-id). The conflicts GitHub showed before the rebase were in internal/goldencorpus/corpus.go and metadata-key-census.txt, where this branch carried #786's additions as its own commits while main carried them squashed, so both sides inserted at the same anchor; the code did not conflict.

What was wrong

Stripe (sk_/pk_ × live_/test_), GitLab glpat- and Slack xoxb-/xoxp- tokens were matched as fixed-length windows with no boundary (sk_live_[0-9a-zA-Z]{24}, glpat-[a-zA-Z0-9_-]{20}, Slack tails of exactly 24/32) and typed by predicates locked to the same lengths (len == 32, == 26). No vendor publishes those lengths: Stripe documents prefixes only (its own CLI checks the prefix and "at least 12 characters"); a GitLab PAT is a routable token with a 27–300-character payload plus .vv. and 9 more; Slack says "the final section is the secret". So a longer token was reported as its first N bytes and redacted as its first N bytes, the rest written back as text.

Measured on v2.5.4 with the issue's repro.py: a 107-byte Stripe key kept 75 bytes in the clear in 9 of 11 realistic shapes, under every strategy, exit 0, empty stderr; a 51-byte GitLab routable token kept 25; a Slack token with a 34-byte tail kept 2. On a keyword line a second, generic finding over the whole value hid the leak. Separately, task_live_DashboardWidgetRefreshInterval and four identifiers like it were STRIPE_API_KEY at 70–100.

The same facts were hand-copied into five places (regexes, predicates, CalculateConfidence, containsSecretIndicators, GetCheckInfo), which is how they drifted.

What changed

One table. providerRules in internal/validators/secrets/provider_rules.go — type, prefixes, head (Slack's digit groups), body charset, minimum, mode, trailer, base confidence, help text — and everything else is derived: the regexes compileKeywordPatterns appends, the anchored predicate behind getSecretType and CalculateConfidence (providerTypeOf), the containsSecretIndicators admits, the --help lines. TestProviderRulesAreTheOnlySource fails on a keyword regex carrying a provider prefix that the table did not generate, a row missing from the compiled set, a prefix the pre-check does not admit, a described row absent from GetCheckInfo, or a predicate disagreeing with its row's mode.

Two modes, decided by the vendor. modeExact for AWS AKIA (16), GitHub gh?_ (36), Google AIza (35), Docker dckr_pat_ (36): the window is the key, the regex engine's fixed window is exactly that grammar, unchanged. modeAtLeast for Stripe (≥24), GitLab (≥20, then the optional routable suffix \.[0-9a-z]{2}\.[0-9a-z]{9}), Slack (tail ≥24 / ≥32): found by findProviderRuns, a hand-rolled scan, because two rules a regex cannot express are needed:

• Straddle cut. sk_live_<99>pk_live_<99> is one run of body bytes as far as the charset knows (_ stops it two bytes late); a regex would consume pk into the first key and never see pk_live_. After a run is taken, its last len(prefix)-1 bytes are checked for the start of any provider prefix that continues past the end; if one is there and the shortened token still meets its minimum, the run is cut there and the scan resumes at the cut. Only the tail is searched: a prefix wholly inside a run (sk_live_<20>AKIA<16>) is body bytes. Provider spans therefore never partly overlap.
• Stripe left-glue. A Stripe run whose prefix is glued to a word byte on its left, and whose body has no digit, is an identifier and is dropped (issued keys carry digits an author cannot remove; a random 24-byte body has none with probability (52/62)^24 ≈ 1.5%, the residual below). An escape tail — JSON \n, \t, \u0022, URL %22 — is not glue, and neither is the end of another provider token (two glued no-digit keys both report). Stripe-only, because the other AtLeast prefixes contain -, which no identifier is spelt with. The issue measured the alternatives: widening only the regex silenced 18 of 84 over-long cases, consuming a boundary silenced 33 of 180.

Minimums did not move, so sk_test_1234567890abcdef and sk_test_abc123def456 stay unreported. No row has a maximum: one may annotate a finding, never shorten it.

One consequence of derivation, stated: AWS_ACCESS_KEY's predicate now requires the 16 body bytes to be [0-9A-Z], as its regex always did (isAWSAccessKey was prefix + len == 20 with no class). A 20-byte entropy candidate with a lowercase letter after AKIA is typed generic instead of AWS. No corpus finding moved.

Docs corrected. The awsDocPlaceholderCap comment, the secrets README and CONFIDENCE_CONTRACT.md §7c all said the EXAMPLE rule is safe because the predicate "rejects" a 27-byte key. It never sees one: the window has already cut it to 20, and the key reports at full confidence, uncapped (TestAKIAWindowPins). The conclusion was right — appended bytes are never in the match — and now the reason is too, plus the precondition it depends on: TestProviderMarkerRuleOnlyOnExactRows fails if a type the rule names is ever moved off modeExact. The README's pattern list now says which rows have a vendor length. The "Too long AKIA prefix" unit test is annotated as exercising the predicate only.

The issue's reproduction, main@e8c6fe2 vs this PR

repro.py from #769, verbatim (13 files, --enable-redaction, format_preserving); "left" is the token tail still in the redacted copy.

shape	bytes	main	PR
Bearer, curl -u, log line, markdown, CSV, STRIPE_SK=, stripe_sk: (7)	107	STRIPE@100 (32), 75 left each	STRIPE@100 (107), 0
STRIPE_SECRET_KEY=<sk>	107	API_KEY_OR_SECRET@100 (107) + STRIPE@100 (32), 0	one finding, STRIPE@100 (107), 0
stripe.api_key = "<sk>"	107	STRIPE@100 (32) + generic@85 (107), 0	one, STRIPE@100 (107), 0
test-mode export STRIPE_API_KEY=, stripe.api_key = '…'	107	STRIPE@70 (32), 75 left	STRIPE@95 (107), 0
GitLab routable, prose	51	GITLAB@93 (26), 25 left	GITLAB@93 (51), 0
Slack user, 34-byte tail, prose	78	SLACK@94 (76), 2 left	SLACK@94 (78), 0

Hazard matrix (CLI, both binaries):

line	main	PR
five identifier decoys (task_live_…, helpdesk_live_…, risk_live_…, disk_test_…, kiosk_test_…)	STRIPE@70–100 (32)	NONE
task_live_RefreshIntervalMs500Default (digit in body)	STRIPE@100 (32)	STRIPE@100 (35), the pinned residual
x+key; key after JSON \n, \u0022, \t; after %22	STRIPE (32)	STRIPE (111), whole
two keys with ,, space, or nothing between	2 × (32)	2 × (111), both whole
sk_live_<24>sk_live_<24> glued, no digits	2 × (32)	2 × (32)
glpat-<20>glpat-<20> glued	2 × (26)	1 × (52): GitLab's charset admits - and letters, so two glued tokens are one run by the grammar; every byte is covered, none left in the clear (pinned as the stated shape)
xoxb-…-<24>xoxb-…-<24> glued	2 × (55)	2 × (55)
AKIA<16>XYZ, AIza<35>_backup, doc ID + 1 (@15), AKIA<16>EXAMPLE (@94)	window	identical

Golden corpus

UPDATE_GOLDEN=1 changed no existing snapshot except the metadata key census (which gains the STRIPE_API_KEY, GITLAB_TOKEN, SLACK_TOKEN types, 13 keys each, band 1): 0 findings added, 0 lost, 0 confidence changes. The corpus had no Stripe/GitLab/Slack case, which the issue noted; two are added:

• secrets_provider_token_lengths — 13 lines: the 107-byte key in the five un-saved shapes, the keyword line (now one finding), a quoted key, \u0022, %22 and glued-x escapes, two glued keys, a GitLab routable token, a 34-tail Slack token. 14 findings, every one whole; both redaction snapshots show no body byte. The Stripe values carry the publishable pk_live_ prefix: GitHub push protection refuses any sk_live_/sk_test_ + 24-alphanumeric value in a committed file, snapshots included (the first push was blocked on exactly those), and a publishable key is designed to be public. The four prefixes are one table row with the same charset, minimum, run scan, straddle cut and left-glue rule, so the golden pins the row; the three secret prefixes, test mode and the minimum lengths are exercised by the unit tests, whose values are built at run time and never written to disk.
• secrets_provider_prefix_decoys — the five decoys (empty), the digit residual (reported), both short placeholders (empty), AWS/Google windows with glued bytes, the doc ID + 1 at the 15 ceiling.

make score

PASS, identical to main: recall_all 1.0000, recall_hm 0.9906, prec_hm 0.9589, whole_leak 0.

Suppression (dimension 5), parent binary vs this one

Rules generated by v2.5.4 for exact-length tokens of all 15 patterns: identical hashes on both binaries (15/15), suppress 15→0 in all four generate/apply directions. Rules for over-long Stripe/GitLab/Slack tokens: generated by main, applied by this build, 3→3 — the findings resurface, whole, because the matched text changed; this is the intended, disclosed hash change. The CHANGELOG says to regenerate those rules.

Timing (dimension 6)

N lines of distinct 107-byte Stripe keys in prose, findings = N at every size: 5k 2.66/2.69s, 10k 5.31/5.36s, 20k 10.65/10.67s (main/PR), 2.0x per doubling, unchanged. One line of K glued distinct keys (the straddle path): 2k 0.61/0.60s, 4k 1.19/1.20s, 8k 2.35/2.39s, linear. New complexity target secrets_provider_runs (distinct glued over-long tokens of all three families, ≥1,500 findings at both sizes): 4.05x at 4x input, margin 1.98x against the 8.0 bound.

Tests, and that they fail on main

Test	Pins	On main
TestProviderGrammar	every row × 4 body lengths (Exact: exact + glued suffix) × 11 contexts: exactly one finding of the row's type with the promised text, nothing equal to or inside its span	needs the table (compile)
TestProviderRulesAreTheOnlySource	the derivation guard	needs the table
TestProviderRunHazards	decoys empty, residual reported, placeholders empty, escapes whole, adjacency whole, no partial overlaps, glued GitLab one run, routable suffix included	17 failures: every decoy, every escape, every adjacency, GitLab suffix
TestAKIAWindowPins	doc ID + 1 → 20-byte window @15; random + EXAMPLE → window, uncapped; glued suffix outside	passes on both (preserved behaviour)
TestProviderMarkerRuleOnlyOnExactRows	the EXAMPLE rule reaches only modeExact rows	needs the table
goldencorpus TestProviderTokensRedactWhole	Stripe/GitLab/Slack at min, +1, +8, +75 × 10 contexts × 2 strategies: no 8-byte substring of the body survives; AWS/GitHub exact as controls	the fixed-window leak

Suite: 88 packages ok; the one red is TestGrowthRatioMissesASparseQuadratic, the perfguard host flake (fails on unchanged main on this host 2/5; CI reads 4.6–5.0x). -race on secrets, goldencorpus, pkg/redact, cmd: ok. gofmt/vet/build clean; 7 formats + 2 redaction strategies snapshotted for both new cases.

Not in this PR (sibling audit items, as #769 scopes them)

More rows (rk_live_, ghs_ stateless format, ASIA, Slack digit-group relaxations), the header grammar, and the escape-tail catalogue beyond the three shapes measured here.

Test plan

-- changelog                       RAN — entry present under [Unreleased] (41 line(s)); required because it changes what a release 
1 gofmt                            RAN — clean
1 go vet                           RAN — clean
1 go build                         RAN — ok
2 full suite (-count=1)            FAIL — --- FAIL: TestGrowthRatioMissesASparseQuadratic (0.62s) FAIL FAIL	github.com/awslabs/ferret-scan/v2/internal/goldencorpus	27.861s 
3 golden regen                     RAN — 0 files changed
4 redaction                        NEEDS-MANUAL — scan a fixture with --enable-redaction for EVERY strategy; for xlsx/docx read the part INSIDE the zip (grepping the compressed bytes returns 0 and looks like a pass)
5 suppression                      NEEDS-MANUAL — generate with the PARENT binary, apply with YOURS — same-binary round-tripping passes even when the hash inputs changed
6 timing / O(n^2)                  NEEDS-MANUAL — scaling curve at N/2N/4N with a NON-VACUITY floor (assert findings > 0 at every size) and distinct values (identical repeats let a quadratic measure as linear)
7 all 7 formats                    NEEDS-MANUAL — text json yaml csv junit gitlab-sast sarif
8 web / CSP                        N/A — no web/template/asset file touched
10 non-vacuity                     NEEDS-MANUAL — new tests MUST fail on the parent commit; prove it by reverting the production change (or the guard) and re-running
11 cross-platform vet              RAN — linux darwin windows
12 flake                           FAIL — a repeat failed
13 race (whole module)             RAN — 89 packages ok
-- -short mode                     RAN — ok
-- tree clean                      RAN — this run dirtied nothing

(2 and 12 are the one host flake named above; 4, 5, 6, 7 and 10 are the sections above.)

[sectoramen]

@rustic this one is stacked on #786 (same validator.go); the first three commits are #786's and the review target is the last commit only. Once #786 lands I'll rebase and the diff collapses to it. A note for the record: the first push was blocked by GitHub push protection on the sk_live_/sk_test_-shaped finding texts in the new golden snapshots, so the corpus case uses the publishable pk_live_ prefix (same table row, same rules) and the secret prefixes are exercised by the run-time-built unit tests; the PR body explains.

[github-actions]

🔍 Ferret Scan Results

✅ No sensitive data detected in the changed files.

---

🤖 This comment was automatically generated by Ferret Scan

[sectoramen]

@rustic #786 is in, so this is rebased onto main: one commit (366bc34), diff byte-identical to a04d1b2 (same git patch-id). The conflicts GitHub flagged were the two goldencorpus files where this branch carried #786's additions as separate commits while main has them squashed; nothing in the code conflicted. Build, vet, goldencorpus and the validators packages pass on the rebased head; CI is re-running.

[github-actions]

🔍 Ferret Scan Results

✅ No sensitive data detected in the changed files.

---

🤖 This comment was automatically generated by Ferret Scan