Regex Alone Cannot Find Your Secrets

Every secret scanner starts with pattern matching, and every scanner that stops there produces the same result: a long list of findings, a developer who skims it once, and a check that gets switched off two sprints later. The detection problem is largely solved. The judgement problem is not.

Why patterns are the right place to start

Many credentials have a predictable shape. An AWS access key ID has a known prefix and a fixed length. Stripe secrets, GitHub tokens, Slack webhooks, and most cloud provider keys follow documented formats with checksums or recognisable structure.

Regex over a whole repository is also cheap. It runs in milliseconds per file, needs no network, and never gets tired. Any serious scanner should keep this layer, and should keep it generous, because missing a live credential is far worse than passing an extra candidate to the next stage.

Where pattern matching stops being useful

A pattern can tell you that a string looks like a credential. It cannot tell you whether the credential is real, whether it is still valid, or whether anything reads it. The same forty-character shape appears in a test fixture, in a README example, in a mocked API response, and in a production config file that is imported at startup.

Three of those are noise. The fourth is an incident. Reported with equal weight, they teach the reader that findings are not worth opening.

The failure mode is volume, not blindness

Regex-only scanning rarely fails by finding too little. It fails by finding too much, and burying the handful of real problems under fixtures, samples, and rotated keys that were never cleaned up.

What context adds

A model that reads the code around a match can see the things a pattern cannot:

• The file is a test and the value is asserted against a fixed expectation.

• The identifier is named example_key and sits inside a documentation block.

• The literal is a fallback for an environment variable that is normally set at runtime.

• The string is loaded by a config module that the application imports on boot.

• The credential belongs to a provider whose blast radius covers the entire account.

None of that is visible in the matched string. All of it changes what you should do next.

A scanner that reports forty issues where three matter is not more thorough. It is less useful, because it spends attention it did not earn.

Two layers, not one

The practical architecture keeps the fast layer and adds judgement on top. Patterns produce candidates; a language model reviews each candidate with its surrounding code and assigns a severity and a remediation step. You keep regex recall and gain something close to human triage.

The cost is real: a model pass per candidate takes time and tokens. That is exactly why the pattern layer stays. Running a model over an entire repository would be slow and expensive, while running it over a few dozen candidates is neither.

The cost of a noisy report

False positives are not a cosmetic problem. They have a measurable effect on behaviour: the first noisy report gets read carefully, the second gets skimmed, and by the third the team has learned that findings are someone else problem.

Once that happens, the scanner is worse than nothing, because it provides the appearance of coverage while the real findings scroll past unread. Every additional rule that fires on test fixtures makes the tool marginally less likely to catch the credential that matters.

Tuning is not the answer either

The usual response is an allowlist: ignore this path, ignore this pattern, ignore this file extension. It works for a month and then hides a real finding, because the exception was written for a directory layout that has since changed. Judgement applied per finding ages better than rules applied per path.

How to evaluate any scanner in an afternoon

1. Point it at a repository you know well, including its history.

2. Count the findings you would actually act on today.

3. Count the findings you would dismiss without reading twice.

4. Check whether the ones that matter appear at the top, not on page three.

The second and third numbers matter more than the total. A report you finish reading is worth more than a report that is technically complete.

KAIKI runs both layers by default: more than fifty detection patterns for recall, then a context pass that rates each candidate and explains the reasoning, so the list you get is a work queue rather than an inventory.