How Two-Layer Scanning Cuts False Positives

Every secret scanner has the same problem, and it is not detection. It is confidence. Finding strings that look like credentials is cheap. Deciding which ones matter is what determines whether a report gets worked or ignored.

KAIKI runs two layers for exactly that reason. This is what each one does, and what the combination changes about the report you read.

Layer one: patterns, because they are fast

The first pass is regex across the whole repository. More than fifty patterns cover cloud providers, payment services, communication tools, databases, version control, and AI providers, including the newer token formats that older rule sets miss.

This layer runs in milliseconds per file and is deliberately generous. Missing a real credential is far worse than passing an extra candidate to the next stage, so the pattern layer errs toward flagging and lets the second layer decide.

Why we do not stop there

A generous pattern layer produces a long list, and most of that list is not actionable. The same forty-character shape appears in a test fixture, in a README example, in a mocked API response, and in a production config file imported at startup.

Report all four with equal weight and you have taught the developer to skim. That is how scanners get switched off: not because they were wrong, but because they were indiscriminate.

Layer two: read the code around the match

Each candidate goes to a KAIKI Alpha model together with its surrounding context. The model does not have to find secrets, because layer one already did. It answers a narrower question: given this code, is this value a real credential, and what would happen if it were public.

Narrow questions are where models are reliable. There is no open-ended search, no whole-repository reasoning, and no dependence on the model noticing something a regex missed.

Signals that push a finding down

• The file is a test, fixture, or example and the value is asserted or illustrative.

• The identifier is named placeholder, sample, dummy, or example.

• The literal is a fallback for an environment variable normally set at runtime.

• The value is malformed or truncated in a way real credentials are not.

Signals that push a finding up

• The value sits in a config file imported at application startup.

• It appears in a committed .env, Dockerfile, or Terraform variable file.

• It is a cloud provider or payment credential, where blast radius is wide.

• It is well formed and matches a provider checksum or prefix convention.

The measure of a scanner is not how many findings it produces. It is how many of them you act on.

Severity is the output that matters

The second layer produces the rating: Critical for live production secrets that need rotation now, High for sensitive credentials or committed configuration, Medium for values worth investigating, Low for hygiene issues and recommendations.

Each finding also carries the file, the line, the credential type, and a remediation suggestion, so the report reads as a queue of tasks rather than a list of complaints. Knowing a value is a Stripe restricted key rather than a generic string tells you which console to open before you finish reading the line.

What this costs, and what it buys

Adding a model pass costs time and tokens on every candidate, which is precisely why the pattern layer stays. Running a model over an entire repository would be slow and expensive; running it over a few dozen candidates is neither, and it happens while you are still looking at the screen.

What it buys is a report a developer will finish reading. In practice that changes behaviour more than any single detection rule, because a short accurate list gets worked the same day.

What the model is not asked to do

It is worth being precise about the boundary. The model does not browse the repository, does not decide which files to open, and is never the only thing standing between a credential and a missed finding. If layer one flags a candidate, it reaches a human report one way or another.

That boundary is what makes the design predictable. A pattern layer with a fixed rule set gives you repeatable recall, and a bounded question per candidate gives you triage that behaves consistently across languages and file types.

Where it still gets things wrong

Two cases remain genuinely hard: a real credential deliberately written to look like a placeholder, and a fixture value that happens to be a valid sandbox key. Both are rare, both are visible in the finding explanation, and both are why every rating ships with its reasoning rather than a bare label.

Try it on a repository you know

The honest way to evaluate any scanner is to point it at code you understand well, including its history, and count the findings you would act on today. Then count the ones you would dismiss without reading twice. The second number is the one that decides whether the tool survives contact with a busy week.

KAIKI is free during early access. Connect a public repository, run a scan, and compare the result against what you already believe about your own code.