Predicting the next vulnerability from a codebase's own fix history

Our last post was about attacking. Point a capable enough model at your own codebase and it finds real vulnerabilities, including the ones your scanners, your tests and your reviewers all missed. We spent a month doing that to ourselves, and it worked.

This post is about defending. If finding bugs that way is now cheap and repeatable for anyone, the useful question is whether you can predict where the next one will be, and get there first. Our answer starts with a record every project already keeps of its own mistakes: its history of security fixes, and the regressions that followed them. Below is the research that says this should work, the pipeline we built on it, and what it found when we pointed it at our own repositories.

The thesis and supporting research

A codebase’s own history of security fixes is the highest-signal map of where its next vulnerability lives. Bugs cluster in the same components, sinks and classes, and a fix on one path is usually missing on a sibling path.

Most security tooling treats a repository as a snapshot: here is the code today, here are the patterns that look dangerous. That discards the richest source of security information a project has, which is the record of every time somebody already got it wrong.

What matters in a fix commit is the shape of the mistake. Which invariant was violated, which sink was reachable, which guard was missing, and, the part that generalizes, which other call paths reach that same sink without the guard.

None of this is our idea. Four separate lines of work support the claim.

1. Defect clustering. Decades of defect-prediction research show that bugs concentrate in a minority of files, and that past fixes in a file predict future faults there better than size or cyclomatic complexity do. Ostrand and Weyuker’s study of a large industrial system found faults heavily concentrated in a small share of files (ISSTA 2002), and the follow-up with Bell built a predictor on that signal (IEEE TSE 2005). This is why hot components are weighted by recent fix count rather than by lines of code.

2. Variant analysis. Standing practice at Google Project Zero: once a bug is fixed, systematically hunt the same pattern everywhere else. Their 2021 year-in-review of in-the-wild 0-days found that a large share were variants of previously disclosed bugs, the same defect reachable through a path the original fix did not cover. It works, and it is mostly done by hand with bespoke queries written per bug class. What is newly available is automating the fingerprint-to-match step with a model instead of hand-writing the query.

3. Incomplete fixes are common. A large fraction of CVEs are follow-ups to an earlier, partial patch. CVE-2022-25365 following CVE-2022-23774 is a typical example. If the first fix was incomplete, the surrounding code is already known to be the right place to look.

4. Commit history plus a model is a productive substrate. The literature has converged on this. LLM4VFD reports large F1 gains over pre-trained-model baselines for fix detection by combining commit intent with retrieval of semantically-similar past fixes. CommitShield detects both vulnerability introduction and fix. Vercation pins the introduce-to-fix window with program slicing. PatchSeeker maps NVD records back to their fixing commits.

That last line is where the field splits. Most of the work measures the mining half: can a model tell a security fix from an ordinary bug fix? That is now well benchmarked. The hunting half, whether the mined map can find a new bug, is harder and much less benchmarked. That is the half we cared about, so it is the half we built verification for.

Two projects shaped the design directly. Pwno’s Rolling in the Diffs treats each commit diff as a model-sized unit of human intent and sweeps every one of them with sanitizer-gated proof, a discovery engine. Security Context compiles the set of past fix commits into a reusable map, a targeting layer. They are complementary, and neither closes the loop alone. Our pipeline implements both halves plus the verification step that the targeting half deliberately leaves open.

What we built

We built two things: a new piece of evidence, and the data pipeline that produces it.

The evidence is an AI Security Context. Analyze a repository’s whole history of security fixes, compile the result, and you get a machine-readable map of where that codebase tends to go wrong. In Chainloop it ships as a built-in workflow called AI Code Security Analysis, and the output is a signed CHAINLOOP_AI_SECURITY_CONTEXT attestation.

Because it is a first-party material, it lands beside everything else Chainloop already holds about that project: the SBOMs, the scan results, the attestations, the AI coding sessions behind the changes. It carries the same provenance and tamper-evidence as any of them. On its own the map is a useful record of what a project has had to fix, where, and what check made it right. Sitting next to the rest of the SDLC evidence, it is something a pipeline can act on.

The pipeline is strata. It walks a repository’s commit history, classifies every diff, hands the survivors to a deeper investigation, and compiles what it confirms into that map.

Three things follow from having that map.

• Warn people and agents before the change lands. Each confirmed fix records the invariant it established, in plain language. Once that map exists, a pull request can be checked against it: this diff touches a component the project has been burned in before, or reaches a sink that a past fix put a guard on, or breaks the sentence that fix established. Whoever is writing the code, a human or a coding agent, finds out before it merges.
• Find first-party vulnerabilities, not just known CVEs. Scanners find published CVEs in your dependencies and pattern matchers find what their rules describe. Neither finds a novel flaw in your own code, which is the class that causes incidents. The map turns that open-ended hunt into a targeted one.
• Measure security posture over time. Every confirmed fix is dated and classified, so the context also shows how quickly this team finds and patches things, how long bugs stayed exposed, and whether the risk profile is improving or drifting.

The first one arrives in the pull request itself:

The map, applied to a diff. It names the past fixes in the file being touched, the invariant each one established, and the guards to confirm are still on every path — including the 26 other entry points that share the surface.

Note what the first entry says: partial fix. The original patch forced two plugins onto a public-only client and left the generic webhook and Dependency-Track paths guarded only behind a setting. That is the incomplete-fix case from the research section, surfaced on a pull request while somebody can still act on it.

The third use is the whole history at once, since every confirmed fix carries a date, a class and a verified origin:

Posture over three and a half years: when flaws were introduced, how many sat open at once, and how long the median one survived before somebody fixed it.

A median exposure window of 277 days is not a flattering number. Two caveats printed on the panel keep it honest: only fixed flaws appear here, so this is resolved debt rather than live risk, and the two fixes without a verified origin are marked unmeasurable rather than counted as zero.

How it is built

strata runs in three phases across two commands, and is written in Go. It drives sandboxed Codex sessions over JSON-RPC for the agentic stages. It is internal, lives in our platform monorepo and is not something you can clone, but nothing about how it is assembled is secret. From the outside it looks like this:

# Illustrative — strata is internal, so these commands will not run for you.

# Phases 1+2 — mine history into a security context
strata scan  --last 200 --out security-context.json

# Phase 3 — audit the current code, seeded by that context
strata audit --context security-context.json

The two commands are deliberately not chained. The context is the expensive, reusable artifact; audits seeded by it are cheap and frequent. A convenience flag that used to run both in one process was removed, because it created a second Phase-3 call site that quietly disagreed with the real one about the working directory and never wrote its findings file.

Four ideas shape it, and they apply to any agentic audit pipeline. Two are visible in the walkthrough below: every phase has one job and a prompt written for that job, and phases hand each other typed JSON instead of sharing one long conversation. The other two are easier to miss:

• Model routing. A cheap model triages thousands of commits; a strong model adjudicates and audits the survivors. Spending frontier tokens on the 98.5% of commits that are refactors is where these pipelines waste most of their budget.
• Validation gates. Every finding must survive adversarial review and be reproduced before it is reported. Agentic audits generate plausible, well-written, wrong findings, and this is the step that catches them. We took both ideas from Harnessing Harnesses.

We wrote it in Go because a security audit is a long-running, concurrent, I/O-bound process: call a model, run tools, stay alive for minutes to hours. That fits Go’s runtime, and the audit loop is small enough to own directly.

The pipeline

A recall-first funnel: cheap and permissive at the top, expensive and precise in the middle, deterministic at the bottom.

These are the engine’s names for the stages. In the Chainloop product the same four steps are called Classify, Investigate, Compile and Hunt.

Phase 1 — triage

Goal: catch every security fix, cheaply, accepting false positives.

For each commit, a cheap classifier at temperature zero answers one question: does this diff fix a real, pre-existing, exploitable security vulnerability, or is it an ordinary bug fix, feature, refactor, test or docs change? One word back, YES or NO. No tools and no agent loop, because the model only has to read the diff and answer.

Two details in the input profile mattered more than the prompt did.

The classifier sees the diff, not the commit message. We strip it. This was measured rather than assumed: a routine-sounding subject line was observed to suppress a YES that the diff alone produces. Developers are often vague about security fixes, sometimes deliberately, and the diff is not.

Merges are diffed against their first parent only, and non-code files are excluded, so a merge does not drag an entire branch’s worth of unrelated change into one classification.

Oversize diffs are skipped rather than silently dropped. They go into a skipped list with their byte count, so the numbers still add up at the end.

Each flagged commit also gets a git patch-id, a hash of the diff hunks alone. Unlike the commit SHA, it survives a rebase, squash or clean cherry-pick, so a later scan can still recognize a fix it has already seen after somebody rewrites history.

At a base rate around 1.5%, this stage is tuned hard for recall, and precision is left to the stages below it.

Phase 2 — adjudicate

Goal: for each flagged fix, confirm it was real and emit a reusable fingerprint, or abstain.

Each survivor gets one sandboxed agent session with read-only access to the repository. It is not told the answer; it investigates, and it is free to conclude that triage was wrong. Most of the funnel’s cost lives here, and so does the false-positive gate: features, refactors, and hardening against something that was never reachable all get thrown out at this stage.

What comes out is a fingerprint, the structured description of the mistake:

• class and cwe — what kind of bug it was
• severity and fix_shape — how bad, and what the fix actually did
• invariant — the rule the code must hold, stated in one sentence
• anchors — exact file, line range and quoted span, on both the parent and the fixed revision
• fix_completeness and reachability_delta — was the patch complete, and did it narrow or widen what is reachable

The invariant field is the one worth the effort. It states, in one sentence, the rule the code has to hold. You can go looking for violations of a sentence like that. You cannot do the same with a CWE number.

Here is one from a fix in our own open-source repository:

The invariant is the reusable part. If a later change breaks that sentence, the fix is undone even though the commit is still in history.

This is the record that produced the pull-request warning further up: the hint on the diff was a projection of these fields.

Phase 2 does not wait for Phase 1 to finish. A commit goes to adjudication the moment triage flags it. That is safe because each session sees only its own commit and never another’s verdict, so the order they run in cannot change the result. Output order is handled separately, by sorting fingerprints by SHA before they are returned. On a real scan this saves the whole of Phase 1’s wall clock.

Compile

Goal: turn a pile of fingerprints into a map. Pure Go, deterministic, no model.

This is the step that produces the evidence. The compiler weights the fingerprints by recency and emits the AI Security Context: the components carrying the most risk, the attack surfaces that several fixes have in common, and how the fixes break down by class. That payload is a first-party Chainloop material with a published JSON schema, so it is validated and signed like any other piece of evidence rather than being a blob we happen to write.

Two properties of that schema are worth pulling out. Every array in it is sorted, so two scans over identical history with identical model output produce byte-identical artifacts, which is what lets you diff a context across scans and see what actually changed. And it stores facts rather than prose: there is no summary or agent brief in the payload, because each is a projection of the fields underneath it and is rebuilt on demand. What does live there is the model’s judgment in the places no template could reconstruct it, like the sink, the root cause, the invariant and the check hint.

Here is the shape, abridged, on getkin/kin-openapi from 776 commits scanned and 25 confirmed fixes:

{
  "top_risks": [
    {
      "component": "openapi3filter/validate_request.go",
      "severity": "high",
      "fix_count": 10,
      "classes": ["authentication", "information_disclosure",
                  "input_validation", "resource_exhaustion"]
    }
  ],
  "shared_surfaces": [
    {
      "surface": "Input Validation",
      "entry_points": ["... 16 paths that reach the shared sink ..."],
      "check_hint": "A request must not contain a schema property marked
                     readOnly, and a response must not contain one marked
                     writeOnly.",
      "support": 8
    }
  ]
}

Ten security fixes in one file, and sixteen entry points sharing one surface with a stated check to run at each of them. The two counts measure different things: fix_count counts the fixes that landed in that component, while support counts how many of them back this particular surface. That is a targeting list, derived entirely from the project’s own history, with no rule authored by us and no pattern database behind it.

Phase 3 — audit

Goal: find something new.

The audit is seeded with that context and pointed at HEAD. The brief is explicit that re-confirming an already-fixed issue is not a finding, while an unguarded sibling of one is.

The run is planned by the host, not by the model. Each hunter is given one attack class and one hypothesis to chase, instead of the model deciding for itself what to delegate and when.

The benefit is that coverage gets recorded. The run artifact says what happened to every class, so “nobody looked at this one” stays distinguishable from “we looked and it was clean”. Most classes go unhunted in any single run, and we would rather know that than read a short findings list as an all-clear.

A candidate is not reported as a finding until it has been reproduced, by probe, by traced reading or by fuzzing, and marked verdict=verified. Unverified candidates stay pending and are never counted as findings. Runs are bounded by a spend ceiling and a hard wall clock, and a run stopped by either keeps everything it found, marked as a partial.

The results

We ran it on ourselves first. Across four repositories, the scan funnel comes out like this:

Repo	Commits scanned	Triage candidates	Confirmed fixes
chainloop-platform	5,262	556	75
chainloop (oss)	2,363	331	58
agent-sandbox	790	77	29
getkin/kin-openapi	776	54	25

Phase 1 keeps roughly a tenth of what it sees, and adjudication cuts that by most of an order of magnitude again. The last column counts confirmed past fixes, the material the map is built from, not new vulnerabilities. Those come from Phase 3, further down.

Here is the open-source repository’s full history as it appears in the product:

The compiled map: which classes this project keeps having to fix, and which components keep needing the fixes.

Both columns point the same way. Access Control is the largest class by a wide margin, at 27 fixes — more than twice the next one — and the component at the top of the right-hand column is authz.go, with nine fixes of its own. The authorization layer is this codebase’s most-repaired code, and the map says so from history alone.

That is the thesis checking its own work. In the previous post we described three high-severity bugs that a month of agentic auditing found in our own code, and one of them was a broken-access-control path in exactly that authorization layer. The map knows nothing about that audit. It is built only from commits that were already in the repository, and it still ranks the same code first. Nothing in the ranking came from a rule we wrote; it is what this repository’s history says about itself.

The abstentions are worth showing too, because a stage that never says “I don’t know” is a stage that is guessing:

An inconclusive verdict with its reasoning attached. The adjudicator is free to conclude that triage was wrong, and to say when the evidence does not reach a verdict.

Does the history-mining earn its cost?

This claim needed a control, so we ran one. On getkin/kin-openapi, a complete audit returned 36 verified findings (12 high, 14 medium, 10 low). A general-purpose agentic auditor, one that sweeps the current code without any history grounding, returned 2 on the same repository. Our findings were dynamically reproduced, 24 by probe, 8 by reading and 4 by fuzz, and all carried verdict=verified.

Half of them, 18 of 36, were grounded, meaning they were seeded by the mined security-context.json rather than found by sweeping the code. A tool without commit-history mining cannot produce those.

The cost side: that run took roughly 4.6 times the wall clock (138 minutes against 30), about 3.5 times the tokens, and somewhere between $50 and $90 against $7.30. The benchmark is also n=1 complete, one repository and one full run, so widen it before treating any of those ratios as fixed. It does not settle the question.

The design goal from here is to keep the depth and halve the time and cost. Most of the volume sits in Phase 1, which is a classification problem rather than a generation one: it reads a diff and returns one bit. That is the shape TypeSafe’s System One models are built for. Their first model, Jev, gives up string generation in exchange for sampling structured outputs in parallel, and the published figures put it around two orders of magnitude faster and cheaper than a frontier LLM on this kind of task, with a calibrated confidence attached to every answer instead of a bare YES.

Both of those matter for a stage that has to look at every commit in a repository’s history and whose whole job is to sit at a chosen recall threshold. We are testing it.

It is in Chainloop now, in preview

The question the last post left open was where to point these models. Aiming a frontier model at every file in a large repository is expensive, and it tends to produce a report full of the word “potentially”. The fix history answers that: it is free, already in the repository, and specific to your code rather than to code in general.

AI Code Security Analysis is in preview now, and you can turn it on today.

You enable it per project and Chainloop does the rest. It is a built-in workflow template, so you pick it in the Create Project wizard or add it later from a project’s workflow settings (ai-code-analysis in the CLI). Chainloop clones the repository connected through GitHub or GitLab into a sandbox it manages and runs the whole thing server-side. No CI changes are required.

Those posture numbers are an aggregate, and every one of them drills down. Here is the single fix from earlier, traced back to the commits that introduced it:

All four commits that introduced the SSRF flaw, one per plugin, and the 3.3 years between the earliest of them and the fix. That span is one of the 63 behind the median above.

Four separate commits, added over two years, each extending the same unguarded pattern to another plugin. That is defect clustering and variant analysis in one panel. The fourth commit would have been a good place to ask whether the first three had a guard.

Every claim in the context is tied to a byte-exact span of real source, with a revision, file, line range and a hash of the quoted code, so anyone can re-check that the cited code says what the context claims. Spans that could not be verified are marked as such rather than quietly dropped. Scans are incremental, so a later run only walks commits it has not seen, and the context is queryable through Chainloop’s MCP tools, so an assistant can ask a project what it tends to get wrong.

A caveat worth repeating: Phase 3 reproduces a finding before reporting it, but reproducing a defect is not the same as proving it exploitable in your deployment. What lands in the project’s Security tab is a set of leads grounded in real history, pointing review at the highest-probability areas. You still record the formal decision there, affected or not, with justification and evidence attached, versioned and signed.

This is a preview feature and further changes are expected. The docs are here, and if you want a hand switching it on, talk to us.

Chainloop’s core is open source, and fixes from this work land there in public: github.com/chainloop-dev/chainloop.

References

Defect clustering

• T. J. Ostrand and E. J. Weyuker, “The distribution of faults in a large industrial software system”, ISSTA 2002.
• T. J. Ostrand, E. J. Weyuker and R. M. Bell, “Predicting the location and number of faults in large software systems”, IEEE Transactions on Software Engineering, 2005.

Variant analysis

• Google Project Zero, “The More You Know, The More You Know You Don’t Know”, a year in review of 0-days exploited in the wild, 2022.

LLMs over commit history

• “Code Change Intention, Development Artifact and History Vulnerability: Putting Them Together for Vulnerability Fix Detection by LLM”, the LLM4VFD framework: commit intent plus retrieval of semantically-similar past fixes.
• “CommitShield: Tracking Vulnerability Introduction and Fix in Version Control Systems”.
• “VERCATION: Precise Vulnerable Open-source Software Version Identification based on Static Analysis and LLM”.
• “PatchSeeker: Mapping NVD Records to their Vulnerability-fixing Commits with LLM Generated Commits and Embeddings”.

The two projects that shaped the design

• Pwno, Rolling in the Diffs, the discovery engine: sweep every diff, gate on sanitizer proof.
• Security Context, the targeting layer: compile the set of fix commits into a reusable map.

Harness design

• A. Gill (ZephrSec), Harnessing Harnesses — Climbing the LLM Hills, on staged work, typed artifacts, model routing and validation gates.
• Zep, Agentic Development in Go, on why a long-running, concurrent, I/O-bound agent fits Go’s runtime.