AI Coverage Closure: What Actually Closes Bins

Milan Kubavat · · -- min read · AI, AI Coverage Closure, Coverage, Verification,

Every claim in the vendor decks is true, and most of them are not about closing coverage. That is the puzzle this post untangles. AI tools for coverage closure are real, deployed, and in a few verified cases spectacular — one production SoC interconnect went from a 79.91% plateau to 100% functional coverage at 18x less compute. But the marquee numbers you have seen — 16x, 10x, 5x — are almost all compression results: the same coverage, reached cheaper. Closing new coverage is a different problem, the published wins at it are rarer and more conditional, and the conditions are the interesting part. This post maps the manual closure loop you already run, sorts the AI offerings into three tiers by what they actually touch, walks through the verified production results, and then spends equal time on what none of them automate — because the honest boundary of these tools is exactly where your judgment still lives.

The Last-Mile Problem

Start with the loop as every DV team actually runs it, stated with unusual candor by an NVIDIA team at DVCon: identify covergroups, code the bins, run regressions, generate reports and analyze, "modify constraints manually and rerun the regressions," and — step six — "repeat steps 4 and 5 until we hit 100%." Behind that "repeat" sits the whole machinery: merge the coverage databases (urg, imc, vcover merge), rank the tests, triage the holes, categorize each one — needs a new test, needs a constraint tweak, genuinely unreachable, or waivable — write exclusions, get them reviewed, rerun. The loop is not broken. It just has a cost curve that turns hostile precisely when you need it most.

The best production dataset ever published on that curve comes from a Samsung + Cadence paper on a mobile application processor's SoC interconnect — hundreds of masters, near a thousand slaves, hundreds of thousands of functional coverage elements (DVCon US 2025). One regression pass: 274 runs, 1,744 CPU-hours, 26.74% coverage. Twenty iterations: 5,480 runs, 33,751 CPU-hours, 94.84%. The next four iterations — 1,096 more runs, 7,408 more CPU-hours — bought 1.43 points. Do the division and the efficiency collapse is about 15x: the first 94.84% cost roughly 356 CPU-hours per coverage point; the tail cost about 5,180. After 41,158 CPU-hours, 4,478 bins were still open.

A software engineer will recognize the shape instantly: it is a feedback loop with exponentially decaying reward. Constrained-random stimulus re-hits the easy bins the way CI re-runs already-passing tests — every regression pays full price to reconfirm what the last one proved, and the probability of landing on an unhit bin shrinks as the unhit set does. Infineon quantified the redundancy on a production radar DSP in their Aurix line: closing coverage took on the order of two million tests, a thousand machines and licenses running nearly continuously for six months — and an industry-standard ranking algorithm afterward showed that about 3,000 of those tests would have sufficed to hold 100% coverage (DVCon US 2021). Over 550 tests simulated for every one that added coverage.

This is the pressure the AI tools are selling into, and the pressure is real. The 2024 Wilson Research Group study — still the latest as of this writing — has first-silicon success at 14%, the lowest in two decades of tracking; 75% of IC/ASIC projects behind schedule; design engineers spending 49% of their time on verification. Nobody needs convincing that the loop is expensive. What needs examining is which part of it each tool actually touches.

Your Regression Is 30–60x Bigger Than Its Coverage

Before any AI enters the picture, understand what plain test ranking already proves, because it is the baseline every ML claim should be measured against — and in the one independent study that did measure, the baseline nearly tied.

Ranking is the greedy algorithm your coverage tools have shipped for years: given a merged, test-associated coverage database, select the minimal set of test-seed pairs that reproduces the merged coverage. Infineon ran it across three real projects with Cadence (DVCon) and the numbers are startling: a microprocessor IP regression of 260 runs compressed to 8 runs at 100% coverage regain — 32.5x. A mixed-signal SoC's 5,124 runs compressed to 1,204, again at full regain. Your regression is 30–60x larger than its minimal coverage-proving subset, and the tooling to prove it is already in your license.

Two caveats turn this from a party trick into engineering judgment. First, ranking hits exactly the bins the original regression hit — never one more. It is compression by construction, and a compressed regression cannot close a hole. Second — and this is the finding teams skip past — the same study showed the compressed regressions changed the failure profile: one stage went from 3 failing runs in the original to 23 in the optimized set. The redundant tests were not worthless; they were soak. The coverage-proving subset and the bug-finding regression are different objects, and a tool that optimizes the first while you silently assume it preserved the second is how a "more efficient" flow ships a bug.

When the same study benchmarked Cadence's Xcelium ML against this non-ML ranking baseline, the result was deflationary and useful: "Both Xcelium ML and Ranking methods gave comparable compression & speedup factors around 3 consistently" — with ranking sometimes compressing more. The ML tool's one structural advantage: its regenerated regressions occasionally exercised genuinely new scenarios, regaining more than 100% of the original coverage (101–108% in several configurations) — something ranking cannot do by construction. Hold that thought; it is the entire difference between the next two sections.

One more piece of the baseline: hole triage has five buckets, not four. Needs a new test; needs a constraint or seed change; genuinely unreachable (route it to formal, not to a human); waivable with review; and — the bucket most flows don't have — the coverage model itself is wrong. A Samsung Memory team measured that last bucket on a production cache-managing IP: 14.1% of their coverage holes were bins that were never defined at all — the initial hand-written model covered only 53.5% of the true bin space (DVCon). Keep that number in mind when a tool promises to close your holes. Some of your holes are not holes.

Three Tiers of "AI Coverage" — Read the Fine Print

Every commercial "AI coverage" offering does one of three things, and the tier determines what the tool can possibly deliver. Sorting the market this way is the single most useful filter you can apply to a vendor deck.

Tier A — selects or ranks existing tests. Same stimulus pool, fewer cycles. AMD's SNUG result with Synopsys VSO.ai — 1.5–16x fewer tests to the same coverage across four designs — lives here, as do Renesas's 2.2x/3.6x Xcelium ML compressions, VSO.ai's regression-ROI ordering, and the change-based smoke-suite selection Intel presented in 2026. A Tier A tool, by construction, cannot close a coverage hole. It can only make the coverage you already reach cheaper — which is genuinely valuable, and is where nearly every marquee number comes from.

Tier B — steers the constrained-random distribution. The tool reaches into the randomization kernel or constraint solver and re-weights what stimulus gets generated, from the same testbench and the same constraints. Xcelium ML's accelerated-closure mode, Cadence's Verisium SimAI, and VSO.ai's in-simulator coverage-directed solving live here. Tier B can hit bins that plain random rarely reaches — this is where the real closure results live — but it cannot reach anything your constraints exclude. It explores your legal space more cleverly; it does not enlarge it.

Tier C — authors new verification artifacts. Tests, sequences, properties that did not exist before. As of mid-2026 no dedicated coverage-closure product from the big three is in this tier. What is here: the research wave (agentic property generation, LLM testbench synthesis) and the vendors' new agentic layers — Cadence's ChipStack, Synopsys's AgentEngineer, Siemens's Questa One Agentic Toolkit, all announced within a single month in early 2026, all early-access, all pitched at bring-up and productivity rather than last-mile closure.

The reader's rule that falls out: for every number in a vendor deck, ask did coverage go up, or did the same coverage get cheaper? Renesas's split is also worth carrying with you — 3.6x compression on a derivative design versus 2.2x on the original, because the ML had regression history to learn from. These tools eat your data, and a team with deep regression archives has a moat a fresh project does not. As one DAC 2026 wrap-up put it, the EDA companies build the tools, but the training data belongs to the companies building chips.

What Actually Closed New Coverage

Now the wins that survive fact-checking — each one primary-sourced, each with its condition attached.

The Samsung + Cadence interconnect result is the strongest closure number in public. On the second covergroup category — the one where the traditional flow plateaued at 79.91% after 41,158 CPU-hours — the SimAI-guided flow reached 100% in 2,261 CPU-hours: 18.2x less compute and 20.09 points more coverage, from the same testbench. On the first category, where the baseline had already clawed to 96.27%, the gain was 4.92x. Read those two numbers together: the advantage was largest where the tail was worst. The tail is where these methods earn their keep, not where they fail. ("Up to 18x" is doing real work in the abstract, though — the two categories are the whole spread.)

A second Samsung team reported the same shape on a production camera-interface IP: 134 cross covergroups, 14,616 bins; the ML-driven flow reached 100% in about 75,000 tests while the baseline regression "fails to exceed 70% coverage rate, even after more than 100,000 runs."

Infineon's novelty-driven selection is the best-documented Tier A result: an autoencoder ranks candidate tests by reconstruction error — novelty, in effect anomaly detection pointed at your own stimulus — and simulation order follows novelty. 60% fewer tests to reach 99.5% coverage, still 40% fewer at 99.95%, projecting the six-month closure campaign to under three months. Two honesty flags the paper itself carries: the time saving is a projection from an offline replay of 85,470 already-generated tests, not a measured deployment; and the savings decay from 60% to 40% exactly in the final half-percent, where the hard bins live.

NVIDIA's VSO.ai deployment — designs exceeding 100 million coverage targets — reported 33% more functional coverage in the same number of runs, alongside a 5x regression-suite reduction. Their published flow combines test grading, formal unreachability analysis, and VSO.ai; the often-quoted 17%-more-coverage-at-3.5x-compression figure belongs to that combined flow, not to the AI tool alone. Their lesson learned is also on the record: the tool was first tried late in a project, and the vendor now recommends deploying at early milestones, while stimulus is immature — late-stage deployment underperforms.

On the research side, one result is worth more than all the benchmark tables: LLM4DV, the Cambridge/Imperial/lowRISC benchmark for LLM-driven stimulus generation (FCCM 2025). In its 2023 version, GPT-3.5 managed 5.61% coverage on the hardest DUT, an Ibex CPU. In the current version, on identical scaffolding, Claude 3.5 Sonnet reaches 100% on that same CPU — against a constrained-random baseline of 15.31%. Across the benchmark the per-model spread runs from 7.93% to 98.84% on the same design with the same framework. The framework was never the binding constraint. The model was. Whatever you concluded about LLM stimulus generation from a 2023-era evaluation, the conclusion has a shelf life measured in model releases.

The Catch: A Human Still Writes the Targets

Here is the pattern connecting every win above, and it is the thesis of this post: the automation is in reaching specified targets efficiently, not in discovering what to target. The coverage model and the scenario list remain human artifacts, and every failure mode in this section is a way of forgetting that.

The Samsung + Cadence paper says it outright: the DV team supplies the target scenario specification, and "if the target specification is incomplete… the proposed approach may not hit the bins even though all the test scenarios of the provided specification are satisfied." Deriving the specification automatically is listed as future work. The 18x result is a solver being steered brilliantly toward targets a human enumerated.

Nokia and MathWorks hit the boundary from the other side (DVCon Europe 2023). Their autoencoder test selection cut tests-to-closure by up to 43% — but the coverage goal was capped at 67%, "the maximum possible for the current test randomization constraint configuration." No selection strategy, however intelligent, could touch the remaining third, because the constraints excluded it. Selection cannot fix a constraint problem. (The paper's abstract says "2x speedup"; its own results section reports that wall-clock regression time was not improved — doubled, in the worst case — because each iteration restarted the simulation environment. Cite this paper carefully.)

The Samsung missing-bins result gives the model-side version: 14.1% of holes were bins nobody wrote. An AI aimed at your holes is aimed at the defined bin set. The gap between the bins you wrote and the bins you should have written is invisible to every tool in this post — as the Verilab crew put it in the best paper ever written on coverage quality, "if something is missing from the model, it does not appear as a coverage hole, it's simply invisible." Their companion warning belongs on a wall: functional coverage modeling "is essentially a software task. As such, models will likely have bugs… Given the trust we put in functional coverage results for tapeout decisions, this is an oddly overlooked requirement." Mark Litterick's Lies, Damned Lies, and Coverage names the failure taxonomy — deception, omission, fabrication — and the reason coverage bugs survive: "if you make a mistake in stimulus or checks, there is a good chance you will kill the regression; if you mess up coverage there is no comeback." Goodhart's law — when a measure becomes a target it ceases to be a good measure — comes from economics via the software-testing literature, but DV built its own sharper version first.

Now put an AI in that loop and watch the metric detach from the goal. Infineon's agentic formal-coverage work (arXiv:2603.03147) is admirably honest about what happened: LLM agents read Jasper coverage reports, characterized the uncovered RTL, and generated new SVA properties, lifting formal coverage by roughly 10–20% across five designs. And: "in some cases… the proven rate for generated properties decreased after the coverage agents' workflow, even though overall coverage increased." The coverage number went up while the fraction of properties that could actually be proven went down — and the published results ran with no human review in the loop. That is the metric improving while assurance does not, measured and self-reported.

The sharpest 2026 datapoint on where AI stimulus generation actually stops comes from a hole-by-hole taxonomy of everything an agentic flow failed to close across 19 designs (arXiv:2604.15657). Under 7% of the residual holes were genuinely unhittable — tied-off integration logic, defensive dead code, infeasible boundaries. About 92% were reasoning frontiers: multi-module pipeline warm-up sequences (49.9% of the frontier bucket) and protocol sequencing (40.2%) — holes that require building a Wishbone burst model or an MDIO responder to reach. The authors' key finding: "the agent correctly diagnoses these problems, [but] it fails to implement the solutions." The agent can read a coverage report and explain the hole like a staff engineer; it cannot yet write the protocol machinery to hit it. Diagnosis has been automated. Generation, at protocol depth, has not.

So the reformulated claim that survives all the evidence: AI earns its keep on tail bins that are reachable and correctly specified. The unreachable and the unenumerated stay yours.

Formal UNR: The Workhorse With Its Own Wall

The least glamorous automation in this story predates the AI wave and out-delivers most of it. Unreachability analysis takes your partial coverage database plus the RTL, formally proves which uncovered targets cannot be hit under any stimulus, and emits an exclusion file back into your coverage flow — Jasper's UNR app, VC Formal's FCA invoked natively from the VCS shell, Questa CoverCheck. A Questa team's DVCon tutorial reported a PCIe-bench UNR run of three hours that saved an estimated three weeks of manual analysis; Synopsys's blog cites Cisco seeing a 9% coverage improvement from pruning noise. One design in that same tutorial had over 3,000 unreachable coverage elements — at even 15 minutes of human triage each, that is 4.5 person-months of analysis a formal engine did before lunch.

Two things keep this section honest. First, nobody has a consistent answer to "how much does UNR save": Synopsys's own materials claim 40–80% verification-effort savings in one publication and 8–80% in another. The technique is real; the aggregate number is marketing. Second, a Qualcomm engineer's account from a VC Formal SIG punctures the assumption that UNR is a solved deployment: on their larger configuration — over 200 million coverage goals — the analysis produces claims at a scale where "millions… may need manual review by design experts, an impractical expectation only manageable through engineer-defined blanket exceptions which undermine the integrity of the analysis." And the exceptions don't port across configurations or even successive RTL drops. UNR "still is not as mainstream as you might imagine."

One reframe worth stealing from Synopsys's FCA documentation: an unreachable target you expected to be reachable is not a waiver candidate — it is a design bug wearing a coverage costume. And note that UNR-versus-ML is a false rivalry: NVIDIA's published flow runs test grading, UNR, and VSO.ai together. The formal engine prunes the impossible; the ML steers toward the merely improbable.

Exclusions Are Code

Every closure flow — manual, formal, or AI — terminates in the same artifact: an exclusion list that redefines what 100% means. Treat that artifact with the same rigor as RTL, because it carries the same tapeout risk. The best public, enforced example is OpenTitan's DV methodology:

  • Every exclusion carries a standardized annotation prefix — UNR, NON_RTL, UNSUPPORTED, EXTERNAL, LOW_RISK — making the exclusion base greppable and auditable. LOW_RISK is the honest bucket: a named, reviewed "we chose not to chase this," instead of a silent one.
  • Designers sign off on exclusions in PR review — the person who wrote the RTL certifies the code is genuinely uncoverable, not the person whose schedule benefits from the waiver.
  • "If any RTL changes happen to the design after the coverage exclusion file has been created, it needs to be redone and re-reviewed." Exclusion work starts only after design freeze, precisely because of this rule. The Questa tutorial gave the failure mode a name worth adopting: waiver rot — manually generated waivers "have to be maintained as the code changes."
  • Coverage is one line item among many: the V3 signoff gate also requires all assertions proven, no unreachable properties, and a nightly regression 100% passing with a week of soak. Closure is a portfolio, not a number.

The AI angle lands directly here. At least one new-entrant "coverage agent" product's headline capability is recommending exclusions with supporting evidence. Read that plainly: it closes coverage by removing targets. That may be exactly right — a good UNR-plus-triage assistant is valuable — but an AI-recommended exclusion must enter the same gate as a human one: annotated, designer-signed, invalidated on RTL change. An agent that can edit your exclusion file has write access to the definition of done.

Build Your Own Loop: The API Surfaces That Matter

Suppose you want the loop the papers describe — a model reading coverage, choosing what to run next — without waiting for a product. What can you actually build against, today? The answer has a clean structure, and it starts with zero APIs at all.

The loop you can write this afternoon lives entirely inside IEEE 1800. Every covergroup, coverpoint, and cross has a get_coverage() method, and pre_randomize() is the standard-blessed hook that runs before every randomize() call. Put them together and the testbench biases its own stimulus toward whatever is least covered:

function real calc_weight(opcode_t op);
  real cov;
  case (op)
    nop_op:  cov = covunit.cg.op_nop.get_coverage();
    load_op: cov = covunit.cg.op_load.get_coverage();
  endcase
  return (100 - cov) * 0.5;   // colder coverpoint => heavier weight
endfunction

function void pre_randomize();
  weight_nop  += calc_weight(nop_op);
  weight_load += calc_weight(load_op);
endfunction

No vendor dependency, works on all three simulators, and it is a real coverage-feedback controller — a proportional controller, in control-theory terms. It also teaches you the standard's load-bearing limitation by running into it: the LRM gives you coverage percentages, never bin identities. §19.9 defines exactly three covergroup system tasks ($set_coverage_db_name, $load_coverage_db, $get_coverage) plus the get_coverage methods; there is no standard way to ask which bins are empty from inside a running simulation. Everything more ambitious than proportional weighting needs the coverage database — which means it happens between runs, not during them.

The coverage database is the real interface, and one vendor documents it. Siemens publishes the full Questa UCDB C API — a 200+ page reference with worked examples shipped in the install — and the UCDB test record turns out to already be an ML training record: it stores the test name, the seed (verbatim from -sv_seed), a command field the docs describe as capturing "knob settings for parameterizable tests," CPU and simulation time, and pass/fail status. Stimulus knobs, seed, cost, outcome — the (action, cost, reward) tuple, persisted by the tool you already run, surviving merges. The Tcl layer above it is equally direct: vcover merge -testassociated (nothing per-test works without it), then coverage analyze -select cover -eq 0 — hole extraction as a query — then coverage ranktest, which emits ranktest.contrib and ranktest.noncontrib: machine-parseable lists of contributing and redundant tests, which is to say, labeled training data for a compression model, generated by a shipping tool. There is even a sanctioned place for an agent to stamp its own metadata into the database: coverage attribute -trendable.

The other two vendors gate their equivalents — Synopsys's coverage C API manual is marked Confidential; Cadence's IMC reference and vManager REST API live behind support portals. The practical trick: open-source consumers are the real documentation. OpenTitan's production fpv.tcl is a better JasperGold coverage reference than anything public from Cadence (check_cov -init before design load, -measure with a time limit, -report to parseable output — and waivers are just a Tcl file, which means an agent's exclusions are just a file it writes, subject to the previous section's gate). The Jenkins vManager plugin documents the vAPI REST surface by using it. And imc -execcmd "help report" makes the tool document itself.

On machine-readable output, the ground truth is humbling. The VCS manual documents exactly two urg output formats — text and HTML; zero occurrences of XML, JSON, or CSV. Cadence's flow, per OpenTitan's production scripts, emits text summaries and HTML (plus a native rank command). Across the industry, the practical egress contract for coverage data is parsing text reports — which retroactively explains the most striking pattern in the published work: neither industrial AI-closure paper used a vendor API. Nokia shelled out to ExecMan and parsed results; Infineon's agents shell out to Jasper and parse reports. The standard that was supposed to fix this — Accellera UCIS — was ratified in 2012 and never revised; its working group is inactive, no vendor publicly documents a conformant UCIS shared library, and the most credible open implementation (pyucis) had to write its own. The working loops route around the standard: Infineon's ISCAS 2025 flow skipped database extraction entirely and had the PyVSC coverage callback in a PyUVM monitor append stimulus values and bin hit/miss flags to a CSV during the run. Their stated reason is the thesis of the open-source path: "PyUVM testbenches offer a significant advantage in data collection compared to SystemVerilog-UVM testbenches." Python's edge is not nicer constraints. It is data egress — the testbench already lives in the language the ML lives in.

For tools with no API at all, two bridge patterns cover everything. Inside the simulation: DPI-C plus a TCP socket — both published RL closure loops (a DQN closing a compression encoder's CAM bins, and Infineon's Gym-based agents) independently built the same bridge, SystemVerilog importing a DPI function whose C side talks to a Python agent over a socket. One discipline point if you try it: the foreign call blocks the simulator, so make decisions in pre_randomize() or between transactions — a model consulted inside a sampling path perturbs timing and breaks seed-stability. Outside the simulation: a Tcl socket listener running inside any Tcl-shelled tool (vsim, ucli, Jasper, IMC) with a thin MCP server outside — a pattern already demonstrated by a third-party Xcelium debug server, and generalizable to every tool in your flow. No vendor cooperation required.

And the economics lesson that decides whether any of it pays: Nokia's loop reduced simulated tests by 43% and still failed to improve wall-clock time — worst case, regression time doubled — because every iteration tore down and restarted the simulation environment. The integration point that determines whether an AI loop is economical is not the coverage API. It is whether you can keep a warm simulator between decisions. Design for that first.

One boundary to scope your ambitions honestly: the commercial Tier B tools plug into the constraint solver and randomization kernel — a layer of the stack with no public API on any simulator. You cannot build VSO.ai or SimAI from outside. Everything upstream of the solver (test selection, knob choice, seed allocation) and downstream of it (hole extraction, triage, ranking, property generation, exclusion drafting) is buildable today with what this section named.

Piloting Without Getting Burned

If the post has a single operating principle, it is Mike Bartley's: separate productivity gains from assurance gains. A tool that halves your regression bill has improved productivity; whether your verification got better is a different question with different evidence, and the Infineon proven-rate result shows how easily a coverage metric can rise while assurance falls. His pilot guardrails compose into a checklist that fits on an index card: bound the pilot's scope; define success in engineering terms before deployment; version your datasets with traceability to DUT revision — "if the regression environment cannot reliably map failures to DUT revision, scenario, configuration, and known bug state, the model will learn noise"; route generated artifacts (tests, properties, exclusions) through the same review path as human-authored ones; and keep sign-off authority human.

Add the deployment lessons the case studies paid for: deploy at early milestones, not late (NVIDIA's late-project trial underperformed; the vendor now says so). Expect results proportional to your regression history — the derivative-design effect is real, and a team with years of archives will see numbers a fresh project will not. And run the trial on your design with a defined baseline, because the independent-evaluation landscape is thin enough to be its own finding: for the most heavily marketed tool in this space, every public number still routes through the vendor, and the one independent multi-project study found the ML tool roughly tied with plain seed ranking on its headline metric. That is not a reason to skip these tools. It is a reason to measure them the way you would measure anything else you were about to trust with tapeout: on your data, against your baseline, with the metric and the goal kept honestly apart.

The last mile of coverage closure has always been where verification stops being mechanical and starts being judgment — deciding what the model should contain, what the constraints should allow, what the design can never do, and what you are willing to sign. The verified wins in this post are real, and none of them moved that boundary. They cleared the brush on the road to it. Walk the last stretch yourself, and know exactly where it starts.

This post is part of the AI for DV series. Claims above trace to the primary sources linked inline; the research corpus and verification notes live with the series. Related: the Practitioner Playbook and the AI Reading List.

The AI-for-DV Research Reading List (2024-2026)

Milan Kubavat · · -- min read · AI, AI Reading List, Verification,

This is the citation home for the AI page. Every research claim in the Practitioner Playbook and its deep-dive posts traces back to an entry here. The list is grouped by theme, spans 2024–2026, and gets updated as the field moves.

Two kinds of entries: papers you should actually read (marked in the Start Here box), and papers you should know exist so you can find them when a problem lands on your desk. Where a named system has no standalone paper link, the entry points at the survey that covers it.

Start Here — Three Papers

  • The ASPDAC 2026 survey (Surveys, below) — the best current map of LLM-assisted verification: assertion generation, testbench automation, and RTL debug in one method tree.
  • CVDP (Code Generation, below) — the 783-problem hardware benchmark behind the 34% pass@1 ceiling. Read it before believing any codegen demo.
  • The Prompt Report (Prompt & Context, below) — the systematic survey the practical prompting patterns are drawn from.

Surveys & Field Maps

  • LLM-Assisted Circuit Verification: A Comprehensive Survey (ASPDAC 2026) — the field map: SVA generation (prompting / RAG / training-based branches), testbench and test automation, automated RTL debugging, and collaborative verification frameworks.
  • arxiv 2512.23189 — The Dawn of Agentic EDA — three-tier method taxonomy (prompt-based, fine-tuned, multi-agent), the "unit-test fallacy" critique of module-level benchmarks, and the call for an Open Agentic EDA Standard.

Prompt & Context Engineering

  • Anthropic, Effective Context Engineering for AI Agents (2025) — the canonical industry essay on the shift.
  • arxiv 2406.06608 — The Prompt Report, 58 techniques, PRISMA-grounded.
  • arxiv 2402.07927 — Systematic Survey of Prompt Engineering.
  • arxiv 2407.12994 — Prompt Engineering Methods for NLP Tasks survey.
  • arxiv 2509.21361 — Maximum Effective Context Window.
  • arxiv 2603.04814 — Beyond the Context Window, fact-based memory vs. long-context.
  • arxiv 2601.01954 — Reporting LLM Prompting in Automated SE.

Pair Programming & Debugging

Code Generation

  • arxiv 2604.24621 — Evaluation of LLM-Based SE Tools.
  • arxiv 2503.01245 — LLMs for Code Generation Comprehensive Survey.
  • arxiv 2505.02133 — Multi-Agent Collaboration and Runtime Debugging.
  • arxiv 2506.14074 — CVDP, 783-problem hardware benchmark.
  • arxiv 2506.07945 — ProtocolLLM, SV testbench benchmark.

Assertions & Formal Verification

  • arxiv 2410.23299 — FVEval (NVIDIA) — the first comprehensive benchmark for LLMs on formal verification: NL-to-SVA translation and direct assertion suggestion from RTL, in graded tiers.
  • AssertLLM (ASPDAC 2025) — multi-LLM pipeline generating SVAs directly from full specification documents; reported 89% of generated assertions syntactically and functionally correct on its evaluated design.
  • AutoSVA2, ChIRAAG, LASSO, AssertionForge, Hybrid-NL2SVA — the prompting / RAG / fine-tuned branches of the SVA-generation method tree; see the ASPDAC 2026 survey above for the full map.

Coverage & Benchmarks

  • Revisiting VerilogEval (ACM TODAES) — a year of LLM progress on the canonical RTL-generation benchmark, extended to spec-to-RTL tasks and failure classification.
  • arxiv 2311.00176 — ChipNeMo (NVIDIA) — domain-adapted LLMs for chip design; the reference point for the fine-tuned-model route.
  • LLM4DV (FCCM 2025) and VerilogReader (LAD 2024) — coverage-directed stimulus generation with feedback loops on uncovered bins; both covered in the ASPDAC 2026 survey above.

Technical Debt

  • arxiv 2601.06266 — SATD in LLM Software, three new debt types.
  • arxiv 2507.03536 — ACE: Validated LLM Refactorings.
  • arxiv 2501.09888 — Automated SATD Repayment.

Agentic Design Patterns

  • arxiv 2601.12560 — Agentic AI Architectures & Evaluation.
  • arxiv 2510.09244 — Fundamentals of Building Autonomous LLM Agents.
  • arxiv 2604.00835 — Agentic Tool Use.
  • arxiv 2510.25445 — Agentic AI Survey.
  • arxiv 2604.27643 — HAVEN, UVM testbench synthesis.
  • arxiv 2504.19959 — UVM².
  • arxiv 2605.04704 — UVMarvel, subsystem-level UVM testbench construction.

Security & IP Safety

  • arxiv 2604.01572 — VTS survey of AI-assisted hardware security verification — the five-stage pipeline from asset identification through countermeasure reasoning; covers SV-LLM and SoCureLLM (HOST 2025).
  • arxiv 2503.13116 — IP leakage from fine-tuning on in-house Verilog; reports up to 46.52% of generated code similar to the source IP.
  • arxiv 2405.07061 — LLMs and the Future of Chip Design — security risks and building trust in AI silicon flows.

Limits & Pitfalls

  • arxiv 2411.09916 — "Should I Give Up Now?" LLM Pitfalls in SE.

Conference Proceedings Worth Browsing

  • DVCon US 2025 proceedings — agentic verification, coverage closure with AI, and formal + GenAI flows from practitioners.
  • DVCon Europe 2025 program — three full sessions on AI in verification, including an industry-practice paper on RL-driven coverage closure.

Back to the AI page · the Practitioner Playbook is the guided tour of how these papers turn into daily practice.

Context Engineering for DV: From Failure Bundles to Agent Memory

Milan Kubavat · · -- min read · AI, AI Playbook, Context Engineering, Debug, Prompt Engineering, SystemVerilog, UVM, Verification,

In 2025, an estimated 65% of enterprise AI failures were attributed to context drift or memory loss during multi-step reasoning. That is not an edge case — it is the dominant failure mode. The field has responded by reorganizing around context engineering: the discipline of curating what tokens reach the model at each step, what the model remembers across steps, and how it retrieves what it needs without drowning in what it does not.

This post is the deep-dive on what that means for Design Verification. Eleven sections cover the effective-context-window research, the context taxonomy, minimum-viable-context discipline, memory systems for long-running DV agents, compression techniques, tool design, prompt caching economics, structured output as implicit context, the bundle anatomy that makes DV-specific context work, agent memory for long-running workflows, and the honest limits of all of it.

The structure follows the same pattern as the rest of this AI series — each section opens with a research callout citing the specific paper, numbered playbook patterns provide the takeaway, and DV-application blocks ground each technique in your UVM testbench or RTL workflow. Cross-links to the AI Playbook, DV Prompts, Full Pipeline, and Structured Logging posts thread throughout where appropriate.

1. The Effective Context Window Crisis

The single most actionable finding the DV community has not internalized: advertised context windows are not effective context windows. The gap is large, the failure modes are predictable, and pretending otherwise is the #1 source of disappointing LLM-debug experiences.

Research arxiv 2509.21361 Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs — effective capacity is typically 60-70% of advertised; some tasks degrade by up to 99%. The Context Rot analysis (Morph LLM, 2026) names three compounding degradation mechanisms. The lost-in-the-middle phenomenon, first documented by Stanford/UC Berkeley researchers in 2023 and refined repeatedly since, shows 30%+ accuracy drops for information positioned in the middle of context vs. the start or end. The 2026 enterprise-AI failure analyses attribute the majority of multi-step reasoning failures to context drift.
  1. Measure your model's real MECW on your workload. Do not trust the marketing number. Build a 10-prompt eval set with varying context sizes (1K, 5K, 20K, 50K, 100K tokens) and measure accuracy degradation. The curve is what tells you where to cap.
  2. Place critical information at the start and end. The lost-in-the-middle research is unambiguous: middle-positioned facts are functionally ignored at large context sizes. The failing event in a bundle goes at the top; the system instructions stay at the top of the system prompt; long mid-prompt context dumps are anti-patterns.
  3. Cap at 50-60% of the advertised window. If your model claims 200K tokens, treat 120K as the soft ceiling and 60K as the comfortable working zone. The marginal token past 60% is doing less work than the first one.
  4. Avoid mid-context noise. Distractor interference — semantically similar but irrelevant content — is a measurable third degradation mechanism. The fix is not to include “helpful background” that is not load-bearing for the task.
  5. Re-measure on every model upgrade. MECW changes when the underlying model changes. The eval set from step 1 is your regression suite; run it on every provider update.
Apply to DV
  • Never paste raw regression logs into a chat. The 2,000-line log dump is the textbook trigger of all three degradation mechanisms.
  • In your failure bundles (Section 9), the failure event lives at the top of the bundle JSON, before the 200-event context window. The agent sees the punchline first.
  • If your team uses a 200K model, set the bundle composer's max_tokens at 60-80K, not 200K. The compose-and-pray approach kills accuracy long before it kills cost.

2. Context Taxonomy: What Lives in the Window

Not all context is the same. A clean taxonomy of what types of information compete for the window lets you optimize each independently — cache the stable, prune the ephemeral, retrieve the on-demand.

Research The Anthropic Effective Context Engineering for AI Agents essay (September 2025) and the LangChain context-engineering writeup converge on a six-part taxonomy. arxiv 2603.09619 Context Engineering: From Prompts to Corporate Multi-Agent Architecture formalizes the lifecycle distinction: stable, semi-stable, and ephemeral context have different caching, retrieval, and pruning strategies.
  1. System instructions. Stable across the session. Cache hot. This is your prompt template, role priming, output schema. Changes only on team-level prompt evolution.
  2. In-context examples (few-shot). Semi-stable. Swap the examples by task family (AXI debug, PCIe debug, scoreboard generation). Cache by task family; refresh when the team adds canonical examples.
  3. Retrieved knowledge. Ephemeral. Just-in-time. RTL excerpts, spec sections, past-bug context come in via tool calls when the agent asks for them — never preemptively loaded.
  4. Tool outputs. Ephemeral. Summarize aggressively and drop the raw form. The 800-line log slice from a query_log call should be reduced to 30 key events before the next reasoning step.
  5. Conversation history. Semi-stable but compounds dangerously. Prune aggressively. At session checkpoints (every 8-10 turns), re-summarize the conversation into a 200-token state synopsis and let the rest fall off.
  6. Scratchpad / think tool. Ephemeral. Fresh per agent step. The reasoning output is not preserved across iterations — only the conclusions feed forward.
Apply to DV
  • System instructions: your senior-DV-engineer role priming, the bundle schema description, the output JSON shape. Cache hot.
  • Examples: keep a per-protocol example library (axi, pcie, usb, custom). The bundle composer picks the relevant examples based on the failure's protocol family.
  • Retrieved: RTL excerpts via get_rtl_excerpt, past bugs via query_fingerprint_db, design constraints via query_spec. None loaded upfront.
  • Tool outputs: run_smoke returns pass/fail + log digest, not the full log. The agent does not need 50K tokens of new log to learn the smoke passed.

3. The Minimum-Viable-Context Principle

The 2025-2026 industry consensus has shifted from “more tokens equal more capability” to just-in-time retrieval and progressive disclosure. The agent should fetch what it needs when it needs it, not start the session with everything that might be relevant.

Research Anthropic's Effective Context Engineering for AI Agents (September 2025) is the canonical industry essay and is explicit about just-in-time retrieval as the recommended strategy for long-running agents. Claude Code is the canonical implementation: maintain lightweight identifiers (file paths, line ranges, IDs) and let tools like grep/glob/read dynamically load only what the agent requests. The State of Context Engineering 2026 (Aurimas Griciunas) and the LogRocket LLM context problem analysis (2026) corroborate: progressive disclosure beats upfront loading on every measured workload.
  1. Lightweight references over data dumps. The bundle composer should emit file_paths, signal_names, fingerprint_ids, commit_hashes — not the full contents. The agent fetches contents via tools when it decides the reference is relevant.
  2. Tools for on-demand load. get_rtl_excerpt(file, line_range), query_log(filter, max_events), get_commit_diff(hash). Each tool is the equivalent of grep/glob/read for the DV domain.
  3. Progressive disclosure: broad to narrow. The first tool call returns a summary (e.g., 5 candidate RTL files); subsequent calls drill into the one the agent picks. Avoid round-trips on broad scans.
  4. Verify the agent did not over-ask. Cap tool-call budgets per session (8-12 calls maximum). When the agent is over budget, that is your signal the task scope is wrong — not that you should raise the budget.
Apply to DV
  • Your failure bundle (Section 9) carries identifiers, not data: failing transaction txn_id, suspect signal names, candidate RTL file paths, fingerprint hash, run header. The agent calls tools to fetch what it actually needs.
  • For SoC-scale debug, the first tool call returns the per-IP failure summary; the agent picks the IP; subsequent calls drill into that IP only. Never pre-load all IPs.
  • Cap your debug agent at 10 tool calls per session. If it needs more, the bundle was wrong or the task was too big — not the agent's fault, but yours.

4. Memory Systems for DV Agents

Memory has become a first-class architectural component — not an afterthought of the model's context window. The 2024-2026 research has settled on a four-type taxonomy borrowed from cognitive science, and the DV-side mapping is direct once you see it.

Research arxiv 2502.06975 Episodic Memory is the Missing Piece for Long-Term LLM Agents — the foundational argument. arxiv 2604.04853 MemMachine: A Ground-Truth-Preserving Memory System for Personalized AI Agents (2026) is the canonical implementation. The 2026 State of AI Agent Memory report (Mem0) and the GitHub Memory in the Age of AI Agents survey converge on four memory types. Recent 2026 papers (MemRL, Agentic Memory) extend with self-evolving and unified short-long memory management.
  1. Short-term memory (STM) is the context window itself. Do not fight it — treat it as scarce. Apply Sections 1-3 ruthlessly: cap context, prune mid-window noise, use just-in-time retrieval.
  2. Episodic memory: specific past experiences. The vector database of past failure bundles + their resolutions. When a new failure arrives, semantic-similarity-search the episodic store to find “we have seen something like this before.” The bundle composer surfaces matches as fingerprint_siblings.
  3. Semantic memory: distilled facts and constraints. Protocol invariants (AXI burst constraints, PCIe LTSSM rules), design constraints (clock-domain mappings, power-state transitions), team conventions (UVM agent patterns, naming standards). Updated rarely; consulted often.
  4. Procedural memory: learned strategies. Debug recipes that worked. “For SCB_MISMATCH fingerprints involving D3 power state, check power-controller reset domain first.” Built from the postmortem stage of the Full Pipeline; consulted by the bundle composer at agent invocation.
  5. Memory consolidation: episodic to semantic over time. The MemMachine pattern: episodic stores raw experience; a periodic consolidation pass distills recurring patterns into semantic facts. For DV: monthly job that converts “these 47 past bugs all involved CDC issues on clk_b” into a semantic-memory entry “clk_b CDC is a frequent failure class — check first.”
Apply to DV
  • Stand up an episodic memory store (SQLite + vector index): one row per resolved fingerprint, embedding of the bundle, link to the fix commit. The Full Pipeline post's Stage 6 already writes this; this is its purpose.
  • Stand up a semantic memory store: a structured YAML or JSON file per IP capturing “known invariants” (signal width assumptions, FSM transition rules, power-state semantics). Updated on spec change; consulted on every debug session.
  • Stand up a procedural memory store: a runbook indexed by failure-class fingerprint — “for fingerprint class X, run plusarg Y first.” Build this empirically from successful debug sessions.
  • Memory consolidation: a quarterly review that asks “what episodic entries clustered into recurring patterns?” and writes them into semantic memory. This is the maintenance step most teams skip and most regret.

5. Context Compression Techniques

Compression is the engineering response to the effective-window crisis: keep the load-bearing tokens, shed the rest. The 2026 research has converged on three concrete approaches, each with measured token-reduction numbers in the 26-54% range without task-success degradation.

Research arxiv 2601.07190 Active Context Compression: Autonomous Memory Management in LLM Agents (Focus architecture, slime-mold-inspired exploration) — addresses “context bloat” in long-horizon SWE tasks by autonomous pruning. arxiv 2510.00615 ACON: Optimizing Context Compression for Long-Horizon LLM Agents — failure-driven, task-aware compression guideline optimization, reducing peak tokens by 26-54% while preserving task success. arxiv 2510.08907 Autoencoding-Free Context Compression via Contextual Semantic Anchors (SAC) consistently outperforms prior compression methods.
  1. Anchored iterative summarization. At checkpoints, summarize prior turns into a 200-token state synopsis anchored to key entities (failure event, hypothesis, last tool result). Subsequent turns reference the synopsis, not the raw history.
  2. Failure-driven guideline optimization (ACON). When the agent fails to converge, the meta-loop asks “what part of the context was load-bearing? What was noise?” and updates the compression policy for next time. Compression learns from misuse.
  3. Autonomous pruning (Focus). Agents decide for themselves when to consolidate raw history into a persistent “knowledge block” and drop the rest. The slime-mold metaphor: keep the trails to food, prune the dead ends.
  4. Semantic anchor compression (SAC). Instead of summarizing prose, compress around named entities (the failing signal, the suspect FSM, the relevant txn_id) and let the model reconstruct details from anchors at inference time.
  5. Provider-native compaction. Anthropic and OpenAI both offer compaction APIs that summarize prior turns automatically. Cheaper than DIY; less control. Use when your eval suite says quality is preserved.
Apply to DV
  • For long debug sessions: at every 5 tool calls, re-summarize the session into a state synopsis — failing event, current hypothesis, what has been falsified, what is unknown. The next 5 calls operate on the synopsis, not the raw trace.
  • Compress regression history into per-fingerprint summaries rather than carrying every past failure. A summary line per fingerprint with hit-count and last-resolution outperforms the raw event stream for context purposes.
  • Use semantic anchors for DV: the txn_id is the natural anchor for a transaction; the fingerprint is the natural anchor for a failure class; the signal name is the natural anchor for a debug investigation. Compress around these.

6. Tool Design as Context Discipline

Tool design is context engineering by other means. A well-designed tool returns minimum-viable context; a poorly-designed tool floods the window with noise and burns the budget you spent the previous five sections protecting. Anthropic's 2025-2026 guidance is unusually concrete.

Research Anthropic Writing Effective Tools for AI Agents (2025) — core principles: tools need different design than human-facing APIs because they must be protected against the LLM “chaos monkey”; choose high-leverage tools; clear distinct names; human-readable fields beat raw IDs; pagination, truncation, filtering as defaults. arxiv 2505.18135 Tool Preferences in Agentic LLMs are Unreliable — agents under-use even well-designed tools, over-use poorly-named ones. arxiv 2412.04093 Practical Considerations for Agentic LLM Systems covers token-aware tool design at scale.
  1. High-leverage tools, not thin wrappers. A tool that expands agent capability is worth a tool slot; a tool that just relays an API call is not. query_log(filter, max_events) with a focused filter beats read_file(path) applied to a 50MB log.
  2. Clear, distinct names. No two tools should be plausibly confusable. get_rtl_excerpt vs read_rtl_file is exactly the confusion to avoid — pick one verb, document the contract, deprecate the other.
  3. Human-readable fields beat raw IDs. Return {"signal": "axi_wb_inflight", "value": 0, "at_ns": 4960}, not {"s": 42, "v": 0, "t": 4960}. The agent reasons about names, not opaque integers.
  4. Pagination, truncation, filtering as defaults. Every tool should return at most a few hundred tokens by default. A tool that could return 50K tokens must have explicit pagination parameters and the agent must opt in.
  5. Defensive against misuse. Validate parameters at the tool boundary; return clear error messages the model can recover from. The agent will pass nonsense; your tool should reject it gracefully without filling the window with stack traces.
Apply to DV
  • The DV agent tool catalog: query_log(filter, max_events=200), get_rtl_excerpt(file, line, context_lines=20), run_smoke(test, seed, plusargs, timeout=300), query_fingerprint_db(fingerprint), get_commit_diff(hash), list_signal_history(signal, time_range). Six tools, each with explicit limits.
  • Tool outputs are digests, not raw artifacts. run_smoke returns {pass: true/false, duration_s, error_class, last_10_events}, not the full 50K-line simulation log.
  • If a tool has more than 5 parameters, split it. If a tool's name needs a paragraph to explain, rename it.

7. Prompt Caching Economics for DV Loops

Prompt caching is the highest-ROI cost lever for any agentic workload that re-queries the same context dozens of times per session. UVM debug loops are exactly that workload. For teams running this at scale, not using prompt caching is not a cost optimization to skip — it is unaffordable not to use.

Research Anthropic prompt caching: writes cost 1.25x normal input rate; reads cost 10% of normal input rate (a 90% discount). OpenAI's cached prefix is billed at 50% of normal. Real-world reports show 70-90% token cost reduction in agent loops; Anthropic claims up to 85% latency reduction for long prompts with TTFT dropping from 5s to under 200ms. arxiv 2601.06007 Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks — cache hit rate becomes the dominant cost factor at scale.
  1. Order context for cacheability. Stable content first, volatile content last. The system prompt, role priming, output schema, and codebase examples go at the top — they do not change between requests in the session. The failure event and per-request bundle slice go at the bottom — they change every call.
  2. Cache breakpoint design (Anthropic). Anthropic gives you explicit cache breakpoints with configurable TTL (5 minutes default). Place breakpoints after each stable block. The agent gets a 10% read on everything before the breakpoint, full price only on the new tokens after.
  3. Cache invalidation strategy. When does the cache expire? When the system prompt changes, when the schema version bumps, when the team examples are refreshed. Plan invalidation around team release cadence, not silently.
  4. Track cache hit rate as a first-class metric. If your agent's cache hit rate is below 60%, your context ordering is wrong. If it is above 90%, you are doing context engineering right.
Apply to DV
  • For UVM debug sessions where the agent re-queries the same bundle context across 5-15 ReAct iterations, prompt caching cuts per-iteration cost by ~90% on cached tokens. A $0.30 debug session becomes a $0.05 debug session at no quality cost.
  • Order your debug agent context: [stable: system prompt + role + bundle schema + protocol examples] — cache breakpoint — [volatile: failure event + 200 context events + tool outputs from this turn].
  • Add cache-hit-rate to your debug-pipeline dashboard. If you do not measure it, the cache is silently degrading and you will discover only when the bill arrives.

8. Structured Output as Implicit Context

Schema-enforced structured output is not just a parsing convenience — it is implicit context. The schema you pin to the response tells the model exactly what fields to produce, in what order, with what types. That schema is doing context-engineering work without consuming context tokens.

Research The 2026 industry consensus identifies three reliability levels: prompt engineering for structured output (80-95% valid), function calling/tool use (95-99%), and native structured output with constrained decoding (100% schema-valid by construction). OpenAI Strict Mode (August 2024), Anthropic native structured output (late 2025), and Google Gemini equivalent capability all use finite-state-machine compilation from JSON Schema. XGrammar (March 2026) is the default structured-generation backend for vLLM, SGLang, and TensorRT-LLM at <40 microseconds per token. Outlines is the open-source Python library pioneering grammar-based generation.
  1. Three reliability levels — know which one you are using. If your prompt says “return JSON with fields X, Y, Z” you are at level 1 (80-95%). If you use the provider's tool-call interface, you are at level 2 (95-99%). If you use native structured output with FSM enforcement, you are at level 3 (100% schema-valid).
  2. Schema-as-context. The schema describes the output shape and the model treats it as instruction. A well-named schema field (fingerprint_hash) is more useful than an instruction-prose equivalent (“include a fingerprint hash here”).
  3. Schema versioning is part of context engineering. When you change a schema, you change the implicit context. Version your schemas; include the version in cache invalidation; include it in your prompt eval suite.
  4. Validate even with constrained decoding. Schema-valid does not mean semantically correct. A schema-valid coverage plan can still be a bad coverage plan. Constrained decoding eliminates one class of errors; downstream validation handles the rest.
Apply to DV
  • Maintain a DV schema library: hypothesis_rank.schema.json, coverage_plan.schema.json, fix_proposal.schema.json, refactor_proposal.schema.json. Version each.
  • Use level-3 native structured output for any agent response that feeds downstream tooling. Use level-1 only when humans are the only consumer.
  • The schema definitions themselves are part of your team's context-engineering surface area. Treat them with the same review discipline as production code.

9. DV-Specific Context Patterns: Bundle Anatomy

Everything above sets up this section. The DV-specific context pattern — the failure bundle — is what makes context engineering land for hardware verification. This is the deep-dive on bundle anatomy, RTL excerpt selection, signal slice formatting, DUT state representation, and the multi-IP coordination patterns that emerge at SoC scale.

Research The Anthropic context-engineering essay applied to coding agents (Claude Code) is the architectural template; the DV adaptation builds on the Structured Logging JSONL substrate and the Full Pipeline bundle pattern. The 2026 industry coverage of Siemens' agentic verification toolkit shows MCP-based context exposure: real-time design and verification state available to agents through a standardized protocol. arxiv 2512.06247 DUET: Agentic Design Understanding via Experimentation and Testing demonstrates agentic dynamic exploration of RTL with the agent generating its own experiments to build context.
  1. Bundle anatomy — six well-typed sections. The DV failure bundle is a JSON document with: (a) failure — the triggering event, (b) context_events — ring-buffer dump of pre-failure events (Structured Logging Pattern 9), (c) rtl_excerpts — relevant RTL code snippets with reasons for inclusion, (d) fingerprint_siblings — past bugs with the same fingerprint plus their resolutions, (e) recent_commits — RTL git log for the last N days scoped to the failing IP, (f) run_header — seed, plusargs, tool version, RTL hash for replay.
  2. RTL excerpt selection — symbol-indexed, not text-grep. Build a Verible / sv-parser / Surelog index of your RTL once, cached. For each failure, the bundle composer queries the index by signal names appearing in the context events, scoped to the failing IP. Returns the file path, line range, and 25 surrounding lines for each match, ranked by hit count. Never blind grep across the SoC.
  3. Signal slices as DUT_STATE events. Your bind-based harness (see Debug page Pattern 11) emits DUT_STATE JSONL events on key triggers: FSM transitions, queue depth changes, register writes. These appear as first-class events in the context_events stream — the agent sees them inline, not as a separate waveform query.
  4. DUT state representation. A snapshot DUT_STATE event contains the current state of the things that change rarely but matter critically: FSM states, queue depths, key configuration registers, power state, clock domain, mode/profile. Format as flat key-value pairs in the JSONL event; the agent reads them with zero ceremony.
  5. Per-IP bundle scoping for SoC debug. At SoC scale, never compose a bundle that spans IPs by default. The first-pass bundle is per-IP; cross-IP correlation happens via shared txn_id threading (Structured Logging Pattern 3). The agent decides when it needs to cross IP boundaries and fetches additional per-IP bundles via tool calls.
  6. MCP-based context exposure (Siemens pattern). The Model Context Protocol gives you a standardized server interface for exposing DV tools to any MCP-compatible client (Claude Code, IDE assistants, custom agents). Each tool from Section 6 becomes an MCP endpoint; the protocol handles the rest.
  7. Bundle hash for deterministic replay. Every bundle gets a content hash. Cache prompt+response by hash. The same failure replays to the same agent response, every time. Reproducibility is non-negotiable when the agent's output goes anywhere near production silicon decisions.
Apply to DV
  • If you build only one piece of context-engineering infrastructure this quarter, build the bundle composer. Everything else — agent design, memory, prompt caching — assumes the bundle exists.
  • Adopt MCP early. The 2026 vendor landscape (Siemens, Cadence experimental, Synopsys announced) is converging on MCP as the integration protocol. Build your DV tool catalog as MCP-compatible from day one.
  • The bundle composer is the single most important piece of Python in your DV-AI stack. Treat it accordingly: code review, unit tests, schema versioning, performance budget.
  • For multi-team SoC environments, agree on a shared bundle schema across teams. The schema is a contract; each IP team produces bundles to it, agents consume bundles in the same shape regardless of source.

10. Agent Memory for Long-Running DV Workflows

Section 4 laid out the memory taxonomy. This section grounds it in the long-running workflows DV teams actually run: regression seasons that span weeks, projects that span quarters, IP families that span years. Memory is what turns the agent from a smart one-shot tool into a teammate that gets better with the project.

Research arxiv 2604.04853 MemMachine — the canonical ground-truth-preserving memory system architecture, applied to personalized agents. The 2026 State of AI Agent Memory report (Mem0) and Best AI Agent Memory Frameworks in 2026 (Atlan) survey production-grade options: Mem0, Letta, Zep, Cognee, Memori. arxiv 2604.16548 Survey on the Security of Long-Term Memory in LLM Agents covers the security and integrity dimensions that matter for regulated DV flows.
  1. Episodic store: failure bundle archive. Every resolved bundle gets stored in a vector DB (Chroma, Qdrant, pgvector). Key fields: bundle hash, embedding, fingerprint, resolution file/line, resolving commit, owning team. Retrieved by semantic similarity when new bundles arrive.
  2. Semantic store: per-IP design book. A structured YAML/JSON per IP capturing invariants, conventions, protocol-specific rules. Updated on spec change. Loaded into the bundle composer's “known constraints” context block. Example fields for an AXI IP: burst_length_max, id_width, supported_burst_types, back_pressure_protocol.
  3. Procedural store: debug runbook. Per fingerprint class, the canonical first-checks. “For SCB_MISMATCH fingerprints with D3 in the state signature, first run +NO_D3_TRANSITIONS=1. If passes, the bug is power-gating related; consult pwr_controller.” The agent consults this before generating its own hypotheses.
  4. Memory consolidation cycle. Monthly: scan episodic memory for clusters (k-means on embeddings); promote recurring patterns into semantic memory. “47 past bugs involved CDC on clk_b” becomes “clk_b CDC is a known failure class — check first.” This is the maintenance most teams skip and most regret.
  5. Memory hygiene and audit. Stale episodic entries (bug fixed two release cycles ago, IP redesigned) should be pruned. Every memory write logs author + timestamp + provenance. For regulated flows, every memory consultation that influenced an agent decision is auditable after the fact.
Apply to DV
  • Start with episodic memory only. SQLite + a vector index over bundle hashes is a one-day build. Procedural and semantic stores can wait until you have enough episodic data to feed them.
  • The Full Pipeline post's Stage 6 (postmortem) is exactly the write path into episodic memory. If you have Stage 6, you already have episodic memory; the read path (similarity search in bundle composer) is the integration work.
  • Plan the memory consolidation cycle into your quarterly cadence. The episodic-to-semantic distillation is high-value but only if someone owns it — designate the owner before building the store.
  • For regulated DV flows, the audit dimension is not optional. Memory writes and consultations are first-class events in your structured log, with the same provenance discipline as the testbench logs themselves.

11. What Does Not Work (Honest Limits)

The anti-patterns. Each of these is a failure mode the research has measured and the practitioners have lived. Knowing them in advance is how you save the afternoon you would otherwise spend on a context strategy that was never going to work.

Research A composite synthesis of the failure findings cited across this post: lost-in-the-middle (2509.21361, Context Rot 2026), context bloat in long-horizon tasks (2601.07190 Active Context Compression), tool preference unreliability (2505.18135), cache-bypass anti-patterns (2601.06007 Don't Break the Cache), and memory security issues (2604.16548).
  1. Pasting raw regression logs into a chat. The textbook trigger of all three context-degradation mechanisms. The bundle pattern exists specifically to prevent this.
  2. Trusting mid-context information without verification. The agent “cited” a line that was 80K tokens deep in a 200K-token prompt. The lost-in-the-middle research says that citation is unreliable. Verify before trusting.
  3. Assuming long context equals good context. The marketing slide says 1M tokens; the effective window is closer to 130K. Cap your bundle composer at 50-60% of advertised, not 100%.
  4. Cache-bypass anti-patterns. Putting volatile content (timestamps, request IDs, dynamic counters) at the top of your prompt invalidates the cache on every request. Stable content first, volatile last — not negotiable.
  5. Unbounded memory growth. Episodic store grows forever; semantic store accumulates contradictory entries; procedural store fills with one-off recipes that never apply again. Plan for pruning from day one.
  6. Tool results dumped into context without summarization. The 50K-line log returned by run_smoke belongs in a file, not in the next turn's context. Always summarize tool outputs before the next reasoning step.
  7. One agent doing everything. The generalist agent over-fits to common cases and under-performs on rare ones. UVMarvel's multi-agent-per-protocol pattern is the prior art; specialize where the failure modes differ.
  8. Schema validation as the only check. Schema-valid is not semantically correct. A schema-valid hypothesis-rank response can still be three wrong hypotheses. Pair structured output with at least one of: smoke verification, human review, or formal proof.
Apply to DV
  • Print this list. The eight anti-patterns are the ones a junior engineer is most likely to learn the hard way; making them visible is cheaper than the lessons.
  • Add a context-engineering review item to your debug-pipeline retrospectives: which of these eight bit us this month? The recurring offenders tell you where to invest the next round of infrastructure.

Reading List

Context Engineering Foundations

  • Anthropic, Effective Context Engineering for AI Agents (September 2025) — the canonical industry essay
  • Anthropic, Writing Effective Tools for AI Agents (2025) — tool design principles
  • arxiv 2603.09619 — Context Engineering: From Prompts to Corporate Multi-Agent Architecture
  • LangChain blog, Context Engineering for Agents (2026)
  • State of Context Engineering 2026 (Aurimas Griciunas)

Effective Context Window & Degradation

  • arxiv 2509.21361 — Maximum Effective Context Window
  • arxiv 2407.03651 — Working Memory Test & Inference-time Correction
  • Morph LLM, Context Rot: Why LLMs Degrade as Context Grows (2026)
  • Stanford / UC Berkeley (2023), Lost in the Middle: How Language Models Use Long Contexts

Memory Systems

  • arxiv 2502.06975 — Episodic Memory is the Missing Piece for Long-Term LLM Agents
  • arxiv 2604.04853 — MemMachine: Ground-Truth-Preserving Memory
  • arxiv 2604.16548 — Survey on the Security of Long-Term Memory in LLM Agents
  • Mem0, State of AI Agent Memory 2026
  • Atlan, Best AI Agent Memory Frameworks in 2026

Context Compression

  • arxiv 2601.07190 — Active Context Compression (Focus architecture)
  • arxiv 2510.00615 — ACON: Optimizing Context Compression for Long-Horizon LLM Agents
  • arxiv 2510.08907 — Autoencoding-Free Context Compression via Semantic Anchors

Just-in-Time Retrieval & Tool Design

  • arxiv 2505.18135 — Tool Preferences in Agentic LLMs are Unreliable
  • arxiv 2412.04093 — Practical Considerations for Agentic LLM Systems
  • LogRocket, The LLM Context Problem in 2026

Prompt Caching

  • Anthropic prompt caching API documentation
  • OpenAI cached prefix pricing documentation
  • arxiv 2601.06007 — Don't Break the Cache: Prompt Caching for Long-Horizon Agentic Tasks

Structured Output

  • XGrammar (March 2026) — default backend for vLLM, SGLang, TensorRT-LLM
  • Outlines library — grammar-based generation with JSON Schema
  • OpenAI Strict Mode documentation (August 2024)
  • Anthropic native structured output documentation (late 2025)

Hardware Agent Context

  • arxiv 2512.06247 — DUET: Agentic Design Understanding via Experimentation
  • arxiv 2604.01572 — AI-Assisted Hardware Security Verification (IEEE VTS 2026)
  • Semiconductor Engineering, Human-Centered Agentic AI Comes to RTL Verification
  • EE Times, Agentic AI Tackles RTL Verification's Productivity Gap
  • Embedded.com, Siemens Agentic Toolkit Automates Chip Verification Workflows

For the prompting techniques layer, see Better Prompts for DV. For the broader AI-in-DV map see the AI page. For the debug-specific deep-dive see the Debug page. The bundle composer this post depends on is built incrementally through the Full Pipeline and Structured Logging posts.

Better Prompts for DV: A Researcher Guide to Prompt Engineering in Hardware Verification

Milan Kubavat · · -- min read · AI, AI Playbook, Debug, DV Prompts, Prompt Engineering, SystemVerilog, UVM, Verification,

Most prompting research is generic. Most DV blog content about AI is hype. This post sits in the small intersection: a researcher guide to prompt engineering techniques mapped explicitly to Design Verification problems, with research callouts naming the papers, numbered playbook patterns as the practitioner takeaway, and DV-application blocks grounding each technique in your UVM testbench or RTL workflow.

The differentiator is Section 10 — the systematic correlation between SWE prompting strategies (where the empirical research has been done) and the DV adaptations they imply. Section 12 closes with six ready-to-paste prompt templates, each annotated with the research pattern it borrows from.

A note on what this post is not. It is not a survey of LLM model capabilities (those change quarterly). It is not a tutorial on Anthropic vs OpenAI vs your local model (the workflow is provider-agnostic). It is not advocacy for AI replacing DV engineers (the research consistently shows the human-in-the-loop is what wins). It is the durable methodology layer beneath whichever model and provider you happen to use today.

1. Foundations: Zero-shot, Few-shot, Role

Before the fancy techniques, the basics that ground everything else. Each of the three primitives below has a clean role in DV; mixing them up is a leading source of frustration.

Research arxiv 2406.06608 The Prompt Report — PRISMA-grounded systematic survey of 58 prompting techniques. arxiv 2402.07927 Systematic Survey of Prompt Engineering. The 2025 MDPI review of 42 peer-reviewed SE studies clusters prompting into four research patterns: manual crafting, RAG, chain-of-thought, automated tuning. The Liu et al. 2026 taxonomy formalizes 58 distinct techniques across modalities.
  1. Zero-shot. A single instruction, no examples. Right for: tasks the model has clearly seen many times (generate a docstring, summarize a paragraph, classify by sentiment). Wrong for: anything that involves your team's idioms, your codebase's style, or a non-standard output schema.
  2. Few-shot in-context learning. 2-5 worked examples before the actual ask. The model picks up the shape of the desired output. Critically: examples should come from your own codebase, not from a generic library — the model already knows generic libraries.
  3. Role-prompting. "You are a senior DV engineer reviewing a 10-year-old testbench." The role routes the model into a more conservative, idiom-aware mode than the default helpful-assistant frame. The 2025 SE literature notes role prompts can be removed without much loss once you have good few-shot examples; until then, they are doing real work.
Apply to DV
  • Zero-shot for repetitive boilerplate: covergroup skeletons, factory registration, default field automation macros.
  • Few-shot for anything style-sensitive: your team's sequence library idioms, your scoreboard pattern, your monitor structure.
  • Role for tasks where the default helpful-assistant tone would be wrong: code review (assistant is too lenient), architectural critique (assistant defers), and triage (assistant hedges).

2. Chain-of-Thought for DV: The Caveat Section

CoT is the most famous prompting technique. It is also the one most likely to disappoint you on RTL tasks if you apply it naively. The fix is to scaffold the reasoning with HW-meaningful intermediate steps, not generic "let's think step by step."

Research Wei et al. 2022 (arxiv 2201.11903) — CoT activates reasoning in large models; without it even 540B-parameter models behave like much smaller ones. The 2026 RTL survey (Preprints 202509.1681) explicitly states that naive chain-of-thought has been largely ineffective in automating IC design workflows because the step granularity and reasoning direction do not align with expert RTL knowledge. arxiv 2504.06939 (FeedbackEval) shows that for code repair, removing structured-reasoning cues causes severe degradation.
  1. HW-aware CoT scaffolding. Replace "think step by step" with "walk through this cycle by cycle." The model needs the right intermediate representation; for hardware that is signal traces, FSM transitions, timing windows, and clock-domain crossings — not generic prose.
  2. Trace-conditioned CoT. If you have a waveform or structured log, paste the relevant signal slice into the prompt before asking for analysis. arxiv 2505.04441 shows trace-conditioned prompts consistently beat trace-free prompts for SWE code repair; the DV analog is direct.
  3. Bounded CoT. Ask for reasoning in N steps, not free-form. "Identify the failing cycle. Identify the violating signal. Identify the upstream cause. Propose the fix." Four bounded steps beat an open-ended "think about this."
  4. Avoid CoT for structural generation. RTL module generation rarely benefits from CoT — the model needs to emit a structured artifact, not reason about it. Use decomposition (Section 5) instead.
Apply to DV
  • HW-aware CoT for timing-related debug: assertion failures involving multi-cycle properties, arbitration races, CDC investigations.
  • Trace-conditioned CoT for any failure where the structured log captures the relevant signal evolution — combine with the JSON ring-buffer dump pattern.
  • Skip CoT entirely when asking for boilerplate or scaffolding; you want the artifact, not the model's commentary on producing it.

3. Self-Consistency and Self-Verification

A complex problem usually admits multiple correct reasoning paths. Sample many paths, vote on the consistent answer. The technique is dramatic on hard problems and almost free when the model supports temperature sampling.

Research Wang et al. 2022 (arxiv 2203.11171) introduces self-consistency: sample diverse reasoning paths, take the most consistent answer. arxiv 2510.01069 Typed Chain-of-Thought / Certified Self-Consistency (CSC, 2026) aggregates only over experiments satisfying typing constraints — 69.8% accuracy on GSM8K versus 19.6% baseline. arxiv 2603.08999 Confidence-Aware Self-Consistency maintains comparable accuracy with up to 80% fewer tokens. Self-Verification (Weng et al. 2022, arxiv 2212.09561): generate forward, verify backward.
  1. Vote on hypothesis-rank. When triaging a critical bug, run the hypothesis-rank prompt three times with temperature > 0. Take the hypothesis that appears highest in all three runs. Cheaper than it sounds; catches obviously-wrong single-shot answers.
  2. Verify forward, check backward. Generated a fix proposal? Ask the model to derive the test that would fail without the fix. If the model cannot, the proposal is suspect.
  3. Certified self-consistency for typed outputs. When the answer must conform to a schema (a coverage plan, a JSON fix proposal), aggregate only over candidates that pass type/schema validation. The 2026 CSC paper is the formal version of what good DV teams already do informally.
  4. Confidence-aware sampling. When the first two samples agree strongly, stop. When they disagree, sample more. The 2026 CASC paper formalizes the trade-off; the practical takeaway is to avoid blindly running N samples when 2-3 suffice.
Apply to DV
  • Self-consistency on RCA for silicon-escape bugs — the stakes justify the extra samples.
  • Self-verification on every proposed RTL fix before the human reads it: ask the model to derive a failing test for the unpatched version.
  • Certified self-consistency for coverage-plan generation: aggregate only over plans that parse against your covergroup schema.

4. Tree-of-Thoughts for Hard RCA

When chain-of-thought gets stuck on a wrong path, Tree-of-Thoughts explores multiple branches in parallel and prunes. The headline result is dramatic; the DV use case is hard root-cause analysis where 3+ hypotheses are equally plausible.

Research Yao et al. 2023 (arxiv 2305.10601) introduces Tree-of-Thoughts. The headline: on Game-of-24, GPT-4 with standard CoT solved 4% of problems; with ToT, 74%. arxiv 2401.14295 (Besta et al., Demystifying Chains, Trees, and Graphs of Thoughts) extends to graph-shaped reasoning. The pattern works because branching defers the commitment to a single reasoning trajectory.
  1. Branch the hypothesis space. "Generate three independent root-cause hypotheses, each starting from a different evidence anchor in the bundle." Force the model to start from different observations, not extend one chain of reasoning.
  2. Score and prune. After branching, ask the model to score each branch by "evidence weight" and propose which to prune. The model is often better at evaluating its own branches than picking one upfront.
  3. Iterate on the survivor. Take the highest-scored branch and apply CoT or self-debug only to that subtree. Saves the cost of exploring all branches deeply.
  4. Use when bug is "ambiguous." If a senior engineer cannot quickly name the most likely root cause from inspection, that ambiguity is the trigger for ToT. For obvious bugs, vanilla hypothesis-rank is cheaper.
Apply to DV
  • ToT on multi-IP SoC bugs where the failure could plausibly involve cache, fabric, or peripheral — let the model branch by subsystem.
  • ToT on protocol-error bugs where the violation could be the master, the slave, or the interconnect — branch by node.
  • Skip ToT for clear-cut bugs — the cost is real and the marginal benefit is zero when the answer is obvious.

5. Decomposition: Least-to-Most and Decomposed Prompting

Complex problems become a sequence of simpler ones. The technique transferred wholesale from theorem-proving to LLM prompting and remains one of the highest-leverage moves in your toolkit for any task larger than a single function.

Research Zhou et al. 2022 (arxiv 2205.10625) Least-to-Most Prompting — 16% to 99% on SCAN compositional generalization. arxiv 2210.02406 Khot et al. Decomposed Prompting — modular subproblems with composable solvers. The 2026 RTL research (PALM, arxiv 2506.09002) extends decomposition to program-analysis-derived path constraints; the DV analog uses coverage-bin-derived path constraints.
  1. Stage 1 — decompose. Ask the model to list the subproblems before solving any. "To verify this IP, list the 6-8 distinct sub-tasks in order of dependency." The decomposition itself is high-value output.
  2. Stage 2 — solve in order. Each subproblem solution becomes context for the next. The model maintains coherence because each step is bounded.
  3. Dependency-aware decomposition. When subtasks have non-linear dependencies, draw the DAG. The model can produce the DAG and then walk it in topological order.
  4. Path-constraint decomposition. For test generation: derive path constraints from coverage bins (analog of program-analysis-derived branching conditions in PALM), use each constraint as a sub-prompt.
Apply to DV
  • Verification plan: spec → coverage plan → sequence plan → scoreboard plan → checker plan → test list. Each step a separate prompt, each consuming the prior.
  • UVM agent scaffold: interface → transaction class → driver → monitor → sequencer → agent → example sequence. Each generated separately, each consistent with the prior.
  • SoC verification: top-level constraints → per-IP test plans → integration scenarios → stress tests. Top-down decomposition mirrors how senior engineers actually plan.

6. Few-Shot Done Right: Example Selection

Few-shot is the most common technique and the most commonly done badly. Random examples are sub-optimal. Text-similarity examples are sub-optimal. The 2024-2026 research has converged on better methods.

Research arxiv 2310.09748 LAIL (LLM-Aware ICL for Code Generation) — LLM labels candidate examples as positive (helpful) or negative (trivial); a model-aware retriever learns the preference. arxiv 2305.14210 Skill-Based Few-Shot Selection — pick examples that share the underlying skill needed, not just textual surface. arxiv 2412.02906 empirically: few-shot helps code synthesis but example quality dominates count.
  1. Examples from your codebase, not from the public web. The model already knows the public web. Your codebase carries your idioms, your naming, your error-handling conventions; the model picks these up from few-shot context cheaply.
  2. Structurally-similar examples beat textually-similar ones. Two AXI sequences that look different on the surface but share the same burst structure are better few-shot fodder than two cosmetically similar sequences with different structures.
  3. Negative examples are valuable. "Here is a similar task done wrong, here is the correct version." The contrast teaches the model what to avoid; bare positive examples cannot.
  4. 2-5 examples, no more. Beyond five, the marginal benefit per token drops sharply and the long-context degradation effect (arxiv 2509.21361) starts to bite.
Apply to DV
  • Maintain a curated examples directory in your TB repo: examples/sequences/, examples/scoreboards/, examples/checkers/. Each example is a known-good reference you few-shot from.
  • Index examples by protocol family and skill (AXI burst, AXI single, PCIe TLP, USB packet) so retrieval is structural.
  • Keep a small set of "canonical wrong" examples for negative few-shot — the seq that deadlocked, the scoreboard that missed an off-by-one. Contrast teaches.

7. Agentic Prompting: ReAct and Reflexion

When the LLM needs to iterate — observe, decide, act, observe again — you are in agentic territory. ReAct and Reflexion are the foundational patterns; the 2025-2026 HW research extends them with verifier-guided refinement.

Research Yao et al. 2022 (arxiv 2210.03629) ReAct — interleave Reasoning steps with Action calls (tool invocations) and Observations. Shinn et al. 2023 (arxiv 2303.11366) Reflexion — the agent self-critiques after each task and improves next time. arxiv 2509.06239 Proof2Silicon — verifier-guided prompt refinement via reinforcement learning, applied to verified hardware code generation.
  1. ReAct loop. Reason → Act (call a tool: query log, run smoke, fetch RTL excerpt) → Observe (tool output) → Reason. Three to five iterations beats one zero-shot answer almost universally for non-trivial problems.
  2. Bounded tool catalog. Per the 2025 Anthropic context-engineering essay: 4-6 self-contained tools beat one mega-tool. Tools must be self-contained, error-robust, and unambiguous. If a human cannot decide which tool to call, the agent cannot either.
  3. Reflexion after task completion. After the loop converges (or fails), ask the agent to write a paragraph about what worked, what did not, what to try differently next time. Cache the reflection; use it in the next session's system prompt.
  4. Verifier-guided refinement. Per Proof2Silicon: when a verifier (smoke test, formal property, lint rule) gives concrete feedback, route it back into the prompt as a hard signal. This is the HW-side analog of test-driven debug from DePro (arxiv 2603.19399): up to 64% fewer attempts and 7.6 minutes saved per problem in the SWE setting.
  5. Checkpoints and human approval. Per the 2025-2026 agentic-AI consensus: free-form agent loops are less reliable in production than graph-based orchestration with explicit state transitions, debuggability, and human-approval gates. Plan for the human checkpoint; do not assume autopilot.
Apply to DV
  • ReAct for triage: query_log, get_rtl_excerpt, run_smoke, query_fingerprint_db as the tool catalog. Anything else is scope creep.
  • Reflexion for postmortem: have the agent draft the "what we learned" section from its own trace. Edit for tone before publishing.
  • Verifier-guided refinement when you have a formal property: let the agent iterate on the property/fix until the formal tool stops complaining. The HW-side feedback signal is unusually strong.
  • Checkpoint on every fix proposal that touches RTL — never let the agent commit autonomously, regardless of how confident it sounds.

8. Constrained Decoding for Structured DV Output

When you need the LLM to produce a syntactically-valid artifact — JSON, Verilog, SVA, a coverage plan in your team's schema — you should be using constrained decoding rather than crossing your fingers. The infrastructure is cheap and the failure mode it eliminates is exactly the one DV engineers complain about most.

Research Constrained decoding modifies the LLM's sampling step via a logit processor: at each token position, valid tokens are computed from a grammar state and invalid tokens are masked. Libraries: llguidance (~50µs CPU per token; arbitrary context-free grammar), Outlines (compiles JSON schemas to O(1) lookup), Anthropic tool-call structured output, OpenAI JSON mode with response_format schemas. arxiv 2603.03305 Draft-Conditioned Constrained Decoding extends this to drafted generation with later refinement.
  1. JSON schema for structured outputs. Coverage plans, fix proposals, hypothesis-rank responses, test lists — all should land as schema-validated JSON, not as parseable-prose. The provider enforces the shape; you skip the defensive parser.
  2. Grammar-guided for HDL fragments. When asking for SVA or a small Verilog snippet, a grammar (or even a regex-shaped constraint) eliminates syntactic invalidity at the source. The model never emits the "almost valid" output that wastes a downstream tool invocation.
  3. Hybrid LLM + template. Per HAVEN (arxiv 2604.27643): the LLM produces a structured architectural plan in JSON; a rule-based generator emits the actual UVM. The split is what gets HAVEN to 100% compile success.
  4. Validate even with constrained decoding. Schema-validity is necessary, not sufficient. A schema-valid coverage plan can still be functionally wrong. Always pair constrained decoding with a downstream validation step.
Apply to DV
  • Define one JSON schema for each recurring DV output: coverage plan, fix proposal, hypothesis rank, test plan, refactor proposal. Reuse across teams.
  • For Verilog or SVA fragments, prefer hybrid generation: LLM produces a structured intermediate (signal list, property predicates, port map), template emits the syntactically-valid code.
  • Wire constrained-decoding errors into your alerting — if the model is failing constraint satisfaction repeatedly on a class of prompt, that is your signal the prompt itself needs work.

9. Meta-Prompting and Prompt Optimization

The prompts you write today are the next thing to refactor. Meta-prompting uses the LLM to rewrite your own prompts, A/B-tested against a small eval set. The 2026 industry consensus is that prompt engineering has matured from craft into versioned-and-tested engineering practice.

Research arxiv 2502.00728 Meta-Prompt Optimization for LLM-Based Sequential Decision Making. The 2026 industry literature (Comet, IntuitionLabs) reports self-refinement loops consistently improve prompt quality by 10-25%. arxiv 2503.02400 Promptware Engineering formalizes prompt engineering as a software engineering discipline: versioning, evaluation, regression testing for prompts.
  1. Build a small eval set. 15-25 representative DV prompts (debug, scaffold, review, refactor, cov-plan) with expected output shapes. This is your prompt regression suite.
  2. Self-refinement loop. Ask the LLM to critique and rewrite your prompt against the eval set. Compare A/B. Keep the winner. Repeat until the curve flattens.
  3. Version prompts like code. Prompts in a repo, semver tags, regression tests on every change. The day the underlying model updates, you re-run the suite to catch silent drift.
Apply to DV
  • Standardize the team's top 6-10 DV prompts in a shared repo. The hypothesis-rank prompt, the scaffolding prompt, the coverage-plan prompt — all versioned, all tested.
  • On every model upgrade or provider switch, re-run the eval suite. Catch the prompt that quietly stopped working before a junior engineer trusts a wrong answer.

10. SWE → DV: Methodological Correlations

This is the section that does not exist elsewhere on the internet. The SWE research community has done the empirical work on prompting for debug, test generation, code review, and refactoring. Each result transfers to DV with a clear adaptation. The table below names the SWE finding, the empirical evidence, the DV analog, and the specific adaptation needed.

Research Six recent SWE-prompting findings anchor this section. FeedbackEval (arxiv 2504.06939) on structured-reasoning code repair. DePro (arxiv 2603.19399) on iterative test-driven debug. PALM (arxiv 2506.09002) on program-analysis-derived path constraints. IntUT (paper covered in 2024-2025 SE literature) on intent-first unit-test generation hitting +94% branch coverage. Meta's semi-formal reasoning template hitting 93% code-review accuracy. Refactor subcategory explanation (arxiv 2411.02320) moving success from 15.6% to 86.7%. Static-analysis-augmented prompts (arxiv 2508.14419) cutting security violations from >40% to 13%.

The seven meta-patterns below each have an empirical SWE-side anchor and a direct DV adaptation. The DV adaptations are where the value is — they are not in the SWE papers because the SWE researchers were not thinking about hardware.

#SWE Meta-PatternSWE EvidenceDV Adaptation
1 Trace-Conditioned Prompting Execution traces in prompts consistently beat trace-free prompts (arxiv 2505.04441) Waveform-conditioned debug: paste FSDB signal slice formatted as "trace events"; Grove (arxiv 2511.x) confirms this for HW debug
2 Iteration-Feedback Loop DePro test-driven debug (64% fewer attempts); IntUT coverage-driven test gen (+94% branch coverage); FeedbackEval structured reasoning Smoke-driven self-debug loop; coverage-bin-driven test generation; formal-feedback loops (Proof2Silicon 2509.06239 is the HW-side prior art)
3 Intent-First Prompting IntUT: explicit test intentions improve branch coverage by 94%, line coverage by 49% Verification-intent-first: every UVM sequence prompt opens with "this is verifying scenario X / covering bin Y"; never "generate a test for this DUT"
4 Structured-Reasoning Templates Meta's semi-formal reasoning (premises → execution → conclusion) reached 93% code-review accuracy SVA-anchored review template: relevant assertions as premises; simulation/formal evidence as execution; coverage-bounded conclusion
5 Multi-Generation + Vote 5 refactorings per input → +28.8% pass rate; self-consistency over CoT samples Multi-fix-proposal + smoke gate: LLM proposes 3 fixes, smoke runs all 3, pick the one that passes. HW has a natural validator the SWE side often lacks — the simulator
6 Hybrid Tool+LLM Pipeline Static-analysis-augmented prompts (security >40% → 13%); PALM program-analysis path constraints Lint-augmented prompts; formal-augmented prompts; coverage-gap-augmented prompts. LASHED (arxiv 2504.21770) and Proof2Silicon are the HW-side prior art
7 Subcategory-Explanation Pattern Refactor success 15.6% → 86.7% from naming the refactoring type upfront (arxiv 2411.02320) UVM refactor categories upfront: extract base class, replace inheritance with composition, virtual-sequence extraction, agent split, factory-override simplification, config-object introduction. Name the category before asking for the change

Two threads tie the seven patterns together. First, every winning SWE prompting move has a feedback signal — tests, coverage, lint, formal — and the DV side has all of those signals available, often more cheaply than SWE does. Second, every winning move provides structure to the LLM upfront — the test intent, the refactor category, the schema — rather than asking the model to infer it. These two themes recur through Section 12's templates and Section 13's honest-limits discussion.

Apply to DV
  • Audit your top 10 prompts against the seven patterns. The patterns missing are usually the lowest-hanging adoption fruit.
  • For each adopted pattern, instrument a metric so you can measure the lift on your eval suite over time. The SWE results are large; the DV results should be too.

11. HW-LLM Frameworks Already in the Wild

The 2024-2026 hardware-LLM research has converged on a small number of named systems. Each one reveals a specific lesson about prompting for HDL/UVM that you can borrow without adopting the whole system.

Research Ten published HW-LLM systems, with the lesson each contributes: HAVEN (arxiv 2604.27643) plan-then-template, 100% compile, 90.6% coverage. UVM² (arxiv 2504.19959) domain-knowledge prompts + syntactic constraints + iterative refinement. UVMarvel (arxiv 2605.04704) multi-agent per protocol, 95.65% coverage. VeriGRAG (arxiv 2510.15914) structure-aware soft prompts. FVDebug (arxiv 2510.15906) causal graph synthesis + agentic exploration for waveform/RTL/spec debug. MEIC (arxiv 2405.06840) iterative debug with bounded progress per turn. LASHED (arxiv 2504.21770) LLM + static analysis for early RTL bug detection. Self-HWDebug (arxiv 2405.12347) self-instructing debug from vulnerable/secure RTL pairs. ReasoningV (arxiv 2504.14560) reasoning specialization for Verilog. Proof2Silicon (arxiv 2509.06239) RL prompt repair from formal feedback. CVDP (arxiv 2506.14074) benchmark establishes the 34% pass@1 ceiling.
  1. Never let the LLM emit HDL directly without scaffolding. This is the single lesson all 10 systems share. HAVEN's split (LLM → plan JSON; template engine → UVM) is the canonical pattern; the 100% compile success and 90.6% coverage numbers are the proof.
  2. Iterate with feedback, not in isolation. UVM², MEIC, and Proof2Silicon all use iterative refinement against a verifier signal (coverage, syntax, formal property). Single-shot generation is what produces the disappointing CVDP numbers; iteration is what closes the gap.
  3. Combine LLM with classical tools. LASHED pairs LLM with static analysis. Proof2Silicon pairs LLM with formal verification. The hybrid systems beat LLM-alone systems consistently.
  4. Specialize agents per protocol or subsystem. UVMarvel uses a different agent per bus protocol and reaches 95.65% coverage. Generalist agents over-fit to common protocols and under-perform on rarer ones.
  5. Borrow the patterns without adopting the systems. You do not need to deploy HAVEN to use its plan-then-template idea in your own prompts. The published systems are demonstrations; the patterns are reusable.
Apply to DV
  • Pick one HW-LLM lesson and apply it to your current workflow this month. Plan-then-template is the highest-leverage starter.
  • Track your own pass@1 number on a small local benchmark. The 34% CVDP ceiling is a public-research average; your number with structured prompts and iteration should be substantially higher on tasks scoped to your IP family.
  • When evaluating a vendor's AI-DV tool, ask which of the 10 patterns above it implements and which it skips. The honest vendors can answer.

12. A Concrete DV Prompt Template Library

Six ready-to-paste prompt templates, each annotated with the research pattern it borrows from. Replace the {{variables}} with your specifics. None of these templates are theoretical; each composes 2-3 of the techniques from Sections 1-10.

Template 1 — RTL Module from Spec (decomposition + few-shot + intent-first)

[SYSTEM]
You are a senior RTL designer writing SystemVerilog for a verification
team to validate. Match the style of the provided examples. Never invent
functionality not present in the spec.

[USER]
Generate a SystemVerilog module implementing the feature below.

Decomposition (handle in order):
1. Identify all signals required from the spec
2. Identify the FSM states (if any)
3. Identify the data flow (combinational vs sequential)
4. Generate the port list
5. Generate the module body
6. State any assumptions in a final comment

Spec section:
{{spec_excerpt}}

Module name: {{module_name}}
Style examples (from our codebase):
{{example_1}}
{{example_2}}

Why this works: The numbered decomposition (Section 5) prevents the model from emitting a monolithic blob. The codebase-derived examples (Section 6) carry team idioms the model would not infer. The explicit "never invent" instruction caps the hallucination surface.

Template 2 — UVM Agent Scaffolding (role + decomposition + few-shot + scope limits)

[SYSTEM]
You are a senior UVM verification engineer scaffolding a new agent.
Match the team's existing patterns. Never invent functionality not
explicitly asked for. Default to non-blocking, event-driven design.

[USER]
Generate UVM agent scaffolding for the interface below.

Interface definition:
{{interface_signature}}

Reference monitor (the team's style guide):
{{reference_monitor_code}}

Output, in order:
1. Transaction class (sequence_item) with rand fields and UUID stamping
2. Driver class with run_phase
3. Monitor class with run_phase and analysis_port
4. Sequencer typedef
5. Agent class with build_phase + connect_phase

Do NOT generate:
- The sequence library (separate prompt)
- The scoreboard (separate prompt)
- Any test class

Why this works: Role priming (Section 1) sets the assistant voice. Decomposition with explicit ordering (Section 5) keeps the agent components consistent. The "do NOT" list prevents the helpful-assistant scope creep that breaks generated TBs.

Template 3 — Failure RCA (Hypothesis-Rank) (CoT + multi-gen self-consistency + constrained-output JSON)

[SYSTEM]
You are a senior DV engineer triaging a UVM regression failure. Your
job is ranked hypotheses with falsifying experiments, not free-form
analysis. Reason from evidence in the bundle, never from generic
knowledge of the protocol.

[USER]
Analyze this failure bundle and propose the 3 most likely root causes
ranked by probability. For each cause:
- one-sentence explanation grounded in a specific bundle field
- ONE falsifying experiment (plusarg, variant, probe) runnable in <5 min
- the bundle field (line/event/signal) that supports the hypothesis

Return JSON only, no prose:
[
  {"cause": "...", "prob": 0.XX, "falsifier": "...", "evidence_field": "..."},
  ...
]

Bundle:
{{bundle_json}}

Why this works: Constrained-decoding output (Section 8) forces the structure to be machine-checkable. Forcing "grounded in a specific bundle field" closes the hallucination escape hatch. Run the same prompt with temperature > 0 three times for self-consistency (Section 3); accept the hypothesis appearing as #1 across all three runs.

Template 4 — Coverage Plan from Spec (decomposition + JSON schema + intent-first)

[SYSTEM]
You are a UVM verification engineer producing a coverage plan. Output
is JSON consumed by a covergroup generator. Cover features, not signals.
Every cross must have a reason.

[USER]
Generate a coverage plan from the spec section below.

Spec section:
{{spec_section}}

Decompose into:
1. Feature categories (top-level groups)
2. Per-feature bins (specific values to cover)
3. Per-feature crosses (combinations - each must have a reason)
4. Per-feature negative scenarios (error and corner cases)

Output JSON shape:
{
  "features": [
    {
      "name": "...",
      "bins": [{"name": "...", "values": [...]}],
      "crosses": [{"name": "...", "with": [...], "reason": "..."}],
      "negatives": [{"name": "...", "scenario": "..."}]
    }
  ]
}

Why this works: "Cover features, not signals" is intent-first prompting (Section 10 pattern 3) at the system level. The "every cross must have a reason" constraint prevents combinatorial blowup of meaningless crosses. JSON shape pinned for downstream tooling.

Template 5 — NL to SVA Translation (structured-reasoning template + bounded output)

[SYSTEM]
You are a formal verification engineer translating natural-language
requirements into SystemVerilog Assertions. Output must be syntactically
valid SVA. Use vacuous-pass-aware patterns. Use semi-formal reasoning:
state premises, trace execution, derive the property.

[USER]
Translate this requirement into one SVA property.

Requirement:
"{{nl_requirement}}"

Available signals:
{{signal_list_with_widths}}

Clock: clk (posedge), Reset: rst_n (active low)

Reasoning structure:
1. Premises: what signals participate and what their roles are
2. Execution: cycle-by-cycle trace of the requirement
3. Derivation: the SVA expression

Output:
- property_name: ...
- property body: ... (assert property)
- cover property: ... (proves the property is reachable, not vacuously true)
- non-vacuity note: one sentence explaining why this property cannot
  be vacuously true

Why this works: Meta's semi-formal reasoning template (Section 10 pattern 4) is the direct ancestor — premises, execution, derivation. The explicit non-vacuity check addresses the most common SVA bug: properties that pass because nothing triggers them.

Template 6 — UVM Refactor (Subcategory-First) (subcategory-explanation pattern + role + scope discipline)

[SYSTEM]
You are a senior UVM engineer refactoring legacy testbench code.
You must FIRST identify the refactoring category, THEN propose the
change. Never refactor without naming the category from the list below.

[USER]
Refactor this UVM code. The refactoring category is: {{category}}.

Categories available:
- extract_base_class: pull common functionality into a base class
- replace_inheritance_with_composition: convert is-a to has-a
- virtual_sequence_extraction: factor sequences from a monolithic test
- agent_split: separate driver/monitor/sequencer responsibilities
- factory_override_simplification: collapse N overrides into config_db
- config_object_introduction: replace string parameters with typed config

Code to refactor:
{{legacy_code}}

Output:
1. Confirm the category and explain in 2 sentences why it applies
2. The refactored code
3. Backwards-compatibility note: what tests or sequences need to change
4. Risk assessment: 1-10, with one sentence justification

Why this works: The subcategory-explanation pattern (Section 10 pattern 7) is what took refactoring success from 15.6% to 86.7% in the SWE literature. The explicit refactoring categories prevent the model from inventing its own. The risk-assessment line forces the model to consider what it is changing — not just produce the change.

Apply to DV
  • Adopt one template this week. The hypothesis-rank template (Template 3) has the highest leverage for engineers actively triaging regressions.
  • Version templates in your team repo. Add them to the eval suite from Section 9 so model upgrades cannot silently break them.
  • Build one new template per quarter as you discover prompts that consistently work for your team's workflow.

13. What Does Not Work in DV (Honest)

An honest accounting of where the research and the lived experience agree the LLM falls short. Knowing these in advance is what keeps you from wasting an afternoon on a prompt that was never going to work.

Research The 2026 RTL survey explicitly names naive CoT as ineffective for IC design. CVDP (arxiv 2506.14074) shows SOTA models hit only 34% pass@1 on hardware-specific tasks. ProtocolLLM (arxiv 2506.07945) shows even syntactic validity is unreliable on novel SystemVerilog testbench code. The effective-context-window paper (arxiv 2509.21361) shows degradation past ~1,000 tokens of operational context even for frontier models with advertised 200K windows.
  1. Naive chain-of-thought for RTL design. "Let's think step by step" produces generic reasoning that does not map to the actual structure of HDL synthesis. The fix is HW-aware CoT scaffolding (Section 2), not abandoning CoT entirely.
  2. Long raw log dumps as context. Pasting 2,000 lines of UVM log into a chat is the textbook trigger of the effective-window degradation. Use the JSON bundle pattern instead — 200 structured events beat 2,000 lines of prose every time.
  3. Asking for "novel" protocol implementations. The model knows AXI, PCIe, USB, and AHB because the training data does. Your team's proprietary or pre-release protocols are not in there; generated code looks right and is wrong.
  4. Trusting LLM-emitted line numbers. Frontier models routinely produce line numbers off by 4-10 lines, even for code in the prompt. Treat the line number as a neighborhood pointer, never as an exact location. The fix is to verify before applying.
  5. Random few-shot examples. The research is unambiguous: random or text-similarity examples are sub-optimal. If you do not invest in example selection (Section 6), you leave significant performance on the table.
  6. One-shot complex IP verification. "Verify this PCIe controller" as a single prompt produces vague boilerplate. Decomposition is non-negotiable for tasks larger than a single component.
  7. Believing published pass@1 numbers as your own ceiling. Benchmarks like CVDP report averages across heterogeneous tasks. Your number on tasks scoped to your IP family with structured prompts and iteration is almost certainly higher — measure it yourself.
  8. Single-sample answers for high-stakes decisions. Any output that goes into shipping silicon (an SVA the design relies on, a coverage point that gates signoff, an RTL fix) should pass either self-consistency, formal verification, or human review — not all three is fine, none of three is not.
Apply to DV
  • Print this section as a poster. The bottom three failure modes (LLM line numbers, one-shot IP verification, single-sample high-stakes) are the ones a junior engineer is most likely to learn the hard way.
  • For tasks the model genuinely cannot do (novel protocols, multi-cycle deep reasoning), stop trying. Fall back to traditional methods; do not waste your morning on iteration #4 of a prompt that was never going to converge.

Reading List

Prompting Foundations & Surveys

  • arxiv 2406.06608 — The Prompt Report (PRISMA-grounded survey of 58 techniques)
  • arxiv 2402.07927 — Systematic Survey of Prompt Engineering
  • arxiv 2407.12994 — Prompt Engineering Methods for NLP Tasks
  • Liu et al. 2026, Frontiers of CS — Comprehensive Taxonomy of Prompt Engineering Techniques

Reasoning (CoT, Self-Consistency, Tree-of-Thoughts)

  • arxiv 2201.11903 — Wei et al., Chain-of-Thought Prompting
  • arxiv 2203.11171 — Wang et al., Self-Consistency Improves CoT
  • arxiv 2305.10601 — Yao et al., Tree-of-Thoughts
  • arxiv 2401.14295 — Besta et al., Chains, Trees, Graphs of Thoughts
  • arxiv 2510.01069 — Typed CoT / Certified Self-Consistency (2026)
  • arxiv 2603.08999 — Confidence-Aware Self-Consistency (2026)

Decomposition & Few-Shot

  • arxiv 2205.10625 — Zhou et al., Least-to-Most Prompting
  • arxiv 2210.02406 — Khot et al., Decomposed Prompting
  • arxiv 2310.09748 — LAIL: LLM-Aware ICL for Code Generation
  • arxiv 2305.14210 — Skill-Based Few-Shot Selection
  • arxiv 2412.02906 — Does Few-Shot Help LLM Code Synthesis?

Agentic Patterns

  • arxiv 2210.03629 — Yao et al., ReAct
  • arxiv 2303.11366 — Shinn et al., Reflexion
  • arxiv 2509.06239 — Proof2Silicon: RL Prompt Repair from Formal Feedback

Constrained Decoding

  • arxiv 2603.03305 — Draft-Conditioned Constrained Decoding
  • llguidance (github.com/guidance-ai/llguidance) — ~50µs/token CFG enforcement
  • Outlines library — JSON-schema-compiled valid-token lookup

Meta-Prompting & Promptware Engineering

  • arxiv 2502.00728 — Meta-Prompt Optimization for Sequential Decision Making
  • arxiv 2503.02400 — Promptware Engineering

SWE Prompting Empirical Studies

  • arxiv 2504.06939 — FeedbackEval (code repair)
  • arxiv 2603.19399 — DePro (debug)
  • arxiv 2506.13186 — Empirical Evaluation of APR
  • arxiv 2505.04441 — Execution Traces for Program Repair
  • arxiv 2506.09002 — PALM (Rust unit test coverage)
  • arxiv 2407.00225 — Prompt Engineering for Unit Test Generation
  • arxiv 2402.00097 — Code-Aware Prompting for Coverage-Guided Tests
  • arxiv 2508.14419 — Static Analysis as Feedback Loop
  • arxiv 2411.02320 — Empirical Study on Code Refactoring (15.6% → 86.7%)
  • arxiv 2303.07839 — ChatGPT Prompt Patterns for Code Quality

HW-LLM Frameworks & Benchmarks

  • arxiv 2604.27643 — HAVEN (UVM TB synthesis, 100% compile, 90.6% coverage)
  • arxiv 2504.19959 — UVM² (LLM-aided UVM machine)
  • arxiv 2605.04704 — UVMarvel (subsystem-level, 95.65% coverage)
  • arxiv 2510.15914 — VeriGRAG (structure-aware soft prompts)
  • arxiv 2510.15906 — FVDebug (waveform + RTL + spec debug)
  • arxiv 2405.06840 — MEIC (iterative RTL debug)
  • arxiv 2504.21770 — LASHED (LLM + static analysis)
  • arxiv 2405.12347 — Self-HWDebug (self-instructing security debug)
  • arxiv 2504.14560 — ReasoningV (efficient Verilog generation)
  • arxiv 2506.14074 — CVDP benchmark (34% SOTA pass@1)
  • arxiv 2506.07945 — ProtocolLLM (SV testbench benchmark)
  • arxiv 2212.11140 — Benchmarking LLMs for Verilog (foundational)

Context Engineering

  • Anthropic, Effective Context Engineering for AI Agents (2025) — the canonical industry essay
  • arxiv 2509.21361 — Maximum Effective Context Window
  • arxiv 2603.04814 — Beyond the Context Window (fact-memory vs long-context)

For the broader AI-in-DV map see the AI page. For the debug-specific deep-dive see the Debug page. For UVM patterns these techniques compose with, see UVM.