The Trust Problem With AI-Designed DNA Constructs
9 min read · Updated July 14, 2026
A growing number of tools now let you describe a cloning or CRISPR experiment in plain language and get back a finished design: a plasmid map, a set of primers, a knock-in donor sequence. Almost all of them share the same failure mode, and it is not a bug that better prompting fixes. Sooner or later, in a long enough multi-step design session, the model hands back a sequence it never actually derived from anything. It pattern-matched a plausible-looking answer instead of computing one.
This is not a criticism of any particular product's engineering effort. It is a structural property of how these systems are usually built: a capable model, a system prompt asking it to be careful, and an assumption that asking is enough. That assumption is worth being skeptical of, and it is worth understanding exactly why it fails before looking at what actually fixes it.
Why a capable model still invents a sequence
A language model generates its next token by predicting what is plausible given everything said so far, not by tracking which specific claims in the conversation are backed by an actual computation and which are not. Nothing about that mechanism changes once the model is wired up to call tools. It changes what plausible answers are available, not what the model is fundamentally doing when it produces one.
Ask a capable agent to design a three-fragment Golden Gate assembly and it will, correctly, call out to design primers, check overhang compatibility, and simulate the ligation across several real tool calls. Ask a follow-up question two turns later — "what's the final sequence around the second junction" — and there is nothing inherent in the model stopping it from answering directly out of its own running impression of the conversation, rather than re-deriving the answer from the actual last tool result. The answer will look completely plausible. It may even happen to be correct. The problem is that "looks right" and "was actually computed" are not the same claim, and only one of them is checkable from the outside.
This gets worse, not better, as a task gets more capable-looking. A model confidently chaining ten tool calls across a complex multi-part design is exactly the situation where it is most likely to, somewhere in the middle, quietly answer one sub-question from memory instead of from a tool, because nothing in the architecture distinguishes those two sources once they are both just tokens in the same context window.
Why "we told it not to" is not a safeguard
The obvious response is to add an instruction: "never state a sequence unless it came from a tool result," or "always verify before reporting success." This helps, and it is not nothing. It is also not a guarantee, for a simple reason: a system prompt is just more text the model conditions on, competing with everything else in a long context, including the user's own increasingly specific and insistent follow-up questions.
An instruction is a preference the model usually follows. It degrades over long conversations, it competes with contradicting pressure from the user ("just give me the sequence, I don't need the whole explanation"), and — most importantly — it depends on the model remembering to apply it under every phrasing of every request, including phrasings nobody happened to test for before shipping. Evaluation suites test known phrasings. Production traffic does not limit itself to known phrasings.
This is a known limitation of prompt-only alignment for agentic, tool-using systems, not a criticism specific to biology tools: asking a sufficiently general model to reliably self-police a narrow behavior, across unbounded conversation shapes, is asking it to be reliably correct about exactly the kind of thing language models are least reliable about. A consequential output — a construct someone is about to spend a week and a reagent order on — needs a guarantee. An instruction is not one.
The only fix that actually holds: move the guarantee outside the model
If the model cannot be relied on to reliably follow an instruction under every possible conversation shape, the instruction is the wrong place to put the guarantee. The alternative is to make the surrounding system — the code the model's output passes through before it ever reaches a user — structurally incapable of reporting an unverified result as trustworthy, regardless of what the model does or says.
Concretely, that means two things, and both have to be enforced by code that runs unconditionally, not by a request the model is trusted to make. First, a sequence should only ever be able to enter the conversation as fact through one channel: an actual executed computation, or the user's own input. The model can reason about it, summarize it, explain it — it should never be the origin of it. Second, the system should refuse to label a design "finished" unless a separate, independent check has already run against the exact final output and reported success — checked in code, after the fact, whether or not the model remembered to ask for that check itself.
Neither of these is a novel idea specific to AI. It is the same fail-closed, least-trust design principle behind a build pipeline that will not ship an artifact past a failing test, regardless of how confident the commit message sounds. The reason most AI tools skip it is not that the principle is obscure — it is that wiring a real guarantee like this through a genuinely open-ended, multi-turn agent loop is considerably harder than writing a good system prompt, so most products stop at the prompt.
What this looks like when it is actually built this way
SeqBench-GPT, currently an early-access feature, is built around exactly this boundary for the guarantee that matters most: it will not report a design as finished on trust. If a session produces a candidate construct or donor sequence, the assistant cannot mark that design complete until an independent verification step has run afterward, in that same session, and reported a pass. That check is plain server-side logic, evaluated every time the assistant tries to stop — it runs whether or not the model remembers to ask for it, and there is no phrasing of "just tell me it's done" that talks its way around it.
The architecture backs this up on the input side, too: every number, sequence, and result that appears in the conversation as a tool's output is the unmodified return value of that actual computation — the model cannot forge or substitute what a tool reports back. What still depends on the model behaving as instructed, rather than being independently checked line by line, is narrower: whether it chooses to answer a passing follow-up question by re-deriving the answer or by describing something from its own memory of the conversation so far. That is a real, ongoing boundary to keep pushing on, not a solved problem — which is exactly why the verification gate is built to not depend on the model's cooperation at all: even a model that gets sloppy mid-conversation still cannot get a design marked "finished" without a real, passing, independently run check on it.
Exactly how the rest of this is wired through the loop — the specific checks involved, the exact tool sequencing, the model behind it — is not the subject of this article. The verification gate is the point; treat the rest as an implementation detail, in the same way you do not need to know a compiler's internals to trust that a type error stops a build.
What this guarantee does not mean
A structural guarantee like this closes one specific failure mode. It is not a blanket claim about correctness, and it is worth being precise about the difference.
- It does not mean the verification step catches every real-world failure mode. A pass is only as meaningful as the specific, bounded thing that check actually looks at — treat "verified" as a defined, checkable claim, not an assurance that nothing else could possibly be wrong.
- It does not mean the model's proposed experimental strategy was a good one. Being unable to report an unverified result as "done" says nothing about whether the underlying design choice — which assembly method, which guide RNA, which primer pair — was actually the right call for the experiment.
- It does not remove the value of a human reviewing the final design before committing bench time or placing a synthesis order. A code-level check removes one common, specific failure mode: a silently unverified or partially hallucinated result being reported as fact. It is not a replacement for scientific judgment.
- It is an early-access feature and should be treated with the same scrutiny you would apply to a first draft from a new lab member: promising, usable, still worth checking yourself before it leaves the building.
A better question to ask any AI lab assistant
Whether or not a given tool discloses how it works, most of these questions are answerable just by using it and trying to break it:
- If you keep pushing back, can you get it to report a design as "finished" anyway? This is testable directly, not something you have to take on faith.
- Is the check something the model decides to run, or something the surrounding system will not let it skip no matter what the conversation looks like?
- Can the sequence in the final answer be traced to one specific computed result, or could it plausibly have been reconstructed from the model's own memory of the conversation so far?
- What does "verified" actually mean here — a specific, named check with a defined, statable scope, or an unspecified claim that "the AI double-checked its work"?
The pattern is bigger than biology
These questions generalize past biology. The underlying pattern — a capable model, a helpful-sounding instruction, and no code-level backstop — shows up anywhere an AI agent is trusted to report its own work as done. The stakes here happen to be a wasted week of bench time and a reagent order that did not need to happen, which is exactly specific and expensive enough to be worth asking about before you trust the answer.
Frequently asked questions
Can an AI hallucinate a DNA sequence?
Yes. A language model generates plausible-sounding continuations of a conversation, and nothing about that mechanism inherently distinguishes a sequence it actually computed from one it reconstructed from its own running impression of the discussion. In a long, multi-step design session this becomes more likely, not less, since there are more opportunities for a sub-answer to come from memory instead of from an actual tool call.
Is prompt engineering enough to stop an AI agent from hallucinating in a multi-step task?
Not reliably. An instruction in a system prompt is a preference the model usually follows, not a guarantee — it competes with the rest of a long context, degrades under user pressure to skip steps, and depends on the model applying it correctly under every phrasing of every request, including ones nobody tested for. Consequential outputs need a check enforced in code after the fact, not just an instruction asked for in advance.
What does it mean for an AI-designed construct to be "verified"?
It should mean a specific, named, code-executed check ran and reported a pass — not a general assurance that the AI "looked it over." Ask what the check actually covers and how directly it's tied to that specific design, since a passing verification is a bounded claim about whatever that check looks at, not a guarantee against every possible error.
What is SeqBench-GPT?
SeqBench's AI design copilot, currently an early-access feature. It plans and executes real tool calls across a cloning or CRISPR design session, and if that session produces a candidate construct or donor sequence, it will not report the design finished until a passing, independent verification step has run afterward in that same session.
Should I still manually check a construct an AI assistant says it verified?
Yes. A "verified" claim is only as broad as whatever specific check produced it. Treat it the same way you would treat a colleague telling you they checked something: useful and generally trustworthy, but worth a final look yourself before you commit bench time or place a synthesis order, especially for anything novel or unusual.
How is a code-enforced verification gate different from just telling a chatbot to double-check its work?
Telling a model to double-check is still an instruction it can fail to follow under some conversation shape nobody anticipated. A code-enforced gate is checked by the surrounding system after the fact, unconditionally, whether or not the model remembered to ask for it — the difference between a rule the model is trusted to remember and a rule it is structurally unable to skip.