Who Signs When the Machine Is Wrong?
AI tools fail differently than people do. They fail fluently, confidently, and without a tell, and the legal profession's entire accountability structure was built for a kind of mistake that announces itself. A note from inside the deployments on what the signature on the filing now certifies.
A while ago I was reviewing the output of a contract analysis deployment, the kind of thing I do constantly now. The tool had been pointed at a stack of agreements and asked to pull and summarize a particular category of clause. The summaries came back clean. They were well organized, fluent, correctly formatted, and confident. One of them was wrong. The tool had described a provision as a mutual obligation when the underlying language made it unmistakably one-sided, and it had done so in the same calm, competent register it used for the forty summaries that were correct.
What stayed with me was not that the tool made a mistake. Tools make mistakes. What stayed with me was that the wrong summary looked exactly like the right ones. There was no tremor in it. There was no hedge, no "this one is unusual," no softening of the verbs, none of the small linguistic tells that a human produces without meaning to when they are unsure. The error was delivered with the same evenness as the truth, and if I had not happened to open that particular underlying document, I would have had no way to know from the output alone that anything was off.
I want to write about that evenness, because I think it is the most important and least discussed property of the systems we are putting into legal work, and because the profession's entire structure for catching mistakes was built for a kind of mistake that does not behave this way.
Here is the thesis, stated plainly. The machine does not fail the way a person fails. A person's failure announces itself. The machine's failure does not. And nearly everything we use to hold the practice of law accountable, from the duty of competence to the supervising partner to the signature on the filing, was designed around the assumption that a failure, if you are paying attention, will give you something to catch. That assumption is quietly false now, and we have not reckoned with what follows from it.
The mistake that announces itself
Think about how a junior lawyer is wrong.
A second-year associate who is unsure of a research question produces a memo that shows the uncertainty. The verbs go soft. The argument hedges. There is a footnote that says but see, or a sentence that begins it is unclear whether, or, most tellingly, the associate comes to your office and asks. Uncertainty in a person leaks. It leaks into the prose, into the body language, into the questions they ask and the ones they avoid. The whole apprenticeship I have written about elsewhere — the slow formation that the profession is now dismantling — is, among other things, a long training in how to read those leaks, in yourself and in the people you supervise.
The accountability structure of the profession is built on top of that legibility. The supervising attorney reviews the junior's work on the premise that careful reading will surface the weak spots, because the weak spots show. The duty of competence assumes a lawyer can know the boundaries of what they know, because a diligent person generally can feel where their understanding thins. When a lawyer signs a filing, the signature certifies that a reasonable inquiry was made, and the entire premise of that certification is that reasonable inquiry is capable of catching the error. Read it carefully enough, check the citations, pressure-test the argument, and the mistake reveals itself, because human mistakes, under scrutiny, tend to.
This is not a small assumption. It is the load-bearing assumption underneath the whole edifice. Review works because error is legible. Supervision works because uncertainty leaks. The signature means something because diligence can find what is wrong. Take away the premise that mistakes announce themselves under scrutiny, and you have not weakened one rule. You have pulled the floor out from under all of them at once.
How the machine fails instead
The systems I deploy do not produce legible uncertainty, and it is worth being precise about why, because the why is not a temporary flaw that the next model release will fix.
A large language model generates its output token by token, each one the most probable continuation given everything before it. The correct summary and the wrong summary are produced by the identical process. There is no separate faculty that generates truth and a different, shakier faculty that generates error, the way a person has a confident register and a hesitant one. The model has one register, and it is fluent, because fluency is what the process optimizes for. The wrongness is not a degradation of the output. It is the same machinery running on a case where the most probable continuation happens not to correspond to the fact of the matter. The output that is wrong and the output that is right are, at the level of how they are made, indistinguishable. That is why they look the same on the page. They look the same because, mechanically, they are the same kind of thing.
These systems can be made to express something that looks like confidence levels, and there is real work on calibration, on getting a model's stated certainty to track its actual reliability. But the default behavior of the tools as they are actually deployed, on the actual Tuesday afternoon, is to answer in a single even voice regardless of whether the answer is solid or invented. The uncertainty, to the extent the system has any internal analog of it, does not leak into the prose. It has to be deliberately extracted, and most deployments do not extract it, and most users would not know to ask.
So the failure mode is not that the machine is wrong more often than a person. On many tasks it is wrong far less often than a person. The failure mode is that when it is wrong, it is wrong silently, and the silence is the dangerous part. A junior associate who is wrong forty times will telegraph thirty-eight of them. The model that is wrong once in forty gives you no telegraph at all. You can review its output as carefully as you have ever reviewed anything and the careful review will slide right over the error, because the error is not sitting in a hedge or a soft verb or a nervous footnote. It is sitting in a sentence that reads exactly like the truth.
The signature was built for a different mistake
Now put those two things next to each other. An accountability structure that assumes mistakes announce themselves under scrutiny, and a tool whose mistakes specifically do not.
The signature still has to be signed. Someone certifies the filing, vouches for the memo, sends the advice to the client. The professional responsibility did not go anywhere. What changed is the relationship between the diligence the signature certifies and the error the diligence is supposed to catch. With a human work product, careful review is reasonably likely to surface the weak spots, because the weak spots show. With a machine work product, careful review of the output is not reliably capable of surfacing the weak spots, because the output is uniform whether or not it is sound. The only way to actually verify the machine's summary of a document is to go back to the document, the thing the tool was brought in to spare you from reading. The verification cannot be done at the level of the output. It has to be done against the source, every time, or it has not really been done.
This is the part that nobody has priced in. The efficiency case for these tools is built on not having to do the underlying work. The accountability case requires doing enough of the underlying work to catch a silent failure that gives you no hint of where it is hiding. Those two cases are in direct tension, and the tension does not resolve by reading the output more carefully, because reading the output more carefully is exactly the move that does not work here. We have a verification burden that has not shrunk. It has relocated, from the output back to the source, and intensified, because you are now hunting for an error that has been specifically stripped of every tell a hunter relies on.
And the people who used to build the instinct for this kind of hunting are the people the apprenticeship was forming, slowly, before we started taking the apprenticeship apart. The skill of feeling that something in a document is off before you can say why is precisely the skill that the slow work used to build and that the slow work no longer exists to build. So we are removing the formation that produced the verification instinct at the same moment we are deploying the tools that make the verification instinct matter more than it ever has. I do not think that is a coincidence so much as the same event seen from two sides.
What I learned building something to fail loudly
I build things on nights and weekends. The main one is an autonomous trading system I have been working on for a couple of years. I did not build it to make money, or not only that. I built it to see whether I could make something that learns from its own mistakes without lying to itself about them, and the single hardest and most important design problem in the whole system turned out to be exactly the problem I am describing here.
A trading system that is confidently wrong and silent about it will destroy itself. It will take a position it should not take, with a certainty it has not earned, and it will keep taking that position because nothing in its output told anyone, including itself, that it did not actually know. So I built the thing around a principle I now think is the most important property an automated system can have, and the one the market rewards least. I made it fail loudly. I made it separate the part that predicts from the part that acts, so that a confident-sounding prediction cannot quietly become a trade. I made it surface the boundary of its own knowledge, so that "I do not know" is a first-class output and not something you have to infer from a loss. The whole architecture is organized around the conviction that a system which fails silently is more dangerous than a system which fails often, because the silent failure is the one that compounds before anyone catches it.
Almost none of the legal AI tools I deploy are built this way. They are built to be fluent, helpful, and confident, because fluent, helpful, and confident is what demos well and what sells. The property I think matters most, the loud failure, the visible boundary, the honest "I am not sure about this one," is the property that makes a tool look weaker in a procurement bake-off and so it is the property that does not get built. We are optimizing these systems for the exact characteristic that makes their mistakes hard to catch, and we are doing it inside a profession whose accountability structure depends on mistakes being easy to catch.
This is the seam I keep finding myself standing on. The trading world and the legal world look nothing alike, but the underlying problem is identical, because it is not a problem about law or about markets. It is a problem about automated systems that fail without announcing it, deployed into human structures that assume failure announces itself. Any field that hands consequential judgment to a confident, silent system and keeps an accountability regime designed for legible human error is going to discover the same gap. Law is just the field where I happen to be standing when the floor gives.
What we actually owe
I do not have a clean fix, and I am suspicious, as always, of anyone who says they do. But I think the shape of the answer is at least visible, and it is not the answer the market is currently producing.
We owe ourselves tools that fail loudly. Systems that surface their own uncertainty as a first-class part of the output, that flag the summary they are least sure of instead of burying it in the same even voice as the rest, that make the boundary of their knowledge visible rather than leaving the user to discover it by being burned. This is buildable. It is mostly not built, because it is not what wins the demo. That is a choice the people procuring these systems can refuse to keep making, if they understand what they are choosing.
We owe the work a verification discipline that is honest about where the verification actually has to happen. Not a glance at the output, which we now know does not surface the silent error, but a designed, resourced, deliberate check against the source, with the understanding that some of the efficiency the tool promised has to be spent back on confirming the tool. A firm that pretends otherwise is not capturing efficiency. It is accumulating undetected error and calling it efficiency, and the bill comes due later, in a venue, with a client's name on it.
And we owe the people coming up the honest version of what the signature now means. It does not mean what it used to mean. It used to certify that a diligent person reviewed a work product whose flaws, under diligence, would show. It now certifies something harder: that a diligent person verified, against the source, the output of a system specifically built to give diligence nothing to catch. That is a heavier obligation wearing the same word, and the lawyers signing their names to machine-assisted work should know that the word got heavier even though it did not change.
I keep coming back to that one wrong summary, sitting in the stack, reading exactly like the right ones. The tool did not know it was wrong. That is forgivable; tools do not know things. What is not forgivable is for us to keep building and deploying systems whose mistakes are invisible, inside a profession that survives on mistakes being visible, and to keep signing our names as though nothing about the signature had changed.
The machine does not hedge. Someone still has to.