What Training Legal AI Models Taught Me About AI's Blind Spots

Since 2025, part of my work has been training AI models on legal reasoning. As an independent contractor for several companies building these systems, I write and evaluate legal problems, judge the model's answers, and explain in detail where its reasoning holds up and where it breaks down. After twenty years of practicing law, it's an unusual way to apply that experience. It has also given me a clear view of how these tools think, or appear to.

I can't discuss any company's confidential work, but I can share the general lessons. They're useful for any lawyer or business owner deciding how much to trust AI with legal questions.

Lesson one: fluency is not accuracy

The first thing you notice is how confident the models sound. The writing is clean, the structure is professional, and the tone is authoritative. That fluency is genuinely impressive, and it's also the biggest risk. A model can produce a beautifully written analysis that rests on a wrong rule, a misapplied standard, or a case that doesn't say what the model claims.

Experienced lawyers learn to distrust work that sounds too certain. AI output deserves exactly that skepticism. The polish tells you nothing about whether the law is right.

Lesson two: the models are best at the middle of the distribution

AI handles common, well-documented legal questions remarkably well. Standard contract issues, black-letter rules, frequently litigated doctrines. Where there's a lot of consistent material for the model to learn from, it performs well.

The trouble starts at the edges: unusual fact patterns, minority rules, recent changes, procedural quirks in a specific court, and the intersection of several doctrines at once. That's precisely where real legal problems tend to live. Clients rarely pay lawyers for the easy middle. They pay for the edge cases.

Lesson three: jurisdiction and currency are weak spots

Law is local and it changes. A model may blend rules from different jurisdictions or confidently state a rule that has since been amended. In training work, some of the most common errors I flag involve applying a general or majority rule where a specific jurisdiction does something different, or missing that a statute or rule has changed.

For practitioners, the takeaway is simple: never rely on AI for what the law currently is in your jurisdiction without checking a primary source.

Lesson four: reasoning can look right while skipping steps

A good legal analysis identifies the issue, states the rule, applies it to the facts, and reaches a conclusion. Models often produce something that looks like that structure while quietly skipping the hard part, the application. They restate the facts and the rule side by side and then leap to a conclusion without actually doing the analysis that connects them.

Much of the value of human training is pushing models to show their work, to engage with the specific facts, and to acknowledge when the answer genuinely depends on something unknown. Lawyers reviewing AI output should look for the same thing. Does the analysis actually apply the rule to these facts, or does it just sound like it does?

Lesson five: the models want to give you an answer

AI systems are built to be helpful, and that creates a bias toward giving an answer even when the honest answer is "it depends" or "we need more facts." Good lawyers know that identifying what you don't know is often the most valuable part of the advice. Models are improving here, but they still tend to resolve uncertainty that should be flagged.

When you use these tools, ask them directly what facts would change the answer and what assumptions they're making. You'll often get a more useful response, and you'll see where the soft spots are.

Lesson six: it is getting better, quickly

None of this means AI is useless for legal work. The opposite. The improvement I've seen in a short period is real, and the tools are already valuable for drafting, summarizing, organizing and issue-spotting. The work of training them is precisely about closing these gaps, and the gaps are narrowing.

But "better" doesn't mean "reliable without supervision." The right mental model for legal AI today is a very fast, very well-read junior colleague who is sometimes confidently wrong and doesn't always know it. You'd never file that colleague's work without review. Treat AI output the same way.

What this means for small firms

Small firms have the most to gain from AI, because they have the least capacity to spare. They also have the least margin for an expensive mistake. The answer isn't to avoid these tools. It's to use them where they're strong, verify them where they're weak, and keep a lawyer accountable for everything that leaves the office.

That's the approach I bring to helping firms adopt AI: practical, specific about the risks, and grounded in seeing firsthand how these systems actually reason.

Want to use AI in your practice with your eyes open?

Book a consultation and I'll help you put it to work where it's strong and guard against where it isn't.