What AI Actually Changes About Legal Document Search (And What It Doesn't)

Keyword search fails in a specific way, and the shape of that failure determines what a replacement has to do.

If a document says "indemnification" and you search for "hold harmless," keyword search returns nothing. The document isn't ranked badly — it's absent. Lawyers have compensated for decades by constructing Boolean queries with every synonym they can think of, which works to the extent they think of the right ones.

Semantic search genuinely fixes that. Embedding-based retrieval matches on meaning rather than tokens, so the hold-harmless clause surfaces. It's a real, measurable capability and the strongest case for these tools. If your users' main complaint is that they can't find documents they know exist, it will help.

The line falls directly after. Semantic search improves recall — finding what's relevant. It does much less for precision — excluding what isn't. In a legal corpus precision is usually the harder problem, because the corpus is full of near-identical documents that differ in ways that matter enormously.

Where the legal domain fights the technology

The documents are long, and relevance is sectional. A ninety-page agreement isn't relevant or irrelevant as a unit — the limitation-of-liability clause is what you want and the other eighty-nine pages are noise. Systems that embed whole documents dilute the signal past usefulness. Chunking helps, and chunking legal text is harder than chunking prose, because a clause's meaning often depends on defined terms established fifty pages earlier. "Company shall not be liable" means little without knowing which entity "Company" is in this instrument.

Legal language is precise in ways embeddings flatten. "Shall" and "may" sit close in vector space and are opposite in effect. "Best efforts" and "commercially reasonable efforts" look near-identical to a model and are the subject of actual litigation. A system optimized for semantic similarity treats these as the same, which is exactly wrong here.

Negation is a persistent weakness. Finding agreements that do not contain an arbitration clause is a reasonable, common request, and similarity search handles absence poorly. That's structural, not a tuning problem.

Demo conditions versus your corpus

The demo runs against a clean corpus. Yours is not clean, and the differences are the ones that matter.

Your DMS contains eleven versions of the same agreement, plus the executed copy, plus a scanned PDF of the executed copy someone filed separately. Semantic search will return all of them as highly similar, because they are highly similar. Ranking the executed version above the drafts requires metadata and business rules, not better embeddings — and that metadata is exactly the part of your DMS that's inconsistent.

Your corpus contains scanned documents with imperfect OCR. It contains documents where the substance is in tracked changes or comments. It contains email attachments filed without the email that explains them.

Then there's the part vendors demonstrate least: security trimming at scale. Results have to be filtered to what each user is permitted to see, and in a firm with ethical walls that's not a simple filter. Applying it after retrieval means a partner and an associate running the same query get different result counts and different ranking quality, which users notice and interpret as the tool being unreliable. Applying it before retrieval, correctly, at speed, is a genuinely hard engineering problem and is where several products quietly underperform.

Ask about ethical walls specifically, and ask how the filtering is applied relative to retrieval.

Evaluating it properly

Don't evaluate on searches you invent. Collect thirty real queries from the people who will use this, including the vague ones, and have each person identify what a correct result set looks like. That takes a couple of days and is worth more than every vendor demonstration combined.

Run them against your own corpus, not a sample. Vendors resist this, and it's the only test that means anything, because the failure modes above are all properties of real corpora.

Then look at the failures rather than the successes. Every system gets most of the thirty right. What distinguishes them is the rest: does it return nothing, or something plausible and wrong? The second is far worse in legal work, and it's the characteristic mode of generative systems, which are built to produce fluent output rather than to decline.

Finally, ask how the system reports uncertainty. A tool that can flag a weak match lets the user decide how far to trust it. One that presents every result identically gives them no way to distinguish a strong match from a marginal one, which pushes the entire burden of judgment onto someone who came to the tool for help with exactly that.