How to test an AI drawing review before you trust it

The pitch is always the same. Hand the model the drawing set, get back a list of code problems. I have watched enough of these demos to want a real answer about whether it works, and a demo has never once given me one.

The question worth asking is narrower than “is AI good at code review.” Can this model do this particular check, on my drawings, and can I tell the difference between an answer it verified and an answer it guessed?

Answering it takes a real test, run the way you would run any other test: hold everything constant, vary one thing, and build an answer key before you look at the results. This is the version I have been using on architectural review, which mostly holds up and breaks in a few places worth knowing about.

Who this is for: anyone in a design office deciding whether an AI review belongs in a workflow. What you need: a closed-out project with issued redlines, a licensed reviewer willing to argue with you, and about two weeks. What you do not need: new software, at least for the first pass.

Two things to settle before you run anything

Client drawings are client documents. So are consultant models, specifications, and the comment letter from plan review, which usually carries a reviewer’s name and phone number on it. Whatever your firm’s rule is about putting client material into an AI tool, this sits squarely inside it. Get the approval in writing, or build the pilot on a closed-out project the firm owns and whose sensitivity has lapsed.

The second thing has to be written into the instructions themselves, not just understood. Every finding is a draft for a licensed reviewer, checked against the edition and amendments the authority has actually adopted before it changes a drawing or goes out in a response. The model never closes a finding and never signs anything. If that sentence is not in the file, the file is not finished.

Sort every check into one of two piles

Before writing a line of instructions, take your list of checks and sort each one. This sort does more for safety than anything else in the method.

The first pile is arithmetic. Measure something, look up a number, compare. Occupant load is area divided by a load factor. Is the corridor 44 inches clear where it serves 50 people or more. Is the door 32 inches clear with the leaf open 90 degrees. Does measured travel distance beat the table limit. Do the fixture counts on the plan match the schedule, and do both match what the code requires. A model can answer these.

The second pile is judgment. What occupancy is this space, when it has food service, seating, and a back of house behind it? Is that a corridor, an aisle, or an exit passageway? Does this alteration trigger the accessibility provisions? Does the authority in your county read that provision the way your office reads it? The model does not answer these. It raises them, lists the options, says what each one would change downstream, and stops.

The reason for the split is the failure mode that should worry you most, which has nothing to do with invented answers. A model reads the wrong object, pulls perfectly real data out of it, and then finds real ways downstream to confirm the wrong reading.

Occupancy classification is that wrong object sitting at the top of a life safety plan. Get it wrong and the load factor is wrong, so the occupant load is wrong, so the exit count and the travel limit and the fixture count are all wrong, every one of them computed correctly. They will cross-check perfectly, because they all inherit the same mistake.

So pin the premise by hand. The first thing your instructions do is make the model read back what it thinks it is working from: occupancy per space, construction type, sprinkler status, code edition and amendments, alteration level. Then it stops until somebody confirms. Thirty seconds, and the worst failure mode becomes a question.

Write down how you actually do one check

Most people skip this step, and it does more work than everything around it.

Pick one check. Do it by hand across a full set rather than a sample, because you are not trying to produce answers here. You are trying to watch yourself work. Then write out every step and every thought in the order they happen, which will look much less like a rule than like somebody hunting around a drawing.

Mine for a single egress door goes something like this:

Find the door tag on the plan. Note the leaf size. Ask whether that door is actually in the means of egress or whether it is a closet, because half the doors on the sheet are not my problem. If it is in the path, go to the door schedule and confirm the leaf size matches the tag, because they disagree more often than anyone admits. Then stop and remember that clear is not nominal. A three foot leaf does not give three feet of clear width. Take off the door thickness and the stop and you land around 33 and a half inches, which passes. A two foot ten leaf lands around 31 and a half, which fails. So I need the real leaf, not the opening size. Then check the hardware set, because panic hardware and closers stick out into the clear width. Then check the swing, because the door has to not eat the corridor when it is open, and it has to have maneuvering clearance on both sides.

That paragraph holds six sub-checks, two cross-references, one judgment call, and the trap that catches people. None of it survives being compressed to “check door clear width, 32 inches minimum,” which is why the transcription has to be what you hand over, with the checklist line serving as its index.

Then iterate. Take one instance whose answer you already know and ask the model for that one instance. When it is wrong, ask why it concluded that, and fix the gap it actually had instead of the gap you assumed. When it is right, ask how it knows. On a compliant-or-not question, a coin flip looks exactly like competence at n=1, and a right answer reached by guessing is a failure that has not happened yet. Run the loop against more than one model, or you will write instructions that only fit one model’s habits.

Use the answer keys you already have

The obvious way to build ground truth is to do the work yourself several times and reconcile the differences. That is fine for pulling known values into a spreadsheet. It works badly for finding defects, because doing it four times tells you nothing about what you missed all four times.

An architecture office has better keys sitting in the archive.

Back-checked redlines on a superseded revision. Take a set where redlines were issued and the corrections were made, then run the model against the drawings from before the corrections. Those redlines are an answer key written by a human reviewer at the time, with no idea you would ever use it this way, which makes it a genuine blind test on errors that really happened, and it costs nothing to produce.

One caveat belongs in your results. Redlines are a lower bound, since anything the human reviewer also missed is absent from the key and will look like a false positive when it might be a genuine catch. Every finding that is not on the key gets judged on its merits before you score it.

Closed-out comment letters from plan review. A comment letter is an answer key written by the only office whose opinion counts. Run the model against the set as submitted and score whether it predicted the comments. This also quietly changes the question from “is this compliant,” which is an argument, to “what will plan review flag,” which is the thing anyone actually wants to know.

Seeded defects. Take a clean set and break it on purpose: shrink a door leaf, delete a dimension, change a schedule value so it disagrees with the plan, move a fixture so the clear floor space fails. Twenty or thirty of these, weighted toward what you fear most, is the only key guaranteed to be complete, which makes it the only one that gives you a true catch rate.

Keep all of it off the machine. That sounds too obvious to write down, right up until you remember that these tools carry context between sessions, and that you have almost certainly discussed this project with one before.

Build at least one check that works a different way

This is the rule I would hold onto hardest. Your checks have to be independent in kind rather than merely independent in location, because two readings of the same wrong number are indistinguishable from confirmation. A model reads a dimension off a schedule, finds the same figure printed on a section, and reports it verified. One error, sighted twice.

Architectural sets are full of that trap. A wrong occupant load travels from the area schedule into the life safety legend into the egress calculation, and all three agree, because all three are the same number wearing different clothes. Cross-referencing catches copying mistakes, and a bad derivation gets caught only by a check built on a different kind of evidence.

Measure the drafted geometry instead of reading the dimension. Measure the line, convert with the sheet scale, compare to the annotation. This catches the defect class nobody catches systematically: the note says one thing and the geometry says another. A string reading 44 inches over geometry drafted at 41. A door tagged three feet with a two foot ten leaf drawn. It is tedious enough that no human does it on every door, and nearly free for something running overnight. Fair warning, this layer usually needs software IT has to approve, so scope it early and say so in your results if you ran without it.

Derive the same number three ways. Occupant load as printed in the legend, area from the schedule divided by the load factor, and area measured off the plan polygon divided by the load factor. The third is the only genuinely independent derivation. The first two both trace back to the same area figure, so their agreement means nothing on its own.

Close the sums and bound the impossible. Room areas add up to gross area within a wall thickness. Exit capacities add up to at least the occupant load. Egress width never gets narrower in the direction of travel. Door count on plan equals door count in the schedule equals door count in the hardware sets. And nothing gets past a two-hour assembly six inches thick, a ramp at 1:8 labeled accessible, or nine-inch risers.

Make “I don’t know” a real answer

Give the output five states instead of two:

• compliant and verified
• not compliant and verified
• cannot be determined
• needs interpretation, which is your second pile
• uncited, meaning no traceable source, which gets thrown away and logged

The third state is where this gets interesting, because in a drawing review it is frequently the finding itself.

If the model cannot work out a door’s clear width because the leaf size is not on the plan, not in the schedule, and not in the spec, it has found a hole in the drawings, which is what a QA review is for. Plan review will ask the same question, in writing, on their schedule instead of yours.

So every “cannot determine” gets sorted by hand before you score anything. Information missing from the set, score it as a catch. Information present and the model could not find it, score it as a miss. That half hour of sorting separates “the AI is weak here” from “our drawings are weak here.”

Watch the count, too. A model reporting zero unknowns on a real drawing set is telling you it is uncalibrated, and I would treat that as a defect rather than a good score. A reviewer can work with eight findings and five open questions, because the five are a map and a person can be put on them. Thirteen confident assertions of unknown reliability leave you re-checking all of it or trusting all of it.

Make every code claim name its edition

Everything above is about getting the drawing right. Code review has a second and nastier axis, which is the citation, and the failure there usually takes the form of a real provision that does not govern here.

Model training data is saturated with base model code text, and your jurisdiction is not on the base model code. Mine deletes an entire chapter of the IBC and runs accessibility through a state program plus the ADA standards instead, so a model answering from memory will cite a chapter that does not apply here. Mine runs an NFPA-based fire code, so IFC sections come back fluently and confidently and do not exist in the governing document. Editions drift as well: provisions move, renumber, and change substantively between them.

Every one of those produces a finding that is fluent, specifically cited, internally consistent, and wrong.

So require an edition, a document, and a section on every citation. Route code questions through a reference you have already verified rather than through the model’s memory, and flag anything outside that verified set as needing interpretation instead of answering it. Then score citation accuracy on its own, separate from whether the conclusion was right, because a correct conclusion reached from the wrong section will not be correct the next time you ask.

Score the misses, not the accuracy

Drop the single accuracy percentage. It suits a quantity count, where a wrong cell is a wrong cell, and it suits a review badly, because the two kinds of error cost wildly different amounts. A missed non-compliance goes to permit, sometimes to bid, occasionally to the field. A false positive costs a reviewer ten minutes, and at volume it costs you their trust in the tool. Both of those matter, and they are nowhere near the same size.

Report two numbers separately and never blend them: how much of the key it caught, and how much of what it flagged was real. Then report citation accuracy, premise accuracy, and the split between drawing-side and model-side unknowns, each on its own. Set your floor for false positives before you run, not after you have seen the results.

Declare your tolerances in writing first as well. Three feet zero equals 36 inches. Whether 32.0 and 2’-8” count as the same answer. Where you round fractional occupants, which is a real argument and not a formatting nit. How far measured geometry can sit from the annotation before it becomes a finding, because otherwise that layer will bury you in noise. Anything you did not declare in advance is an error.

What to run first

Pick one check class. Egress is the right first choice, because it is mostly arithmetic and it has a right answer. Pick a closed-out project with issued redlines. Build the schema from the QA checklist your firm already has rather than one you invented, because digitizing your own checklist is more defensible and it makes the output comparable to a human reviewer’s. Transcribe your reasoning, iterate on single known instances, then run one model, score it, sort the unknowns, and map which checks it fails.

Success at this stage is a defensible answer key and a score you could reproduce. Whether the model did well is the less important half. If the method does not hold up with one model, comparing five of them means nothing.

After that, hold everything constant and vary only the model, add accessibility, add the seeded defects, and report cost and runtime alongside accuracy. What comes out works less like a leaderboard than like a routing table, telling you which model to use for egress arithmetic, which for dimensional checks, and which for citation discipline.

Coordination review comes last. Comparing the architectural set against structural or mechanical has no code table to check against, only two drawings that might each be wrong, and the output is a list of unknown length where every conflict needs a severity ranking or nobody can use it. Come to it with a method that already works.

One last thing about those error maps, which I would expect to be more useful than the scores. Human errors scatter, and I would bet a given model’s errors cluster instead. You cannot predict which cell a tired reviewer will fat-finger, but if a model reliably blows door clear widths you can learn that once and hand the column to a person every time after. A model with a stable, known failure signature is worth more to me than a slightly more accurate one whose mistakes are everywhere.

The short version

• Get the confidentiality question answered in writing before any client material moves.
• Write the licensed-review line into the instructions, not just into your intentions.
• Sort every check into arithmetic or judgment. The model answers the first pile only.
• Make it read back its premises and stop for confirmation before it computes anything.
• Transcribe how you actually do one check, in full, before you write instructions.
• Build ground truth from redlines, comment letters, and seeded defects, and keep all three off the machine.
• Include at least one check that runs on a different kind of evidence.
• Require an abstain state, and treat zero abstentions as a defect.
• No code citation without edition, document, and section.
• Declare tolerances and rounding before the run.
• Report catches and false positives separately, and never as one number.