The Interface Layer, part 3: measuring whether it worked

Part 2 ended with two artifacts every handover produces on its own: an approval record, decision by decision, and a queue of what the system could not resolve. Those are the instruments. This chapter is about reading them. It is deliberately dry, because the alternative to measuring is believing, and believing is how companies end up paying for a feeling.

The six numbers

Each of these has a plain definition, a source you already have, and a reason it exists. Four are about the system; two are about the company.

Resolution precision. Of the times the system said "this invoice matches that order" or "this is the same supplier", how often was it actually right, judged by the final outcome? A proposal corrected at approval counts as wrong, and so does one that was accepted and found wrong later. Computed per class of decision from the approval record plus the escapes traced back to it. Measured per class, never as one blended number, because a blended number hides the class that is failing.

Approval rate. Of the proposals a person saw, how many did they accept without touching, at the moment they saw them? This looks like precision but measures behavior rather than truth: a person can wave through a wrong proposal, which shows up later as an escape, or correct a right one out of habit. What it tells you is whether the judgment is actually being taken and trusted. A system can be precise on what it proposes and still propose so little, or so cautiously, that the person is doing the job with extra steps. A rising approval rate on a class is the evidence that earns it sampled autonomy; a falling one is the earliest warning you get.

Escape rate. How many wrong values reached the ledger, the client, or the authority? This is the number that matters most and is the hardest to see, because an escape is by definition something nobody caught at the time. It surfaces later: a supplier's complaint, a correction at month close, a query from the accountant. Count each one and trace it back to the decision that produced it. A system that never reports escapes is a system whose escapes nobody is counting.

Time to close. Per item, from arrival to recorded, with the human step included. The system's part is seconds; the human's part is where the number lives. If time to close does not fall after handover, the proposals are not being trusted or the boundary is in the wrong place.

Queue age. The unresolved queue has a size and an age. Size tells you how much judgment the system is declining to take; age tells you whether anyone is working the queue. A queue that grows and ages is a handover that stopped halfway.

Hours returned. The business number, and the only one an owner will ask for. The seam's monthly hours from the inventory, minus the hours the approver now spends on it. Measured honestly it includes the queue work and the sample reviews. It should be reported next to escape rate every time, because hours returned with escapes rising is a loss dressed as a saving.

One more, derived: cost per correct decision. Everything spent on the seam in a month, model calls and people included, divided by the number of decisions that were correct. This is the unit economics of the interface layer, and it is what "cheaper model" or "more automation" must be judged against. A cheaper model that lowers precision often raises this number.

The baseline you get for free

The question "did it work" needs a "compared to what". The shadow stage from part 2 provides it without any extra effort: while the person keeps doing the seam the old way and the system proposes in parallel, you are measuring both at once. The person's time per item, their own error rate found at month close, and the system's precision against the person's decisions all come out of the same weeks.

Record the baseline before the assisted stage begins and keep it. Six months later, when the seam feels normal and nobody remembers the eleven minutes per invoice, the baseline is the only thing that shows what changed.

What a demo cannot show

Every vendor demo, ours included, shows a document read correctly and a fluent answer. That demonstrates capability, which is real and is not the question. The question is what happens on your documents, with your spelling variants, over a month that contains a format change, a new supplier, and a week when the approver was on holiday. Precision on your variants, drift, the queue, and escapes are all things only a month shows. There is no way to shorten this, and a vendor who claims there is has not measured their own product.

Reading the patterns

The numbers are more useful together than apart. A few combinations recur.

Precision high, approval rate falling. The system is right but the person has stopped trusting it, usually because of one visible mistake or because proposals are arriving without their evidence. Look at the last correction they made and what the proposal showed them.

Approval rate high, escape rate rising. The dangerous one. Approvals have become a reflex, the boundary has turned into theatre, and errors are passing through with a signature on them. Reduce the volume the person sees by moving safe classes to sampled autonomy, so their attention returns to the decisions that need it.

Queue growing, precision stable. The system is declining more cases than before. Usually a source changed: a new invoice layout, a renamed field, a partner switching formats. This is drift, and the queue caught it as designed.

Hours returned flat, everything else fine. The approver is re-doing the work before approving, either from habit or because the proposal is not laid out in the order they work. Watch them for an hour.

Precision falling on one class only. Something specific: a supplier whose invoices break the pattern, a rule in memory that was right in March and wrong now. Per-class numbers exist for exactly this moment.

The cadence

Weekly during shadow and assisted stages, on one page, reviewed with the approver present. Monthly after that. The review is a fifteen-minute conversation about the numbers, the queue, and the one thing to change; it is not a report to be filed.

The evaluation protocol

These are the questions to put to any vendor of an interface layer, including us. They are questions a serious one will welcome.

• Define your metrics. Ask for their definitions of precision, approval, escape, and time to close. If the definitions are not per class of decision, the numbers will hide the failing class.
• Run in shadow on our data. Two weeks minimum, on the seam we chose, against the person who does it today. The output is a baseline and a per-class precision number, or it is a sales call.
• Show us the queue. Where does "I could not determine this" go, who sees it, and how old is it allowed to get?
• Show us a correction becoming memory. The approver fixes one line; what happens next time?
• Show us an escape. What is the path from "a wrong value reached the ledger" back to the decision and the evidence that produced it? How long does it take?
• Where does the boundary live? In code that cannot be talked past, or in instructions to the model?
• What can this credential do? Read what, write what, and what happens if it leaks?

A vendor that answers all seven with specifics is one you can measure. A vendor that answers with a demo is one you can only believe.

The limits of measuring

Small numbers lie. A class with fourteen decisions a month cannot show a precision difference between two setups; wait for the numbers or merge classes carefully. Seasonality matters: the month with the year-end invoices is not the month of August. And the oldest trap of all, the measured number becoming the goal: an approval rate rewarded on its own will rise for the wrong reason. Pair it with escape rate, always, and let the two argue.

Measurement will not make the interface layer work. It will tell you, week by week, whether it is working and where it is not, which is the difference between a system you run and a system that runs you. Part 4 turns to the other side of the same layer: the forms your people and your customers fill, and what replaces them.