How to evaluate a document automation solution with a pilot

Seven weeks, in an order that matters more than the length. The baseline comes first, because once the vendor's system is running nobody measures the old way any more. The holdout is labelled before the vendor configures anything, so the ground truth cannot be bent toward the output. The blind run comes before the shadow run, so the scored numbers come from documents the vendor never saw. The shadow week then tests what a static set cannot: live volume, latency, new senders and the review queue under load.

Two weeks of baseline cover the weekly cycle (Monday backlogs, Friday cut-offs). If your volume has a month-end or seasonal peak that changes the document mix, include it or sample from it. A vendor that can show a working proof of concept sooner compresses weeks 3 and 4; it should not compress the baseline or the blind test.

One workflow, named by what goes in and where it lands. "Automate our documents" cannot be scored; "purchase orders received in the orders mailbox, entered as sales orders in the ERP" can. Write down the documents included and excluded, the target system and the exact fields written, and mark the critical ones. Three examples:

The sector guides set out each of these workflows in detail: sales order automation, freight quote automation, CargoWise data entry automation and subcontractor bid leveling. For estimating departments choosing which workflow to start with, the estimating department guide covers that choice and a pilot on a bid already priced.

The baseline is what every pilot figure is compared with, so it is measured the same way: per document, all in. The common mistake is to time the keying alone. The time a document really costs includes checking it, chasing what is missing (a call to the customer, an email to the carrier, a question to the subcontractor), and correcting what went wrong after entry.

A shared sheet with one row per document and four time columns is enough. Ask people to log as they go, not at the end of the day, and tell them why: the baseline is the comparison, not a productivity review. Twelve months of volume from the system log will show whether the two weeks were typical.

There is no universal sample size. What a sample can tell you depends on how precise the answer must be, and two numbers make that concrete. A rate measured on a sample comes with an interval, and the interval narrows slowly as the sample grows. And when you observe no critical error at all, the most you can say is that the true rate is below a bound that depends only on the sample size: about 3 divided by the number of documents. Both come from the standard methods for a proportion in the NIST/SEMATECH e-Handbook (Wilson interval; exact binomial limit, here one-sided).

Read it backwards to size your holdout: if you need to show that critical errors stay below 1% and you expect none, you need about 300 in-scope documents. A holdout of 50 documents, common in demos, cannot show anything below about 6%.

How to draw it

1. Take every document received over a recent continuous period, from four to eight weeks, not the ones someone remembers as typical.
2. Tag each one with a stratum (table below) and a sender.
3. Draw at random within each stratum, keeping the natural mix, and add extra documents from the rare strata so that each has enough to be reported on its own.
4. Split at random into a development set (about a third) for the vendor to configure on, and a holdout set (the rest) that you keep.
5. Make sure the holdout contains senders that also appear in the development set and some that do not: the second group is what the system will meet in production.

A single accuracy number treats a wrong phone number like a wrong ship-to address. Grade errors by field and by consequence, and do it in week 0, before any output exists, so the grid cannot be adjusted to fit a favourite vendor. The buyer defines it; the vendor can comment.

Give each critical field a cost from your own records: what a re-delivery, a customs amendment, a credit note or a bid error has cost on average. Where the consequence cannot be priced (a safety document, a regulatory filing), treat the field as zero tolerance: it is always reviewed, and the decision is about how fast the review is, not whether to skip it.

One sheet, the same for every vendor and for the team's own entries on the holdout. Every metric has its definition, its formula and the person who measures it. Publish the definitions with the numbers in the readout: a rate without its definition cannot be compared with anything.

The licence is the visible line. The pilot is where the others become measurable.

Compute the same figure for the current process: baseline hours × loaded hourly cost, plus the team's own escaped errors × the same error costs. That is the number the automated process has to beat, with the same definitions on both sides.

The automation ROI calculator starts from example values. After a pilot, each one has a measured replacement:

Two limits to keep in mind. The calculator values hours only: add the escaped error cost from the pilot separately, because it can reverse the result, as vendor A shows. And it counts every document outside the straight-through share as fully manual; when assisted review is much faster than keying, the real gain is larger. Note the difference in the business case rather than raising the share to absorb it. For freight specifically, building the business case for freight document automation goes further into the cost lines.

A system that worked on the holdout drifts when customers change formats, carriers add fields or the price list changes. Decide who does what before go-live, and write the vendor's half into the contract.

Keep the holdout. After launch it becomes the regression test: any change is re-run on it and scored with the same sheet before it reaches production.

Settle what happens at the end before the start. Five things are produced during the pilot and the run, and the contract should say who owns each one and how it leaves:

Documents such as orders and shipping files carry personal data (names, emails, phone numbers). Where the GDPR applies, the processing contract must require the vendor, at your choice, to delete or return that data at the end of the service and delete existing copies, and to make available the information needed to demonstrate compliance and allow audits (GDPR, Article 28(3)(g) and (h)). In the EU, the Data Act, applicable since 12 September 2025, also sets a framework for switching between providers of data processing services; whether your contract falls under it is a question for counsel.

The exit clause, in practice

• A notice period and an assistance period during which the vendor keeps running while you switch.
• Export of extracted data, logs and configuration in open formats, on request and at the end, with any fee stated in advance.
• Return or deletion of documents and data at your choice, with written confirmation.
• A parallel run with the next system, on the holdout and on live documents, before the old one stops.

Where documents are processed, who holds the keys and how access is removed are part of the same review; Mirage sets out its own answers on its security page.