"We'll do a pilot" is often a way of deferring a decision rather than making one. A pilot that has not been designed to answer something specific will produce a demo, some enthusiasm, and no basis for the next decision.
Write the question down first
One sentence, decided before anything is built. For example: does automating first-response to inbound enquiries reduce our average response time from hours to minutes, on our real enquiry volume, within a fixed budget?
That sentence sets the scope, the measure and the exit. Without it, a pilot ends with everyone describing a different outcome.
Choose a measure you already track
This is the part most often skipped, and skipping it is fatal. If the measure is new, you have no baseline, and without a baseline the result is unfalsifiable — any number can be described as an improvement.
Good measures are boring and already in a system somewhere: time from enquiry to first response, number of records typed in by hand, hours a week on a named task, percentage of calls answered within thirty seconds.
Agree the exit before you start
Both exits. What happens if it works, and what happens if it does not.
A pilot with no defined failure state does not fail — it lingers, gets extended, and quietly consumes the budget that should have funded the next attempt. Being able to stop cleanly is what makes a pilot low-risk, and it is worth more than any discount.
What a pilot genuinely proves
- Feasibility on your data. Not on demo data. Yours, with its gaps and its inconsistencies.
- Direction and rough size of the effect. Whether the number moves, and whether it moves enough to matter.
- What the work is actually like. Whether the vendor asks good questions, whether they surface problems early or late.
What it does not prove
- That it scales. Behaviour at ten times the volume is a different question, and a pilot deliberately avoids it.
- That people will adopt it. A pilot runs with attention on it. Adoption is tested when the attention moves elsewhere.
- Long-term reliability. Six weeks tells you nothing about what breaks in month eight when an upstream API changes.
Those are real risks. The answer is to name them as things the pilot does not cover, rather than to pretend it covered them.
The failure mode nobody names
The pilot works, everyone is pleased, and nothing happens next — because no one agreed in advance who would own the system, whose budget would carry it, or what "in production" means here.
Decide that at the start, while it is an easy conversation. After a successful pilot it becomes a negotiation.
Questions
- How long should a pilot run?
- Long enough to see the measure move and to hit a normal bad week — often a few weeks rather than a few days. Running it only across a quiet period produces a flattering result you cannot rely on.
- Should a pilot be free?
- A free pilot tends to get the attention a free thing gets, on both sides. A small fixed price with a defined outcome and a clean exit is usually a better structure than free, for the buyer as much as the vendor.
- What if the pilot shows it does not work?
- That is a successful pilot. It answered the question for a fraction of the cost of finding out during a full build, and it usually reveals the adjacent problem that was worth solving instead.
Want this working on your numbers rather than in an article? We scope it on one call.
Book a Consultation Call