From AI idea to pilot.A guide

A prototype can be built quickly these days. It shows that something can work. It does not show whether it works in everyday operations. That is the gap a pilot is meant to close.

This guide describes the path from an AI idea to the decision after the pilot. It is organised in stages that build on one another. Each stage comes with a question that should be answered before the next one begins. The guide starts at the point where a use case has already been selected.

The prototype convinced everyone. So why can we still not say whether to go ahead?

Why is a successful prototype not enough?

A prototype answers a technical question: can a model handle this task in principle? An investment decision needs more. How good is the output compared with today’s process? How much checking does it require? Who is accountable, and what will ongoing operation cost?

A pilot that starts without these questions can easily end with a positive impression and no decision. The solution then remains in limbo, neither used nor retired.

Which task should the pilot address?

Start with a concrete task, not a technology. Describe it precisely enough for an outsider to understand: who does it today, what inputs are available, what output is produced and who works with it next.

Limit the scope deliberately. A pilot covering one process step in one team yields clearer insights than one that tries to change several processes at once. Also name a process owner. This person carries the pilot on the business side and shares in the final decision.

It helps to treat this description like a requirement: agreed with the stakeholders involved, written down and changed only deliberately.

How will you measure whether the pilot succeeded?

Success criteria are set before the start, not after the first results. They should be testable, much like acceptance criteria in software development, and phrased in the language of the business.

Typical criteria concern:

• the quality of the output compared with today’s process,
• the effort required for human review,
• acceptance among the people who work with the solution,
• the expected effort for operation and further development.

To do this, first describe how the task is handled today. Without that starting point, there is nothing to compare against.

Stop criteria are just as important. Define the conditions under which the pilot ends early, for example if the data needed turns out to be unavailable or errors cannot be detected reliably.

What data does the pilot need, and may you use it?

A pilot is only as meaningful as the cases it works with. Use real, representative cases, including the difficult and incomplete ones. Carefully selected examples make a solution look better than it will be in everyday use.

Clarify early which data the pilot will access, where it will be processed and who approves access. Information security and privacy belong at the start. Questions raised late can otherwise hold a pilot up just when it is due to go live.

Sensitive data is often not needed for the preparation. Anonymised or purpose-built examples are enough to clarify the task and the process.

Where does a person check, and who is accountable?

Human oversight is part of the process, not a safeguard added afterwards. Define where a person reviews, confirms or corrects the output, and what happens to errors.

Three questions help:

• Which outputs may be used without review, and which may not?
• Who is accountable if a faulty output slips through?
• How do corrections feed back so that the solution can be improved?

The answers determine not only the risk but also the value. A solution whose every output must be checked saves less time than expected. That should be clear before the start, not after.

How do you run the pilot?

The pilot runs for a fixed period, with real cases and with the people who do the task today. Depending on the risk, the solution works within the actual process or alongside it. That way, results can be compared without putting operations at risk.

Agree fixed checkpoints. At each one, check whether the criteria are still within reach and whether a stop condition has been met. Record observations as you go, including the unexpected ones.

Changes to scope are allowed, but not in passing. If the task is extended or changed during the pilot, those responsible decide this deliberately and adjust the criteria. Otherwise the pilot ends up measuring something other than what was planned.

How do you make the decision at the end?

The pilot ends with a decision meeting, not a presentation. It is clear in advance who decides and on what basis. The results are set against the agreed criteria, including observations that do not fit any of them.

Three outcomes are possible, and each is equally valid if it is well founded:

• Scale. The criteria are met, and the conditions for operation can be put in place.
• Rework. The approach is sound, but the task, the data or the checkpoints need adjusting. This leads to a new, tightly scoped pilot or an extension with revised criteria.
• Deliberately stop. Value, risk and effort do not fit together. The reasons and lessons are recorded so that they inform future assessments.

Deliberately stopping, in particular, needs the backing of leadership. A pilot ended with a clear rationale is worth more than one that carries on without a decision.

What does the move into operations require?

If you intend to scale, clarify the prerequisites during the pilot. They include:

• business and technical ownership for ongoing operation,
• integration with existing systems instead of manual workarounds,
• regular checks that output quality remains stable,
• rules on who approves changes to the solution,
• training and support for the people who work with the solution,
• a realistic view of the running costs.

The move into operations does not start after the pilot but in its planning. If you think these points through early, the decision at the end is one you can actually implement.