A Pilot Isn’t a Program Yet
Every pilot is run under conditions the organization will never offer again. There is a dedicated coordinator, protected time, a sponsor who answers email the same day, and participants who get a little extra attention because everyone is watching. Those conditions are not a flaw. They are how new ideas get a fair test. They are also the reason a promising pilot result says less than it seems to about what will happen next.
A pilot answers one question: can this idea work when it has help? A program has to answer a harder one: can it work on an ordinary Tuesday, with ordinary staffing, inside the systems everyone already uses? The space between those questions is where many good ideas quietly fade.
Success under special conditions
Consider what a typical pilot borrows. Approvals move faster because a leader is personally involved. Data is tracked in a separate spreadsheet by someone whose job is to track it. Staff make exceptions to normal rules because the project has permission to. A referral partner gives extra attention because the relationship is new and visible.
When the pilot ends, each of those supports goes away at once. The coordinator returns to a previous role. Temporary funding runs out. The spreadsheet has no owner. What looked like a working service turns out to have depended on the scaffolding around it.
None of this means the idea failed. It means the pilot measured something different from what leaders will be deciding on.
Write down how it is supposed to work
The best protection is to describe the program plainly before it starts. The CDC Program Evaluation Framework makes describing the program an early step, and for good reason. A short written logic model, covering the resources, the activities, who they reach, and the short-term changes that should appear before the long-term result, does two things.
First, it separates the essential parts from the incidental ones. A particular meeting, form, or job title may not need to survive, as long as the function it serves does. Second, it shows which resources are temporary. If the logic model depends on a grant-funded position, that dependency is now on paper, and someone has to say what replaces it.
AHRQ’s guidance on planning for sustainability recommends creating the plan at the outset. By the end of a pilot, many of the choices that determine whether it can last have already been made.
Define every indicator before anyone sees a number
Pilots are often judged by measures that were never fully defined. “Follow-up rate” can mean five different things depending on who is counting. A measure is ready to support a decision only when the team has written down:
- The numerator and the denominator.
- The data source.
- How often it will be reported.
- Who is responsible for reviewing it.
- What the team will do if it rises, falls, or differs across groups.
That last item is the discipline most often skipped. A measure with no possible response is decoration, and it tends to crowd out the few measures that matter. A good pilot report is short because each number in it is tied to an action.
Set the criteria before the results arrive
Here is the core argument: leaders should decide what would make them continue, adapt, or stop a pilot before they see the results.
The reason is human, not technical. Once a pilot has a team, a story, and early participants who benefited, it develops a constituency. Results that are mixed get read generously. Results that are poor get explained by start-up problems. Without criteria set in advance, the decision drifts toward whatever avoids an awkward conversation, which is usually continuing at a reduced level that helps no one well.
Criteria set in advance can be simple:
- Continue if the core result improved by a stated amount and the work fits within normal staffing.
- Adapt if the result improved but the model depended on conditions that won’t last, and name what would need to change.
- Stop if the result did not move, or moved only for the people who needed help least.
Stopping is a legitimate outcome. So is redesign. A program that cannot be sustained uses resources another approach could put to better use, and the people it serves are the ones who notice when it disappears.
Test it in ordinary conditions early
One way to close the gap is to remove some of the special conditions while the pilot is still running. Have regular staff do the work during normal hours for part of the period. Move data collection into existing systems. Ask what happens during a vacancy or a busy season. Ask whether the work depends on one unusually committed person, and, if it does, write down who reviews performance, who trains new staff, and who maintains partner relationships if that person leaves.
Funding is part of the picture, but not all of it. A 2024 study in Preventing Chronic Disease examined state tobacco control programs across eight sustainability domains, including funding stability, partnerships, organizational capacity, program evaluation, and strategic planning. Its setting was specific, but it is a useful reminder that a renewed grant cannot make up indefinitely for unclear ownership or a model that doesn’t fit routine work.
Calling it what it is
Organizations would make better decisions if they used the word “program” more carefully. A pilot is an experiment with a question attached. It becomes a program when the question has been answered under ordinary conditions, with defined measures, against criteria chosen in advance. Until then, the most honest label is the modest one.
Sources
- CDC: Program Evaluation Framework
- AHRQ: Planning for Sustainability
- Preventing Chronic Disease: Barriers and Facilitators to Program Sustainability Among State Tobacco Control Programs
About the author
Sherry Sayari, MPH, is an Operational Standards Evaluator who has led population health, community outreach, and social services programs. More on the About page.
