Why Your AI Project Needs a PoC First (And How to Run One)
Published 2026-09-26 · Updated 2026-10-13 · 7 min read · ShooWork (FREECO CO., LTD.)
A two-week proof of concept is the cheapest insurance an AI project can buy. What to test, how to run it, and the three signals to kill the build.
Short version: you run a proof of concept before an AI build because two weeks and a small test budget can tell you what six months and a six-figure spend would otherwise teach you the painful way. A PoC is not a demo, and it is not a prototype you keep. It is a disposable experiment whose only real deliverable is a number-backed yes or no.
The disaster script barely changes from company to company. Three months writing specs. Six months building. The budget gone. Then launch day arrives, the model meets real customer data for the first time, and performance looks nothing like the vendor deck. Nobody lied, exactly. The demo ran on ten clean records. Your database holds a hundred thousand messy ones, and the gap between those two numbers is where AI projects go to die.
We run our own AI platform at ShooWork — ShooWork Clip for video editing, ShooWork Write for SEO articles, ShooWork Ads for campaign work — and every single feature goes through a PoC before anyone writes production code. Not because we are unusually disciplined. Because we pay the model bill and the engineering hours out of our own pocket, and we have eaten this lesson more than once.
What a PoC Actually Tests
Most people think a PoC exists to prove the technology works. It doesn't. Anyone can make the technology look like it works — hand-pick ten tidy records and every model on the market looks like magic. A useful PoC answers three uncomfortable questions instead.
- Accuracy on your actual data. Not the vendor's sample set. Not the ten examples someone curated for the kickoff meeting. Pull 100 to 200 raw records straight out of your own systems — including the ones with broken formatting, typos, half-empty fields, mixed languages, and the free-text notes somebody typed at 2am. That is the distribution your system will actually live in. If accuracy collapses there, nothing downstream saves you.
- Unit cost. How many tokens, how many cents, how many seconds per record? Then multiply by your real monthly volume. That multiplication is where a lot of exciting projects quietly die. We had a feature that looked terrific in testing until we did the arithmetic — at production volume the model bill would have been larger than the value the feature could possibly create. We swapped the big model for a small one plus a rules engine and shipped it for a fraction of the cost. The savings paid several months of salary. We broke down that math in AI project costs: the one-time build vs. the monthly model bill.
- The shape of the errors. When the model is wrong, how is it wrong? Obviously wrong is cheap — you can filter it, flag it, route it to a human. Confidently, fluently, plausibly wrong is expensive, because nobody catches it until a customer does. Error shape determines how much guardrail engineering you need, and guardrails are routinely a bigger line item than the model integration itself.
Notice that none of these is "can it be built?" The answer to that is almost always yes. The question that matters is whether it can be built at an accuracy and a price that leave you better off than the status quo.
The Two-Week PoC, Day by Day
Two weeks is not an arbitrary number. It is short enough that nobody gets emotionally attached to the code, and long enough to build an honest test set. Here is the schedule we use:
- Days 1-2 — Write down what passing means. In plain text, before any code: accuracy must exceed X percent, cost per record must stay under Y, latency under Z seconds. Fire the arrow first, then paint the target, and you will find a way to call anything a success. This is the single most-skipped step and the reason so many PoCs end in "feels pretty good?"
- Days 3-5 — Build the test set. Sample 100 to 200 real records and have a human label the correct answer for each. This part is boring, and it is the part that makes everything else meaningful. A PoC without ground truth produces vibes, not evidence. Budget a real person's real hours for it.
- Days 6-10 — Wire up the model the laziest way possible. No architecture. No admin panel. No database schema. One script that reads the test set, calls the API, and writes results to a CSV. This code is a consumable. Making it elegant is waste, and worse, elegant code creates pressure to keep it. If you are still choosing between providers, run two and compare — our checklist for that is in how to choose an LLM model.
- Days 11-14 — Score, price, inspect, decide. Compute accuracy against your labels. Compute cost per record and multiply by volume. Read 20 or 30 of the failures by hand — not the summary statistics, the actual failures. Then hold the results up against what you wrote on day one and make the call.
The most valuable output of a PoC is not the demo. It is a build-or-don't-build decision with numbers behind it.
Three Signals That Should Make You Kill It
Killing a project is not a PoC failing. It is a PoC doing its job — that is literally what you paid for. These are our stop-loss signals:
Accuracy is more than ten points below your threshold. Prompt tuning, few-shot examples, and better retrieval typically buy you a few percentage points. They do not close a twenty-point gap. If you are that far off, the honest conclusion is that this task is not ready for this model generation, not that you need three more weeks of prompt engineering.
Unit cost times volume exceeds what the humans currently cost. If automation is more expensive than the people doing it today, and it is not faster or better, then you are buying a science project. Sometimes the fix is a smaller model, a cheaper retrieval step, or narrowing the scope to the 20 percent of cases that carry 80 percent of the volume. Sometimes there is no fix. Either way, you learned it in week two, not month eight.
The errors are confident nonsense and the use case can't absorb it. Anything touching money, contracts, medical detail, or regulatory language has a low tolerance for fluent fabrication. If your error analysis shows the model fails in ways a reviewer can't easily spot, and the domain punishes mistakes hard, stop. Or shrink the scope until a human sits in the loop on every output that matters.
When we hit one of these with a client, we say so directly: don't build this yet. Wait a model generation, or come back with a narrower slice. We lose the contract that month. We keep the relationship, which is worth considerably more.
What a Passing PoC Leaves Behind
A PoC that clears the bar hands you three assets, and they are worth more than the prototype code you are about to throw away.
First, a feasibility write-up with actual numbers in it — the thing that lets a finance conversation happen without hand-waving. Second, your labelled test set, which becomes the regression suite for the real build. Every time you change a prompt, swap a model, or tighten a retrieval step, you re-run those 200 records and see whether you improved things or quietly broke them. Teams that skip this spend the next year guessing. Third, a real grip on your cost structure, which drives pricing, margin, and how aggressively you can scale. That last one feeds straight into how to calculate AI ROI — saved hours only turn into money if you know what the automation costs to run.
Those three assets typically cut the production build's wasted effort roughly in half, because the expensive unknowns got resolved while they were still cheap to resolve.
How Teams Fake a PoC Without Meaning To
The most common failure mode is not a bad PoC. It is a PoC that was never really a PoC. A few patterns to watch for:
Success criteria written after the results came in. If the threshold moved, you ran a demo, not a test. Timestamp the criteria if you have to.
Test data that a human tidied up first. Cleaning the inputs is the same as deleting the hard cases. Whoever pulls the sample should not be the person who wants the project approved.
The PoC code becoming the product. This happens constantly. Something works on day nine, everyone is excited, and the throwaway script ships. Then it turns out there is no error handling, no logging, no cost ceiling, and no one can explain what happens when the API times out. Decide up front that the code dies at day 14.
Only testing the happy path of a multi-step process. If you are chaining steps — retrieve, then reason, then act — small per-step error rates compound fast. Four steps at 90 percent each gives you 66 percent end to end. Test the chain, not the links. That compounding is exactly why we are cautious about where we hand work to autonomous systems, which we covered in what an AI agent is and which processes to hand it.
The whole argument compresses to one line: in an AI project, a two-week PoC is the cheapest insurance you will ever buy. Scope it as its own phase with its own budget, and give everyone — including yourself — explicit permission to stop at the end of it.
FAQ
Q: How much should a PoC cost?
Think in terms of a small fraction of the full build — typically the model API spend for a couple hundred test records plus one or two people for two weeks. The API portion is usually the smallest part; labelling time is the real expense. We can't quote numbers for your case, but check the official pricing page of whichever model provider you are testing and multiply by your record count. If the PoC costs more than a few percent of the projected build, you have over-scoped it.
Q: Can we skip the PoC if the vendor already has a working demo?
No, and vendor demos are precisely the reason PoCs exist. A demo shows the model on data chosen to make it look good. Your PoC shows the model on data chosen at random from your own systems. Those two results are frequently twenty or thirty accuracy points apart. Run the vendor's tool against your test set — a vendor who refuses to let you do that has told you something useful.
Q: How many test records are actually enough?
100 to 200 is the practical sweet spot for most business tasks. Below about 50 you cannot distinguish a real difference from noise. Above 300 the labelling cost grows faster than your confidence does. If your data has distinct segments — different customer types, document formats, languages — make sure each segment has at least 30 records, because averages hide segment-level failures.
Q: What if the PoC passes but only just?
Treat a marginal pass as a scope-reduction signal, not a green light. Find the subset of cases where accuracy is comfortably above threshold and ship only that, with everything else routed to a human. Narrow-and-reliable beats broad-and-shaky every time, and you can widen the scope later once the regression suite gives you confidence.
Q: Who should run the PoC — the vendor or us?
Whoever runs it, you own the test set and the success criteria. Those two artefacts are the entire experiment; the code is incidental. If a partner is doing the engineering, insist that the labelled data lives with you and that the criteria are agreed in writing before work starts. You can see how we structure this across our own tools at ShooWork tools.