Building got too easy
More than two years ago, I started an incubation team. The deal was simple: six weeks to prove an idea was both feasible and useful. Call it a prototype or proof of concept. The goal wasn’t just to show something to others, but to convince yourself.
The trouble with prototypes is that they mean different things to different people - too many times I've seen directors just asking to improve the prototype. 6 months later the team still prototypes. That's not s strategy - it's scope creep and lack of vision.
The question I asked instead to prototype creators was: Does this give you enough confidence that it will work - that you could spend the next 12 months of your life making it real? And then: Does it convince others enough that they’d put a million dollars behind it?
If the answer was no, the idea either was too early, or not good enough (or we didn't have enough information to act on). In that case, it didn’t make sense to commit 2 of ML engineers to try to built it.
Today, with vibe coding and fast prototyping, the hard part isn’t building anymore. The real challenge is building conviction. Two things matter most:
1) Evaluating quality. A large share of the six weeks went into designing reliable, unbiased evaluations. A simple baseline - often just prompting an LLM - was judged with a few hundred high-quality samples and the LLM-as-a-judge.
2) Assessing usefulness. We had a group of 50+ people (from outside the immediate team) to get feedback from. Online surveys (Qualtrics) helped reach the right user profiles. In-person interviews required empathy and listening, and a reminder to avoid a wish-list trap. In both cases, it was a substitute for talking to real users. A hard feat when the company wants to keep the feature in the stealth mode.
When building becomes cheap and fast, the bottleneck shifts. What’s truly hard is knowing whether what you’ve prototyped is any good -- especially when everyone else is out there building, too.