On Oct 10, 2026, I backtested my own AI's verdicts on what's worth building.
On Oct 10, 2026, I backtested my own AI's verdicts on what's worth building. The surprise was how little it actually knew.
Setup: standing at April 2024 and April 2025, with only the data available then, the same research pass judged 50 user needs at each point — 100 calls in all — as worth building or not. Then I checked which needs actually grew a new, non-clone product past 10,000 users within 12 months.
Result: its worth-building / not-worth-it calls scored AUC 0.58, against a coin flip's 0.5. Of the 15 needs that later succeeded, it had flagged only 2 in advance.
Reframing the question as "probability this succeeds" lifted AUC to 0.88. A control exposed why: ranking by scale alone reached 0.92, and within the same scale band it scored only 0.70 (53 pairs). The model mostly learned "bigger wins"; beyond scale, its judgment is thin.
So now the AI doesn't make that call on its own: every verdict is still "build" or "don't build", but it states the premises it rests on and the risks that would flip it. The sample is small (100 calls), so this stays an observation, not a rule.
Do you read "big market" as "worth building"?
10-16 correction: this paragraph originally said we no longer give a build-or-not verdict. In practice every free report ends with a "build" or "don't build" verdict, with its premises stated. The text now says so.