回测结果Backtest results
E28 历史回放与 E29 研判回测 · 实验完成于 2026-10-09 / 2026-10-10E28 historical replay and E29 verdict backtest · experiments completed 2026-10-09 / 2026-10-10
这页把回测怎么跑、结果是什么、结论的边界照实写出来。出处列在第 5 节：E29 有公开文章可以直接打开，E28 的实验原文暂未公开。能公开打开的部分，你可以自己核对、自己下判断。This page sets out how the backtest ran, what it found and where the conclusions stop. The sources are listed in section 5: the E29 article is public and can be opened, while the original E28 write-up is not yet public. Where a source is public, you can check it and judge for yourself.
1. 怎么跑1. How the replay ran
- 两个回放点：2024-04-01 和 2025-04-01。每组只看那一天能拿到的商店数据，按默认评估标准排出当天的前 20 个需求。Two replay points: 2024-04-01 and 2025-04-01. At each point we used only the store data available on that day and took the top 20 needs by our default evaluation criteria.
- 每个回放点跑 2 次、每次 20 个需求，合计 80 个需求。命中 = 这个需求在回放点之后 12 个月内，出现了用户数达到 1 万、且不是重传或克隆的新产品。Each replay point was run twice with 20 needs per run, 80 needs in total. A hit means that within 12 months after the replay point, the need produced a new product that reached 10,000 users and was not a re-upload or a clone.
- 对照是同规模需求：每个被选中的需求，和同一规模档里随机抽的需求比，避免“大市场本来就更容易冒出新产品”把倍数夸大。The comparison is against same-size needs: each selected need is compared with random needs in the same size band, so that “big markets naturally produce more new products” does not inflate the multiple.
- 这一轮回放只回放了信号和范围。评分项、形态和研判都没有进入回放名单。This replay covered signals and range only. Scoring items, product shape and the verdict step were not part of the replay list.
2. 核心结果：评估标准回放约 1.8 倍2. Core result: about 1.8× on the evaluation-criteria list
80个历史需求historical needs
28命中hits
15.2同规模期望命中expected hits by chance at the same size
1.84×提升倍数lift
| 选的需求Needs selected | 命中Hits | 同规模期望命中Expected by chance | 提升倍数Lift | 90% 区间90% interval | 判定Verdict |
|---|
| 80 | 28 | 15.2 | 1.84 | 1.21–2.37 | 有效Effective |
评估标准回放约 1.8 倍——80 个历史需求里命中 28 个，随机同规模期望 15.2 个，90% 区间 1.2–2.4。Across 80 historical needs, 28 surfaced by our evaluation criteria later succeeded — about 1.8× the 15.2 expected by chance (90% interval 1.2–2.4).
边界：同一次实验里，“加工数据 + 方法论说明”的方案回放是 2.60 倍，“加工数据 + 查询”是 1.63 倍。1.84 倍说明这份名单有效，但它不是实验里的最好成绩。The limits: in the same experiment the “processed data + methodology notes” setup replayed at 2.60× and “processed data + SQL” at 1.63×. The 1.84× shows that this list works; it is not the best result in the experiment.
3. 研判这一步没有增量3. The verdict step added nothing
我们另外测了“做 / 不做”的研判。同一批 100 个需求里，研判只说 8 个“做”，其中 2 个后来成功；这个样本量按跑前标准不足以判“有效”（“做”组命中要 ≥10 个、需求要 ≥15 个）。We also tested the “build / don’t build” verdict. Among the same 100 needs it said “build” for only 8, and 2 of those later succeeded. Under the criteria fixed before the run, that sample is too small to call the verdict effective (the “build” group needed at least 10 hits and 15 needs).
区分能力：AUC 0.58（0.5 = 瞎猜）。在评估标准前 20 名内部，研判说“做”的 5 个里成功 2 个（40%），说“不做”的 35 个里成功 12 个（34%），只差 6 个百分点；跑前定的增量门槛是 10 个百分点，所以判“无增量”——功劳在名单，不在研判。Its discrimination was weak: AUC 0.58 (0.5 = guessing). Inside the top 20 of the evaluation criteria, 2 of the 5 needs it said “build” for succeeded (40%), against 12 of the 35 it said “don’t build” for (34%) — a gap of 6 percentage points. The increase we had fixed before the run had to be at least 10 points, so the verdict step was judged to add nothing: the list did the work, not the verdict.
改问“12 个月内出现 ≥1 万用户新来者的可能性”后（E29b），可能性分数的 AUC 是 0.88；但只按需求规模排序，AUC 就有 0.92，比可能性分数还高；同一规模档内部，研判的 AUC 只有 0.70（53 对，样本小）。也就是说，可能性分数主要学到的是规模效应，而不是规模以外的判断力。When we changed the question to “the likelihood that a newcomer reaching 10,000 users appears within 12 months” (E29b), the likelihood score reached AUC 0.88. But ranking by need size alone already gave AUC 0.92, higher than the likelihood score; within the same size band the verdict’s AUC was only 0.70 (53 pairs, a small sample). In other words, the likelihood score mostly learned the size effect, not judgement beyond size.
| 指标（E29b）Measure (E29b) | 结果Result |
|---|
| 可能性分数 AUCLikelihood score AUC | 0.88 |
| 只按需求规模排序 AUCAUC of ranking by need size alone | 0.92 |
| 同一规模档内部 AUCAUC within the same size band | 0.70 |
E29b 按跑前标准也只是“观察”，不是“有效”：前两条门槛过了，第三条（最高三分之一 ÷ 最低三分之一 ≥1.5）只有 1.15，没有过。我们没有在事后改门槛。Under the criteria fixed before the run, E29b was only “observed”, not “effective”: two thresholds were met, but the third (top third ÷ bottom third ≥ 1.5) reached only 1.15. We did not change the criteria after seeing the result.
4. 成功口径的局限4. What the success definition can and cannot say
- 成功口径是「12 个月内 ≥ 1 万用户新来者」：一个需求在 12 个月内出现用户数达到 1 万、不是重传或克隆的新产品。Success is defined as “a newcomer with at least 10,000 users within 12 months”: a new product that reaches 10,000 users within 12 months and is not a re-upload or a clone.
- 它衡量的是新来者能不能起量，不完全等于「对我们这个小团队值不值得做」——后者还要看收入、维护和竞争成本。It measures whether a newcomer can reach scale. It is not the same as “whether this is worth building for a team our size”: that also depends on revenue, maintenance and competition costs.
- 这个口径在大市场里更容易达到，所以对照按需求规模分组；即便如此，E29b 里只按规模排序就有 0.92 的 AUC，说明口径本身被规模主导。The definition is easier to meet in large markets, which is why the comparison is grouped by need size. Even so, size alone ranked at AUC 0.92 in E29b, so the definition itself is dominated by size.
- 因此对外只报评估标准名单的倍数（1.8 倍），不把研判说成准确率或预测率，也不把 1.8 倍说成研判的效果。研判结论在服务里只作风险与切口说明，不作硬筛子。So we report only the multiple for the evaluation-criteria list (1.8×). We do not present the verdict step as an accuracy or prediction rate, and we do not present 1.8× as the verdict’s effect. In the service the verdict is used only as a note on risk and entry angle, not as a hard filter.
- 这页也不主张我们比通用智能体或 Chrome-Stats 更准：E28 的三类题门槛没有全部达到。1.8 倍只适用于这次回放的样本和窗口。This page also does not claim we are more accurate than general-purpose agents or Chrome-Stats: not all three task thresholds in E28 were met. The 1.8× applies only to this replay’s sample and window.
5. 出处5. Sources
- E28 方案对照实验：结果 —— 第 3 节“T2 历史回放”表（核心结果 1.84 倍那行）。实验原文暂未公开。E28 comparison experiment: results — section 3, the T2 historical replay table (the 1.84× row). The original write-up is not yet public.
- E29 研判回测：结果与跑前设计 —— 公开文章：/articles/e29/。E29 verdict backtest: results and pre-registered design — public article: /articles/e29/.