Before we let an AI score anything in our pipeline, we have it score the same thing twice.

Before we let an AI score anything in our pipeline, we have it score the same thing twice.

I'm letting AI decide what software I build, starting with Chrome extensions. Of the six things we score for each opportunity, one, "product advantage", is a judgment call, so a model makes it.

The question I didn't want to skip: give it the same evidence again, do you get the same score?

So we tested it. 40 pairs, same evidence, scored twice: • 95% differed by at most 1 point • 70% were identical

Our bar was 80%. It passed, so it went into the daily pipeline. When the evidence hasn't changed, we reuse the cached score instead of asking again, so the number can't drift on its own.

It's the consistency check you'd run on human reviewers. It doesn't tell you the score is right. It tells you the score isn't noise, which is the minimum before you build decisions on it.

If you use AI to rate or rank things: have you checked whether it agrees with itself?

Free verdict on a direction you're weighing, 1 a day: numberandreason.com/?src=en-li#join