Confusion matrix example for an AI detector threshold

[ethanjamescolez]

Worked example: interpreting a detector threshold with a confusion matrix.

Suppose a class has 100 submitted drafts. In this hypothetical example:

- 15 drafts are actually AI-assisted
- 85 drafts are not AI-assisted
- a detector flags 80% of the AI-assisted drafts
- it also falsely flags 10% of the non-AI drafts

Calculate the flagged group.

True positives:

15 x 0.80 = 12

False positives:

85 x 0.10 = 8.5

Since a real count has to be whole, this would be about 8 or 9 false positives in a group of 100. Using the expected value:

total flagged = 12 + 8.5 = 20.5

The share of flagged drafts that are actually AI-assisted is:

12 / 20.5 = 0.585..., or about 58.5%.

So even with 80% sensitivity, a flag is not proof. The false-positive rate and the starting mix of drafts both affect the meaning of the result.

Is this a fair worked example for showing why a detector score should be treated as a probability signal rather than a final answer?

No product link is needed. This is just a probability and expected-value example.