A model does not decide. You do.
A classifier outputs a number between zero and one. Somebody then picks the point above which action is taken — and that choice, not the model, determines who is helped and who is harmed. It is the least technical decision in machine learning and usually the most consequential.
6 min · Intermediate · Playable
Two ways to be wrong, and they are not equal
A test can be wrong in two directions. It can raise an alarm when nothing is happening — a false positive. Or it can stay quiet when something is. Every threshold trades one for the other: you cannot reduce both at once, only choose which you would rather have.
False positive
Flagged, but innocent. A declined card, a callback for a biopsy, a job offer in the spam folder.
False negative
Missed. Fraud goes through, a tumour is not spotted, spam reaches the inbox.
In plain words
Nobody can tell you the right threshold from the maths alone. It depends on which of those two sentences describes a worse day for a real person — which makes it an ethical judgement wearing a decimal point.
Move the line, and watch who it hits
The chart shows every case the model scored. Grey above the line is genuinely negative; gold below is genuinely positive. They overlap — they always overlap — and the slider decides where you cut. Start with card fraud, then try cancer screening without changing anything else.
correctly flagged as fraudulent
A real customer’s card is declined at the till. Annoying, recoverable, and they may leave.
A fraudulent charge goes through. The money is gone and the bank refunds it.
left alone, and right to be
Human cost
0arbitrary units · lower is betterMissing fraud costs roughly fifteen times more than a false alarm, so the threshold belongs low — catch more, accept the complaints.
Six hundred simulated transactions per scenario, with costs chosen to reflect the real asymmetry rather than any particular organisation’s numbers. The mechanism — one slider, four outcomes, a cost that depends entirely on which mistake hurts more — is exact.
In plain words
Watch the accuracy figure as you move the slider on the cancer scenario. It stays high almost everywhere, including at settings that miss most of the tumours — because 96% of the scans are clear, and a model that flags nothing at all is 96% accurate. That is why accuracy alone is close to useless on rare events.
The same model, four different answers
Press “Find the cheapest threshold” on each scenario in turn. The optimum moves dramatically — low for fraud and cancer where missing is catastrophic, high for spam where a lost email matters more than a nuisance one. Nothing about the model changed. Only what the mistakes cost.
Precision answers: when it flags, is it right?
Raise the threshold and precision climbs — you only flag the obvious cases. It is the number you care about when acting on a flag is expensive or intrusive.
Recall answers: of the real cases, how many did we catch?
Lower the threshold and recall climbs. It is the number you care about when missing one is the thing you cannot live with.
You cannot maximise both
Every point you gain on one, you pay for on the other. Reporting only the flattering one is the oldest trick in machine learning demos.
The loan case is the hard one
Costs are nearly equal, and the false positives are people quietly refused credit who never learn why. Invisible harm does not show up in a dashboard unless somebody insists on measuring it.
What this explains
Why “99% accurate” means nothing on its own
Ask what the base rate is. On a problem where 1% of cases are positive, a model that says no to everything is 99% accurate and completely worthless.
Why the threshold belongs to the business
The data team can produce the curve. Only the people who own the consequences can say which point on it is acceptable — and that conversation should happen before launch, not after the first complaint.
Why this is the classic interview question
Ask a candidate to choose a threshold and explain it. Anyone who answers 0.5 without asking what the mistakes cost has not shipped a model that mattered.
Why fairness lives here
Run the same threshold across two groups whose score distributions differ and you get different error rates for each. Most real-world algorithmic bias arrives through this door, not through a malicious feature.
Next
Choosing this number is the job
Module II ends with exactly this: a model, a cost matrix, and a written justification for where you put the line. It is the assignment most students say changed how they think.