Precision and Recall (where to set the spam filter’s bar)

How Do You Know It’s Good?, on the How LLMs Work track. The board splits every task by how checkable it is, and puts “spam or not spam” firmly on the checkable side: every email has one right answer, so a filter can be graded exactly. This lesson is that exact grading, done by hand, on the oldest machine-learning job most people have met: the spam folder.

Before you start. You need What a Neural Network Is and Vectors, Dot Products and Matrix Multiply. The first showed that one neuron is a weighted sum: each input times its weight, added up, plus a starting value. The second showed that a weighted sum is a dot product, one row of numbers times another. This lesson is that one neuron, used to answer a yes-or-no question, and then the arithmetic of how to tell whether its answers are any good. It feeds Evaluating Models (how do you *know* it's good?) and Evals for Apps (the test suite for a program that is a string), which grade bigger models with the same habits.

The idea in 30 seconds

One neuron can score a yes-or-no question: a weighted sum of the evidence, pushed through a sigmoid, which turns any sum into a probability between 0 and 1.
A probability becomes a yes-or-no only when you pick a threshold, and the threshold is a choice about which mistake is cheaper.
A false positive is a real no wrongly flagged as yes. A false negative is a real yes that was missed. Every prediction lands in one of four boxes: right yes, right no, and those two mistakes.
Precision asks “when it says yes, how often is it right?”; recall asks “of all the real yeses, how many did it find?”
Lowering the threshold raises recall and lowers precision, and false alarms climb.
The same model has lower precision on data where the thing you look for is rarer, even though its recall does not change.

Read the story · Skip to the numbers