The accuracy trap, the confusion matrix, and the threshold-free metrics underneath them.
The notebook carries the source, the install step and every run below, so nothing has to be on your machine. Open it, then choose Runtime › Run all.
Colab opens it read only. Click Copy to Drive to keep
your changes. You can also download
classification.ipynb, or run it
locally: python3 classification.py.
python3 classification.py
The whole evaluation, from counts to macro-F1.
python3 classification.py --trap
A four-class model scoring 0.90 accuracy and 0.24 macro-F1 by always answering the majority class. Three classes score exactly zero and accuracy does not notice.
python3 classification.py --read
The hardest class and the most confused pair, computed rather than judged. government has the lowest F1 at 0.6364, and news and government exchange 5 documents.
python3 classification.py --threshold
Precision, recall and F1 describe one cut through a ranking. The same ordering scores ROC-AUC 0.9490 and PR-AUC 0.2667 once the positive class is rare, and only the second describes what a user would experience.
python3 classification.py --calibration
A score of 0.9 should mean right nine times in ten. The most confident bin here promises 0.95 and delivers 0.40, an expected calibration error of 0.247, and no ranking metric would notice.
Download classification.py
· served verbatim at https://nlp.jcrlabz.com/code/classification.py
"""Evaluating a classifier, worked as Chapter 14 works it.
Six things, all reproducing the chapter's tables:
the accuracy trap a useless model with 90 per cent accuracy
the confusion matrix per-class precision, recall and F1, computed
reading the matrix hardest class and most-confused pair, derived
macro against micro why they differ, and when they cannot
the threshold sweep ROC and PR from one ranked list, and their areas
calibration a model that ranks well and lies about probability
python3 classification.py # all six
python3 classification.py --trap # the accuracy trap
python3 classification.py --matrix # the full per-class table
python3 classification.py --read # hardest class, confused pairs
python3 classification.py --averages # macro, micro and accuracy
python3 classification.py --threshold # ROC-AUC against PR-AUC
python3 classification.py --calibration# scores that are not probabilities
Install: nothing, the Python standard library is enough
"""
import argparse
# The genre confusion matrix from the lectures. Rows are the true class,
# columns the predicted class. Twelve documents of each genre.
GENRES = ["news", "fiction", "government", "learned"]
MATRIX = [
[9, 0, 2, 1], # news
[0, 11, 0, 1], # fiction
[3, 0, 7, 2], # government
[1, 1, 1, 9], # learned
]
def counts(M, i):
"""TP, FP, FN for class i, treating the task as one against the rest."""
tp = M[i][i]
fp = sum(M[r][i] for r in range(len(M))) - tp
fn = sum(M[i]) - tp
return tp, fp, fn
def prf(tp, fp, fn):
p = tp / (tp + fp) if tp + fp else 0.0
r = tp / (tp + fn) if tp + fn else 0.0
f = 2 * p * r / (p + r) if p + r else 0.0
return p, r, f
def macro_f1(M):
return sum(prf(*counts(M, i))[2] for i in range(len(M))) / len(M)
def accuracy(M):
return sum(M[i][i] for i in range(len(M))) / sum(sum(r) for r in M)
# ------------------------------------------------------------ the trap
def show_trap():
"""Imbalanced classes, and a model that has learned nothing."""
support = [90, 5, 3, 2]
labels = ["news", "fiction", "government", "learned"]
# Everything predicted as the majority class.
M = [[0] * 4 for _ in range(4)]
for i, n in enumerate(support):
M[i][0] = n
print("\nSTEP 1 why accuracy is not enough\n")
print(" A hundred test documents, unevenly distributed:")
print(" " + ", ".join(f"{l} {n}" for l, n in zip(labels, support)))
print("\n The model is a single line of code: always answer 'news'.\n")
print(f" {'true class':<12}" + "".join(f"{l[:7]:>10}" for l in labels))
print(" " + "-" * 52)
for i, l in enumerate(labels):
print(f" {l:<12}" + "".join(f"{v:>10}" for v in M[i]))
print(f"\n {'class':<12}{'P':>9}{'R':>9}{'F1':>9}")
print(" " + "-" * 40)
for i, l in enumerate(labels):
p, r, f = prf(*counts(M, i))
print(f" {l:<12}{p:>9.4f}{r:>9.4f}{f:>9.4f}")
print(f"\n accuracy {accuracy(M):.4f}")
print(f" macro-F1 {macro_f1(M):.4f}")
print("\n Ninety per cent accuracy, and the model cannot tell any two")
print(" documents apart. Three of the four classes score exactly zero.")
print("\n Macro-F1 sees it immediately, because it averages the per-class")
print(" scores and a class the model ignores contributes 0 to that mean.")
print("\n This is why Assignment 5's leaderboard uses macro-F1. You cannot")
print(" win it by chasing the majority class.")
# -------------------------------------------------------- the real matrix
def show_matrix():
print("\n\nSTEP 2 a real confusion matrix, scored\n")
print(" Rows are the true class, columns the prediction.")
print(" The diagonal is correct. Everything else is an error.\n")
print(f" {'true / pred':<13}" + "".join(f"{g[:7]:>10}" for g in GENRES)
+ f"{'total':>9}")
print(" " + "-" * 62)
for i, g in enumerate(GENRES):
cells = "".join(f"{v:>10}" for v in MATRIX[i])
print(f" {g:<13}{cells}{sum(MATRIX[i]):>9}")
print(f"\n Now one row per class, treating each as one against the rest.\n")
print(f" {'class':<12}{'TP':>5}{'FP':>5}{'FN':>5}{'P':>9}{'R':>9}"
f"{'F1':>9}")
print(" " + "-" * 56)
for i, g in enumerate(GENRES):
tp, fp, fn = counts(MATRIX, i)
p, r, f = prf(tp, fp, fn)
print(f" {g:<12}{tp:>5}{fp:>5}{fn:>5}{p:>9.4f}{r:>9.4f}{f:>9.4f}")
print(f"\n accuracy {accuracy(MATRIX):.4f}")
print(f" macro-F1 {macro_f1(MATRIX):.4f}")
print("\n Here the classes are balanced at twelve each, so accuracy is")
print(" not misleading. It is still less informative: it says 0.75 and")
print(" stops, where the per-class table says which genre is failing.")
print("\n Look at 'government'. Its recall is 0.5833, the lowest in the")
print(" table, because five of its twelve documents went elsewhere.")
print(" Precision is 0.7000, because three documents from other genres")
print(" were labelled government. Both directions are wrong at once.")
# ------------------------------------------------------- reading the matrix
def show_read():
print("\n\nSTEP 3 two diagnoses the matrix hands you\n")
f1 = [(prf(*counts(MATRIX, i))[2], GENRES[i]) for i in range(len(GENRES))]
f1.sort()
print(" HARDEST CLASS is the lowest per-class F1.\n")
for f, g in f1:
print(f" {g:<12}{f:>9.4f}")
print(f"\n hardest: {f1[0][1]}")
print("\n MOST-CONFUSED PAIR is the unordered pair with the most cross")
print(" errors. Add the two off-diagonal cells that join them.\n")
pairs = []
for i in range(len(GENRES)):
for j in range(i + 1, len(GENRES)):
total = MATRIX[i][j] + MATRIX[j][i]
pairs.append((total, GENRES[i], GENRES[j],
MATRIX[i][j], MATRIX[j][i]))
pairs.sort(reverse=True)
print(f" {'pair':<26}{'i->j':>6}{'j->i':>6}{'total':>8}")
print(" " + "-" * 48)
for t, a, b, ab, ba in pairs:
print(f" {a + ' / ' + b:<26}{ab:>6}{ba:>6}{t:>8}")
print(f"\n most confused: {pairs[0][1]} and {pairs[0][2]}, {pairs[0][0]}"
f" errors")
print("\n Both answers are properties of the matrix, not opinions. Two")
print(" people reading the same matrix must reach the same pair.")
print("\n Note that the errors are directional. Government loses 3")
print(" documents to news and takes 2 back, which are different")
print(" mistakes with different causes.")
# --------------------------------------------------------- macro and micro
def show_averages():
print("\n\nSTEP 4 macro, micro, and why one of them is redundant\n")
tp = sum(counts(MATRIX, i)[0] for i in range(len(GENRES)))
fp = sum(counts(MATRIX, i)[1] for i in range(len(GENRES)))
fn = sum(counts(MATRIX, i)[2] for i in range(len(GENRES)))
micro = prf(tp, fp, fn)
print(" MACRO averages the per-class F1 scores. Every class counts once,")
print(" however rare it is.\n")
print(" MICRO pools every decision first, then computes one F1. Frequent")
print(" classes therefore dominate it.\n")
print(f" pooled TP = {tp}, FP = {fp}, FN = {fn}")
print(f"\n {'measure':<14}{'value':>9}")
print(" " + "-" * 26)
print(f" {'accuracy':<14}{accuracy(MATRIX):>9.4f}")
print(f" {'micro-F1':<14}{micro[2]:>9.4f}")
print(f" {'macro-F1':<14}{macro_f1(MATRIX):>9.4f}")
print("\n Micro-F1 and accuracy are the same number, and that is not a")
print(" coincidence. When every document gets exactly one label, every")
print(" error is one FP and one FN at the same time, so the pooled")
print(" precision and recall are both the fraction correct.")
print("\n So on single-label tasks micro-F1 tells you nothing accuracy")
print(" did not. Report macro-F1, or report the per-class table.")
print("\n\n ABLATION DISCIPLINE\n")
runs = [("unigrams, raw counts", 0.612),
("unigrams, tf-idf", 0.681),
("unigrams + bigrams, tf-idf", 0.724),
("+ min document frequency 2", 0.748),
("+ lowercasing", 0.741)]
print(f" {'configuration':<32}{'validation macro-F1':>20}")
print(" " + "-" * 54)
best = max(runs, key=lambda r: r[1])
for name, score in runs:
mark = " <- best" if (name, score) == best else ""
print(f" {name:<32}{score:>20.3f}{mark}")
print("\n One factor changes per row. That is what makes it an ablation")
print(" rather than a list of guesses.")
print("\n The last row went down, and reporting it is the point. An")
print(" ablation is a record of runs you did, not of runs that worked.")
print("\n The best row must be the model you actually submit. If it is")
print(" not, your experiment log and your system have diverged.")
# --------------------------------------------------------- threshold sweep
# Ten documents, scored by the model and ranked. 1 marks the rare class.
RANKED = [(0.95, 1), (0.90, 1), (0.80, 0), (0.70, 1), (0.60, 0),
(0.50, 0), (0.40, 0), (0.30, 0), (0.20, 0), (0.10, 0)]
# The same model's behaviour on a realistically rare class: 2 positives in
# 100 documents, ranked 3rd and 10th.
RARE_RANKS, RARE_N, RARE_P = [3, 10], 100, 2
def roc_auc(ranked):
"""Area under ROC, by the rank identity rather than by trapezoids."""
pos = [i for i, (_, y) in enumerate(sorted(ranked), 1) if y == 1]
P, N = len(pos), len(ranked) - len(pos)
return (sum(pos) - P * (P + 1) / 2) / (P * N)
def pr_points(ranked):
"""Precision and recall at every threshold, walking down the ranking."""
P = sum(y for _, y in ranked)
tp = fp = 0
out = []
for score, y in ranked:
tp, fp = tp + (y == 1), fp + (y == 0)
out.append((score, tp, fp, tp / (tp + fp), tp / P))
return out
def average_precision(ranked):
"""Precision averaged over the thresholds where recall actually moves."""
P = sum(y for _, y in ranked)
tp = fp = 0
total = 0.0
for _, y in ranked:
tp, fp = tp + (y == 1), fp + (y == 0)
if y == 1:
total += tp / (tp + fp)
return total / P
def show_threshold():
print("\nSTEP 5 the threshold is a separate decision\n")
print(" Precision, recall and F1 all describe one cut through a ranking.")
print(" Move the cut and every one of them moves. The model itself only")
print(" produces an order, so judge the order.\n")
print(" score label TP FP precision recall")
print(" " + "-" * 46)
for (score, tp, fp, pr, rc), (_, y) in zip(pr_points(RANKED), RANKED):
print(f" {score:5.2f} {y:5d} {tp:4d}{fp:4d} {pr:8.3f} {rc:6.3f}")
print(" " + "-" * 46)
a_roc, a_pr = roc_auc(RANKED), average_precision(RANKED)
print(f"\n ROC-AUC {a_roc:.4f} PR-AUC {a_pr:.4f}")
print("\n Both look strong, and on a balanced problem they usually agree.")
print(" Now make the positive class rare, which is the normal case.\n")
rare = [(0.0, 1) if i + 1 in RARE_RANKS else (0.0, 0) for i in range(RARE_N)]
# ranks are given from the top of the ranking
pos_from_bottom = [RARE_N - r + 1 for r in RARE_RANKS]
a_roc2 = (sum(pos_from_bottom) - RARE_P * (RARE_P + 1) / 2) / (RARE_P * (RARE_N - RARE_P))
ap2 = sum((i + 1) / r for i, r in enumerate(RARE_RANKS)) / RARE_P
print(f" {RARE_P} positives among {RARE_N} documents, ranked {RARE_RANKS[0]} and {RARE_RANKS[1]}:\n")
print(f" ROC-AUC {a_roc2:.4f} still excellent")
print(f" PR-AUC {ap2:.4f} barely better than guessing")
print("\n The same ranking. ROC-AUC counts how many negatives sit below each")
print(f" positive, and {RARE_N - RARE_P} negatives is a lot of credit. PR-AUC asks what")
print(" fraction of what you flagged was right, and at rank 10 that is")
print(f" {RARE_P}/{RARE_RANKS[1]} = {RARE_P / RARE_RANKS[1]:.2f}. Under imbalance, report PR-AUC.")
# ------------------------------------------------------------- calibration
# Three confidence bins. The model orders documents perfectly inside every
# bin, and its numbers still do not mean what they say.
BINS = [("0.6 - 0.7", 0.65, 40, 26), ("0.7 - 0.9", 0.80, 35, 17),
("0.9 - 1.0", 0.95, 25, 10)]
def show_calibration():
print("\n\nSTEP 6 ranking well is not the same as being right\n")
print(" A score of 0.9 should mean the model is right nine times in ten.")
print(" Nothing in cross-entropy training guarantees it, and nothing in")
print(" ROC-AUC or F1 would notice if it were false.\n")
print(" bin mean score n correct observed gap")
print(" " + "-" * 56)
tot_gap = tot_n = 0
for name, mean, n, correct in BINS:
obs = correct / n
gap = mean - obs
tot_gap += abs(gap) * n
tot_n += n
print(f" {name:11} {mean:10.2f} {n:6d} {correct:9d} {obs:10.3f} {gap:+6.3f}")
print(" " + "-" * 56)
print(f" expected calibration error{' ' * 26}{tot_gap / tot_n:.3f}")
print("\n The least confident bin is honest. The other two are not, and the")
print(f" most confident is the worst: it promises {BINS[-1][1]:.2f} and delivers "
f"{BINS[-1][3] / BINS[-1][2]:.2f}.")
print(" Over-confidence grows with the score, which is the usual shape.")
print(" The ordering may still be perfect, so a ranking metric sees nothing")
print(" wrong. Any decision with a cost attached needs the probability, not")
print(" the order, and that is when this table is the one that matters.")
def main():
ap = argparse.ArgumentParser(description=__doc__)
for f in ("trap", "matrix", "read", "averages", "threshold", "calibration"):
ap.add_argument(f"--{f}", action="store_true")
a = ap.parse_args()
picked = (a.trap or a.matrix or a.read or a.averages
or a.threshold or a.calibration)
if a.trap or not picked:
show_trap()
if a.matrix or not picked:
show_matrix()
if a.read or not picked:
show_read()
if a.averages or not picked:
show_averages()
if a.threshold or not picked:
show_threshold()
if a.calibration or not picked:
show_calibration()
print()
if __name__ == "__main__":
main()