Here's a problem that undermines a lot of AI benchmark claims, and it's rarely discussed outside research circles: data contamination. If a benchmark's questions and answers ended up in a model's training data, a high score proves memorization, not capability.
How contamination happens
Models train on enormous web scrapes. Benchmarks are published online — questions, answers, and discussions of them. It's easy for a benchmark to leak into the training set, especially popular ones that get quoted, blogged about, and posted in forums. When that happens, the model isn't solving the test at evaluation time; it's partly recalling answers it already saw. The score looks great and means little.
A benchmark only measures skill if the model hasn't seen the answers. Once the test is in the training data, the test is broken.
Why it's hard to catch
You can't easily inspect a trillion-token training set for every benchmark item, and paraphrased or partial leakage is even harder to detect. Labs try to filter known benchmarks out, but coverage is imperfect, and new contamination appears as benchmarks age. This is a big reason a model can top a public leaderboard yet underwhelm on your real task — the leaderboard may be partly measuring memorization.
What to do about it
The defenses: use fresh or private benchmarks the model couldn't have seen, rotate evaluation sets, and — most importantly for practitioners — test on your own data. Your real tasks are almost certainly not in any training set, so they're the one benchmark that can't be contaminated. Whenever you see an impressive benchmark number, the first question to ask is: could the model have already seen this test?
