There's a stubborn myth in machine learning: pour in more data, get a better model. It was true for a while. In 2026, curation matters more than volume.
Garbage in, garbage out — at scale
Web-scraped text is full of duplication, spam, low-quality boilerplate, and noise. Training on all of it teaches the model the noise along with the signal. A carefully filtered, deduplicated, high-quality dataset — even if much smaller — often produces a stronger model, because every token it learns from is worth learning.
Ten million excellent examples beat a billion mediocre ones. The model can only be as good as what it reads.
What "quality" means
Good data curation involves deduplication (repeated text over-weights whatever it says), quality filtering (removing spam and low-value pages), diversity (broad coverage without redundancy), and increasingly, high-signal sources — textbooks, documentation, curated code, expert writing. Small models trained on excellent data now rival much larger models trained on raw scale.
The practical lesson
This reframes how to build a capable model, especially a specialized one. Instead of chasing the biggest dataset, invest in curating the right one for your task. For fine-tuning, a few thousand excellent examples often beat tens of thousands of mediocre ones. The bottleneck moved from "can we get enough data?" to "can we get good data?" — and that's a curation problem, not a scraping one.
