A search index with thirty million documents in it is not a bigger version of a search index with thirty thousand. Somewhere between those two numbers, every weak assumption you made becomes load-bearing and then becomes visible. One of our principals spent five years building this at a national newsroom: the archive, the extraction pipelines, and the fifteen production AI applications that sat on top of it. The lesson that survived all five years: retrieval quality is an evaluation problem long before it is an infrastructure problem.
Teams reach for infrastructure first because infrastructure is legible: swap the index, add a vector store, tune the analyzer. But none of those moves can be judged without a fixed set of real queries and judged answers. Build that eval set first, from the questions people actually ask, including the ugly ones with typos and half-remembered names. Fifty honest queries with graded results will steer an architecture better than any benchmark blog post, because they encode your corpus, your users, and your definition of correct.
The corpus itself is where most of the quality lives. At newsroom scale, extraction is not a preprocessing step. It is the product's foundation. Optical character recognition on decades of scans, deduplication across syndicated copies, entity normalisation so a person's four name variants resolve to one. We have hit the same wall in every domain that keeps records: sixty-six gigabytes of retail catalogues, fund documents in private equity, nine-thousand-plus records in an agricultural research archive. Different industries, same truth: if the extraction is sloppy, no retriever downstream can be smart enough to compensate.
If the extraction is sloppy, no retriever downstream can be smart enough to compensate.
What made the newsroom work count was not the index. It was that editors stopped spending their mornings hunting for context. Measured against the prior workflow, the document systems were saving on the order of five hundred hours in a thirty-day window. That number is the point. Retrieval systems justify themselves in recovered hours and faster decisions, and if you cannot trace the connection from relevance scores to someone's Tuesday, the system is not done.
So the order of operations we hold: eval set, then extraction, then index, then model. Most teams run it exactly backwards. They pick the most interesting technology first and discover at the end that nobody can say whether the answers got better. At three hundred documents you can get away with that. At thirty million, the corpus will collect the debt.