How to Inventory and Clean Up Content Before Deploying AI Search or RAG

Teams point a shiny new AI search or RAG pipeline at their content, run the first query, and get back a confidently wrong answer sourced from a three-year-old draft that was never supposed to be public. The retrieval layer worked perfectly. It faithfully surfaced the mess it was pointed at: duplicated pages, orphaned PDFs, half-finished migrations, and conflicting versions of the same policy. Garbage in, authoritative-sounding garbage out, and now it is wearing the credibility of an assistant.

The uncomfortable truth is that most RAG failures are not model failures or embedding failures. They are content-hygiene failures that existed long before anyone typed "vector database" into a planning doc. If your knowledge base is unstructured, undated, and ungoverned, chunking and embedding it just industrializes the confusion. Sanity is the Content Operating System for the AI era, an intelligent backend that treats your content as queryable, typed, versioned data rather than a pile of documents to be scraped, which is exactly the foundation an inventory and cleanup effort is trying to reach.

This guide walks the practical pre-flight: how to inventory what you actually have, score it for retrieval-readiness, deduplicate and re-structure it, and set up the governance that keeps it clean after go-live.

Why dirty content breaks RAG in ways models cannot fix

A retrieval-augmented pipeline is only as trustworthy as its lowest-quality retrievable document. When you embed everything indiscriminately, you are making an implicit claim that every chunk is a valid source of truth. The vector store does not know that one document is the current pricing page and another is a superseded draft; cosine similarity treats them as equally quotable. The result is a class of failure that no amount of prompt engineering resolves, because the wrong answer is technically well-grounded in a real source.

Consider a support assistant trained over a help center that still contains the deprecated 2022 refund policy alongside the current one. Both mention refunds, both score highly for a refund query, and the model has no reliable signal for which is authoritative. It will sometimes cite the wrong one, and because it cites a real document, the answer looks defensible right up until a customer acts on it. This is worse than no answer. A blank response fails visibly; a confidently wrong grounded answer fails silently and at scale.

The deeper problem is that traditional content stores were built for publishing, not for machine retrieval. A page is HTML with navigation, boilerplate, and mixed intent. Chunking that page splits a table from its caption, or merges a footer disclaimer into the middle of a procedure. Content that was legible to a human reader becomes noise to a retriever. The fix is not a better embedding model. It is treating content as structured, typed data with clear boundaries, which is the shift from scraping documents to querying a content model. That reframing, not the vector index, is where retrieval quality actually comes from.

Building a real content inventory before you embed anything

An inventory is not a sitemap. A sitemap tells you what URLs exist; an inventory tells you what each piece of content is, who owns it, when it was last verified, what it supersedes, and whether it is fit to be quoted by a machine. Start by enumerating sources across every silo: the CMS, the help center, shared drives, PDFs, wiki spaces, and the inevitable folder of Google Docs that half the company treats as canonical. The goal of the first pass is coverage, not judgment. You cannot clean what you have not found.

For each item, capture a minimum viable metadata set: content type, owner, last-reviewed date, canonical status (authoritative, draft, deprecated, or duplicate), audience, and sensitivity classification. That last field matters more than teams expect. A retrieval system with no sensitivity labels will happily surface internal salary bands or an unreleased roadmap in response to an innocent query, because it has no notion that some content should never be retrievable in a given context.

This is where a structured backend earns its place. In Sanity, content is not a flat list of pages but typed documents defined with `defineType` schemas, so an inventory attribute like reviewedAt, canonicalStatus, or sensitivity becomes a real field you can query, not a spreadsheet column that drifts out of sync. A GROQ query can return every document of a given type that has not been reviewed in twelve months, or every document missing an owner, in one round trip. The inventory stops being a one-time audit artifact and becomes a live view of your content estate that the same query can regenerate on any given day.

Scoring content for retrieval-readiness, not just freshness

Freshness is the metric everyone reaches for first, and it is necessary but nowhere near sufficient. A recently updated document can still be terrible for retrieval if it is ambiguous, duplicative, or structurally tangled. Retrieval-readiness is a composite score, and it pays to make the dimensions explicit so cleanup work can be prioritized instead of feeling infinite.

Score each candidate document on at least four axes. Authority: is this the single canonical source for its claim, or one of several competing versions? Structure: is it cleanly chunkable, with headings, self-contained sections, and captions that travel with their tables and images? Specificity: does it make concrete, quotable statements, or is it marketing prose that says little a customer could act on? Safety: is it correctly classified for the audience that will be able to retrieve it? A document that fails authority or safety should be excluded from the index entirely, not merely down-ranked, because a small probability of a catastrophic citation is not a risk worth carrying.

Structure is the axis most improved by the underlying content format. Portable Text represents rich text as a typed, structured array rather than an HTML blob, which means annotations, marks, and block boundaries are explicit and machine-readable. A retriever consuming Portable Text can respect those boundaries instead of guessing where a section starts, and it can carry an annotation such as a legal disclaimer or a product reference through to the chunk that needs it. Content authored as structured data is retrieval-ready by construction, which is a very different starting point from cleaning up exported HTML after the fact.

Deduplicating, consolidating, and setting canonical sources

Duplication is the silent killer of grounded answers. It rarely looks like exact copies; it looks like near-duplicates that diverge in one important detail, so a retriever pulls two chunks that disagree and the model has to pick. The cleanup goal is a single canonical source per claim, with everything else either merged into it, explicitly linked as a variant, or retired.

Start by clustering. Semantic similarity is genuinely useful here, ironically the same technique the RAG system will use later. Group documents that are close in embedding space and review the clusters by hand, because the machine can find candidates but only a human owner can declare which one is canonical. For each cluster, designate the authoritative document, redirect or archive the rest, and record the decision so it survives staff turnover. Undocumented canonical decisions decay within a quarter.

A content model with references turns canonicalization into something you can enforce rather than merely intend. In Sanity, a document can point at its canonical source through a reference field, and GROQ resolves that reference with the `->` operator in the same query that fetches the content, so a retrieval export can dereference to the authoritative version automatically. You can also express relationships like supersededBy, so a deprecated policy carries a machine-followable pointer to the version that replaced it. Deduplication stops being a one-time purge and becomes a modeled relationship, which means the next audit starts from a known-good graph instead of a fresh archaeological dig.

Governing content so it stays clean after go-live

An inventory is a snapshot; a governed workflow is a guarantee. The reason knowledge bases rot is that publishing and reviewing are decoupled from each other and from any notion of expiry. Content gets published and then nobody is accountable for confirming it is still true. For a RAG system, that decay is not cosmetic, because every stale document remains fully retrievable and fully quotable until someone notices.

The governance primitives that matter are ownership, review cadence, staged releases, and an audit trail. Every retrievable document needs a named owner and a review date, and the system needs to surface what is overdue rather than waiting for a human to remember. Changes that affect what the assistant will say should move through a reviewable, revertible workflow, not a silent overwrite, so a bad edit does not immediately propagate into every future answer.

Sanity provides these as first-class surfaces rather than bolt-ons. Content Releases let you stage and schedule coordinated changes and review them together before they go live, which maps neatly onto retrieval where you want to re-index a curated batch rather than a trickle of unreviewed edits. Roles & Permissions constrain who can change canonical content, Audit logs record who changed what and when, and Functions can run serverless automation such as flagging a document whose reviewedAt date has lapsed or re-triggering an embedding job when a canonical source changes. Governance becomes the mechanism that keeps the expensive cleanup you just finished from silently undoing itself, and it is enforced in the same backend that serves the content, not in a separate process everyone forgets to run.

Wiring clean content into the retrieval pipeline

Once content is inventoried, scored, deduplicated, and governed, the connection to the actual retrieval layer should be boring, and boring is the goal. The two decisions that matter are what you export for embedding and how you keep the index in sync with the source of truth. Both are easier when the source is a queryable content store rather than a crawl of rendered pages.

Export should be a projection, not a scrape. You want the fields relevant to retrieval, the body as structured text, the canonical status, the sensitivity label, and the metadata you will filter on at query time, and nothing else. This is precisely what a query language built for shaping responses is for. A single GROQ query can select only canonical, in-audience documents, project exactly the fields your chunker needs, dereference the canonical source with `->`, and return the whole thing in one round trip, so your embedding job consumes a clean, filtered, predictable payload instead of raw HTML it has to sanitize.

Keeping the index fresh is where a real-time backend pays off. Rather than re-embedding the entire corpus on a cron schedule and hoping, you can subscribe to changes and re-index only what moved. The Live Content API surfaces content changes as they happen, and a Function can respond to a canonical document changing by enqueuing a re-embed of just that document and its dependents. Combined with hybrid retrieval, where GROQ can blend `text::semanticSimilarity()` with structured filters and `match()` in the same query, the boundary between your governed content estate and your retrieval layer stops being an ETL pipeline you babysit and becomes a query you trust. The cleanup you did up front is what makes that trust warranted.