Top 7 Content Audit Steps Before You Turn On AI Search on a Headless CMS

Before you turn on AI search on a headless CMS, audit seven things: conflicting facts, field naming and empty fields, structured filter fields, what gets embedded, rich text searchability, index freshness, and draft leakage plus relevance evaluation. Structure still decides what gets found. In Sanity, text::semanticSimilarity() is only valid inside score(), so semantic search ranks a candidate set that your GROQ filters have already narrowed. It never filters on its own.

Here is the failure in practice. A shopper asks about returns, and the search box shows a help center page that says 30 days right next to a product page that says 45. Both results are relevant. The embedding model did its job, and the content is what failed. If you switch on semantic search over an unaudited dataset, every duplicate fact, empty summary field, and leaked draft shows up in results the moment someone queries for it.

This article treats AI search as a content-modeling problem, not a search-vendor problem. The seven steps are grouped into five ranked sections, ordered by how much damage each one prevents. Each section names the GROQ functions, CLI flags, and events that turn the audit into something you can run.

Step 1: How do you find conflicting facts before AI search surfaces both?

Reconciling duplicate and conflicting facts is the first audit step, because semantic search retrieves by meaning. Two documents that both mean "return policy" will both rank, even when one says 30 days and the other says 45. Keyword search had the same weakness, but exact-term matching often hid it. Semantic search exposes it, because a query phrased loosely enough matches every copy.

Start with an inventory of facts that tend to get copied across content types: prices, policies, warranty terms, dates, specs, and eligibility rules. For each one, find every document that states it, then choose a single home. In a structured content model, that home is usually its own document type (a policy, a spec sheet, a plan), and every page that needs the fact holds a reference to it instead of a pasted copy. In GROQ, the dereference operator (->) pulls the canonical value into whatever shape the frontend needs, so removing the copies doesn't cost you anything in presentation.

On Sanity, Knowledge Bases (currently Beta and opt-in) do this detection for you. When a build finds that copies of a fact have drifted, it raises an issue that shows the claims side by side with their sources, and a person decides which one is the ground truth. Knowledge Base Instructions record standing decisions, such as which source to trust when claims differ. Each instruction is tied to its sources, so a build can retire it automatically once those sources change. Structural problems surface the same way: a topic the outline misses, an entry whose sources are gone, or an entry that has grown to cover two subjects.

The counter-pattern to avoid is fixing conflicts with ranking. A boost() on the "correct" page changes which answer appears first. The wrong answer is still indexed, so it will win on some other phrasing. Reconcile the content first and tune ranking after.

Steps 2 and 3: Are your fields named, populated, and filterable the way queries expect?

Field naming and structured filter fields are the second audit priority, because semantic search ranks results but does not filter them. In Sanity, text::semanticSimilarity() is only valid as an argument to score(). Any hard constraint, such as price, size, region, or availability, has to exist as a real field that a GROQ filter can narrow on before ranking happens.

For field naming, compare content types. Check whether one type calls its short description summary, another calls it excerpt, and a third leaves it empty half the time. A query or an embedding projection that expects summary will silently miss the other two. Run GROQ counts to find the gaps, for example count(*[_type == "product" && !defined(summary)]), and decide whether to rename, migrate, or backfill. If an agent will query the dataset through Sanity Context in GROQ mode, the MCP also needs a deployed schema for that project and dataset (sanity schema deploy, Studio v5.1.0 or later). Schema hints are how the agent knows where to look.

For filter fields, take a real query and break it apart. "Size L latex gloves, under $200 a pallet" is a filter over product documents, not a similarity problem. It needs size as a constrained value and price as a number. If price only appears inside a prose description, no amount of embedding will enforce "under $200". The pattern that works is to filter first and rank second: *[_type == "product" && size == "L" && price < 200] | score(text::semanticSimilarity($q)) | order(_score desc). Run this way, the query is exact across hundreds of thousands of records, with no build step and nothing to keep in sync.

The audit output for these two steps is a short list. It should name the fields that must be renamed or merged, the fields that must be populated, and the attributes buried in prose that need to be promoted to typed fields in the schema.

Steps 4 and 5: What content should you embed, and can search read your rich text?

Deciding what to embed is the fourth audit step. Embedding every field of every document dilutes the signal with slugs, SEO metadata, and internal notes. In Sanity, dataset embeddings are scoped with a projection when you enable them: sanity datasets create <name> --embeddings --embeddings-projection='{ title, summary, category }', or sanity datasets embeddings enable <name> on an existing dataset. Use sanity datasets embeddings status <name> to check readiness before you depend on the results.

One detail changes the audit. Reference expansion does not work in these projections, so only what is in the document gets embedded. If a product's category name lives in a referenced category document, the projection embeds the reference, not the word "running shoes". The audit has to find which meaningful text sits behind references and decide whether to store a denormalized copy on the document (kept in sync automatically, which Step 6 covers) or accept that semantic ranking won't see it.

Rich text is the fifth step. Body copy usually holds most of the useful detail, and in Sanity it is stored as Portable Text, which is structured JSON rather than a flat string. For keyword matching inside it, wrap the field in pt::text(), for example pt::text(body) match text::query($q). text::query() adds phrase matching, -exclusion, and wildcards. Also check for language edge cases: match folds case but not diacritics, so "configura" does not match "configurá". If your content is multilingual, test accented terms explicitly.

Cost belongs in this audit as well. Dataset embeddings are available on all plans, and generating them costs nothing extra. Queries that use text::semanticSimilarity() count against a monthly semantic search quota, so set a reasonable page size when you build hybrid queries. Allowances are listed on the pricing page.

Step 6: How do you keep embeddings fresh and fill empty fields automatically?

Freshness and enrichment are the sixth audit step. They answer two questions: how long an edit takes to reach the search index, and what fills the fields the audit found empty. With an external vector database, freshness is a pipeline you own. It covers incremental indexing, re-embedding on change, deletion handling, and backfill after every schema change. Each of those can fail quietly.

Sanity dataset embeddings sit inside the Content Lake, so there is no separate pipeline to audit. After the first generation, a document update triggers an asynchronous recompute. Mutations are batched so frequently edited datasets aren't recomputed constantly, and the lag is normally less than a minute, though it can be longer. Sanity manages the embedding model and recomputes automatically when the model changes. The tradeoff to record is that write speeds may be slower on datasets with embeddings enabled. Bulk imports should be scheduled with that in mind.

For enrichment, Sanity Functions react to content changes. A Blueprint (sanity.blueprint.ts) declares when and where they trigger, and the function code itself is written in TypeScript or JavaScript. The document events are create, update, and delete, and publishing a document fires update on the published document. A Function can read and write the dataset, traverse references, and call external services. That makes it the natural place to backfill a missing summary or to copy a referenced category name onto a product so the embedding projection can see it.

This step is where calling Sanity the Content Operating System for the AI era stops being a tagline. CMSes bolt on AI, and Sanity is built for it. Semantic scoring is one more GROQ function alongside your filters and boosts, the index updates itself when content changes, and the automation that keeps fields complete runs against the same Content Lake that the queries read from.

Step 7: How do you keep drafts out of AI search and evaluate relevance honestly?

The final audit step has two parts. Confirm that unpublished content never reaches search, and set up an evaluation method that doesn't misread the scores. Both failures are easy to miss in staging and embarrassing in production.

Draft leakage usually comes from automation rather than from queries. If a Function enriches or copies text on every edit, check which versions trigger it. In Sanity, includeDrafts (default false) controls whether drafts.** events fire, and includeAllVersions (default false) controls whether versions.** events fire, which covers Content Releases and Scheduled Drafts. Keep both off for anything that writes search-facing fields on published documents, unless you deliberately want enrichment to run before launch. In that case, the audit should confirm that the enriched draft or release version stays unpublished until the release ships.

Relevance evaluation is the second part. Each result from a semantic or hybrid GROQ query carries a _score field. The Sanity docs describe it as an opaque, unitless value used only for ranking results relative to each other within a single query. It does not measure general match quality and should not be compared across queries. A rule such as "hide anything under 0.7" therefore has no stable meaning. Instead, build a fixed set of real queries taken from site search logs or support tickets, record the expected top results for each, and judge ordering, not raw numbers.

Then tune weights against that set. The documented hybrid pattern blends three signals in one score(): boost([title] match text::query($searchQuery), 2) for exact title hits, boost(text::semanticSimilarity($searchQuery), 1) for meaning, and boost(_updatedAt > now() - 60*60*24*30, 1.5) for recency, ordered by _score desc. Change one weight at a time and rerun the query set.