How to Manage Content Embeddings at Scale
You ship a semantic search feature, it works beautifully in the demo, and three months later it is quietly wrong.
You ship a semantic search feature, it works beautifully in the demo, and three months later it is quietly wrong. An editor changed a product description, a translation shipped, a page got unpublished, and none of those edits reached your vector index. Your embeddings now describe content that no longer exists, and your retrieval surfaces stale or deleted answers to users and, increasingly, to the agents querying on their behalf. The failure is silent, which is what makes it expensive.
Managing embeddings at scale is not a modeling problem, it is a synchronization and governance problem. The moment your source content and your vectors live in two systems, you own a reconciliation job forever: re-embedding on every edit, chunking consistently, versioning models, and keeping deletes honest. Sanity is the Content Operating System for the AI era, an intelligent backend that keeps embeddings tied to the structured content they describe instead of drifting off into a separate database nobody owns.
This guide treats embeddings as a lifecycle, not an artifact. We cover chunking strategy, the re-embedding pipeline, drift and eventual consistency, cost control, and where keeping vectors next to your content in the Content Lake changes the math versus bolting a standalone vector store onto a headless stack.
The real problem is drift, not dimensions
Teams obsess over embedding model choice and vector dimensions, then lose their whole quarter to a boring operational failure: the index and the content fall out of sync. Every embedding is a snapshot of a piece of content at a moment in time. Content is not static. An editor rewrites a paragraph, legal appends a disclaimer, a localized variant ships, an item gets archived. Each of those events invalidates one or more vectors, and unless something reliably fires a re-embed, your index accumulates lies.
This drift is invisible in exactly the way that hurts most. Retrieval still returns results, they are just subtly wrong: the answer cites a price that changed, a policy that was retracted, or a page that returns a 404 when the user clicks through. In a plain keyword system a stale record is obvious. In a vector system it is a plausible, confident, wrong neighbor. As RAG and agent workflows put embeddings on the critical path for answers rather than a nice-to-have search box, the cost of drift moves from mild annoyance to reputational risk.
The root cause is architectural. When content lives in a CMS and vectors live in a separate store, no single system knows the truth about what changed. You are hand-rolling change data capture across a boundary that neither side was designed to guard. The fix is to treat re-embedding as a first-class reaction to content events, driven by the same system that owns the content mutation. That reframing, embeddings as a derived, event-driven projection of governed content, is what makes scale tractable. Legacy CMSes stop at publishing; the work of keeping a derived index honest has to be owned somewhere, and the content backend is the only place that sees every mutation.
Chunking is a content-modeling decision
The unit you embed determines the quality of everything downstream, and most teams pick it arbitrarily. Embed a whole 4,000-word article and retrieval returns a blob where the relevant sentence is diluted by everything around it. Embed by fixed 512-token windows and you slice mid-sentence, splitting a definition from its example and stranding a heading from the paragraph it introduces. Chunking looks like a preprocessing detail; it is actually a content-modeling decision, and it should be made where the model lives.
Structured content gives you natural seams. If your rich text is stored as discrete, typed blocks rather than an opaque HTML string, you can chunk on semantic boundaries: a section, a FAQ entry, a step in a procedure, a single product spec. Sanity's Portable Text represents rich text as an array of typed blocks with marks and annotations rather than a flat markup blob, so a chunker can walk the structure and cut on real boundaries instead of guessing from byte offsets. The same structure that makes content portable across channels makes it cleanly sliceable for embedding.
There is a governance dimension too. When chunks map to modeled fields and blocks, you can attach metadata to each vector: the document type, language, release status, and the field it came from. That metadata is what lets you filter before you rank, so a query for English support docs never retrieves a draft German marketing block. Chunk on arbitrary character windows and you throw that context away at the exact moment you need it most. Model your content well and chunking stops being a lossy preprocessing step and becomes a projection of a structure you already trust.
Building a re-embedding pipeline that survives edits
A durable pipeline has four moving parts: detect the change, decide what to re-embed, generate the new vector, and reconcile the index including deletes. The naive version, re-embed everything nightly, is both expensive and stale by lunchtime. The correct version is incremental and event-driven: a content mutation emits an event, the pipeline diffs which chunks actually changed, and only those get re-embedded.
This is where owning the mutation stream pays off. Sanity Functions run serverless logic in response to content events, so a publish, an update, or an unpublish can trigger exactly the enrichment work that event requires without you standing up a separate queue and CDC layer. Pair that with the Live Content API and downstream consumers see changes propagate in real time rather than on a batch cadence. The pipeline becomes a reaction to the same events that drive your editorial workflow, which means it fails loudly and locally instead of silently drifting.
Deletes and unpublishes are where most homegrown pipelines rot. It is easy to add a vector on create and update it on edit, and easy to forget that unpublishing a document must also purge its vectors. An orphaned vector is worse than a missing one because it actively surfaces content that governance already retracted. Model the lifecycle explicitly: every state transition that hides content from users must also hide it from retrieval. Because Content Releases and scheduling govern when content goes live, the same signals can gate when a vector becomes eligible for retrieval, so an embargoed release is not leaking through your semantic search a week before launch.
Keeping vectors next to content versus a standalone store
The default architecture in most tutorials is a dedicated vector database sitting beside your CMS: Pinecone, Weaviate, or pgvector in your own Postgres. It is a reasonable starting point and it scales, but it comes with a permanent tax. You now operate two sources of truth, you own the sync between them, and every query that needs both a semantic match and a structured filter has to fan out to two systems and stitch the results back together in application code.
The alternative is to keep the vector next to the content it describes and query both in one language. In Sanity, GROQ can blend semantic similarity with structured filters in a single query: text::semanticSimilarity() to rank by meaning, match() and score() to fold in keyword and field signals, and ordinary filters and projections to constrain by type, language, or release status and return exactly the shape your frontend needs in one round trip. Hybrid retrieval, the thing you would otherwise build by hand across two databases, becomes one expression against the Content Lake.
The honest tradeoff: a purpose-built vector store gives you knobs for index type, recall-versus-latency tuning, and billion-scale ANN that a content platform will not match at the extreme high end. But most content-driven applications are not operating at billion-vector scale; they are operating at governance-and-freshness scale, where the expensive problem is keeping vectors correct and filtered, not squeezing the last millisecond out of approximate nearest neighbor. For that regime, collapsing two systems into one, where a shared foundation replaces two silos, removes the entire class of drift bugs that the separate-store architecture manufactures.
Governance, access control, and compliance for vectors
Vectors inherit the sensitivity of the content they encode, and teams routinely forget this. An embedding of a confidential contract is still confidential; a vector for a draft that has not cleared legal review is still a draft. When your vectors live in a separate store with its own, weaker access model, you have effectively created a side channel that leaks governed content past the permissions you carefully set on the source. Retrieval that ignores your roles is a compliance incident waiting for an auditor.
The fix is to make retrieval subject to the same governance as the rest of your content operation. That means vectors carry the metadata to enforce access at query time, that Roles & Permissions constrain who and what can retrieve which chunks, and that Audit logs record retrieval and mutation so you can answer the question every regulated buyer asks: who saw what, and when. When embeddings are a projection of governed content rather than a copy in an ungoverned system, access control is inherited rather than re-implemented.
Compliance posture matters here because AI features are increasingly the thing security review focuses on. Sanity is SOC 2 Type II compliant, supports GDPR obligations, offers regional data residency for where content and its derived data are hosted, and publishes its sub-processor list so you can vet the full chain your embeddings pass through. That last point is easy to miss: every embedding call ships content to a model provider, and if you cannot name your sub-processors you cannot honestly answer a data-residency question. Governing the pipeline end to end, from source content through the embedding call to the stored vector, is the difference between an AI feature that passes review and one that stalls in it.
Controlling cost and versioning models at scale
Embeddings are cheap per call and ruinous in aggregate. A catalog of a few hundred thousand documents, chunked several ways and re-embedded on every edit, is a metered API bill that grows with your editorial velocity. The teams that keep this sane do three things: they embed incrementally so an edit re-embeds only affected chunks, they cache and hash chunk content so an unchanged block never pays for a redundant call, and they batch background re-embeds off the critical path so a spike in editing does not become a spike in latency.
Model versioning is the quieter cost. When you upgrade to a newer or higher-dimensional embedding model, every existing vector is now in an incompatible space, and a mixed index silently degrades relevance because you are comparing distances across two coordinate systems. A real migration re-embeds the whole corpus into a new field or dataset, validates recall against the old one, and cuts over atomically. This is a content operation, not a one-off script, and it wants the same versioning discipline you give schema changes.
This is where owning the corpus in one queryable store earns its keep. Because the Content Lake holds both the source content and the derived vectors, a re-embed is a governed transformation over content you already have rather than an export-transform-reimport dance across systems. Functions can run the migration incrementally, TypeGen keeps the schema for your new vector field type-safe end to end, and the whole job is observable in the same place your editors work. Rigid CMSes force you to scale people to keep an index current; making re-embedding an automated, event-driven projection scales output instead, so a corpus that doubles does not double the humans babysitting the pipeline.