Why No CMS Is Built for the AI Era (And What We Built Instead)

When the Société Royale d'Histoire et d'Archéologie de Tournai (SRHAT), a Belgian historical society founded in 1845, needed to digitize their archives, the requirement seemed simple: make hundreds of scanned PDF bulletins searchable through semantic search.

What started as a straightforward project revealed a gap: excellent CMS platforms exist (Sanity being a prime example), and powerful RAG tools exist, but nothing combined them natively. The integration work—connecting editorial workflows to OCR pipelines to vector databases—had to be built from scratch.

The result is a system that treats documents as both publishable assets and retrievable knowledge units, with Sanity providing the content management foundation and custom integrations handling the AI-powered search pipeline.

This article documents the architecture, integration challenges, and lessons learned.

The Challenge: Connecting Two Worlds

The tooling for document management and the tooling for AI-powered search have evolved separately:

• Content management systems (Sanity, Strapi, Directus) excel at structured content, editorial workflows, and collaboration
• RAG platforms (LangChain, Dify, RAGFlow) excel at ingestion, embedding, and retrieval
• Vector databases (Qdrant, Pinecone, Weaviate) provide the search infrastructure

Each category is mature and well-designed for its purpose. The challenge was connecting them: making a CMS-managed document automatically flow through OCR, chunking, embedding, and into a vector database—while keeping editors in control and providing real-time feedback.

This meant building the integration layer from scratch.

The Architecture

The system runs almost entirely on Cloudflare's edge infrastructure:

┌─────────────────────────────────────────────────────────────┐
│  Sanity Studio (Cloudflare Pages)                           │
│  - Content management + custom R2 upload plugin             │
│  - Real-time progress tracking                              │
└────────────────┬────────────────────────────────────────────┘
                 │
                 ▼
┌─────────────────────────────────────────────────────────────┐
│  Cloudflare Worker (Hono.js) + Cloudflare Queues            │
│  - Webhook handlers + async queue processing                │
│  - OCR orchestration + embedding generation                 │
└────┬────────────────────┬──────────────────┬────────────────┘
     │                    │                  │
     ▼                    ▼                  ▼
┌─────────┐      ┌──────────────┐    ┌─────────────┐
│   R2    │      │   Google     │    │   Qdrant    │
│ Storage │      │ Document AI  │    │ Vector DB   │
└─────────┘      └──────────────┘    └─────────────┘

Why this stack?

• Cloudflare Workers: Zero cold starts (critical for webhooks), built-in queues, R2 integration without egress fees
• Sanity CMS: Real document management with structured content, editorial workflows, and real-time collaboration
• Google Document AI: 90%+ OCR accuracy on 19th-century French documents (Tesseract yields around 60%)
• Qdrant: Open source, excellent filtering, payload storage

Why Sanity? A Real Document Management System

Most RAG tutorials skip the content management problem entirely. They assume documents magically appear in a folder, already tagged with metadata. In reality, someone needs to:

• Upload documents with proper metadata (title, authors, publication year, category)
• Review and correct OCR results
• Manage publication workflows (draft → review → published)
• Control who can access what
• Share documents with proper attribution

This is what a CMS does. Sanity was chosen specifically because:

Structured content with validation: Each publication has a defined schema with required fields, validation rules, and relationships (authors, categories). This isn't a folder of PDFs—it's a proper archive.

// Sanity schema enforces data quality
defineField({
  name: 'publicationYear',
  type: 'number',
  validation: (rule) => rule.min(1000).max(new Date().getFullYear()).integer(),
}),
defineField({
  name: 'trustLevel',
  type: 'number',
  description: 'Source reliability: 1 (low) to 5 (high)',
  validation: (rule) => rule.required().min(1).max(5),
}),

Real-time collaboration: Multiple historians can work simultaneously. Changes sync instantly across all connected browsers.

Editorial workflow: Documents can be drafted, reviewed, and published. The custom publish action ensures lastContentEdit is tracked properly for the re-processing logic.

Self-hostable Studio: The entire admin interface deploys to Cloudflare Pages. No vendor lock-in for the editorial experience.

Webhook integration: Every document change triggers a webhook, enabling the automated OCR → embedding pipeline while keeping humans in control of content decisions.

The result is a system where historians manage their archive through a familiar CMS interface, while AI-powered search happens transparently in the background.

Cost Breakdown (Monthly)

Note: Processing costs scale with volume. A quiet month might be $10; a heavy digitization push could hit $40+.

Challenge 1: The 500MB PDF Problem

Historical bulletins aren't small. A single SRHAT volume can be 500MB with 800+ pages. Every service has limits:

The solution: adaptive processing based on file size.

Small files (<5MB): Base64 encode and send directly to Document AI's sync API. Fast, simple.

Medium files (5-20MB): Upload to Google Cloud Storage first, then call Document AI with a GCS URI.

Large files (>50MB): Split into 15-page chunks using a Python Cloud Function, then process sequentially through the queue:

# Cloud Function splits PDF using qpdf
def split_pdf(input_path, output_dir, pages_per_part=15):
    page_count = get_pdf_page_count(input_path)
    parts = []
    start_page = 1
    
    while start_page <= page_count:
        end_page = min(start_page + pages_per_part - 1, page_count)
        subprocess.run([
            'qpdf', input_path, '--pages', '.', 
            f'{start_page}-{end_page}', '--', output_path
        ])
        # ... queue each part for processing
        start_page = end_page + 1
    
    return parts

The Worker then processes chunks one at a time through Cloudflare Queues:

if (action === 'extract-part' && splitInfo) {
  const { text, pageCount } = await extractTextFromPdf(env, partBuffer);
  await env.R2_BUCKET.put(partTextKey, text);
  
  if (splitInfo.currentPart < splitInfo.totalParts) {
    // Queue next part
    await env.EMBEDDING_QUEUE.send({
      action: 'extract-part',
      splitInfo: { ...splitInfo, currentPart: splitInfo.currentPart + 1 }
    });
  } else {
    // Last part: combine all texts, queue embedding
    const fullText = await combineAllParts(splitInfo);
    await queueEmbedding(publicationId, fullText);
  }
}

The trade-off: Processing a 500MB file takes 20-30 minutes instead of 5 minutes with parallel batch processing. But sequential processing is 99% reliable instead of 60% reliable. For a digital heritage project, reliability wins.

Challenge 2: The Webhook Loop Problem

When the Worker updates a document's embeddingStatus in Sanity, it triggers a webhook back to the Worker, which updates the status again... infinite loop.

Failed approach #1: Sanity's webhook filters can't distinguish between user edits and API mutations.

Failed approach #2: Timestamp comparison is unreliable due to clock skew.

What worked: A lastContentEdit field that's only updated by a custom publish action in Sanity Studio:

// Custom publish action in Sanity Studio
export const SetLastContentEditAction = (props) => {
  return {
    ...defaultPublish,
    onHandle: async () => {
      await props.patch.execute([
        { set: { lastContentEdit: new Date().toISOString() } }
      ]);
      await defaultPublish.onHandle();
    }
  };
};

Then in the webhook handler:

if (event.embeddingStatus === 'completed') {
  const lastEdit = new Date(event.lastContentEdit).getTime();
  const lastEmbed = new Date(event.embeddedAt).getTime();
  
  if (lastEdit <= lastEmbed) {
    // No new content edits since last embedding - skip
    return c.json({ message: 'Already embedded' });
  }
  // User edited content after last embedding - re-index
}

This separates user intent from automation state. Webhook loops work fine in testing (small files process quickly) but break catastrophically in production.

Smart Re-processing: Don't Repeat Expensive Work

A key design principle: never re-run expensive operations unnecessarily.

OCR is slow and costs money. Re-embedding is faster but still not free. The system tracks the pipeline state at each stage:

// Sanity schema tracks both pipelines independently
{
  textExtractionStatus: 'pending' | 'processing' | 'completed' | 'failed',
  extractedTextKey: string,      // R2 path to extracted text
  
  embeddingStatus: 'pending' | 'processing' | 'completed' | 'failed',
  embeddedAt: datetime,
  lastContentEdit: datetime,     // Only set by user actions
}

This enables intelligent decisions:

• User changes the publication year? Metadata changed → re-embed (text chunks need updated payload) → but skip OCR (PDF hasn't changed)
• User uploads a new PDF? New file → re-OCR → then re-embed
• User fixes a typo in the title? Metadata changed → re-embed only
• Automation updates status fields? No content change → do nothing

The webhook handler checks what actually changed:

// Already have extracted text for this exact PDF?
const hasExtractedText = event.extractedTextKey === expectedTextKey 
                      && event.textExtractionStatus === 'completed';

if (hasExtractedText) {
  // Skip OCR, go directly to embedding
  await queueEmbedding(publicationId, event.extractedTextKey, metadata);
} else {
  // Need OCR first
  await queueExtraction(publicationId, pdfKey, metadata);
}

For a 500-page document, this means:

• Re-OCR avoided: saves 15-30 minutes + ~$0.75 in API costs
• Re-embedding only: ~60 seconds + ~$0.01

Over hundreds of documents with iterative metadata corrections, this adds up significantly.

Challenge 3: Custom File Storage

Sanity's asset system isn't designed for 500MB PDFs. The solution was a custom R2 asset plugin:

// Worker generates signed URLs
r2Upload.post('/signed-url', async (c) => {
  const { filename, contentType } = await c.req.json();
  const key = `publications/${Date.now()}-${sanitize(filename)}`;
  
  const uploadUrl = await generateSignedUploadUrl(c.env, key, contentType);
  const downloadUrl = await generateSignedDownloadUrl(c.env, key, 86400);
  
  return c.json({ uploadUrl, key, downloadUrl });
});

Files upload directly from the browser to R2. Metadata lives in Sanity. Download URLs are signed with 24-hour expiry. The UX feels native to Sanity Studio users.

The Chunking Problem

Standard sentence-based chunking doesn't work well on OCR'd historical documents:

• OCR errors create artificial sentence boundaries
• 19th-century French uses different punctuation conventions
• Tables and lists break sentence detection

The solution was word-based chunking:

export function chunkText(text: string, chunkSize = 500, overlap = 50) {
  const words = text.replace(/\s+/g, ' ').trim().split(' ');
  const chunks = [];
  let currentChunk = [];
  
  for (const word of words) {
    if (currentChunk.join(' ').length + word.length > chunkSize) {
      chunks.push(currentChunk.join(' '));
      // Keep last N words for overlap
      currentChunk = currentChunk.slice(-overlap);
    }
    currentChunk.push(word);
  }
  
  return chunks;
}

Less semantically elegant, but robust to messy OCR output.

Expected Outcomes

Based on testing and initial runs:

Processing times (estimated):

• Small files (<5MB): ~30 seconds
• Medium files (5-20MB): 2-5 minutes
• Large files (>50MB): 15-30 minutes

Lessons Learned

1. Reliability beats speed. Parallel batch processing is faster but hits quota limits constantly. Sequential processing takes 6x longer but succeeds 99% of the time. For archival work, this is the right trade-off.

2. Progress feedback is non-negotiable. Users tolerate slow processing if they see progress. A 15-minute job without feedback feels broken; with granular updates (10%, 25%, 50%), users understand the system is working.

3. Webhooks need idempotency guards. Webhook loops are insidious—they work fine in testing but create infinite loops in production. Always track user intent separately from automation state.

4. Design for service limits from day one. Every service has limits. The three-tier processing strategy (inline/GCS/split) handles everything from 1MB to 500MB files because it was designed around constraints, not despite them.

5. Adapt algorithms to data characteristics. Word-based chunking is "wrong" according to RAG best practices, but it's right for OCR'd historical documents with inconsistent punctuation.

What's Not Complete Yet

Transparency about scope:

• Multi-user permissions: Currently single-admin. Role-based access (admin/editor/viewer) is scaffolded but not fully tested.
• SSO integration: Clerk authentication works but could be tighter.

What's Complete: Search Exclusion

One feature that is fully implemented: excluding documents from search results.

This matters more than it might seem. Historical societies deal with documents of varying reliability:

• Unverified transcriptions that might contain errors
• Documents with disputed authorship or dating
• Duplicate versions pending review
• Sensitive materials that should be catalogued but not surfaced in public search

The system allows editors to flag any document as "excluded from search" directly in Sanity Studio. The document remains in the CMS—fully catalogued, with all metadata—but is filtered out at the Qdrant query level. When doubts are resolved, one toggle brings it back into search results.

For an organization like SRHAT managing 180 years of historical publications, this is essential. Not every digitized document should immediately be searchable. Academic rigor requires the ability to say: "We have this, we're cataloguing it, but we're not yet confident enough to include it in research results."

Future Improvements

Several enhancements are planned:

• Streaming OCR: Stream large PDFs directly to GCS instead of loading into memory
• Circuit breakers: Detect service outages and pause processing gracefully
• Retry budgets: Smarter retry logic based on error types (permanent vs transient vs quota)
• Document versioning: Keep history of PDF uploads

The Broader Point

There's a gap in the market for systems that treat content as both a publishable asset and a retrievable knowledge unit. Museums, archives, legal departments, research organizations—they all face this problem.

The stack exists: headless CMS + OCR service + embedding API + vector database. But the integration work is non-trivial:

• File size strategies across service limits
• Webhook idempotency guards
• Progress tracking as a first-class feature
• Graceful degradation when services fail

This is the unglamorous plumbing that makes RAG systems actually work in production.

Tech Stack: Cloudflare Workers, Cloudflare R2, Cloudflare Queues, Sanity CMS, Google Document AI, Mistral AI (embeddings), Qdrant, Clerk (auth), Python/qpdf (PDF splitting)