Building RAG Chatbots on Cloudflare Workers AI + Vectorize
We built Lex — an AI Scrabble coach that answers strategy questions using 1,000+ blog articles as its knowledge base. Here's the architecture, the code patterns, and the lessons learned from deploying a production RAG chatbot entirely on Cloudflare's edge.
What is RAG and Why Does It Matter?
Standard LLMs hallucinate. They confidently make up facts because they have no access to your specific data. RAG (Retrieval-Augmented Generation) solves this by giving the AI real context before it generates a response.
💡 The RAG Flow
User asks a question → Embed the question into a vector → Search your knowledge base for similar vectors → Return the top 5 most relevant chunks → Feed those chunks + the question to the LLM → LLM generates an answer grounded in your actual data.
The result: answers that are accurate, cite-able, and specific to your business — not generic training data.
Architecture Overview
Our stack runs entirely on Cloudflare — no external dependencies, no cold starts, no region selection:
Workers AI
LLM inference (Llama 3 8B) and embedding generation (bge-base-en). Runs on the same edge as your Worker.
Vectorize
Vector database for semantic search. Stores 768-dimension embeddings with metadata (source URL, title).
D1 (SQLite)
Conversation history, user sessions, rate limiting. Persistent storage at the edge.
Astro (SSR)
Frontend chat UI with streaming responses. Server-side API route handles the RAG pipeline.
Step 1: Building the Knowledge Base
Before the chatbot can answer questions, you need to embed your content into vectors. We wrote a Python script that:
🧩 Embedding Pipeline
Reads all blog .astro files and extracts text content (strips HTML tags)
Chunks content into ~500 token segments with 50-token overlap
Calls Workers AI embedding model (bge-base-en-v1.5) to get 768-dim vectors
Upserts vectors to Vectorize index with metadata (title, URL, category)
For ScrabbleWordsFinder, we embedded 1,000+ articles — roughly 3,500 vector chunks. The entire indexing process takes about 10 minutes.
Step 2: The Query Pipeline
When a user sends a message, the API route does this:
⚡ Request Flow (~300ms total)
1. Embed the user's question → ~30ms
2. Query Vectorize for top 5 similar chunks → ~20ms
3. Build system prompt + context + question → ~1ms
4. Stream LLM response back to client → ~250ms first token
Step 3: Prompt Engineering
The system prompt is critical. Ours includes:
Role definition: "You are Lex, an AI Scrabble coach. Only answer questions about Scrabble, word games, and vocabulary."
Context injection: "Use ONLY the following sources to answer. If the answer isn't in the sources, say so."
Guardrails: Topic boundaries, response length limits (under 200 words), no personal advice, safe fallback behaviours.
Tone control: Friendly, knowledgeable, concise. Uses British English. Doesn't over-explain.
Lessons from Production
✓ Chunk size matters
500 tokens with overlap works better than whole-document embeddings. Too large = diluted relevance. Too small = missing context.
✓ Metadata is essential
Store title, URL, and category with each vector. This lets you filter results and provide citation links in responses.
✗ Don't skip guardrails
Without topic boundaries, users will ask the chatbot to write code, tell jokes, or generate harmful content. Define boundaries early.
✗ Don't trust cosine similarity alone
A 0.85 similarity score doesn't guarantee relevance. Set a minimum threshold (we use 0.75) and return "I don't know" below it.
Want a RAG chatbot for your business?
We build and deploy AI chatbots in 3-6 weeks. Trained on your data, deployed on Cloudflare's edge.
Get in Touch →