Meaning has shape.
From LLMs to RAG
Srikanth Doddi
I design & build AI/ML, full-stack and cloud systems, and lead the team that ships them across R&D, delivery and presales.
building
AI / ML & GenAI
RAG, LLMs, embeddings, vision; architected end-to-end
Full-stack
React · Angular · Node · Python / FastAPI
Cloud & platform
AWS · Azure · GCP · IaC · Kubernetes
Lead · R&D · Presales
~20 engineers, delivery & solutioning
You'll know why your chatbot lies, and you'll have watched the fix get built.
One hour to understand. Two to build.
How GenAI broke through
Where it is now
Why LLMs lie
RAG: chunk → embed → retrieve → generate
Where RAG breaks
We build one, live
In two months, AI went from research to everywhere.
ChatGPT reached 100 million users faster than any app in history.
The tech wasn't new. The access was.
Transformers shipped in 2017. The capability built quietly for years. What changed in 2022 wasn't invention, it was access. Three things converged:
Actually capable
Good enough to feel like magic, not a toy, it held a conversation and wrote code.
Zero friction
Free, in a browser, no signup, no manual. You just typed a sentence.
Conversational
Plain language in, plain language out. Your grandmother could use it, and did.
From "can it generate?" to "can it do the work?"
Agentic AI Likely
Models plan, call tools, and recover from failure, not just chat. ~40% of enterprise apps embedding agents by end of 2026.
RAG went enterprise Certain
From a trick to governed "truth systems" with cited, auditable answers. Today's topic.
Multimodal by default Likely
Text, image, audio and documents in one workflow.
Small + on-device Likely
Efficient models run locally; frontier models reserved for hard reasoning.
LLMs are confident liars.
It only knows its training data, frozen at a cutoff. Ask about your data or recent events and it won't refuse, it guesses, fluently, and sounds just as sure as when it's right.
LIVE, DO THIS ON STAGE, DON'T DESCRIBE IT:
Ask a plain LLM something it cannot know: a detail from your college's own handbook, or a very recent event after its cutoff. Watch it answer confidently and wrongly. That failure is the entire reason RAG exists.
Meaning becomes geometry.
Every sentence is a point; similar meaning lands close. Embed a question and the nearest points are the passages most likely to answer it. The green question drops in, beams shoot to its neighbours, that is retrieval, live.
RAG: give the model an open book before it answers.
Chunk
Embed
Retrieve
Generate
Closed-book vs open-book. Steps 1-2 happen once (indexing). Steps 3-4 happen on every question. Next: watch each step happen.
Slice the document into passages.
One long document is too big to search. We cut it into bite-sized passages, with a little overlap so a sentence is never split across the seam.
First, the text becomes tokens.
The model never sees letters, only these token IDs. They flow through the network, and the final layer collapses the whole sequence into one vector. (Long or rare words split into sub-word pieces.)
Each passage becomes a point in space.
An embedding model reads a passage and emits a long list of numbers, a vector. That vector is a single point. Passages about the same thing land in the same neighbourhood.
A vector is just numbers. Stack them in a DB.
Each passage becomes a fixed list of floats, a point in 384-dimensional space. Every passage is one row, and the rows pour into the vector database: stored, indexed, ready to search in milliseconds.
Embed the question. Grab the nearest points.
The question becomes a point too. We measure distance to every stored point and pull the closest k, those are the passages most likely to hold the answer. The vector DB does this in milliseconds.
Hand the passages to the LLM. Answer from them.
The retrieved passages and the question flow into the model as context. It writes the answer from that text, grounded, specific, and able to cite its source, instead of guessing from memory.
Two pipelines, one shared memory.
Indexing fills the Vector DB once. Every question embeds, reads the nearest passages from that same store, and the LLM answers. The model is never retrained.
The entire retrieve-and-answer loop.
# 1. INDEX (once)chunks = split(docs, size=400, overlap=50)index = embed(chunks) # -> vectors # 2. ANSWER (every question)def answer(q): qv = embed(q) hits = index.top_k(qv, k=4) # cosine context = "\n".join(hits) return llm(f"Answer using ONLY:\n{context}\n\nQ: {q}")
No fine-tuning
The model is never retrained, you only change the prompt.
No GPU
Embed a few docs + call an API, runs on Colab's free tier.
Swap any piece
Chunk size, embed model, k, LLM, the shape stays the same.
Same question. Same model. One has the page.
Prompt it, show it, retrieve it, or retrain it.
Just ask. Fast and free, but ungrounded.
A few examples in the prompt, teaches format & style.
Retrieve facts into the prompt, grounds in your data.
Retrain the weights. Powerful, but expensive.
Few-shot teaches HOW to answer. RAG supplies WHAT to answer from. Often you stack them.
Chunk, embedding, vector, what's what?
model
A chunk is the words. A vector is the numbers. Embedding is the model that turns one into the other, and people usually say "embedding" to mean the vector it produces.
How a chunk becomes one vector.
Every token starts as a generic vector. Attention rewrites each one using the words around it, so "bank" by "river" ≠ "bank" by "money". Averaging collapses the whole passage into a single point, trained so similar meanings land in similar directions.
A token's vector is looked up, not computed.
The starting vector isn't calculated, it's looked up. The model learned one row of numbers per token during training. (Formally: a one-hot token ID × the embedding matrix selects exactly that row.) Attention then rewrites those numbers using the surrounding words.
From tokens to one vector, the math.
Closeness = cosine similarity.
It's just the angle between two vectors. Same direction → +1.0 (same meaning). Right angle → 0 (unrelated). Opposite → −1. Retrieval scores every passage this way and keeps the top few.
RAG fails at retrieval, not generation.
Chunking
One chunk holds the whole exam-rules page: attendance, fees, re-eval, all 5 topics. The question needs 1; the other 4 are noise that drowns the match.
Retrieval
"attendance leave" and "medical leave" embed close together. Top-k pulls the medical-leave rule for an attendance question. The right passage is never seen.
Ranking
Right passage is in the index, but ranks #7 when k=4. It existed, it just didn't make the cut. Raising k adds noise; that's the tradeoff.
Chunking: how you cut matters most.
Fixed-size
Every N tokens. Dead simple, but it slices sentences across the cut.
By sentence
Break on sentence boundaries. Clean, but chunk sizes vary a lot.
Semantic
Cut where the topic shifts. Each chunk = one coherent idea. Best recall.
Recursive
Paragraphs → sentences → words, until it fits. A solid default.
Bad chunks are the #1 cause of bad retrieval. Start with recursive; switch to semantic if recall lags. Always overlap ~10-15%.
Hybrid search, then re-rank.
The question splits two ways: vector search for meaning (the smooth stream) and keyword search for exact terms, names, codes, IDs that embeddings blur (the blocky stream). Both pour into one candidate pool, then a cross-encoder re-ranks it down to the top few.
You can't improve what you don't measure.
Of your test questions, how often is the right passage in the top-k retrieved?
Is every claim in the answer actually supported by the retrieved text? Catches hallucination.
Does the answer address what was asked, not just sound related?
Keep a golden set, fixed question → expected-answer pairs, and re-run it on every change. Add human spot-checks. Numbers catch regressions before users do.
What it takes to run RAG in production.
🎯 Retrieval quality
- Smart chunking, semantic splits, right size + overlap
- Pick the embedding model for your domain
- Re-rank the top-k with a cross-encoder
- Hybrid search: keyword + vector
- Measure recall@k, did we fetch the right passage?
⚙️ Operations
- Latency, cache embeddings & answers
- Cost, batch calls, right-size the model
- Freshness, re-index when documents change
- Monitor queries, retrieved chunks, answers
- Scale the vector store (ANN index, sharding)
🛡 Trust & safety
- Always cite sources, keep answers verifiable
- Refuse when no good context is found
- Respect document access control & PII
- Guard against prompt injection hidden in docs
- Human eval on a golden question set
A RAG bot, built live from an empty notebook.
Our document set
A bot that answers questions over a document the base model can't know, so you can SEE grounding work and verify the answers yourselves.
Default pick: your college's exam-rules / handbook PDF (relatable, the model genuinely doesn't know it, and the room can fact-check it live). Easy swaps: the AIML syllabus, a club's docs, or any PDF you bring.
The stack
Three ways in: pick the one your setup allows.
Everyone
Watch it built live, call out the next step. No device needed.
On your phone
Scan the QR for the hosted bot. Try to trip it up.
On a laptop
Open the Colab notebook with your Groq key, run as we go.
Didn't set up a Groq key? No problem, you'll leave with the notebook and a link to do it at home.