Teaching Materials RAG
Search across 86,527 passages of my own course material, fully local.
I built a private search system over years of my own lecture decks, cases and handouts, using open models that run entirely on my laptop. It finds the right slide or passage in seconds, and lets Claude assemble new decks and handouts from my own material.
The problem
After years of teaching at IIM Calcutta, my material sits in thousands of files: lecture decks, cases, handouts, spreadsheets, scanned pages. When I prepare a session, I know a slide exists somewhere. Finding it by folder name or file name often fails, because I filed things by course and year, not by idea.
Two constraints shaped the build. The collection includes third-party teaching cases and some student submissions, so it cannot be uploaded to a hosted service. And an ordinary document search matches words, while I usually remember a concept. I needed retrieval by meaning and by keyword, over my own material only.
What I built
The pipeline starts with extraction. PyMuPDF, python-pptx, python-docx and openpyxl read PDFs, decks, documents and spreadsheets. LibreOffice converts older formats, and Tesseract OCR handles scans. I chunk by natural unit: one chunk per slide or page, each carrying a context header (course, file, position) so a passage is not an orphan. Near-duplicates, which are common when a deck is reused across years, are folded together.
Embeddings come from nomic-embed-text running in Ollama. Chunks go into LanceDB, in a table I call chunks_v2, with a full-text index beside the vector index. Retrieval is hybrid: a vector search and a keyword search, with my own material favoured in the ranking. A small command-line tool, tm.py, offers find, file and passages commands. For questions, ask.py answers locally with Qwen3-8B served through Ollama.
On top of the index, a Claude skill lets Claude assemble decks and handouts from passages the index returns. That part uses a frontier model, and I keep it distinct: retrieval and local answering are open and local; the assembly skill is Claude working on retrieved excerpts.
I also built a simpler ChromaDB version for teaching the idea to students.
How it performed
The numbers I have are about scale, not accuracy. I inventoried 7,339 files totalling 29 GB, considered 6,131, and read 5,527. Version 2 holds 86,527 passages. Version 1 held 65,051 chunks across 19.3 million words. A full rebuild takes about two hours on a laptop with an 8 GB GPU. Another 258 lecture videos are queued for transcription.
What I do not have is a measured retrieval score. I have not recorded a recall figure, a precision figure or a held-out query set for this system, so I cannot tell you it finds the right slide in some percentage of cases. What I can say is that it is a personal production tool that I use for my own preparation. No outcome beyond my own use is recorded.
What I learned
Context is what makes a chunk findable. A slide on its own often says almost nothing; with its deck title and position, it can be found. That is why each chunk carries a context header.
Hybrid retrieval earned its place. Vector search finds the idea I half remember, and the keyword index finds the exact term or case name I do remember. Neither alone suited a teaching library full of both.
Duplicates are a quiet problem. Teaching material is reused and lightly edited, so without folding, the top results can be several copies of the same slide from different years. Folding near-duplicates is cheap and worth doing early.
Privacy was a design input, not an afterthought. Because the collection holds licensed and student material, I chose open, local models for embedding and answering from the start. That is a real case for small models: not that they beat frontier models at everything, but that for this job they are enough and the data stays on the machine.
The thing I would add next is a small evaluation set of real queries with known right answers, so that the quality of the index is a number and not an impression.