Practitioners often search across many scanned books, translations, and commentaries to answer one question. Exact keyword search alone misses related teachings, while vector search alone can miss a specific name or verse. A normal chatbot may also answer too early, lose useful context during a long investigation, or cite text that the user cannot verify. AtmaLoka needed both a stronger retrieval engine and an agent able to research before answering.
01 / Applied AI · Agentic RAG · Scripture Research
Scripture-Grounded AI Assistant
Brand: AtmaLoka
An agentic RAG product that searches a multilingual scripture corpus, performs multi-step research, and shows the original passages and citations behind every answer.
Separate passage retrieval from answer research. The retrieval engine uses HyDE, BM25, multilingual embeddings, RRF fusion, and LLM reranking to find strong passages. A custom agent harness then decides whether to search broadly, stay within a book or lineage, locate an exact verse, fetch neighboring text, or retrieve a complete section before writing the answer.
AtmaLoka is live with 100+ signed-up users and more than 250,000 searchable passages. Its retrieval pipeline uses HyDE, BM25, multilingual vectors, rank fusion, and reranking, while a custom agent harness performs multi-step research with source-aware tools.
The product is live and growing. The next step is to measure which answers users find most helpful and what brings them back regularly.
01
User and problem
Who needed this product, and why.
AtmaLoka is for practitioners and serious readers of Sri Vidya, Tantra, and related traditions. They can ask in everyday English, search for a Sanskrit term, paste a verse in Devanagari, Bengali, IAST, or plain Roman script, search within one book or tradition, and request a detailed explanation. The product returns the original passage, translation or explanation, and an available book, page, section, or verse reference.
I built AtmaLoka from the source library to the final answer experience. The product includes a multilingual RAG pipeline and a custom agent harness. The retrieval pipeline prepares and searches the books, expands difficult questions with HyDE, combines BM25 keyword search with multilingual vector search, merges both rankings with RRF, reranks the strongest passages, and restores their full original text. The agent harness can then perform several searches, choose specialized tools, keep useful passages in working memory, fetch exact verses or complete sections, and build a cited answer over multiple steps.
02
Important decisions
Decision 01
Keep the original text intact
- Why it mattered
- Many books were scanned and contained OCR mistakes, mixed languages, and verse layouts that a normal text splitter could damage.
- What I did
- Clean obvious scanning noise, but keep the original passage unchanged. Split Sanskrit at verse and chapter boundaries, and keep the book, page, section, language, edition, and tradition attached to each passage.
- What it required
- Preparing and checking the corpus took more work than simply uploading PDFs to an AI model.
- The result
- Users can read the passage in its original script and open the correct source location. If the scanned text is unclear, the product marks it as unclear instead of guessing.
Decision 02
Build a multi-stage retrieval pipeline
- Why it mattered
- Exact search is useful for names and verses, but it can miss a teaching described with different words. Semantic search finds related meaning, but it may overlook a precise quotation.
- What I did
- Use HyDE to expand difficult questions into language closer to the source material. Run BM25 exact search and multilingual vector search in parallel, combine their rankings with Reciprocal Rank Fusion, and use an LLM reranker to select the passages that best answer the question.
- What it required
- The pipeline has more moving parts than a single vector search. Both indexes, the fusion stage, and the reranker must stay reliable as the library grows.
- The result
- AtmaLoka can find an exact verse, connect an English question to Sanskrit source text, and remove weaker matches before the answer is written.
Decision 03
Build an agent for research, not one-shot chat
- Why it mattered
- One retrieval call is often not enough for a full text explanation, a lineage comparison, or a question that needs several books and nearby passages.
- What I did
- Build a custom multi-step agent harness with specialized tools. The agent can search broadly, restrict research to a book or lineage, locate a verse across scripts, fetch a complete section, expand neighboring passages, assemble ritual sequences, and escalate difficult reasoning when needed.
- What it required
- A tool-using agent needs step limits, timeouts, retries, scope controls, and clear rules for when to stop searching and answer.
- The result
- The assistant can change its research path based on what it finds instead of forcing every question through the same search call.
Decision 04
Preserve context through long answers
- Why it mattered
- Large source passages and repeated tool results can overflow the model context, while long verse-rich answers can be cut off before the explanation is complete.
- What I did
- Keep selected passages in working memory, remove duplicate text already seen, store full tool results outside the prompt, allow passages to be fetched again by ID, and continue long answers across controlled steps. Restore full original text and citation data before writing.
- What it required
- Context management adds orchestration code, disk-backed tool logs, continuation rules, and completeness checks.
- The result
- The agent can research and write longer answers without repeatedly sending the entire corpus context or losing the source passages it needs to cite.
03
What I built
What the product includes and how it works.
- OCR and structured e-text ingestion with verse-aware chunking and source metadata
- Original passages preserved with their book, edition, chapter, page, section, verse, language, and tradition
- Self-hosted multilingual embeddings that search Sanskrit, Bengali, and English in the same meaning space
- HyDE query expansion that turns a difficult question into source-like language to improve semantic recall
- Parallel BM25 and vector retrieval for exact matching and meaning-based discovery
- Reciprocal Rank Fusion that combines both result lists before an LLM reranks the strongest passages
- Cross-script verse matching across Devanagari, Bengali, IAST, and plain Roman text
- A hand-built multi-step agent harness with 15 research and control tools for broad search, book and lineage search, exact verse location, full-section retrieval, neighboring context, ritual assembly, memory, and continuation
- Working memory, duplicate removal, disk-backed tool results, passage refetching, retries, timeouts, and scoped search controls
- Controlled continuation and completeness checks for long, verse-rich answers
- Citation-aware PDF and e-text readers that open the relevant page, section, or verse
- Automated checks for retrieval, citations, agent tools, context handling, and answer behavior
How it works. A question can first be expanded with HyDE, then searched through BM25 and self-hosted multilingual vector indexes. RRF combines both rankings and an LLM reranker selects the strongest passages. A custom agent harness can run additional book, lineage, verse, section, and neighboring-context tools, keep selected passages in working memory, and produce a cited answer over multiple controlled steps.
04
What exists today
100+ people have signed up for AtmaLoka
Signed-up accounts in the live product.The production corpus contains more than 250,000 searchable passages
The September 2026 production record lists 251,636 indexed passages across 694 catalogue entries.The retrieval pipeline uses HyDE, BM25, multilingual vectors, RRF fusion, and LLM reranking
Each stage improves a different part of search: query recall, exact matches, related meaning, ranking fusion, and final relevance.A custom agent harness performs multi-step research before answering
It selects specialized tools, searches across books or traditions, fetches exact verses and full sections, and gathers more context when one search is insufficient.The harness manages working memory and long answers without dropping source context
Important passages can be pinned, repeated text is removed, full tool results can be fetched again, and long responses continue across controlled steps.Users can open the original source behind an answer
Citations connect to an available PDF page or native e-text section and verse, while the original passage remains visible to the user.05
Reflection
What I learned, and what I would test next.
- A strong RAG product needs both a well-prepared corpus and a retrieval pipeline designed for the ways users actually search.
- HyDE, BM25, multilingual vectors, rank fusion, and reranking solve different retrieval failures; no single stage replaces the others.
- An agent harness becomes valuable when it can choose the right research path, preserve useful context, and know when one search is not enough.
- Context engineering is a product requirement for long-form AI research, not only a backend optimization.
- AI-added translations and metadata should improve discovery without replacing the original source text.
Let’s discuss the product problem.