Case Study
YouTube Growth AI Chatbot (Genzzz)
RAG chatbot over YouTube transcripts. Hybrid BM25+dense retrieval with cross-encoder reranking lifted Recall@1 to 0.673 and MRR to 0.788 on a 55-query eval set (from 0.509/0.684 dense-only), with a CI gate to prevent regressions. 44 tests.
Executive Summary
Genzzz is a RAG chatbot over YouTube creator transcripts. It began as a 7-day FAISS + Gemini build; it now ships hybrid BM25+dense retrieval fused with reciprocal-rank fusion and a cross-encoder reranker — and, more importantly, a retrieval eval harness that measures every change. On the 55-query eval set, hybrid+rerank lifts Recall@1 from 0.509 to 0.673 and MRR from 0.684 to 0.788 over dense-only, with a CI gate preventing regressions.
Problem & Constraints
YouTube growth knowledge is buried across hours of video. Dense-only retrieval missed exact-term queries (channel names, jargon); the fix had to be measured, not vibes — including publishing the negative results.
Architecture
YT-DLP transcripts → chunking → all-MiniLM-L6-v2 embeddings + FAISS ∥ BM25 → reciprocal-rank fusion → cross-encoder rerank → Gemini answer. eval/queries_expanded.json (55 in-corpus queries) + scripts/eval_retrieval.py drive a CI regression gate.
Methodology
- Hybrid retrieval: BM25 and dense candidates fused with reciprocal-rank fusion, then reranked by a cross-encoder
- Authored a 55-query eval set against the actual transcripts (all targets in-corpus) after finding the legacy 20-query set had 9/20 unreachable targets
- Every config (dense / hybrid / hybrid+rerank) measured on both sets with exact reproducible commands
- CI runs the eval and fails on retrieval regressions
- Index integrity enforced: FAISS index and metadata are a matched pair, checked at load
Results & Metrics
| Set | Config | R@1 | R@5 | MRR |
|---|---|---|---|---|
| expanded-55 | dense | 0.509 | 0.855 | 0.684 |
| expanded-55 | hybrid | 0.509 | 0.909 | 0.688 |
| expanded-55 | hybrid+rerank | 0.673 | 0.964 | 0.788 |
Honest negative, reported as-is: on the legacy 20-query set the reranker leaves recall flat and slightly lowers MRR (0.425 → 0.412) — 9 of its 20 targets aren't in the corpus, which is exactly why the eval set was rebuilt. 44 tests passing.
Tech Stack
Python, LangChain, FAISS, rank-bm25, sentence-transformers (bi- + cross-encoder), Gemini API, Streamlit/Gradio, Hugging Face Spaces
Future Work
Query expansion (HyDE) as a measured experiment on the same harness, multi-channel support.