Hybrid retrieval pipeline

Hybrid retrieval fuses keyword-based BM25 searches with vector-based semantic embeddings to capture both exact terminology and conceptual intent.

Modern RAG systems often fail when relying solely on vector similarity (dense retrieval) or keyword matching (sparse retrieval). Hybrid pipelines solve this by executing both methods in parallel and merging the results using Reciprocal Rank Fusion (RRF). This approach ensures that a query for a specific product ID like 'SKU-9921' succeeds via BM25 while a conceptual query about 'troubleshooting power issues' succeeds via OpenAI's text-embedding-3-small. By weighting these scores (often at a 0.3 sparse to 0.7 dense ratio), developers achieve higher Hit Rates and Mean Reciprocal Rank (MRR) across diverse datasets.

https://python.langchain.com/docs/concepts/retrieval/