- Timeline
- Jan 2026 – present
- Team
- Ten Strands and the UC Berkeley Data Discovery program
- My role
- Retrieval system design and evaluation
- Skills
- Python, BM25, Embeddings, Neural reranking, Evaluation
The goal
California K-12 sustainability work depends on finding the right data sources, and finding them by hand is slow. The goal was a retrieval system that puts the most relevant sources at the top for a given question, so people spend their time using data instead of hunting for it.
Highlights
- 01
Keyword, meaning, then a second opinion
The system combines BM25 keyword search with embedding-based semantic search, then passes the candidates through a neural reranker. Each stage catches what the one before it misses.
- 02
More of the right answers up top
Measured on held-out queries, Recall@5 improved by 35% over the starting approach.
+35%Recall@5
- 03
Time back for the people doing the work
Manual source-finding time dropped by 76%.
−76%manual search time
Looking forward
I'm interested in keeping the evaluation honest as the corpus grows, and in making the results easy for non-technical users to trust.