SalesRook
Senior Technical Lead
Lead generation and qualification platform for letting agents. Handles enquiries on autopilot.
Service
LLM & RAG Systems
Retrieval that returns the right chunk — with the evals to prove it.
Most RAG demos work on ten documents and fall apart on ten thousand. I build retrieval the other way round — starting from a labelled eval set, so every change to chunking, embeddings, or reranking is measured instead of guessed.
Sound familiar?
You probably need this if…
If two or more of these land, this is the right conversation to have.
Your assistant answers confidently from the wrong document, and nobody can say why.
Retrieval quality is a vibe — there's no eval set, so every prompt change is a coin flip.
Token spend climbs every month and nobody can attribute it to a feature.
It works on the demo corpus and falls over on the real one.
Deliverables
What actually gets built.
Artifacts, not slides — all of it yours to keep and run without me.
- 01
A labelled eval set you own
50–200 real questions with known-good answers, versioned in your repo. Every retrieval change is scored against it before it ships.
- 02
A hybrid retrieval pipeline
Dense embeddings for meaning, BM25 for exact terms like SKUs and error codes, and a cross-encoder reranker on top. Chunking tuned to your document shapes, not a tutorial default.
- 03
Grounded generation with citations
Answers that cite the chunks they came from, plus an abstain path so the system says "I don't know" instead of inventing something.
- 04
Observability and cost tracking
Per-request traces, latency and token spend broken down by feature, and alerting when retrieval quality drifts.
- 05
A failure-mode runbook
Written documentation of what breaks, how to spot it in the traces, and what to do about it — so your team can operate the system without me.
Process
How this one runs.
Three phases, with something demoable at the end of every week.
- 01
Baseline
Instrument what already exists and build the eval set. Inside the first week you get a real number for how good retrieval is today.
- 02
Rebuild the pipeline
Chunking, embeddings, hybrid search, reranking — changed one variable at a time, each scored against the eval set.
- 03
Harden and hand over
Caching, cost controls, tracing, alerting, runbook, and a walkthrough with your engineers.
The stack
What I reach for.
Defaults, not dogma — I'll work in yours where it makes more sense.
- Python
- OpenAI
- Anthropic
- LangChain
- Hugging Face
- FastAPI
- PostgreSQL
Proof
Where this has shipped.
Real products, in production, with real users on them.
FAQ
Before you ask.
Do I need a vector database?
Often not. Postgres with pgvector handles most corpora comfortably, and it's one less system to operate. I'll recommend a dedicated vector store only when your scale or filtering needs actually justify it.
We already have a RAG prototype. Can you work with it?
Yes — that's the usual starting point. The first week goes on measuring what you have, so we know which parts are worth keeping.
How do you measure "better"?
recall@k and answer accuracy against the labelled eval set, plus latency and cost per request. Every change lands with before/after numbers attached.
Which models do you use?
Whichever fits the task and the budget. I've shipped on both OpenAI and Anthropic models, with smaller open models for classification and reranking. Model choice is a routing decision, not a religion.
Related
Often paired with.
Need help with LLM & RAG Systems?
Tell me what you're working on. I'll tell you what it takes — and whether it's worth building at all.