20 synthetic profiles
A bounded corpus spanning engineering, product, design, recruiting, marketing, customer success, finance, and operations.
A fixed test for keyword, semantic, and hybrid candidate retrieval.
The benchmark freezes a synthetic candidate corpus, recruiter queries, hard filters, relevance judgments, metrics, and retrieval configuration before comparative interpretation.
Benchmark at a glance
10 recruiter queries
Natural-language searches covering role, domain, skills, leadership context, remote eligibility, and minimum experience.
Four declared metrics
Precision@5, NDCG@5, zero-result rate, and hard-filter pass rate measure different retrieval behaviors.
No real candidate data
Names, profiles, queries, and relevance labels are synthetic; protected attributes are excluded from ranking inputs.
What is fixed before evaluation
Freezing inputs prevents the test from being reshaped around a preferred output.
Candidate corpus
Each profile has a stable identifier, headline, location, remote flag, experience level, and concise searchable evidence.
Recruiter queries and filters
Every query declares its natural-language request alongside explicit remote and minimum-experience requirements.
Relevance rubric
Every query-profile pair uses a 0–3 scale ranging from not relevant to a strong match to the declared information.
Cutoff and metrics
The ranking cutoff is five, and the same four metrics apply to every retrieval method compared.
Declared metrics
| What it measures | Why it matters | |
|---|---|---|
| Precision@5 | The share of the first five results judged relevant. | Recruiters inspect the top of a shortlist first; irrelevant early results create immediate review cost. |
| NDCG@5 | Whether stronger matches appear above weaker or adjacent matches. | Two methods can retrieve similar candidates while ordering the best evidence differently. |
| Zero-result rate | How often a method returns no candidates for a declared query. | Exact-term systems can miss relevant profiles when candidate and recruiter vocabulary differ. |
| Hard-filter pass rate | Whether returned profiles satisfy explicit remote and experience constraints. | A semantically similar profile is still unsuitable when a non-negotiable requirement fails. |
Example recruiter queries
The query set tests meaning, exact evidence, and structured constraints together.
| Hard filters | Evidence a strong result should contain | |
|---|---|---|
| Senior backend engineer who built payments infrastructure | Remote; at least six years of experience | Backend ownership plus concrete payments, ledger, checkout, or reconciliation systems. |
| Engineering leader who stayed hands-on with Node.js | Remote; at least eight years of experience | Direct team-management evidence and continued delivery in Node.js services. |
| Machine-learning engineer for semantic retrieval | Remote; at least four years of experience | NLP, embeddings, evaluation datasets, reranking, and production search evidence. |
| Technical recruiter sourcing passive candidates | Remote; at least four years of experience | Passive sourcing, intake calibration, shortlist review, and hiring-manager feedback. |
Retrieval stages represented
Each stage solves a different part of the retrieval problem and can fail differently.
Lexical retrieval
BM25 rewards exact and repeated term evidence, preserving literal matches for tools, credentials, and proper nouns.
Dense retrieval
Embeddings compare meaning across different wording, helping related experience surface without shared tokens.
Rank fusion and reranking
Reciprocal-rank fusion combines lexical and dense lists before a cross-encoder reassesses query-profile relevance.
Hard filters
Structured constraints remain explicit rather than being approximated as semantic similarity.
Interpretation boundaries
A retrieval test, not a hiring test
The benchmark measures ranking behavior inside a synthetic corpus. It does not measure interviews, offers, retention, or job performance.
A bounded corpus
Twenty profiles make failures inspectable but do not represent every role, geography, industry, or candidate-writing style.
Relevance is query-specific
A profile can be strong for one declared need and unsuitable for another; the benchmark does not assign universal candidate quality.
Production behavior is separate
The benchmark configuration describes the evaluation path. Deployment flags, indexed content, thresholds, and model availability still shape a live system.
Structured benchmark data
The complete machine-readable artifact contains all 20 profiles, 10 queries, filters, draft grades, configuration, metrics, and limitations.
See why retrieval method matters
Compare exact-term matching with meaning-based search, including where each approach still belongs.