Fewer LLM calls, better matches

When a candidate signs up, an LLM turns their application into a structured profile. The engine then ranks open vacancies for that candidate and explains each match, and recruiters approve or reject the suggestions in an internal dashboard. I built the first version of the engine and still lead it, and I am the top contributor to the core API and the approval dashboard behind it.

Version one scored every candidate vacancy with its own LLM call. It worked, but recruiters rejected most of the top suggestions, and one LLM call per vacancy made every run slow and expensive.

• Eligibility before any LLM call

  Version two starts from vacancies that pass hard filters and a location check, so the LLM only sees plausible matches.

  Trade-off Vacancies outside the filters are never considered.

• A learned ranker, not the LLM's score

  LLM scores bunched together and barely separated good matches from weak ones, so a small model trained on recruiter decisions orders the shortlist.

  Trade-off The model needs retraining as recruiter behaviour shifts.

• Hard rules live in code

  Business rules are enforced in code after ranking. The model never gets to break them.

  Trade-off Every new rule is a code change, reviewed and tested.

• Batched LLM calls

  One call evaluates many vacancies, which cut calls by more than half and cost by about a quarter.

  Trade-off Answers can land on the wrong vacancy (see hard problems).

• Idempotent by design

  Jobs are keyed on the event that triggered them, and snapshot tests guard every pipeline configuration.

  Trade-off Snapshot updates are part of every pipeline change.

1. Answers landing on the wrong vacancy

   Batching made the model attribute some answers to the wrong vacancy. Making it echo what it was scoring and repairing partial batches made that rare.

2. Misses that weren't

   Many of the matches the engine seemed to miss were recruiters deliberately matching to closed vacancies. Those vacancies are now part of the pool.

3. Rules that fought each other

   Several ranking rules interacted badly at the cut-offs, until each became an explicit, tested rule.

Every change started as a written spec and was measured against a fixed benchmark with a holdout set before it shipped. Role-split AI agents (architect, executor, reviewer) did much of the implementation inside those guardrails.