Portfolio · Mercor
Evaluating frontier language models on mathematics, agents and data
Three projects for AI research labs: writing mathematics problems that frontier models get wrong and experts can solve, evaluating models and agentic frameworks under constraints, and a STEM review project producing data visualisations with the science checked behind each one.
A frontier model answers a probability question in three confident paragraphs. The answer is wrong. My job was to write the questions where that happens, and to check, as the domain expert, that the right answer really is the right answer.
The challenge
The better language models get, the harder it is to tell real reasoning from pattern matching. Research labs need problems that expose the gap, checks on how models and agents behave when you put constraints on them, and reviewers who can tell whether an output is actually right rather than merely plausible. That has to come from people who know the subject, not from the models being tested.
Three projects
Adversarial problem sets. I wrote mathematical and statistical questions that state-of-the-art models answer wrongly but that a domain expert can solve, each with a verified solution. The topics were the ones I know best: probability (awkward distributions, conditional probability, expectations), statistical inference (hypothesis testing, confidence intervals, causal inference), optimisation (constraints, duality) and turning a messy applied question into something you can actually solve.
Agentic frameworks under constraints. Defining agentic frameworks and the behaviour expected of a model under several constraints at once, then evaluating how models cope when those constraints apply.
STEM review. Exploring datasets and producing data visualisations, diagrams and infographics, and reviewing the science and the data behind every one of them, so that what ends up on the page is correct and not just convincing.
What came out of it
The work fed evaluation frameworks used by leading AI research labs, and helped pin down where state-of-the-art models still fail on reasoning and how they behave under constraints.
What I took away
Good benchmarks come from knowing the subject well enough to know where models go wrong. The most interesting problems are the ones that are hard for a model but routine for an expert; that’s where the capability gap really shows. And problem sets get better the way models do, by being tested and revised, again and again.