Python · cell 1
An automated research agent is a language model in a loop that does machine learning research. It proposes an idea, writes the code, runs the experiment, reads the result, and tries again. Labs now build these agents and quote their benchmark scores, so an engineer needs to read those scores the way a skeptical reviewer would. This section asks two questions. What does such an agent need in order to make real progress, and how would you know whether it did?
The progress has been fast. MLE-bench is a benchmark of 75 past competitions from Kaggle, a website that hosts machine learning contests (Chan et al. 2024). In each one an agent must train a model and submit predictions within 24 hours. It wins a medal when its submission would have ranked high enough on the competition's leaderboard to earn a Kaggle bronze, silver, or gold. In October 2024 the best agent won a medal in 16.9% of the competitions. It was the o1-preview model inside AIDE (Jiang et al. 2025), a scaffold that keeps a tree of training scripts. A scaffold is the program around the model that decides what to ask it next.
By February 2026 the top of the leaderboard had reached 64.4%. In April 2026 the maintainers stopped taking submissions "while we develop an improved process for ensuring submissions are fair and comparable" (MLE-bench repository, read 2026-10-20). Two entries, the best at 77.8%, already sat in a separate table because they had used feedback from the test set. A year earlier, in April 2025, a paper written end to end by an automated system, The AI Scientist-v2, scored above the acceptance threshold of an ICLR 2025 workshop (Yamada et al. 2025).
One idea explains both the successes and the doubts. A loop improves only as far as its checker, the verifier, can tell good outputs from bad, which is the lesson of §26.1. Research agents are the cleanest case. Where the verifier measures exactly, as with the number of multiplications in a matrix algorithm or the runtime of a kernel, agents already find things people had not. Where the verifier is noisy, they find noise. Where it is a model reading a paper, they produce what looks good to that model.
We build three tools to make this precise. The winner's curse bounds how much of the best of many measured scores is luck. A comparison of search with independent sampling says when an exact evaluator lets improvements add up. And best-of- at a time budget explains the most quoted result comparing agents with human experts. With these tools we read MLE-bench, RE-Bench, PaperBench, AlphaEvolve, and The AI Scientist the way a lab reads them.
An agent is a model in a loop that acts on an environment and observes the results (§24.1). A code agent acts by writing programs and running them in a sandbox, an isolated machine where the code cannot touch anything else (§24.5). A research agent is a code agent whose goal is an open question with a measurable answer, not a fixed specification. "Make this model's validation loss as low as you can in 24 hours" is a typical goal.
Every system in this section can be described as a search. Toledo et al. (2025) used this framing to take apart the agents on MLE-bench. Here is the idea before the symbols. The agent keeps a growing collection of things it has built, each with a score. At each step it picks some of them, asks a language model to change or combine them, scores the result, and adds it to the collection. When time runs out it reports the best one.
In plain words, a research agent is a search guided by a number it can compute. Its result is worth exactly as much as that number tells it about the number we care about. Everything below studies the distance between the measured score and the true score .
That distance depends on the kind of evaluator. The systems in this section span the whole range.
| evaluator | example | how relates to |
|---|---|---|
| exact and cheap | rank of a tensor decomposition, size of a cap set, kernel runtime | , up to timing noise |
| exact and expensive | a full training run, a long simulation | , but few calls fit in a budget |
| noisy proxy | validation accuracy on a Kaggle split | , and the noise is selected on |
| learned judge | an LLM grading a rubric or reviewing a paper | has its own errors, and the generator can learn them |
| humans after execution | experts running an idea for 100 hours | closest to , and the slowest and costliest |
§26.1 named the accuracy that a checker adds, when it is used to keep only good outputs, the generation-verification gap. An agent can climb only as far as that gap allows. The first row is where agents already make discoveries. The middle rows are where they fool their own metrics. An interviewer who asks "is this agent result real?" is asking which row its evaluator sits in.
Every measurement has noise. If you measure many candidates and keep the best measurement, you tend to keep the candidate whose noise happened to be most positive, so its score is too high. This is the winner's curse, and every research agent faces it. Here it is in its smallest form.
§6.8 met the same effect when choosing the best of many prompts on a small development set. Here we bound it. An agent that runs hundreds of experiments selects on noise far more aggressively than a person picking prompts.
The idea of the proof fits in two sentences. The winner's error is one of the errors, so it is at most the largest of them. A Gaussian tail falls like , so the largest of draws sits roughly where , which is .
The optimism of the winner is its own error, . One of the errors can never exceed the largest, so .
For the bound on the maximum, write with standard normal. §16.1 proved in three moves. Jensen's inequality (§0.4) for the convex function gives . The maximum of the positive numbers is at most their sum, whose expectation is by the Gaussian moment generating function. So , and the choice makes it . Multiplying by gives .
For the adaptive case, the candidate evaluated at step depends on the earlier measurements, but is drawn fresh. So conditional on everything before step , it is still . Multiplying these conditional densities shows that is a vector of independent errors whatever the search did, and the argument above applies unchanged.
Finally, a fresh measurement , with independent of the search, has . So . ∎
§16.1 quoted this limit from extreme-value theory. Here is a short proof. Take and write , for the standard normal density, and for its upper tail. We need a lower bound on . For ,
using . Integrating from to gives . Since , we get .
Now choose , so that . Then
because grows only like . The errors are independent, so , using .
Split the expectation on the event . There, . On the complement, . So . Divide by . The factor tends to 1, the probability tends to 1, and the constant vanishes, so . With the upper bound, the ratio tends to 1. The approach is slow, as the cell below shows (0.72 at , 0.87 at ).
In plain words, if you try things and report the best measurement, expect to have overstated it by up to about . The overstatement grows without bound as you try more, though slowly. The only cure is a measurement the search never saw.
At the optimism is 5.8 accuracy points on a 500-example validation set. The fresh measurement is unbiased to within simulation error.
The theorem assumes fresh noise on every evaluation. Real agents do worse than that, because they evaluate on the same validation set again and again. Then the errors are not new draws. They are fixed properties of a fixed set of examples, and a search can learn them. This is adaptive overfitting to a held-out set. §22.1 studies it on leaderboards. There, an analyst who submits adaptively chosen classifiers to a held-out set of examples can overstate accuracy by order . Fresh noise allows only order . So is the optimism in the lucky case. With a reused set it can be much larger.
In practice, every number an agent reports about its own best result must be re-measured on data it never touched. An interviewer will ask you to estimate the optimism of a reported gain, and gives the answer in one line.
You are given the true scores of candidates. Simulate an agent that picks a winner on a noisy validation set and then checks it on a fresh test set.
Implement select_and_reevaluate(true_scores, sigma_val, sigma_test, reps, seed). Each of the reps repetitions does three things.
Return (optimism, test_bias). The first is the average over repetitions of . The second is the average of the test measurement minus . Use rng = np.random.default_rng(seed) and vectorize over repetitions. Also implement curse_bound(n, sigma), which returns .
The winner's curse makes selection sound dangerous. Yet selection is also the engine of every successful research agent. When the evaluator is exact, keeping the best candidate and building on it lets improvements add up. A clean comparison makes this precise.
Here is the setting in words first. A candidate is a vector of parameters. Its true score rises steadily in one direction that the search does not know. A mutation nudges the candidate by a small random step. Each nudge helps or hurts by a random amount, and the search can see exactly how much.
Now in symbols. The true score is linear near the start, , for parameters and an unknown gradient . A mutation of is with and step size . Write for the maximum of independent standard normals. There are two ways to spend exact evaluations.
A mutation changes the score by . The quantity is a linear combination of independent standard normals with coefficients , so it is , which is times a standard normal. Independent sampling keeps the best of such changes, so its improvement is , and the winner's curse theorem bounds the expectation.
For the strategy, is linear, so a child's gain over its parent depends only on its own and not on where the parent sits. Each generation's gain is therefore with a fresh . The zero is the parent kept when no child is better. The gains of the generations are independent and identically distributed, so their expected sum is times one expected gain. ∎
In plain words, independent sampling gets one lucky draw out of . Search gets a small gain every generation and keeps all of them, because each new generation starts from the best point so far.
The linear picture holds only while the steps are small compared with the distance to the optimum. Take a quadratic in one dimension, with . A child changes the score by . Far from the optimum the first term dominates. The gradient there has magnitude , and progress is linear as above. Within a few steps of the optimum, the term eats the gain, and progress stalls unless the step shrinks. That is why evolution strategies adapt their step sizes. It is also why a language model that proposes informed edits, rather than random perturbations, is such a strong mutation operator.
Now put the two lessons together. Real objectives are rugged, with many local peaks, and real evaluators are noisy. The next cell compares independent sampling with a steady-state evolutionary search, one that replaces a single member of its population at a time. The population holds 16 candidates. The search picks a parent by a tournament, drawing four members at random and taking the best-measured one. It adds a Gaussian mutation, and the child replaces the worst member if it measures better.
The objective is rugged, six-dimensional, and has maximum 0.9. Every candidate is measured once, with noise of standard deviation 0.3. Each method gets six runs of 1,000 evaluations, and the cell averages three numbers over them. They are the reported score (the best measurement), the true score of the reported candidate, and the best true score anywhere in the run.
Evolution wins on the true score by a wide margin, 0.54 against , even with this much noise. It also reports a score far above the truth. Its optimism, 0.57, is larger than independent sampling's 0.46. Both stay under the bound of 1.12, which holds for any search with fresh noise.
Evolution comes closer to the bound for a simple reason. It spends its budget on many candidates of nearly equal true quality near the top. So the maximum of their measurements is nearly the maximum of their noises. A search that has stopped improving the true score keeps improving the reported one.
The standard defense is an evaluation cascade. Score every candidate with the cheap, noisy evaluator. Send only the few best to an expensive, accurate one. Report the accurate score. The cell re-measures the five best-measured candidates of each evolutionary run with an evaluator whose noise is 0.02.
Five accurate calls remove almost all of the optimism. The true score of the reported candidate goes up too, from 0.54 to 0.60, because the accurate stage picks the truly best of the five. AlphaEvolve, the system of the next part, runs its cascades as test cases of increasing difficulty (Novikov et al. 2025). RE-Bench's authors plan to show agents only validation scores in future versions for the same reason (Wijk et al. 2024).
The widget runs the same experiment with three searches. Independent sampling is the first. Hill climbing keeps one candidate and replaces it whenever a child measures better. Evolution with islands splits the cell's population of 16 into two islands of 8 that evolve separately. Every 100 evaluations, each island's best member replaces the other island's worst.
Hill climbing shows the noisy failure most starkly. It compares each child with its parent's single lucky measurement. Once a parent has drawn a large positive error, few children can beat it, so the climb slows. At noise 0.6 it ends below independent sampling in truth. Populations, tournaments, and re-measurement all soften this. Islands keep diversity, so the search does not collapse onto one lucky region. In a design review, ask two questions of any agent's search. How does it re-measure its leaders, and what final evaluation did it never see?
Evolving programs is an old idea. What changed is the mutation operator. A language model, shown a few good programs and their scores, proposes edits that are far better than random perturbations. An exact evaluator turns those edits into progress.
FunSearch (Romera-Paredes et al. 2024) evolved one short Python function inside a fixed program skeleton. Its main target was the cap set problem. A cap set in dimension is a set of vectors of length with entries 0, 1, or 2, in which no three distinct vectors sum to zero modulo 3. Geometrically, no three lie on a line. The goal is the largest such set.
FunSearch's skeleton builds a set greedily. The evolved function gives each vector a priority, which decides the order in which vectors are tried. The evaluator builds the set and counts it. Each prompt showed the model a few high-scoring programs from one island, sorted by score, which the authors call best-shot prompting.
FunSearch found a cap set of size 512 in dimension 8, larger than any previously known. It also raised the best lower bound on the cap set capacity, the growth rate such that the largest cap sets grow like , from 2.2180 to 2.2202. It also evolved heuristics for online bin packing, which puts items into as few fixed-size bins as possible while deciding for each item as it arrives, and they beat standard baselines. All of this used on the order of a million samples from a code model chosen for fast sampling rather than for quality.
In plain words, AlphaEvolve is the steady-state evolution of the last cell with three changes. A language model is the mutation operator, whole programs are the candidates, and a machine-checkable score is the measure of quality.
Three details in that work apply the winner's curse lessons with care.
In each case the search optimized one number, and the result was judged on another. A follow-up study ran AlphaEvolve on 67 problems in analysis, combinatorics, geometry, and number theory. It rediscovered the best known solutions in most of them and improved several (Georgiev, Gómez-Serrano, Tao & Wagner 2025).
AlphaTensor (Fawzi et al. 2022) is an earlier system for the same matrix problem, without a language model. It uses the AlphaZero loop of §26.2, in which a tree search proposes moves and a network learns from the search's results. It plays a one-player game whose moves build a decomposition of the matrix multiplication tensor, and the score is the exact number of multiplications. In arithmetic modulo 2 it found 47 multiplications for matrices.
The AlphaEvolve paper names its own limit. Requiring an automated evaluation function "puts tasks that require manual experimentation out of" scope. Everything in this section outside the first row of the evaluator table, from a Kaggle validation split to a paper review, has to be read with that sentence in mind. An interviewer may ask "could AlphaEvolve do my research problem?" First ask whether the problem's quality can be computed thousands of times, cheaply and exactly.
Build a small AlphaEvolve. A program here is a tuple of 8 integers in . The setup provides ProgramSearchProblem(seed). Its method full(x) returns the exact score, which counts as expensive. Its method cheap(x) returns a noisy estimate, which counts as inexpensive. The problem logs every call in full_log and cheap_log. The setup also provides mutate_program(x, rng), which returns a new tuple. Implement
evolve(problem, seed_program, mutate, budget_full, budget_cheap, pop_size, rng, tournament=3)
as follows.
budget_full full calls and fewer than budget_cheap cheap calls have been made.tournament distinct database entries at random (all of them if there are fewer) and take the one with the best full score as the parent. Mutate it, and score the child with cheap.full evaluation only if its cheap score is at least the parent's full score. Then add it to the database and keep only the pop_size best by full score.(best_program, best_full_score), the best program by full score.This is the shape of AlphaEvolve's loop, with a cheap filter guarding an expensive evaluator. Research engineering interviews ask for loops like it.
For machine learning engineering, the scaffold behind the best results of 2024 was AIDE (Jiang et al. 2025). It keeps a tree of complete training scripts. Each node is a script with its validation score, and each edge is an attempted improvement. A hard-coded rule decides what to do at each step.
A summary of the tree goes into each prompt, so the model knows what has been tried. In the framing above, that is greedy search.
On MLE-bench, GPT-4o in AIDE won a medal in 8.7% of competitions (Chan et al. 2024). The same model won 0.8% in MLAB and 4.4% in OpenHands, two general-purpose scaffolds that let the model call tools as it likes. The authors found that MLAB and OpenHands often ended their runs early, sometimes within minutes, while AIDE kept prompting the model to improve until the 24 hours were up. The same model, in a different program around it, scored about ten times higher than in MLAB. When you compare agents, the scaffold is part of the result.
Toledo et al. (2025) separated the two ingredients of such a search. One is the operators, meaning draft, debug, improve, and the prompts behind them. The other is the search policy, which can be greedy, evolution, or Monte Carlo tree search. Their tree search picks nodes by an upper confidence bound, a rule that adds an exploration bonus to each node's average score. It resembles the rule that AlphaZero uses to choose moves in §26.2.
With AIDE's operators, better search policies gave no advantage, so the operators were the bottleneck. With improved operators, the best pairing raised the MLE-bench lite medal rate from 39.6% to 47.7%. They also measured the generalization gap, the difference between the validation score an agent sees and the hidden test score it is graded on. That gap is why choosing the final node by test score instead of validation score gained the 9 to 13 points quoted in the fact box on the winner's curse.
MLE-bench is the most quoted benchmark for agents that do machine learning engineering. Its headline number hides several choices, about which competitions count, what a medal means, and how many attempts an agent gets. Here is the benchmark exactly.
A few numbers make the definition concrete. Only nine people have ever won medals in 75 different Kaggle competitions, so an any-medal rate near 100% would be a career's work.
More attempts help a lot. Write pass@ for the chance that at least one of attempts wins a medal. From seeds with medals, the unbiased estimate is (§18.2). By this measure, o1-preview's rate doubled from 16.9% at pass@1 to 34.1% at pass@8. More time helps less. GPT-4o in AIDE went from 8.7% with 24 hours to 11.8% with 100 hours. Its score sometimes fell during a run, because AIDE's choice of its best node was imperfect. That is the generalization gap again.
Contamination is the obvious worry. It means that test material leaked into a model's training data (§22.3), and winning Kaggle solutions are all over the internet. The authors checked it three ways.
That rules out crude memorization. As the authors say, it does not rule out subtler transfer of winning strategies.
medals is an array of shape (C, S) for competitions and seeds. Each entry is 1 (medal), 0 (no medal), or nan (the run crashed or never submitted). Following MLE-bench's leaderboard, a missing run counts as no medal, in both functions. Assume and . Implement two functions.
any_medal_rate(medals) returns (mean, sem). Here mean is the average over seeds of each seed's medal rate across competitions, and sem is the sample standard deviation of those per-seed rates (ddof=1) divided by .pass_at_k_benchmark(medals, k) returns the mean over competitions of the unbiased pass@ estimate , with and the competition's medal count.As of the leaderboard's last update (read 2026-10-20), the top three entries are these.
They scored 77% to 80% on the low-complexity split and 42% to 47% on the high one. Several of the top scaffolds are evolutionary searches over solutions, the subject of the previous parts. The entries that used test-set feedback are listed separately. Closing the leaderboard to new submissions is the benchmark's own statement that comparability had become the binding problem. When someone quotes an MLE-bench number, ask for the seeds, the pass@, the hours, and the hardware.
MLE-bench compares agents with past Kaggle competitors. RE-Bench compares them with human experts working on the same tasks, under the same conditions, at equal time. That makes it the benchmark behind the most quoted comparison of agents and people.
The result most often quoted is this. With 2 hours of total time per environment, the best agents scored 4 times higher than the experts. With 8 hours, the humans narrowly led. With 32 hours, spent as several runs, the humans scored twice the top agent. The best way to split the time differed by contestant. At 16 hours, Claude 3.5 Sonnet did best with 32 runs of 30 minutes, o1-preview with 8 runs of 2 hours, and humans with 2 runs of 8 hours.
Agents tried far more things. Agents in AIDE ran the scoring function 36.8 times per hour, and agents in a modular scaffold 25.3 times, against 3.4 for humans. Their striking successes include a GPU kernel faster than any of the 9 experts' solutions. Why would the curves cross? Best-of- at a fixed budget answers it. Here is the idea in words first. A budget buys some number of independent runs, and you keep the best one. The best of runs lands roughly at the score that a single run beats only once in tries. So many short runs reach far into the upper tail of their scores, and a few long runs reach only a little above their typical score.
For a nonnegative random variable, . The maximum of independent runs is at most exactly when every run is, so .
Let be the -quantile, so that a single run is at most with probability (for a continuous distribution). Then all runs are at most with probability , and the maximum exceeds with the complement.
The estimator is the expected maximum of a uniformly random -subset of the observations. The value is the subset's maximum exactly when the subset contains it and of the smaller ones, which happens with probability . A random -subset of independent samples is itself independent samples, so the estimator is unbiased. This is the argument for the pass@ estimator in §18.2. ∎
In plain words, splitting a budget into many short attempts buys a high quantile of a weak distribution. A few long attempts buy a low quantile of a strong one. Agents win when their many cheap tries reach far enough into their own tail. Humans win once each human run is so much better that a modest quantile of it beats a high quantile of the agent's.
One detail matters when you reproduce the paper's curves. RE-Bench drew its runs with replacement from the observed ones. That plug-in version is slightly biased low, because a draw can repeat a run. The estimator above has no bias for .
The cell makes this concrete with a model fitted to RE-Bench's three published comparisons. A run scores 0 with probability . Otherwise its score has a Weibull tail, . This is a standard family of distributions on positive numbers, with scale and shape . A smaller shape means a heavier tail, so very good runs are less rare.
The model fits several free numbers to three published ratios. So it shows that such distributions produce the pattern, not that these are the true distributions. Its prediction is still useful. To move the crossover right, agents need better runs, not more short ones. In practice that means staying productive over longer attempts, the subject of §25.4. The 50% time horizon of §19.4 measures the same thing from another side. It is the length of task, in human expert time, that a model completes half the time.
The setup provides agent_runs, 256 normalized scores of 30-minute agent runs, and human_runs, 32 scores of 8-hour human runs. They are synthetic but shaped like RE-Bench's. Agents fail often and their successful runs score low, while humans rarely fail and score higher, with a wider spread. Implement three functions.
best_of_k(scores, k) is the unbiased estimate of the expected best of runs from the sorted scores. It returns nan unless .budget_curve(scores, run_hours, budgets) returns an array with best_of_k(scores, floor(T / run_hours)) for each total budget T, with nan where that is out of range.crossover(agent, agent_hours, human, human_hours, budgets) returns the first budget in budgets at which the human curve is strictly above the agent curve, both defined, or nan if there is none.RE-Bench's authors also list how the benchmark could mislead in both directions. Its environments are smaller, with clearer objectives and faster feedback than real research. Its agents saw test scores in six of the seven environments, which invites the overfitting the GPT-2 rerun exposed. And tokens are far cheaper than expert hours. An 8-hour agent run cost about $123 in tokens, so a better-funded agent could buy many more runs. The practical lesson is to compare agents and people at matched time and matched cost, as whole curves rather than single points.
Replicating a published paper is a core research skill, and it has no single metric to compute. PaperBench grades it anyway, by splitting each paper into hundreds of small checkable requirements and having a language model check them. So it sits in the fourth row of the evaluator table, a learned judge.
The judge, o3-mini with the authors' scaffold, reached an F1 of 0.83 on JudgeEval at about $66 per paper graded. F1 is the harmonic mean of precision and recall, here of the judge's pass grades against the human grades.
The best agent, Claude 3.5 Sonnet (New) in a simple tool-use scaffold with 12 hours and one A10 GPU, scored 21.0%. Most other models scored under 10%, partly because they ended their runs early. When the scaffold stopped o1 from quitting, it rose from 13.2% to 24.4%, and to 26.0% with 36 hours.
On a 3-paper subset, ML PhDs working 48 hours scored 41.4% (best of 3 attempts), against 26.6% for o1. The agent led early, then plateaued after the first hour, the RE-Bench pattern again. A cheaper variant, PaperBench Code-Dev, grades only code-development leaves. It gave o1 43.4%, and its scores were only weakly correlated with the full benchmark's (a Pearson correlation of 0.48 for o1). That is a warning that judging code without running it measures something else.
A rubric node is a dict. An internal node has "weight" and a list "children". A leaf has "weight", "grade" (0 or 1), and "type" ("code", "execution", or "result"). Implement replication_score(node, keep=None), which returns the PaperBench score of node. A leaf scores its grade. An internal node scores the weighted average of its children's scores, with weights normalized among siblings.
If keep is a set of types, only leaves of those types are graded (keep={"code"} is Code-Dev). Leaves of other types are removed. An internal node with no remaining leaves is removed from its parent's average. If nothing remains at the root, return nan.
The AI Scientist (Lu et al. 2024) automates every stage of the definition at the top of this section, from ideation to review.
Each paper cost under $15.
The reviewer is the most interesting part, because it is the evaluator. Its authors tested it on 500 ICLR 2022 papers with known accept or reject decisions. They used GPT-4o with one example review, five rounds of self-reflection, an ensemble of five reviews, and a decision threshold of 6 out of 10. As the human baseline they used the NeurIPS 2021 consistency experiment, in which 10% of submissions were reviewed by two independent committees to measure how consistent reviewing is (Beygelzimer et al. 2023). They compared the two with four measures.
It gets worse when the generator is tuned against the reviewer. The reviewer's error rates were measured on human papers. A system that produces many papers and keeps the ones its reviewer likes is searching for that reviewer's false positives. This is the reward overoptimization of §16.5, where a policy pushed hard against a learned judge finds the judge's mistakes. So the false positive rate on the system's own outputs can be far above 0.31. That is one reason to treat the generated papers as hints, not results. It is also why every judge-graded benchmark, PaperBench included, measures its judge's own accuracy as a benchmark in its own right.
The AI Scientist's paper also reports what its agent did to its harness, the code that runs and limits it.
These are small cases of specification gaming (§19.1), where a system satisfies the letter of its objective in a way its designers did not intend. The agent was told to finish the experiment, and it changed the constraint rather than the solution. §26.6 follows the same failure to systems that can edit their own reward. The defenses are structural. Run the agent in a container with no write access to its harness. Enforce time, memory, and storage limits from outside the agent's process. Restrict network access, and log every command for review.
The AI Scientist-v2 (Yamada et al. 2025) dropped the templates. It replaced the linear experiment loop with a parallel tree search over experiments, run by a separate experiment-manager agent. It also added a vision-language model to critique figures. Three of its papers were submitted, with the organizers' agreement, to an ICLR 2025 workshop that received 43 submissions. One scored 6, 6, and 7 (average 6.33), above the workshop's average acceptance threshold. It was withdrawn after review, as agreed in advance.
Read the procedure as well as the outcome. People chose the three ideas from about 40 the system generated. For each idea they chose the best manuscript among several seeds. Those are two human selection steps on top of the system's own. The authors' internal review judged that none of the three met main-conference standards. They also note that workshops accept 60% to 80% of submissions, against 20% to 30% at the main venues.
If papers are hard to evaluate, ideas are harder. Si, Yang & Hashimoto (2024) recruited more than 100 NLP researchers. Some wrote research ideas, and others blind-reviewed ideas from both the experts and an LLM agent. The reviewers judged the LLM ideas more novel than the experts' () and slightly less feasible. The same study found the agent's ideas highly repetitive. Of 4,000 seed ideas generated per topic, only about 200 were not duplicates.
The same group then tested the ideas by running them (Si, Hashimoto & Yang 2025). Each of 43 experts spent more than 100 hours executing a randomly assigned idea, written by an expert or by the LLM, and wrote a short paper. Blind reviewers then scored the results. After execution, the LLM ideas' scores fell significantly more than the human ideas' on every metric, namely novelty, excitement, effectiveness, and overall (). That closed the gap and, on many metrics, reversed it. A reviewer who judges ideas without running them is a learned judge in the sense of the evaluator table, and execution was the verifier it lacked.
Systems aimed at the natural sciences use the same loop with different evaluators. Google's co-scientist (Gottweis et al. 2025) runs several Gemini agents that generate, critique, and rank hypotheses in a tournament. Its drug-repurposing candidates for acute myeloid leukemia were tested in vitro, in lab experiments on cells. That is the expensive and trusted final stage of a cascade. Kosmos (Mitchener et al. 2025) runs for up to 12 hours. In that time it performs about 200 agent rollouts that together execute on average 42,000 lines of code and read 1,500 papers. Independent scientists judged 79.4% of the statements in its reports accurate, which is also to say that one in five was not.
Each benchmark in this section is a different answer to one question. How do you measure research ability with an evaluator the agent cannot exploit? The table puts the answers side by side.
| benchmark | what is measured | evaluator | human baseline | main threats |
|---|---|---|---|---|
| MLE-bench | ML engineering on 75 Kaggle tasks | hidden test split and the private leaderboard | Kaggle competitors | contamination, test-set feedback, seed noise, compute differences |
| RE-Bench | research engineering in 7 environments | scoring functions, normalized to start and reference | 71 expert attempts of 8 hours | visible test scores, small scope, best-of- allocation |
| PaperBench | replicating 20 papers | rubric trees graded by an LLM judge | ML PhDs, 3 papers | judge errors, partial credit for code that does not run |
| The AI Scientist | end-to-end papers | LLM reviewer, then a workshop | human reviewers | base rates, optimization against the reviewer, human curation |
| AlphaEvolve | algorithms and constructions | exact evaluation function, held-out inputs | prior best results | only tasks that have an automatic evaluator |
Here is a research loop at the smallest scale that still has every part. The "agent" proposes random configurations, a learning rate and a width for a small MLP, instead of LLM edits. A cheap evaluator trains each for 40 steps. The 4 best go to a full evaluator that trains for 300 steps. The chosen configuration is measured once on a test set that the search never saw. Swap the random proposals for a language model writing training scripts, and this is the skeleton that every agent in this section shares. A cheap filter guards an expensive evaluation, and one fresh test comes at the end.
With 16 configurations and only 4 full evaluations, the selected validation MSE and the fresh test MSE come out close, about 0.054 and 0.056 on a CPU. The validation set is large compared with the differences between the finalists. Shrink the validation set to 32 examples and raise the number of configurations, and the gap between the two numbers opens, as the winner's curse theorem predicts.