which optimizer, at what budget?

Tim Menzies timm@ieee.org · timm.fyi · 2026-07-14 · paper (arXiv preprint)

New preprint: “Which Optimizer, At What Budget? A Tournament of Optimizers for Search-Based SE” (Kishan Kumar Ganguly & Tim Menzies). We ran 20 black-box optimizers head-to-head on 106 SE tasks — about 180,000 search runs, 14,000 CPU hours — and the short answer is: there is no best optimizer. But there is a best optimizer for your budget, and you can look it up for free.

the one-line version

At tight labeling budgets (30–50 samples) a tiny geometric active learner (EZR) wins; at larger budgets (100–200) differential evolution takes over. Which one you need is predicted by just two zero-cost task attributes plus the budget — a six-cell table lookup that ties or beats a hindsight oracle on 74% of held-out tasks.

why a tournament?

Every optimizer bakes in assumptions about the fitness landscape: local continuity, building-block decomposability, Pareto incomparability, low intrinsic dimensionality, and so on. We went back to the originating paper of each optimizer family, mapped each to one of seven such assumptions, then built a binary tournament tree where each match isolates one assumption. Quality was scored as distance-to-heaven against a random-search floor, with Scott-Knott + Cliff’s delta over 20 seeds.

what we found

Optimization pays, fast: 30 labels close 31% of the random-to-optimal gap; 200 labels close 74.5%.
The winner migrates: 58% of tasks change their best optimizer at least once as the budget grows. Pick an optimizer at one budget and it falls out of the top tier on up to half the tasks at another.
Popular ≠ efficient: the widely recommended NSGA-II needed ~1000 evaluations to match what EZR achieved in 200 — and no optimizer family led on more than 37 of the 106 tasks.
Cheap beats clever: expensive instance-space clustering predicted the right optimizer 44% of the time; our free two-attribute lookup (objective conflict × input-space shape × budget) scored 74%.

so what?

Budget is a first-class variable. The open question in SE optimization is never whether to optimize, only which optimizer earns those few dozen labels. In practice this suggests a “warm handoff”: start geometric and frugal, switch to evolutionary search only if the labels keep coming. Reproduction package: github.com/KKGanguly/OptimizerTournament.