better together, in the right order

Tim Menzies timm@ieee.org · timm.fyi · 2026-07-14 · paper (arXiv preprint)

New preprint: “Better Together, in the Right Order: Classical-then-LLM Optimization for SE” (Srinath Srinivasan & Tim Menzies). Everyone is bolting LLMs onto optimizers. Prior hybrids put the classical optimizer in charge and asked the LLM for hints. We tried the opposite: let a cheap classical learner go first, then hand its trail to the LLM to finish. Same ingredients, opposite order, eleven points better.

the one-line version

SNAP2 runs the EZR active learner for the first 10 of a 20-label budget, then feeds that labeled trajectory as generation zero to an OPRO-style LLM loop. Result: statistically top-tier on 85% of 105 SE optimization tasks — versus 74–75% for every LLM-first or LLM-only rival — while using ~30% fewer tokens and half the dollar cost of the pure-LLM approach.

why order matters

An LLM prompted cold has to guess the shape of the landscape from generic priors. Seed it with even ten well-chosen labeled examples and it starts in the right region, with real gradients to reason over. The classical stage is the cheap scout; the LLM is the expensive closer. Collision tracking (nudging proposals away from already-explored regions) keeps it from re-asking questions the scout already answered.

the uncomfortable baseline

Honesty compels a footnote that is really a headline: plain EZR — no LLM, zero tokens, zero dollars, three orders of magnitude faster — landed within 2.8 points of SNAP2. So the pragmatic advice is a two-step: run the free thing first; pay for the LLM only when the last few points of optimization matter. (The whole study, including all seven methods over 105 tasks and 20 seeds, was budgeted at $400 — on the principle that an LLM budget should stay below the cost of a graduate research assistant.)

details

Benchmarked: SNAP2, SNAP, BS_LLM, SYNTHCORE, EZR, random, and row-ranking baselines on 105 tasks from the MOOT repository (config tuning, process models, effort/defect/health prediction), scored by distance-to-heaven with Cliff’s delta + Kolmogorov-Smirnov at 95%. LLM: gpt-oss-120b.