your model is unstable (and that’s fixable)

Tim Menzies timm@ieee.org · timm.fyi · 2026-07-14 · paper (arXiv preprint)

New preprint: “Is Model Instability just Noise to be Tolerated or a Property that can be Managed?” (Amirali Rayegan, Lunxiao Li & Tim Menzies). Here is a number that should worry anyone who runs an optimizer once and ships the answer: under default settings, two runs of the same state-of-the-art optimizer on the same data agreed on just 2.9% of 12,700 test cases.

the one-line version

Performance instability is the norm, not the exception — but it is not just noise. It is a measurable, manageable property. A handful of configuration changes raised run-to-run agreement 4.8x and cut the variance of optimization error by 22%, while improving optimization quality.

two kinds of instability

The study (127 multi-objective SE tasks, 20 repeated runs per treatment) separates two things people usually conflate:

Structural instability: the learned trees look different from run to run. This is the Rashomon effect and it is largely irreducible — and harmless. Different-looking trees can give the same advice.
Performance instability: different runs give different recommendations. This one matters, and this one is fixable.

what fixes it

Six candidate causes were tested (budget, acquisition strategy, tree complexity, split criterion, confounder filtering, data locality). The winning recipe: labeling budget 50, “near” acquisition, min-leaf 3, Gini splits. The acquisition function alone lifted stability top-rank rates from under 50% to over 80% of datasets — at zero extra labeling cost. And stability did not tax quality: the refined settings were statistically top-ranked for optimization performance on 119 of 127 datasets, versus 74 for the defaults.

a floor you cannot dig below

Some instability is the data’s fault, not the learner’s. Even with causal confounder-filtering and the friendliest clusterers, agreement never passed ~51% of cases. So the honest position is: instability can be managed down to a data-inherent floor, and past that you should report it, not hide it. We argue stability should be a routinely-reported evaluation axis in SBSE, right next to accuracy. Reproduction package: tinyurl.com/Model-Instability.