less noise, more signal

Tim Menzies timm@ieee.org · timm.fyi · 2026-07-05 · paper (IEEE TSE, to appear)

Good news: our paper “Less Noise, More Signal: the DRR Effect for Better Optimizations of a Range of SE Tasks” (Andre Lustosa & Tim Menzies) was just accepted to IEEE Transactions on Software Engineering. Here is the story, minus the greek.

the one-line version

Before you buy an expensive optimizer, measure your data. If

DRR = 1 - I/R > 0.35

then a 30-sample optimizer will very probably do as well as a state-of-the-art one running 3000 evaluations — two orders of magnitude faster (seconds, not 20+ minutes).

Here R is the number of raw attributes in your data and I is its intrinsic dimensionality (how many underlying dimensions actually matter, computed via a quick fractal-based measure). DRR — the Dimensionality Reduction Ratio — is just the fraction of your attributes that are noise.

why this works

SE data is famously compressible. Software projects report many things, but only a few of them relate to any particular goal. When most attributes are superfluous, the optimization landscape collapses to a few buckets, and elaborate search (evolution, multi-fidelity, iterated racing) is effort spent exploring dimensions that are not there.

We tested this on 50 datasets — software configuration, cloud resource management, project health prediction, effort/defect/schedule process models — tuning a random-forest learner with two optimizers:

LITE: ~30 lines of code, 30 evaluations;
DEHB: state-of-the-art hyperparameter optimization, 3000 evaluations.

38 of 50 SE datasets (76%) had DRR > 0.35. On all of those, LITE was statistically indistinguishable from DEHB (Scott-Knott + Cliff’s delta, 20 repeats), while running about 100 times faster. On low-DRR data — which includes most classic non-SE machine learning benchmarks — the extra 3000 evaluations really do pay off. Which is maybe why AI researchers never noticed this effect: they were looking at different data.

an effect, not a law

DRR is an empirical effect in the tradition of the Pareto principle or Brooks’ Law: a frequently recurring pattern with known exceptions, not a universal truth. Near the threshold there is noise (a couple of DRR ≈ 0.3 datasets were easy anyway). It also corrects prior work: an earlier TSE paper proposed “use simple methods when I < 4”, a rule that mis-advises most of our 50 datasets.

So the recommendation is a cheap diagnostic, not a dogma: check DRR before deploying expensive optimization. A full reproduction package (all 50 datasets, LITE, DEHB, the intrinsic-dimension calculator) is at github.com/timm/ezr.

Congrats to Andre Lustosa (IRL PhD 2025, now Redhat) — this one was years in the making.