hardware-counters

Problem size, not hardware: the rescaled Cirrus campaign

The cross-platform experiment (crossplatform.md) left one question open. Cirrus looked harder to model than ARCHER2, but problem sizes there had been held identical to ARCHER2’s for comparability, which on a 288-core Zen 5 node made many runs sub-second. Was Cirrus intrinsically harder to model, or was that an artefact of the design choice?

Answer: largely the design choice. Rescaling so that work grows with rank count lifts the runs off the sub-second floor and makes the target far narrower, which makes the task easier for every method including the zero-parameter baseline. Two things should be kept apart when reading the results below. The first is that the platform comparison itself has to be made on the 166 configurations that exist on both machines (crossplatform.md): on that paired subset, at the original fixed problem sizes, within-Cirrus RF reaches 1.1248 and GBQ 1.1241 against a constant of 1.1294, both beating the constant and both surviving Holm (adjusted p = 0.047 and 0.0056). The second is the problem-size effect documented here, which is independently measured: rescaling cuts sub-second runs from 47.6% to 16.7% and the spread of the target by a factor of four. This experiment shows how much of the apparent platform difference is attributable to the runtime distribution the design produced.

What the rescaling is, and is not

This sweep carries the word “weak” where it names a file or a directory (cirrus_weak.csv, weak_*.sh), but it is not weak scaling in the strict sense. Under true weak scaling, work per rank is constant and runtime is flat in core count. Measured ratio of runtime at maximum cores to runtime at one core, at the largest per-rank work unit:

Only CoMD and HPCG approximate weak scaling. The HPL rule scales the matrix order as the square root of the rank count, which holds memory per rank constant at 0.13 GB, but HPL’s work is O(N^3) against memory O(N^2), so compute per rank grows as the square root of the rank count. The accurate description of this sweep is therefore “rescaled so that total work grows with rank count”, and that is the wording used throughout. It achieves its purpose, which is to lift runs off the sub-second floor where fixed overheads dominate; the mechanism is “bigger problems” rather than “constant work per rank”, and a true weak-scaling study would need the HPL rule changed to N ~ base * n^(1/3).

Why fixed sizes fail on a wider node

Only some benchmarks are affected, and for an identifiable reason: whether their size knob is global or per rank.

HPCG is never a problem because nx/ny/nz in hpcg.dat are per-rank, so its total work already grows with the job. The rest take global sizes, so a fixed size means strong scaling and collapsing runtime. Below ~1 s the fixed overheads — MPI_Init, I/O, CrayPat instrumentation — dominate wall time, and the efficiency factor η stops reflecting anything the counters measure.

The rescaling rule

scripts/weak_sizes.py scales each benchmark by the exponent appropriate to its knob, so that total work grows with rank count:

HPL at base 4000 holds 0.13 GB/rank from 1 to 288 cores (36.7 GB total at full node) — comfortably inside the 768 GB available.

Validated at 288 cores before committing to the sweep:

165 jobs, three per-rank work levels per application so problem size remains a genuine independent variable. All completed, zero failures.

Result

Rescaling cut sub-second runs from 47.6% to 16.7% and reduced the spread of the target by 4× (0.285 → 0.072). Median η rose to 0.946, i.e. runs now spend most of their wall time doing work rather than paying overheads.

Leave-one-application-out, within each dataset:

Two things should be read from this table, and neither is “the model got better”.

First, the constant baseline improves by almost exactly as much as the best model does: 1.166 → 1.062 for the constant against 1.186 → 1.057 for the model, a difference of 0.005 in a quantity whose seed-to-seed spread is three times that. On the rescaled data no model beats the constant: RF, GBQ and MLP all come back at Holm-adjusted p = 0.37. “Cirrus went from 1.186 to 1.057, essentially matching ARCHER2” is true about the number and says nothing about the modelling. Rescaling made the task easier for every method including the zero-parameter baseline; what changed is that models stopped being actively worse than a constant. Rescaling improved the data, not the models.

Second, the fixed-size Cirrus verdicts come from the paired test, not from a comparison of two marginal medians. On the paired differences, RF on fixed-size Cirrus is significantly better than its constant (Hodges-Lehmann -0.061, Holm p = 0.040), and GBQ is better at raw alpha but does not survive Holm (raw p = 0.0279, Holm 0.0558, the one result in the project lost to multiple-comparison control). See modelling.md, Statistical methodology and the precision of the numbers.

Figure: ../results/figures/fig_weakscaling.png shows the runtime distributions, the log-η distributions and the model-versus-constant comparison for the three datasets side by side.

One further caveat. The full feature set is worse than the intersection on rescaled Cirrus (1.104 versus 1.057). With a narrower target, the extra Zen 5-only features (L1_DCM, VEC_INS, FMA_INS) add variance without signal.

Cross-platform transfer improves too

This is where rescaling does change a modelling verdict, and it is the strongest evidence in the project that the source dataset’s quality rather than its size governs transfer. Training on fixed-size Cirrus and predicting ARCHER2 is significantly worse than a constant (RF Holm p = 0.0020, MLP p = 0.033). Training on rescaled Cirrus and predicting ARCHER2 is significantly better, in all three families and by a wide margin (RF Holm p = 5.5e-09, GBQ 1.2e-09, MLP 2.8e-12, rank-biserial -0.52 to -0.61). The sub-second-dominated fixed-size runs teach the model the wrong η distribution; the rescaled runs do not.

Transfer error remains around 1.14 to 1.18 against about 1.05 within-platform, so unseen hardware still roughly triples the error gap, and that conclusion is unaffected. Figure: ../results/figures/fig_crossplatform.png.

What this means

The headline lesson is methodological, and it is the kind of thing that is easy to get wrong silently:

Comparability and predictability are in tension. Running identical problem sizes across machines is the natural way to make a fair comparison, but it systematically handicaps the faster machine by driving its runs into the fixed-overhead regime, where the efficiency factor stops measuring anything the counters can see. Here it inflated the apparent difficulty of the Cirrus target by a factor of four in the spread of log-η.

There is a second, larger cause of an apparent platform difference that this experiment cannot see, because it is not about problem size at all: comparing two samples that are not the same workload. crossplatform.md handles that by restricting every platform comparison to the paired configurations. The two causes are worth keeping separate. Unmatched samples bias the comparison; small problem sizes degrade the data. Only the second is fixed by rescaling, and rescaling fixes it for the constant baseline just as much as for the models.

For anyone building a cross-platform performance dataset the practical guidance is therefore threefold: verify that the configurations really do match before attributing any difference to the platform, check the runtime distribution per platform before drawing conclusions about model quality, and be explicit about whether the design holds problem size or work per rank constant, because the two answer different questions.

Reproducing

python scripts/weak_sizes.py APP NRANK BASE     # the scaling rule
bash   scripts/weak_worklist.sh                 # generate the worklist
bash   scripts/weak_drip.sh                     # submit within the QOS cap
python analysis/parse_weak.py                   # -> data/cirrus_weak.csv
python analysis/weak_analysis.py                # the comparison above
python analysis/size_diag.py                    # the per-app diagnosis