hardware-counters
rigour.md
Statistical protocol, precision and the learning curve
The protocol behind every comparison in ../README.md, modelling.md,
crossplatform.md and weakscaling.md, with the full tables. It is kept as a
separate document because each of the four checks below is worth reading end
to end, and because the checks generalise beyond this project: whether two
samples are actually paired, what happens to a fold with no test set, how a
paired verdict is labelled, and how many digits a random forest median is
worth.
Referenced outputs live in ../results/model/: stats_corrected.{txt,csv},
matched_crossplatform_{summary.txt,results.csv,stats.csv},
reconcile_rf.txt and learning_curve.{txt,csv}; the learning-curve figure is
../results/figures/fig_learning_curve.png.
Everything here was verified empirically by running the analyses on Cirrus (and, where figures needed reproducing exactly, on ARCHER2).
Throughout, the error metric is the symmetric factor
max(pred/actual, actual/pred), so 1.0 is perfect and 1.05 means “typically
within 5 per cent”.
1. The paired subset
GROMACS and OpenFOAM were never built or run on Cirrus. Cirrus additionally has
125- and 288-core points that a 128-way ARCHER2 node cannot reach. Joining on
(app, size, ncore, nthread) gives 166 configurations present on both
machines:
Both totals being 191 is a coincidence of how far each sweep reached, not a matching.
Why it matters
A comparison of the two full samples is an unpaired comparison of two differently-composed samples, so “platform” is confounded with “which configurations happen to be in this sample”. The confounding has a direction: the 25 Cirrus rows without an ARCHER2 counterpart are exactly its 125- and 288-core points, its widest and shortest runs, where fixed overhead dominates and the efficiency factor collapses. Including them makes Cirrus look intrinsically hard for a reason that has nothing to do with Zen 5.
analysis/matched_crossplatform.py therefore builds a genuinely paired
universe (166 ARCHER2 rows aligned row-for-row with 166 Cirrus rows, verified
by assertion) and reports it side by side with the 191-versus-191 version, so
the cost of the restriction is visible.
What the pairing changes
Once the workload is held fixed:
On the paired set, within-Cirrus RF (1.1248) and GBQ (1.1241) both beat the constant (1.1294) and both survive Holm-Bonferroni (adjusted p = 0.047 and 0.0056). On the two full samples they lose. “No model beats a constant on Cirrus, so Zen 5 is genuinely harder to model” is what the unpaired comparison says, and it is an artefact of sample composition; the statement the paired data support is “with the workload held fixed, tree models beat a constant on Cirrus too, but by a much smaller margin than on ARCHER2”.
The paired descriptive statistics weaken the “Cirrus is harder” story without eliminating it: the log-eta spread ratio is 1.77x (0.2377 versus 0.1342) on the paired data against 2.22x (0.2844 versus 0.1280) on the full samples.
The transfer results (both directions) move by less than 0.02 and their qualitative verdicts are the same either way.
Wording
ARCHER2 contributed 191 configurations across eight applications and Cirrus 191 across six; GROMACS and OpenFOAM could not be built on Cirrus, and Cirrus alone reaches 125 and 288 cores. 166 configurations are present on both machines and form the paired subset used for all platform comparisons.
2. Folds with no test set
The A2->CIR LOAO and CIR->A2 LOAO settings loop over the union of
applications. For gromacs and openfoam the Cirrus test mask is empty. A
fold in that position must not be skipped silently, because three things go
wrong at once when it is: the reported n still reads 191 if it counts the
mask size rather than the number of finite predictions; the constant reference
for the setting is never computed, so a verdict “loses to constant” is a
verdict against a NaN; and the hardest, most interesting setting in the whole
matrix, unseen hardware and unseen application simultaneously, is evaluated on
the six proxy benchmarks only, with the two production codes (a molecular
dynamics code and a CFD code, the two least like the proxies) dropped without a
word.
Every fold is therefore listed with its disposition. From
../results/model/matched_crossplatform_summary.txt:
A2->CIR LOAO, unmatched (191 v 191):
app n_test n_train disposition
comd 28 166 evaluated
gromacs 0 183 NOT EVALUABLE: application absent on the TARGET platform
hpcg 51 146 evaluated
hpl 27 159 evaluated
lulesh 16 178 evaluated
minife 42 155 evaluated
openfoam 0 183 NOT EVALUABLE: application absent on the TARGET platform
stream 27 167 evaluated
-> 6 of 8 folds evaluated, 2 not evaluableThe disposition is not evaluable, application absent on the target
platform. The CIR->A2 LOAO direction is different and is stated as such:
there GROMACS and OpenFOAM do have a non-empty ARCHER2 test set, so all eight
folds are scored, but no Cirrus training rows for those codes exist either, so
the fold is really “predict a code the source platform has never seen in any
form”. That is a legitimate and harder test.
The constant reference is computed on the same folds as the models, so the comparison is like for like and no NaN appears.
Numbers for these settings, paired set, six evaluable folds
Wherever 1.254 is quoted for “ARCHER2 to Cirrus, unseen application too”, it is stated that it is an average over six applications, not eight.
3. Reporting a paired comparison
Three things a paired model-versus-baseline comparison has to get right, and one routine that does all three.
Direction from the right quantity. A paired Wilcoxon signed-rank test is a test about the distribution of the paired differences, and median(a) - median(b) is not median(a - b). On right-skewed error factors the two disagree: a model can shift most pairs down while a few large-error pairs hold its own marginal median up. A model that wins 118 of 191 paired comparisons at p = 0.0004 is better, whatever its marginal median says, and a model at p = 0.31 is not “better” whatever its marginal median says.
Multiple-comparison control. Around twenty model-versus-baseline comparisons at a nominal alpha of 0.05 with no correction put the family-wise error rate at of order 1 - 0.95^20 = 64 per cent.
Uncertainty and effect size. A median quoted to four decimal places with no interval, and “p = 0.0002” attached to a median improvement of about 0.011x that is of little engineering consequence, are both misleading on their own.
analysis/stats_rigour.py provides one routine used for every comparison in
the project:
- Direction from the sign of the Hodges-Lehmann pseudomedian of the paired differences, the location estimate the Wilcoxon signed-rank test actually targets. The plain median of the differences is reported alongside; the two agree everywhere here.
- Bootstrap 95 per cent confidence intervals (percentile, 10,000 resamples) on every arm’s median error factor, and on the paired difference. The Hodges-Lehmann interval uses 2,000 resamples because it is O(n^2) per resample.
- Holm-Bonferroni step-down correction across each declared family, with monotonicity enforced. Holm is uniformly more powerful than plain Bonferroni and valid under arbitrary dependence, which matters because these comparisons share a baseline and are strongly dependent.
- Effect size: matched-pairs rank-biserial correlation, the effect size that belongs with the signed-rank test, plus explicit win/loss pair counts.
Only one significant result is lost to Holm-Bonferroni
Of 28 comparisons, 20 are significant at raw alpha = 0.05 and 19 survive Holm-Bonferroni. The single casualty is:
Cirrus fixed-size LOAO, GBM quantile(0.5) versus constant: raw p = 0.0279, Holm-adjusted p = 0.0558. Not significant after correction.
The substantive findings are not artefacts of multiple testing. So few are lost because the surviving p-values are mostly very small (1e-4 and below), which the correction cannot touch.
Headline table
ARCHER2 LOAO, intersection features, versus the median-constant baseline (constant 1.0654 [1.052, 1.081]); family of 3:
The MLP row is the case the first requirement above is about: its marginal median is above the constant’s, and on the paired differences it is significantly better.
Cirrus rescaled LOAO (constant 1.0617 [1.056, 1.068]): no model beats the constant. RF 1.0603, Holm p = 0.37; GBQ 1.0565, Holm p = 0.37; MLP 1.0807, Holm p = 0.37. “Cirrus went from 1.186 to 1.057, essentially matching ARCHER2’s 1.055” is true as a statement about the number and misleading as a statement about modelling: the constant baseline improved just as much, so the model adds nothing there.
Counter-group ablation on ARCHER2 (RF), versus config-only (1.1040); family of 3. All three survive Holm:
Versus the no-feature constant (1.0654); family of 4. Only the full counter set survives:
This is the sharpest version of the counter-value result and it is worth stating plainly: the full five-counter-set profiling campaign is the only feature configuration that beats a zero-parameter constant, and it does so by 1.5 per cent in median error factor with a large effect size (r_rb = -0.35). Everything cheaper is statistically indistinguishable from, or worse than, guessing the training median.
Cross-platform transfer versus constant, intersection features; family of 12. Nine of twelve survive Holm. Notably:
The asymmetry is real and quantified: transferring from Cirrus fixed-size to ARCHER2 is significantly worse than a constant, whereas transferring from Cirrus rescaled to ARCHER2 is significantly better. The problem is not the hardware, it is that the fixed-size Cirrus runs are too short to carry signal.
4. How many digits a random forest median is worth
The same nominal experiment, “Random Forest, leave-one-application-out, ARCHER2, 191 merged configurations, median error factor”, is run by three scripts, and they do not agree past the third decimal:
Diagnosis
Ruled out first: the data (runs_expanded.csv, md5 befa6840..., identical on
both machines), the merge, the 21 feature values, the 8 LOAO folds and the
estimator (RandomForestRegressor(n_estimators=400, min_samples_leaf=2, random_state=0)) are byte-identical across the three scripts, because
advanced_nn.py and ablation2.py both do
from retrain2 import load_merged, build_features, fac. Thread
non-determinism was also ruled out: the forest is bit-reproducible and gives the
same answer for n_jobs in {1, 2, 8, -1}.
Two incidental implementation differences account for everything.
C1, feature column order (the dominant term). retrain2.py and
advanced_nn.py pass build_features(df) straight through, so columns are in
definition order with log_instr_per_rank and log_t_analytic last.
ablation2.py rebuilds the matrix as Xall[cols] with cols assembled from its
GROUPS dict in cumulative measurement-cost order, which moves log_t_analytic
to position 4 and log_instr_per_rank to position 6. Same 21 columns, same
values, permuted positions. Because max_features="sqrt" draws candidate
feature subsets from a PRNG stream indexed by column position, permuting the
columns builds genuinely different trees even at random_state=0.
C2, preprocessing chain. Each script implements its own scaler:
A forest splits on thresholds, so it is invariant to any strictly increasing
per-feature map in exact arithmetic, and indeed the sign-log and plain
StandardScaler chains give bit-identical results. RobustSmoothScaler does not,
because 3·tanh(z/3) saturates: it maps abs(z) > about 12 into a band
narrower than 1e-5 around plus or minus 3, so values that were distinct collapse
onto the same float64 and the split that separated them disappears.
The reproduction grid (ARCHER2, scikit-learn 1.7.2, where the summaries were generated)
Each script’s figure is reproduced exactly, to six decimal places on the
median, to four significant figures on the p-value and exactly on the win
count: retrain2.py is definition order with the sign-log chain,
advanced_nn.py is definition order with RobustSmoothScaler, and
ablation2.py is its own group order with a plain StandardScaler. No
residual. C1 explains 1.0545 to 1.0554; C2 explains 1.0545 to 1.0548.
The noise floor is larger than the discrepancy
random_state=0 is arbitrary. Sweeping it over ten seeds on the canonical chain
gives 1.0554, 1.0592, 1.0643, 1.0553, 1.0497, 1.0655, 1.0531, 1.0540, 1.0580,
1.0580: mean 1.0572, standard deviation 0.0049, spread 0.0158. The
implementation spread across the whole eight-cell grid is 0.0030. The seed
spread is five times larger, and every one of the three figures lies inside it.
A third source: scikit-learn version. The same code gives 1.054497 on ARCHER2 (1.7.2) and 1.055414 on Cirrus (1.9.0), because tree-builder tie-breaking changed between releases.
Canonical value
Random Forest, leave-one-application-out, ARCHER2, 191 merged configurations, full 21-feature set: median error factor 1.05. Seed-averaged point estimate 1.0572 over 10 seeds (sd 0.0049). Bootstrap 95 per cent CI on the median: [1.044, 1.072].
The canonical pipeline is retrain2.py’s (median imputation, sign-log,
StandardScaler) applied to build_features output in definition order, because
that is the chain retrain2.py documents and that the other scripts follow.
Report 1.05 with the interval. The third and fourth decimals are seed, column order and library version. No conclusion changes either way, since every variant beats the constant with 123 to 126 wins of 191 at raw p of about 2e-4.
Rules that follow
- Scripts that share an experiment import the pipeline constructor from
retrain2.pyrather than re-implement it. - An ablation masks features rather than reordering them, or indexes with a
column list built in
build_featuresorder. - Every reported forest median is seed-averaged over at least ten seeds and quoted to two decimals with an interval.
- The scikit-learn version is pinned in the environment specification.
Learning curve: is 191 configurations enough?
Protocol
The outer loop is unchanged leave-one-application-out on ARCHER2. Inside each fold the training set is subsampled to 25/50/75/100 per cent before fitting. The test fold is never subsampled, so every point on the curve is scored on the same 191 configurations and the points are directly comparable. Twenty random draws per point; median with a percentile bootstrap 95 per cent CI.
Two subsampling schemes are run, because they answer different questions:
- random: subsample training rows. Keeps all seven training applications present, so it measures density of coverage. “Would more runs of the codes I already have help?”
- by-application: subsample training applications (7, 5, 4, 2 codes at 100/75/50/25 per cent) and keep all their rows. Measures coverage of the application space. “Would more codes help?” This is the question that actually governs leave-one-application-out generalisation.
Results
Figure: ../results/figures/fig_learning_curve.png.
Interpretation
On the density axis the dataset is saturated. Under random row subsampling the curve flattens: the 75-to-100 per cent step buys 0.0037 for RF (and is slightly negative for GBQ), and the 75 and 100 per cent confidence intervals overlap. Collecting more configurations of the same eight codes would buy essentially nothing.
On the coverage axis it is badly under-sampled. Under application subsampling the curve is still clearly descending at the right-hand edge: the final step from five to seven training codes is worth 0.0223 for RF, six times the corresponding density step, and the 75 and 100 per cent confidence intervals are disjoint. There is no sign of a plateau.
Answer to “would more data help?”: more runs of the same codes, no; more codes, yes, and substantially. With only eight applications, LOAO has eight folds and every fold trains on seven codes, which sits on the steep part of the coverage curve. This also explains why the LOAO estimate is unstable enough that a random forest median moves in the third decimal between seeds (section 4 above): the quantity being estimated is dominated by which seven codes happen to be in the training set.
Design implication. The profiling budget was spent widening the configuration sweep within each application. The learning curve says it should have gone on more applications with a coarser sweep each. The 25 per cent by-application point (two training codes, mean 47 training rows) is already 1.1333, worse than the constant baseline at that size (1.0881), which is a direct measure of how much the models depend on application diversity rather than sample size.
This also reframes the cross-platform result. Cirrus has six applications, not eight, so every Cirrus and transfer number in the project sits further left on the coverage curve than the ARCHER2 numbers do. Part of the gap between within-ARCHER2 (1.05) and within-Cirrus (1.12 paired) is a training-set coverage effect, not a hardware effect.
Summary
The most consequential of these is the pairing: an unpaired comparison of the two full samples is confounded with sample composition and would support a conclusion about Zen 5 that the paired data do not.