hardware-counters

Figure captions

Figures for the dissertation. fig_ladder.png and fig_perapp.png come from analysis/make_simple_figs.py; fig_weakscaling.png and fig_importance_artefact.png from analysis/make_clear_figs.py; fig_learning_curve.png from analysis/learning_curve.py. The rest predate the current plotting scripts, as ../../REPRODUCING.md explains. Palette: Okabe-Ito colourblind-safe. All PNGs at 170 dpi.

Throughout, the accuracy metric is the symmetric error factor max(t_pred/t_meas, t_meas/t_pred), which is 1.0 for a perfect prediction and is reported as a median over configurations. The zero-parameter reference is the constant-eta baseline: predict t = cycles / f_peak / eta_median, where eta_median is the median efficiency factor of the training applications. It is a genuinely strong baseline, not a strawman.

Note on precision. Medians in these captions are quoted to three decimals because that is what the plotting code emits. The seed-to-seed spread of a random forest median under this protocol is 0.0158, so only two decimals are meaningful and the prose in docs/ quotes two. See docs/modelling.md, Statistical methodology and the precision of the numbers.

fig_ablation.png

Figure X. Hardware counters, not job configuration, carry the predictive signal. Median leave-one-application-out prediction error on ARCHER2 (191 merged configurations, 8 applications) as feature groups are added cumulatively to a random forest. Bars show the median error factor; whiskers show the interquartile range across configurations; the dashed line is the zero-parameter constant-eta baseline (1.065). Configuration alone (core, rank and thread counts) predicts worse than the constant baseline (1.104, and significantly so, Holm-adjusted p = 0.0044), and adding the analytic cycle-count term does not rescue it (1.109). Only when instruction-mix ratios and then memory and dispatch-stall counters are supplied does the model overtake the baseline, reaching 1.057 with all 14 intersection features (1.05 with the full 21). The improvement over configuration-only is highly significant (Wilcoxon signed-rank p = 6e-16; 148 of 191 configurations improve), and the full set is the only rung on the ladder that beats the constant (Holm p = 1.4e-04). The reader should conclude that the measurable benefit of a learned performance model comes from microarchitectural counters, that a model given only job parameters is not worth building, and that nothing cheaper than the full counter-set rotation is worth building either.

fig_learning_curve.png

Figure X. The dataset is saturated in configurations and under-sampled in applications. Median leave-one-application-out error on ARCHER2 as the training set is subsampled to 25, 50, 75 and 100 per cent; the test fold is never subsampled, so every point is scored on the same 191 configurations and the points are directly comparable. Twenty random draws per point; shading is a percentile bootstrap 95 per cent confidence interval. Two schemes are shown. Under random row subsampling (density: more runs of the same eight codes) the random forest goes 1.087 to 1.059 and flattens, with the final quarter worth 0.0037 and the 75 and 100 per cent intervals overlapping. Under by-application subsampling (coverage: more codes, 2, 4, 5 and 7 training applications) it goes 1.133 to 1.056 and is still clearly descending at the right-hand edge, with the final step from five codes to seven worth 0.0223, six times the corresponding density step, and the 75 and 100 per cent intervals disjoint. At two training codes the model (1.133) is worse than the constant baseline at the same size (1.088). The reader should conclude that more runs of the same applications would buy essentially nothing, that more applications would buy a great deal, and therefore that a profiling budget for this kind of study should be spent on breadth of workload rather than on density of configuration sweep.

fig_weakscaling.png

Figure X. Rescaling the problem lifts every method, including the zero-parameter baseline: the Cirrus runtime distribution, not the microarchitecture, was the problem. Three datasets under one protocol: ARCHER2 at its native problem sizes (191 configurations), Cirrus run at identical sizes for comparability (191), and Cirrus rescaled so that work grows with rank count (162). (a) Runtime distributions. Holding sizes fixed while moving to a 288-core Zen 5 node pushes 47.6% of runs below one second, against 24.6% on ARCHER2; rescaling brings this down to 16.7%. (b) Distribution of log10 eta, the quantity a model must predict. The standard deviation collapses from 0.284 (Cirrus, fixed sizes) to 0.072 (Cirrus, rescaled), a fourfold reduction, and the long low-eta tail produced by sub-second runs disappears. (c) Best counter-based model against the constant baseline within each dataset. The bars to read here are the gaps, not the heights: rescaling moves the Cirrus model from 1.179 to 1.056 but moves its constant baseline from 1.166 to 1.062, almost exactly as far, and on the rescaled data no model beats the constant (Holm p = 0.37 for all three families). The reader should conclude that comparability and predictability trade off, that running identical problem sizes across machines drove the faster machine’s runs into a fixed-overhead regime where the efficiency factor stops measuring anything the counters can see, and that fixing this made the task easier for every method rather than making the models better.

fig_crossplatform.png

Figure X. Cost of transferring a performance model between machines. Median error factor for four training and testing directions, using only the 14 features computable on both platforms (the intersection set, which necessarily excludes ARCHER2’s RAPL energy and Zen L3 counters). Left: Cirrus run at ARCHER2 problem sizes. Right: Cirrus rescaled. Grey bars are the constant-eta baseline for that test set; coloured bars are the best of random forest, gradient boosting and MLP. Within-platform residual error is roughly 1.05 to 1.06; every transfer direction is 1.14 to 1.19, so crossing machines costs roughly a factor of three in residual error over perfect prediction. Transfer beats its baseline in three of four directions. The exception is fixed-size Cirrus to ARCHER2, where the model is significantly worse than the baseline (Holm p = 0.0020), because the sub-second-dominated Cirrus training set teaches it the wrong eta distribution; with rescaled Cirrus training data the same direction becomes significantly better (Holm p = 5.5e-09). The reader should conclude that cross-platform transfer is possible but expensive, that it is limited by the counters the two machines have in common, and that the quality of the source dataset, not just its size, determines whether transfer helps at all or actively hurts.

Note. The bars in this figure are computed on the full 191-row samples. Only 166 configurations exist on both machines, so these are unpaired comparisons; the paired versions are in ../model/matched_crossplatform_summary.txt and move the transfer numbers by less than 0.02. The within-Cirrus bar is the one materially affected by the restriction (see docs/crossplatform.md).

fig_eta_vs_runtime.png

Figure X. The fixed-overhead mechanism behind the sub-second failure. Efficiency factor eta = (cycles / f_peak) / t_measured against measured runtime for all 544 configurations across both machines, log runtime axis. The heavy black line is the pooled binned median. Above about one second eta is nearly constant at 0.85 to 0.95, that is, the analytic cycle-count model works and a single scalar suffices. Below one second eta collapses, with a median of 0.69 for sub-second runs against 0.86 for runs over three seconds, and individual values as low as 0.06. The cause is a fixed per-run cost (process launch, MPI initialisation, first-touch and I/O) that the counted cycles do not capture and which is amortised over ever less work. The reader should conclude that eta is not a hardware property but a work-dependent one, and that any benchmark campaign whose runs fall below roughly one second will produce an unmodellable eta distribution regardless of the machine.

fig_pred_vs_actual_xplat.png

Figure X. Predicted against measured runtime for the best cross-platform setting. An MLP trained entirely on the rescaled Cirrus data (Zen 5, 288 cores/node) and applied without retuning to all 191 ARCHER2 configurations (Zen 2, 128 cores/node), using only the 14 shared features. Shaded bands mark 1.25x and 1.5x; the diagonal is exact prediction. Median error factor 1.142 (Holm p = 2.8e-12 against the constant baseline, rank-biserial -0.61); 85% of configurations fall within 1.25x and 93% within 1.5x, over four orders of magnitude of runtime and eight applications never seen at these problem sizes. Accuracy is uniform across applications (per-application medians 1.09 to 1.19). The single serious failure, CoMD at 128 cores, is a 103 ms run in the fixed-overhead regime of fig_eta_vs_runtime, under-predicted 5.8x, the same mechanism appearing here as the model’s only visible outlier. The reader should conclude that a model transferred across two different AMD generations still gives useful (within-1.5x) runtime estimates for essentially all runs long enough to be worth predicting.

fig_scaling_behaviour.png

Figure X. Why holding problem sizes fixed breaks the Cirrus campaign. Median measured runtime against core count, per application, one line per dataset; shading marks the sub-second regime. Running ARCHER2’s problem sizes on Cirrus’s 288-core node (orange) drives CoMD, HPL, LULESH and miniFE well below one second at high core counts: miniFE spends 65% of its Cirrus configurations there and HPL 50%. HPCG is the exception, because its problem is sized per rank, so its total work grows with the job and it stays near 30 s at every core count on both machines, making it the only application never affected. GROMACS and OpenFOAM were run on ARCHER2 only, which is also why only 166 of the 191 configurations per machine are comparable at all. The rescaled Cirrus curves (green) restore long runtimes at the cost of no longer solving the same problem as ARCHER2. The reader should conclude that the sub-second problem is a direct and predictable consequence of strong-scaling a fixed problem onto a wider node, and that per-rank problem sizing (as HPCG uses) avoids it.

fig_importance_artefact.png

Figure X. Impurity feature importance is not interpretable on a feature matrix assembled by counter multiplexing. How a small, single-size, multiplexed table makes core_energy_frac (the PP0/PACKAGE energy ratio) look like the strongest single predictor of eta, at 0.509 random forest impurity importance. On the 191-configuration merged data the same feature carries 0.008 and ranks 15 of 21, while the leaders are log_t_analytic (0.376), flops_per_instr (0.134), log_instr_per_rank (0.112) and stall_fp_frac (0.077). The panels isolate the three compounding causes: the absence of a run-length covariate on the original 40-row table (adding log_t_analytic to those same 40 rows removes 72% of the claimed importance), the single problem size per application that confounded application identity with counter signature, and the structural missingness of the 815-row per-counter-set table, where impurity importance correlates with per-feature observation frequency (Spearman rho = +0.57, p = 0.007) and where the four fully-observed features take 63% of the total importance against 38% on the merged data. The control is the decisive panel: re-imposing the per-set missingness pattern on the merged data reproduces the inflated ranking at Spearman rho = +0.74, from data known to carry the true signal. The reader should conclude that any counter that sits on a separate register, and is therefore recorded in every rotation of a multiplexed counter set, will absorb the importance displaced from the columns that are missing most of the time and will look like a discovery. RAPL energy is the usual such counter.

On the word “weak-scaled”

The cirrus_weak.csv campaign is named weak-scaled, but the runtime at the largest core count is not constant relative to the single-core runtime for every application:

Only HPCG and CoMD are close to ideal weak scaling; HPL’s cubic-in-N flop count means its per-rank sizing grows work super-linearly. The dataset therefore achieves its purpose (keeping runtimes off the sub-second floor) and is described in the text and the figures as “rescaled so that work grows with rank count” rather than as a weak-scaling study. See docs/weakscaling.md.