hardware-counters

Cross-platform prediction: ARCHER2 vs Cirrus

The experiment the project title promises. Everything prior to this was single-platform, so “Performance Projection: Modelling Hardware” had never been tested across machines.

The two platforms

Three CPU generations apart, 2.25× the cores, a 65% higher clock, and a different memory technology, but the same vendor, the same toolchain and the same CrayPat methodology, which is what makes the comparison tractable.

ARCHER2 contributed 191 configurations across eight applications and Cirrus 191 across six: GROMACS and OpenFOAM could not be built or run on Cirrus, and Cirrus alone reaches 125 and 288 cores, which a 128-way ARCHER2 node cannot. Joining on (application, size, cores, threads) gives 166 configurations present on both machines:

Both totals being 191 is a coincidence of how far each sweep reached, not a matching. Every platform comparison in this document is therefore made on the paired 166, so that “platform” is not confounded with “which configurations happen to be in this sample”. The confounding would run in one direction: the 25 Cirrus rows without an ARCHER2 counterpart are exactly its 125- and 288-core points, its widest and shortest runs, where fixed overhead dominates and the efficiency factor collapses (median runtime 1.041 s against 1.203 s for the paired rows), so an unpaired comparison of the two full samples makes Cirrus look intrinsically hard for a reason that has nothing to do with Zen 5. The figures for the two full 191-row samples are shown alongside the paired ones so the cost of the restriction is visible. Analysis: results/model/matched_crossplatform_*; protocol: docs/rigour.md.

Note on the peak clock. Cirrus login nodes are EPYC 9745 and report 2.4 GHz; compute nodes are EPYC 9825 at 3.7146 GHz. Deriving F_PEAK from the login node would have mis-scaled every analytic term on Cirrus.

The feature problem

The machines do not expose the same counters, which constrains what a cross-platform model may use.

Cirrus has no cray_rapl and no cray_zenl3 component; ARCHER2 lacks the Zen 5 presets. The intersection is 14 features, and it excludes the two energy ratios. Quantifying the cost of that is one of the results below; it turns out to be small, consistent with core_energy_frac carrying 0.008 importance (rank 15 of 21) on the 191-configuration data (docs/modelling.md, Feature importance under structural missingness).

Experiment matrix

Six settings × two feature sets × four model families. Error is the multiplicative factor max(pred/act, act/pred); the reference is a zero-parameter constant predicting the median training efficiency factor. Verdicts use the Hodges-Lehmann pseudomedian of the paired differences with Holm-Bonferroni correction within each family (see docs/modelling.md, Statistical methodology and the precision of the numbers).

Intersection features, paired 166 configurations:

LOAO = unseen hardware and unseen application simultaneously.

The same settings on the two full 191-row samples, for comparison, with the delta the restriction costs:

Where a delta is large and negative, the all-191 figure is pessimistic because of the unpaired Cirrus rows rather than because of the hardware. Where it is near zero, the restriction changes nothing. Only the matched column can support a statement of the form “platform A is harder than platform B”, because only it holds the workload fixed. The restriction costs 25 configurations per platform (13.1%) and two of eight applications.

The two applications that cannot be scored

The A2 → CIR LOAO setting loops over the union of applications, and for GROMACS and OpenFOAM the Cirrus test mask is empty. Every fold is listed with its disposition, folds with no test set are reported as not evaluable, application absent on the target platform rather than skipped, and the constant reference is computed on the same folds as the models (a verdict against a constant that was never computed for the same folds is not a verdict). The figure of 1.254 for A2 → CIR LOAO is an average over six applications, not eight, and the two missing ones are a production MD code and a production CFD code, that is, the two least like the proxy benchmarks. The hardest setting in the matrix is evaluated only on the easier half of the application space, and that must be stated wherever the number is quoted. The CIR → A2 direction differs: GROMACS and OpenFOAM do have a non-empty ARCHER2 test set, so all eight folds score, but no Cirrus training rows for them exist either, so the fold is really “predict a code the source platform has never seen in any form”. That is a legitimate and harder test, and it is labelled as one.

Figure: ../results/figures/fig_crossplatform.png, and ../results/figures/fig_pred_vs_actual_xplat.png for the per-configuration scatter in the best transfer direction.

Findings

1. Cross-platform transfer works, but barely

Training on ARCHER2 and predicting Cirrus reaches 1.203× on the paired set, better than a constant fitted on ARCHER2 data (1.220×) and significant after Holm (p = 0.0060), but only just. The reverse direction does not separate from its constant. The honest statement is that a model trained on one machine carries weak but non-zero information about another.

For context, within-platform prediction on ARCHER2 is 1.05×. So moving to unseen hardware roughly triples the error gap over the baseline.

2. The intersection costs surprisingly little

Dropping energy and L3 moves A2 → CIR by 0.4%, and within-ARCHER2 it costs about 0.005 in median error factor, which is a third of the spread across random seeds and therefore not resolvable.

This agrees with the feature-importance ranking: core_energy_frac carries 0.008 impurity importance on the merged data and is worth essentially nothing on held-out permutation importance, so removing it costs essentially nothing. The ablation was computed on the merged data throughout and is independent confirmation of that ranking.

For a practitioner: a cross-platform predictor does not need RAPL.

3. Transfer is asymmetric, and the direction is informative

A2 → CIR beats its baseline; CIR → A2 does not, and on the two full samples it is significantly worse than a constant (RF Holm p = 0.0020, MLP p = 0.033). The asymmetry follows from target difficulty rather than model quality:

Cirrus is 1.77× more variable in the target on the paired data (2.2× on the two full samples) and its runs are far shorter, because 288 cores finish the same fixed problem much faster. Nearly half the Cirrus runs are under a second, where fixed overheads (MPI_Init, I/O, instrumentation) dominate and η collapses; median η is 0.47 below 0.3 s versus 0.83 above 3 s. Figure: ../results/figures/fig_eta_vs_runtime.png.

So a model trained on ARCHER2’s well-behaved data transfers reasonably onto Cirrus, but a model trained on Cirrus’s noisier, overhead-dominated data does not improve on simply predicting a constant for ARCHER2. The decisive evidence that this is about the data and not the machine is that transferring from the rescaled Cirrus dataset to ARCHER2 is significantly better than a constant in all three model families (MLP Holm p = 2.8e-12), while transferring from the fixed-size Cirrus dataset is significantly worse.

4. Cirrus is harder to predict, but not by much

On the paired 166 configurations, within-Cirrus RF reaches 1.1248 and GBQ 1.1241 against a constant of 1.1294. Both beat the constant and both survive Holm-Bonferroni (adjusted p = 0.047 and 0.0056). The statement the data support is: with the workload held fixed, tree models beat a constant on Cirrus too, but by a much smaller margin than on ARCHER2.

Two things account for the residual gap between within-ARCHER2 (about 1.05) and within-Cirrus (about 1.12), and only one of them is hardware. The first is the problem-size design choice, which is real and is the subject of weakscaling.md: sizes were deliberately kept identical to ARCHER2’s for comparability, and on a 288-core Zen 5 node those sizes are too small, so the machine finishes them before the counters say anything useful. That is a correctable design consequence, and comparability and predictability are in tension here; this experiment chose comparability.

The second is coverage, and it was invisible until the learning curve was measured. Cirrus has six applications where ARCHER2 has eight, and under leave-one-application-out the learning curve (modelling.md, Learning curve) is still steeply descending at seven training codes: the step from five to seven is worth 0.0223 in median error. Every Cirrus and transfer number in this project therefore sits further left on that curve than the ARCHER2 numbers do, so part of the platform gap is a training-set coverage effect that would persist even if both machines ran identically-sized problems. This is worth stating because a conclusion about hardware is here partly a conclusion about sample composition.

The per-application implied clock is a genuine hardware observation and is unaffected:

Most applications run near Cirrus’s higher clock as expected, but miniFE regresses — a genuine Zen 5 behaviour worth noting rather than smoothing over.

5. Simultaneous unseen hardware and unseen application

The hardest setting (LOAO rows) reaches 1.254 (A2 → CIR) and 1.173 (CIR → A2) on the paired data. Both are worse than within-platform prediction but remain in the same broad range, which suggests the two difficulties compound roughly additively rather than catastrophically. Neither separates from its constant after Holm, so the honest reading is that in this setting the models add nothing measurable, and the A2 → CIR figure covers six applications rather than eight (see The two applications that cannot be scored, above).

Threats to validity

• Two platforms, same vendor. Both are AMD EPYC with Cray toolchains. An Intel or ARM target would be a stronger test; the counter intersection would likely shrink further.
• 166 paired configurations, and only six applications on both machines. Differences below ~0.02 in median error are not meaningful, and the application count is the binding constraint on accuracy rather than the configuration count (see the learning curve in modelling.md). The two applications that Cirrus lacks are also the two production codes, so the paired subset is proxy-benchmark-heavy.
• Problem sizes were held fixed for comparability, which disadvantages the larger machine (finding 4). weakscaling.md reports the size-scaled replication, which separates “harder hardware” from “too-small problem” and attributes most of the effect to the latter.
• branch_miss_rate is not an intersection feature. PAPI_BR_INS/BR_MSP were collected on Cirrus but are absent from the ARCHER2 data, so the feature is 0% available there and the intersection is 14 features, not 15.
• The two full samples are not paired. The all-191 figures are shown alongside the paired ones so the size and direction of the restriction can be seen, but only the paired figures support a statement about the platform.

Reproducing

# on Cirrus
python analysis/parse_cirrus.py            # CrayPat dirs -> cirrus_persets.csv
python analysis/crossplatform.py           # the 6 x 2 x 4 matrix on the two full samples
python analysis/matched_crossplatform.py   # the paired 166-configuration analysis
python analysis/xplat_diag.py              # the diagnostics behind findings 3 and 4

Results: results/model/matched_crossplatform_summary.txt, matched_crossplatform_results.csv, matched_crossplatform_stats.csv (paired) and crossplatform_results.csv, crossplatform_summary.txt (the two full samples).