hardware-counters

Performance Projection: Modelling Hardware — ARCHER2 Experiments

Profiling harness, datasets and analysis code for the MSc dissertation. The single-platform work was produced on ARCHER2 (HPE Cray EX, 2× AMD EPYC 7742 “Rome”, 128 cores/node); the cross-platform work adds Cirrus (AMD EPYC 9825 “Turin”, 288 cores/node).

Aim

Predict full-application performance from cheap, low-level hardware-counter measurements, so that a new machine can be characterised without running every application on it. Concretely: profile a set of benchmarks across a range of core counts, extract hardware counters with CrayPat, and train regression models that map counter signatures → runtime. docs/scoping.md states what the pipeline turned out to be able to do, which is narrower than that, and prices the difference.

Reproducing this

REPRODUCING.md gives the run order, which script produces which result, and the environment gotchas. Start there if you want to rerun anything.

Directory layout

dissertation/
├── scripts/
│   ├── counter_sets.sh          # emits PAT_RT_PERFCTR for counter sets A–E
│   ├── pq.py                    # near-square P×Q process grid for HPL
│   ├── check_runs.sh            # which sweep directories are complete
│   ├── build/
│   │   └── build_all.sh         # builds + instruments all applications
│   ├── slurm/
│   │   ├── run_sweep.slurm      # core sweep: stream, hpcg, gromacs
│   │   ├── run_sweep2.slurm     # core sweep: hpl (and the cp2k attempt)
│   │   └── run_of.slurm         # core sweep: openfoam (mesh + solve)
│   ├── cirrus/                  # Cirrus port: env preamble, build, sweep, drip
│   └── analysis/
│       ├── parse_expand.py      # CrayPat experiment dirs → runs_expanded.csv
│       ├── parse_cirrus.py      # same, for Cirrus
│       ├── parse_weak.py        # same, for the rescaled Cirrus campaign
│       ├── retrain2.py          # main results: merged data, LOAO, nested selection
│       ├── ablation2.py         # do the counters earn their keep?
│       ├── baseline_ladder.py   # the measurement ladder
│       ├── noncircular.py       # projection with no target-machine measurement
│       ├── matched_crossplatform.py  # ARCHER2 vs Cirrus on the paired configurations
│       ├── weak_analysis.py     # rescaled Cirrus campaign
│       ├── advanced_nn2.py      # PLR / RealMLP / deep ensembling
│       ├── importance_artefact.py  # feature importance under counter multiplexing
│       ├── stats_rigour.py      # the one reporting routine for every comparison
│       ├── learning_curve.py    # density versus coverage
│       └── ...                  # diagnostics and earlier versions; see REPRODUCING.md
├── data/
│   ├── runs_expanded.csv        # per (config, counter set) rows: the ARCHER2 campaign
│   ├── cirrus_persets.csv       # the Cirrus equivalent
│   ├── cirrus_weak.csv          # rescaled Cirrus campaign
│   ├── runs.csv                 # the first 40-row table (5 apps, 1 size each); read by importance_artefact.py
│   ├── runs_persets.csv         # its per-counter-set form
│   └── functions.csv            # per-function profile rows
├── results/
│   ├── reports/                 # pat_report -O hwpc / -O profile text output
│   ├── model/                   # metrics, predictions, summaries
│   ├── superseded/              # results from earlier stages, with a README on what replaced them
│   └── figures/                 # plots for the dissertation
└── docs/
    ├── methodology.md           # data collection: design and pitfalls
    ├── modelling.md             # prediction models: framing, results, feature importance
    ├── scoping.md               # what is predicted, from what, and at what cost
    ├── crossplatform.md         # ARCHER2 vs Cirrus transfer experiments
    ├── weakscaling.md           # problem size, not hardware: the rescaled Cirrus campaign
    ├── rigour.md                # statistical protocol, precision and the learning curve
    ├── literature.md            # literature survey
    └── literature_corpus.md     # the citation corpus behind it

results/figures/ holds the dissertation figures and captions.md, which gives each one a full caption ending in the conclusion the reader should draw. results/model/ holds the results tables: stats_corrected.{txt,csv} (every comparison in the project under the one reporting routine), matched_crossplatform_{summary.txt,results.csv,stats.csv} (the paired 166-configuration analysis), baseline_ladder.txt and noncircular.txt (the scoping experiments), reconcile_rf.txt (how many digits a random forest median is worth) and learning_curve.{txt,csv}.

Raw CrayPat experiment directories (~17 GB) are not committed; they live on /work and are reproducible from the scripts here. results/reports/ holds the distilled text reports instead.

Applications profiled

Eight codes. Five were in the first campaign; three ECP/Mantevo proxies were added in the expanded campaign to cover kernels the first five missed (unstructured implicit FE, explicit hydrodynamics, classical MD).

pat_build is static instrumentation and requires the binary to be compiled with perftools loaded, so it is used wherever the code is built from source. OpenFOAM is only available as a pre-built ARCHER2 module and cannot be recompiled, so it uses pat_run dynamic instrumentation instead. LULESH is restricted to cube numbers of ranks by the code itself.

These deliberately span the workload space: at 128 cores HPL retires ~472 GFLOP while STREAM retires almost none, so the models see genuinely different counter signatures rather than variations on one theme.

CP2K was attempted but is not included: it calls MPI_Init_thread internally in a way that conflicts with pat_run dynamic instrumentation (Cannot call MPI_INIT or MPI_INIT_THREAD more than once), and the ARCHER2 module cannot be rebuilt with perftools to use pat_build instead. Some results/reports/cp2k_* files survive from that attempt and carry no runtime. BabelStream was considered and skipped: it duplicates STREAM’s memory-bandwidth signature and would add rows without adding a kernel.

Experimental design

Single node throughout. The first campaign varied only core count (1, 2, 4, 8, 16, 32, 64, 128). The expanded campaign added two further axes, because with one problem size per application, application identity and problem size are perfectly confounded, which makes any reading of feature importance unreliable (see docs/modelling.md):

• problem size, three or four per code where the code exposes a size knob;
• MPI x OpenMP decomposition at fixed core count (1, 2, 4 and 8 threads per rank for HPCG and miniFE), which changes cache pressure and NUMA locality for the same code and the same problem.

That gives 191 distinct configurations across 8 applications.

The EPYC 7742 exposes only 5 hardware counters simultaneously, so a wide feature set is collected by repeating each configuration with 5 rotating counter sets (A–E). RAPL energy and Zen L3 counters live in separate components and are appended to every set for free.

Running 5 sets per configuration also yields 5 wall-clock samples, so runtime variance is measured rather than assumed.

Reproducing

All analysis scripts resolve their data and output roots from DISS_ROOT, which defaults to /work/project/project/user. Set it to relocate the whole pipeline.

export DISS_ROOT=/work/project/project/user

# 1. build and instrument everything (login node, ~40 min, mostly GROMACS)
bash scripts/build/build_all.sh          # stream, hpcg, hpl, gromacs, openfoam case
bash scripts/build/build_proxies.sh      # minife, lulesh, comd

# 2. submit the expanded sweep: size x decomposition x 8 codes
bash scripts/slurm/submit_expand.sh      # wraps run_expand.slurm
sbatch -J of --export=ALL,NCORE=128 scripts/slurm/run_of.slurm   # openfoam, pat_run

# 3. build the dataset
module load cray-python/3.10.10
python3 scripts/analysis/parse_expand.py     # -> data/runs_expanded.csv

# 4. environment. --system-site-packages keeps cray-python's numpy/pandas;
#    a plain venv or pip --user pulls a newer numpy that shadows and breaks them.
python3 -m venv --system-site-packages $DISS_ROOT/venv
source $DISS_ROOT/venv/bin/activate
pip install -r requirements.txt

# 5. the analyses, in order
python scripts/analysis/retrain2.py            # main results
python scripts/analysis/ablation2.py           # do the counters earn their keep?
python scripts/analysis/importance_artefact.py # feature importance under counter multiplexing
python scripts/analysis/advanced_nn2.py        # PLR / RealMLP / deep ensembling
python scripts/analysis/crossplatform.py       # ARCHER2 vs Cirrus matrix
python scripts/analysis/weak_analysis.py       # rescaled Cirrus campaign

Earlier versions of several scripts (train_model.py, significance.py, retrain.py, train_nn*.py, ablation.py, make_plots.py) sit alongside the current ones. Their outputs are in results/superseded/, whose README says what each was and what replaced it. Every number in this README comes from the current pipeline.

Headline result: the counters earn their keep, and only in full

191 configurations across 8 applications, core counts 1-128, 3-4 problem sizes and 1-4 MPI/OpenMP decompositions, all on ARCHER2. Validation is leave-one-application-out (train on 7 codes, predict the 8th).

The clearest and most robust result in the project is not that one model beats another, it is that the profiling campaign itself is what buys the accuracy. Feature groups added cumulatively by measurement cost, random forest, against a zero-parameter constant that predicts the median training efficiency factor (1.065):

HL is the Hodges-Lehmann pseudomedian of the paired differences, the location estimate the Wilcoxon signed-rank test actually targets; Holm p is Holm-Bonferroni adjusted within the family of four. Effect size for the full set is a rank-biserial correlation of -0.35 with 130 of 191 configurations improved.

Read plainly: the full five-counter-set rotation is the only feature configuration that beats a zero-parameter constant. Free configuration metadata is significantly worse than guessing the training median, one counter does not rescue it, and one counter set is still indistinguishable from guessing. Everything cheaper than the full campaign fails. That is the result that justifies the campaign, and it is stated first because it is the one the evidence supports most strongly.

Figure: results/figures/fig_ablation.png.

What the models buy on top of that

Much less. On typical accuracy the zero-parameter constant is competitive with everything; the random forest beats it (Holm p = 3.6e-04) but by one percentage point. Where models earn their place is the tail, where p90 falls from 1.53 to 1.27-1.39. The defensible claim is “models cut worst-case error by ~17%, while a constant suffices for typical cases”, which is a statement about when profiling is worth doing. This comparison is reported second because it is nearly null, and reporting a near-null model horse race above a decisive feature-value result would misrepresent where the evidence actually is.

Model medians are quoted to two decimals deliberately. Ten random-forest seeds span 0.0158 in median error factor, five times the spread between different implementations of the same pipeline, so the third and fourth decimals are noise. See docs/modelling.md, Statistical methodology and the precision of the numbers.

Would more data help?

Only more applications. A learning curve over the training set, with the test fold never subsampled, separates two axes:

The density axis is saturated and the coverage axis is not, by a factor of six on the final step. The profiling budget was spent widening the configuration sweep within each application when it should have gone on more applications with a coarser sweep each. Figure: results/figures/fig_learning_curve.png; see docs/modelling.md, Learning curve: is 191 configurations enough?

Neural networks

Pursued through three rounds and never competitive. Plain MLPs, then retuning, then published state of the art (PLR numerical embeddings, robust scaling with smooth clipping, RealMLP, deep ensembling). Every neural approach loses to the zero-parameter constant; PLR, the most-recommended technique, is the worst (p = 3e-06). The cause is the data regime: 191 rows is well below where these methods pay off, and 62% of the target lies in a narrow band, so there is little signal for extra capacity to capture. Reported as a negative result with an identified mechanism. See docs/modelling.md.

Cross-platform results

The project title promises Modelling Hardware, so the models were tested across two machines: ARCHER2 (EPYC 7742, Zen 2, 128 cores/node, 2.25 GHz) and Cirrus (EPYC 9825, Zen 5, 288 cores/node, 3.71 GHz). ARCHER2 contributed 191 configurations across eight applications and Cirrus 191 across six: GROMACS and OpenFOAM could not be built on Cirrus, and Cirrus alone reaches 125 and 288 cores. Joining on (application, size, cores, threads) gives 166 configurations present on both machines, and that paired subset is the basis of every platform comparison. The pairing matters: the 25 Cirrus rows without an ARCHER2 counterpart are its 125- and 288-core points, its widest and shortest runs, where fixed overhead dominates and the efficiency factor collapses, so an unpaired comparison of the two full samples would make Cirrus look harder for a reason that has nothing to do with Zen 5 (docs/crossplatform.md).

The paired 166-configuration results, intersection features:

The last row is an average over six applications, not eight: GROMACS and OpenFOAM have no Cirrus test set, so those folds are not evaluable, and every fold is listed with its disposition in results/model/matched_crossplatform_summary.txt.

Three findings:

1. Transfer works, but weakly. Training on ARCHER2 and predicting Cirrus beats a constant, but only just, and moving to unseen hardware roughly triples the error gap versus within-platform prediction.
2. The counter intersection costs almost nothing. Cirrus has no RAPL and no ZenL3, so energy and L3 features are unavailable. Dropping them changes ARCHER2 → Cirrus by 0.4%, which is consistent with the feature-importance ranking in docs/modelling.md, where energy sits in the bottom half on both the impurity and the permutation measure.
3. Cirrus is somewhat harder, and part of the gap is coverage rather than hardware. On the paired set the log-η spread ratio is 1.77×. Cirrus also has six applications to ARCHER2’s eight, which on the learning curve above is worth roughly 0.02 in median error on its own, so the within-ARCHER2 (1.05) versus within-Cirrus (1.12) gap is partly a training-set coverage effect.

The problem-size design choice is the most useful methodological result in the project. Problem sizes were held identical to ARCHER2’s so both machines ran the same work, the natural way to make a fair comparison, but on a 288-core node that made most runs sub-second. Re-running with sizes rescaled so total work grows with rank count (docs/weakscaling.md; this is not true weak scaling for every code):

The table needs reading with care: the constant baseline improved by almost exactly as much as the model did (1.166 → 1.062, against 1.186 → 1.057), and on rescaled Cirrus no model beats the constant (RF, GBQ and MLP all at Holm p = 0.37). Rescaling made the task easier for every method including the zero-parameter baseline; what changed is that models stopped being actively worse than a constant. Comparability and predictability are in tension, and that is a claim about the difficulty of the target, not about model quality.

See docs/crossplatform.md and docs/weakscaling.md for the experiments, docs/methodology.md for the non-obvious pitfalls encountered, and docs/rigour.md for the statistical protocol behind every comparison.