hardware-counters

Scoping: what is predicted, from what, and at what cost

The pipeline in modelling.md and crossplatform.md reconstructs runtime as t_analytic / eta with t_analytic = PAPI_TOT_CYC / f_peak, and PAPI_TOT_CYC is measured on the very configuration whose runtime is being reported. That is the most important limit on what the project can claim. This document states the scope plainly, quantifies how much of the reported accuracy is attributable to that measurement, states the research question the experiments actually answer, and reports the experiment that tests the stronger claim the feasibility study made.

What the pipeline does and does not do

The pipeline takes one instrumented run of an application at a given configuration on a given machine, extracts twenty-two hardware counters from pat_report, and estimates the dimensionless efficiency factor

    eta = (PAPI_TOT_CYC / f_peak) / runtime

from those counters. Wall time is then reconstructed as t_analytic / eta. What this achieves is an explanation of the gap between the ideal cycles-at-peak-clock time and the observed wall time, expressed as a function of measurable microarchitectural quantities and validated on applications the model has never seen.

What it does not do is predict runtime in advance of running the application. The cycle count that carries almost all of the signal is itself the product of a completed, instrumented execution on the target machine at the target core count and problem size. Anyone in a position to supply PAPI_TOT_CYC for a configuration is already in possession of the wall time for that configuration, because the two come out of the same pat_report output. The reconstruction is therefore a decomposition of a measurement rather than a projection, and no figure in modelling.md should be read as a statement about predicting unrun work.

The feasibility study proposed prediction “without ever running the full application”. The single-platform pipeline does not do that. The experiment in The non-circular experiment below tests the claim directly, and meets it only partially, and weakly.

The baseline ladder

To make the attribution concrete, five predictors of increasing measurement cost were evaluated under one identical protocol: the merged 191-configuration ARCHER2 dataset, leave-one-application-out folds, and the symmetric multiplicative error max(pred/act, act/pred). The script is analysis/baseline_ladder.py and the output is analysis/out/baseline_ladder.txt.

The incremental contributions to the total median gain of 6.533 are: configuration metadata contributes minus 0.238 (it makes matters worse), the measured cycle count contributes 6.588 or 100.8 per cent, the fitted constant efficiency contributes 0.173 or 2.6 per cent, and the entire learned model over twenty-one counter features contributes 0.011 or 0.2 per cent. In the tail the learned model does better, cutting p90 from 1.532 to 1.390, which is the claim already made in modelling.md and which survives this reframing intact.

One qualification on the 0.011, because it looks smaller than the figure modelling.md reports for the same step. These are differences of marginal medians, which is the right quantity for an attribution ladder because each rung is being described on its own terms. The paired difference between rungs 3 and 4 is larger, a Hodges-Lehmann pseudomedian of -0.029 with 130 of 191 configurations improved, because the error-factor distribution is right-skewed and the two quantities are not the same thing. Neither figure is wrong; the ladder measures how much the typical error moves and the paired test measures how consistently it moves. The ladder’s conclusion, that the cycle count does almost all of the work, holds under either.

Rung 1 deserves a word because it is the only rung that could in principle be evaluated before running anything, and it fails. Under leave-one-application-out a model given only core, rank and thread counts must extrapolate a scaling curve for an application it has never seen, and there is nothing in three integers that distinguishes STREAM at 128 cores from GROMACS at 128 cores. It scores 7.826 against the zero-parameter constant’s 7.588, so it is worse than not modelling at all, and it is significantly worse (Wilcoxon p = 0.034). This is not a defect in the model. It is the correct answer: configuration alone carries no information about an unseen application’s runtime, and the profiling campaign exists precisely because of that.

The cycle count’s dominance is not surprising once stated numerically. The log-log correlation between PAPI_TOT_CYC and runtime across the 191 configurations is 0.9929, so t_analytic alone accounts for 98.6 per cent of the variance in log runtime. The efficiency factor is confined to a narrow band (median 0.807, interquartile range 0.681 to 0.839, standard deviation of log10(eta) only 0.128), which is the same fact stated the other way round and is also the reason a zero-parameter constant is so hard to beat, as modelling.md reports.

The research question, restated

The question the project can answer, and which the results above do answer, is this: given one instrumented run of an application on the target machine, how much of the residual inefficiency (the departure of wall time from cycles-at-peak-clock) is explained by hardware counters, and does that explanation transfer to workloads and to machines the model has not seen?

That is a narrower question than “predict runtime without running”, but it is not a trivial one and it has practical content. A practitioner with a profile in hand and a p90 error of 1.39 rather than 1.53 has a better bound on worst-case behaviour, and the counter-group ablation in modelling.md shows the improvement is real and requires the full five-set rotation rather than any cheaper subset. The transfer result in crossplatform.md, that an ARCHER2-trained model reaches 1.203 on the matched Cirrus data against a constant’s 1.220, is likewise a statement about the transferability of the explanation, not about projection onto unrun work. Read that way it is defensible. Read as projection it is not, because it too consumes Cirrus cycle counts.

Every headline figure in modelling.md and crossplatform.md should be understood as conditional on one instrumented target run. None of them should be cited as evidence that runtime can be estimated before the run exists.

The non-circular experiment

The stronger claim deserves a direct test rather than a concession, so one was constructed. The setting is: an application has been profiled on a source machine (ARCHER2), a target machine (Cirrus) is described only by static datasheet facts, and the runtime on the target must be predicted with no Cirrus measurement of any kind entering the feature vector. The script is analysis/noncircular.py and the output is analysis/out/noncircular.txt.

The features are the twenty-one ARCHER2 counter ratios, the ARCHER2 wall time and analytic time, the configuration metadata, and four static ratios between the machines: peak clock (3.71 GHz against 2.25 GHz, a ratio of 1.649), nominal memory bandwidth per socket (576 GB/s for twelve channels of DDR5-6000 against 204.8 GB/s for eight channels of DDR4-3200), cores per node (288 against 128) and Zen generation (5 against 2). Evaluation is leave-one-application-out over the 166 configurations that match exactly on application, problem size, core count, rank count and thread count, covering the six applications present on both machines. GROMACS and OpenFOAM were never run on Cirrus and are excluded.

The pooled median and p90 weight all 166 configurations equally, so HPCG contributes 44 of them and largely determines the pooled figure. The macro column averages the six per-application medians and is the fairer summary here, because HPCG is both over-represented and behaviourally atypical (it is the only benchmark whose size knob is already per rank, so it does not strong-scale into the sub-second regime, and it is the only one whose cycle count rises on Cirrus).

The honest reading is as follows. The best non-circular predictor, ARCHER2 cycles divided by the Cirrus peak clock with efficiency learned from ARCHER2 counters, reaches a median of 1.556 and a macro-average of 1.653. Naive clock-ratio scaling of the ARCHER2 wall time reaches 1.608 and 1.990. The counter-driven model is therefore ahead on both statistics, but the paired Wilcoxon test against clock scaling gives p = 0.993 with 94 wins out of 166, which is no evidence of a difference at all. Predicting the ARCHER2 runtime unchanged (2.289) and predicting a constant Cirrus runtime (5.086) are both soundly beaten, but those are very low bars: the first ignores a 65 per cent clock difference and the second ignores the application entirely.

Set against this, the circular predictor on the same 166 configurations reaches a median of 1.129. The gap between 1.129 and roughly 1.6 is the honest price of not having run on the target. In error-factor terms, one instrumented Cirrus run takes the typical error from about 56 per cent to about 13 per cent.

Two diagnostics explain why. First, the cycle count is not machine invariant, so the analytic transfer cannot be exact even in principle. The ratio cyc_cirrus / cyc_archer2 has a median of 0.651 but an interquartile range of 0.451 to 1.195 and a standard deviation in log10 of 0.269, and the per-application medians run from 0.378 (LULESH) to 1.666 (HPCG). Zen 5 retires the same work in between a third and one and two-thirds of the cycles Zen 2 needs, depending entirely on the application, and no static specification predicts which. Second, the achieved speedup varies far more than the clock ratio does: the median speedup is 2.289 while the clock ratio is 1.649, and the per-application medians span 1.182 (HPCG) to 4.322 (LULESH). An oracle allowed to fit one constant speedup per test application, using that application’s own Cirrus data, would reach a median of 1.130, so essentially all of the achievable accuracy lies in knowing the per-application speedup factor. That factor is what leave-one-application-out withholds, and it is what the ARCHER2 counters fail to supply.

The finding, stated without hedging: non-circular cross-machine projection does not work here. A model given a full ARCHER2 profile and a static description of Cirrus predicts Cirrus runtime no better than dividing the ARCHER2 wall time by the ratio of peak clocks, a calculation requiring no profiling campaign, no counters and no machine learning. This is a negative result with an identified mechanism (the cycle count varies by a factor of four across applications when the microarchitecture changes, and counter ratios measured on the source machine do not predict that variation), and it should be reported as such rather than buried.

One structural limitation must be stated alongside it, because it bounds how strongly the negative result can be generalised. With exactly one target machine, the four static specification features take the same value in every row. They cannot contribute anything beyond what a fitted intercept already captures, so their coefficients are unidentifiable and the experiment cannot distinguish “static machine descriptions do not help” from “static machine descriptions were never given a chance to help”. Separating those requires at minimum three target machines. What the experiment does establish, and establishes cleanly, is that source-machine counters do not predict the application-specific part of the speedup, and that part is where all of the signal is.

Why the scope narrowed

The project as scoped in the feasibility study intended to predict runtime for unrun configurations from a model of the hardware. Four constraints, all discovered during data collection and all documented in methodology.md, pushed it towards the narrower question of explaining measured inefficiency.

The EPYC 7742 exposes only five hardware counters at once. Covering twenty-two counters therefore required five separate runs per configuration with a rotating counter set, which is why the dataset is 191 merged configurations rather than a larger number of independently informative rows, and why the per-run rows had to be merged at all (each individual run measures only its own five counters, leaving roughly 58 per cent of a per-run feature vector missing). That cost, five executions per data point, is what made a broad campaign across many machines and many applications infeasible within the allocation.

The cray_cassini component is disabled on ARCHER2, so no interconnect counters were available. Communication cost, which is exactly the part of the efficiency factor a hardware model would most want to reason about when projecting onto a machine with a different network, is invisible to the features and enters only implicitly through eta. Any cross-machine claim about communication behaviour is therefore unsupported by the data.

CP2K could not be instrumented: it aborts under both pat_run -w and pat_run -S with MPI_Init_thread: Cannot call MPI_INIT or MPI_INIT_THREAD more than once, and no combination of flags avoids it. It was dropped, reducing the application count and removing the one workload in the intended set with a genuinely different communication pattern from the others.

Allocation was single node throughout. Every configuration runs within one node’s 128 cores (or 288 on Cirrus), so inter-node communication is never exercised, and the scaling curves the models learn are intra-node curves. Extrapolating them to multi-node jobs is not supported.

Two of these are hard limits of the platform (the counter budget and the disabled network component), one is a property of an application (CP2K), and one is a resource constraint. Together they left an experiment that can say a good deal about how counters explain single-node inefficiency and rather little about projecting onto hardware that has not been measured. The results reported here are an accurate account of that experiment, not of the one originally proposed.

Reproducing

source $B/venv/bin/activate
python analysis/baseline_ladder.py   # the ladder, table above
python analysis/noncircular.py       # ARCHER2 -> Cirrus without Cirrus counters

Outputs land in analysis/out/baseline_ladder.{txt,csv}, analysis/out/baseline_ladder_perconfig.csv, analysis/out/noncircular.{txt,csv} and analysis/out/noncircular_perconfig.csv.