What hardware counters can and cannot tell us

The processor keeps its own tallies of what it was doing while a program ran, the hardware performance counters, and a lot of the literature offers them as a cheap stand-in for running an application at all. You read the tallies instead of paying for the run.

My campaign covered eight applications, 191 configurations, and ARCHER2, and I then repeated all of it on Cirrus. I also built a measurement ladder to show me which part of all that measuring was giving the accuracy.

Almost all of the accuracy came from the cycle count, a single counter. There is only one way to get that counter, and that is to run the very job whose runtime you’re trying to predict.

Take away every measurement made on the target machine and the error multiplies by 4.3. With one of them a typical prediction lands about 13% off, and without one it lands about 56% off.

Two more results put a ceiling on the method. Asked to predict an application it had never seen, at a scale it had never seen, each learned model was beaten by a straight line with two parameters, fitted through the target application’s own small runs, which are cheap. In cross-machine projection, where measurements from one machine are used to predict the other, a single speedup number fitted per application did as well as the whole apparatus of counters.

All of this is my MSc work at EPCC, written up as the dissertation. Alexei Borissov supervised it.

The rest of this page is more or less the dissertation itself, rewritten for the web. The whole of it takes an hour to read, and the argument takes fifteen minutes, in sections one to four and the ladder.

---

Table of contents

Part I. Setting up

1. The question
2. What people already do
3. The four questions I set out to answer
4. The machines
5. The eight applications
6. Five counters at a time
7. The efficiency factor
8. How I decided whether a model was any good
9. The models

Part II. Building it

10. Getting eight codes to run under instrumentation
11. Nine misleading failures in the toolchain
12. What counts as one observation
13. The features
14. A check before any modelling

Part III. Results

15. How to read these numbers
16. Asking a better question
17. The measurement ladder
18. Do the counters justify their cost?
19. The pipeline explains runtime but doesn’t predict it
20. My p-values are too optimistic
21. Measurement choice dominates model choice
22. Three rounds of neural networks
23. Would more data have helped?
24. The baseline to beat is a straight line
25. Transfer between machines
26. Sub-second runs, not Zen 5
27. Pricing the circularity

Part IV. What it means

28. The overfitting warning, revisited
29. Answering the four questions
30. What a practitioner should do
31. What I got wrong
32. Limitations
33. What I’d do differently

Appendix

34. Every counter collected
35. The rotation schedule
36. The derived features
37. Source code

---

Part I. Setting up

The question

Anyone in high-performance computing hears two questions again and again. The first belongs to a procurement panel. Two machines are on the table, neither one installed yet, and whichever they pick costs several million pounds and, for five years after that, fixes what their researchers are able to compute. Which of the two will run our codes faster?

The second belongs to a user with a job to submit. There’s a twelve-hour queue and a twenty-four-hour queue, and they don’t wait the same length of time. Ask for twelve and the job dies at the wall clock with nothing saved. Ask for twenty-four and it sits a whole day behind every other job. So how long will the job take?

Both people want to know how a code will perform on a machine before it runs there, and the two usual ways of answering don’t help them. One is to benchmark the code on the machine itself. For that the machine has to exist and be free, and every configuration you test spends allocation. The other way is to run it in a simulator of the processor. That needs no machine, but it’s slower than the real machine by a factor of thousands, so the problems it can handle are much smaller than any that anyone wants answered.

There’s a third way in, through the hardware performance counters, and reading them costs next to nothing. They count instructions retired, branch mispredictions, cache misses at each level, and the cycles the pipeline lost stalled on memory. These numbers say the way an application uses a machine. A code that sits stalled on memory for most of its cycles looks nothing like one that keeps its floating-point units busy the whole time.

So it’s fair to hope the signal carries over to other codes and machines. My feasibility study proposed exactly this, and the dissertation was me testing it. Profile a set of benchmarks, work out how their counter signatures relate to runtime, and then use that relation to say how codes and machines beyond the training set will do.

What people already do

There are three traditions in performance modelling, models worked out from what you know of the machine, models fitted to measured runs, and simulation. I want to place a counter-based approach among them, and I’m most interested in how each tradition checks its own claims.

Models derived from the machine

The Roofline model (Williams, Waterman & Patterson, 2009) gives a ceiling. Performance can’t go above the smaller of two things, the chip’s peak floating-point throughput on one side and bandwidth multiplied by arithmetic intensity on the other. The cache-aware descendants (Ilić, Pratas & Sousa, 2014; 2017) of Roofline push the same idea down through the levels of memory and treat the core, the uncore and the package as separate power domains, and it’s still in use, LULESH among the codes people apply it to (Afzal, Hager & Wellein, 2025). What I take from Roofline is that it predicts a bound and nothing more.

The Execution-Cache-Memory model (Hager et al., 2013; Stengel et al., 2015) takes another step, adding the time the core spends computing to the time data spends crossing each cache level, and Kerncraft (Hammer et al., 2017) does the analysis for you. Where they fit the problem, they’re accurate. They want an analysis of the loop nest done statically, plus a model of the target machine. So they stay with kernels and don’t stretch to a whole application. As chips got more complicated, later work had to put real effort into describing the machine side in general terms (Hofmann et al., 2020).

Parallel scaling has its own vocabulary, and it comes from the classical bounds (Amdahl, 1967; Gustafson, 1988) and from the LogP family (Culler et al., 1993; Alexandrov et al., 1997). Extra-P (Calotoiu et al., 2013) takes a modest set of measurements and fits an empirical scaling model to them. Later work has kept most of that accuracy while cutting the profiling cost to a small share of what it was (Ritter et al., 2020; 2026), added neural networks so that noise throws it off less (Ritter et al., 2021), and carried it over to distributed deep learning (Ritter & Wolf, 2023).

For my purposes Extra-P counts for more than anything else in this section. Every learned model I built loses to a trend baseline, and that baseline is a two-parameter version of this very idea.

Models built from counters

Mechanistic CPI-stack models (Eyerman et al., 2006; 2007; 2011) start from measurements instead. Across three Intel generations the reported errors are 9 to 13%. Intel’s top-down method (Yasin, 2014) puts the same idea on commodity counters. The efficiency factor $\eta$ that the whole project stands on comes from that family, a single scalar where a CPI stack gives an itemised list.

Counters can be read the same way on every processor largely thanks to PAPI (Browne et al., 2000). People have spent years asking how far counters can be trusted, and one survey lists what goes wrong when counters are used in security work (Das et al., 2019). The energy counters have been checked the same way (Hackenberg et al., 2015), and I read energy through RAPL, so those weaknesses apply to my data too.

Models fitted to data

The machine-learning literature drops the machine model and puts a fit in its place. The errors people report sit mostly between 10% and 20% (Ding et al., 2020; Sun et al., 2020; Owenson et al., 2019; Malakar et al., 2018). Keep that beside the ~5% figures further down as a yardstick.

Predicting scalability with regression is a line of work on its own (Barnes et al., 2008; 2010), and so is the work on configurable software, which has both deep networks (Ha & Zhang, 2019) and careful strategies for sampling (Kaltenecker et al., 2019; 2020) in it. 1% of target-scale observations was enough for Marathe and colleagues (2017) to find good configurations, by transfer learning. An exploratory study beside it maps where this kind of transfer works and where it breaks down (Jamshidi et al., 2017).

Ardalani and colleagues (2015) predicted how a GPU would perform from CPU-only executions. That is prediction of work nobody has run, and this project doesn’t manage it. There are other routes to cross-platform prediction as well, partial execution (Yang et al., 2005), evaluating a representative region (Ferrerón et al., 2017), and the recent hybrid methods (Mahdavi, 2024), and there’s work that uses public datasets to predict new workloads (Wang et al., 2018).

On proxy applications

Of my eight codes, three are proxies. Whether proxies really behave like their parents has its own literature, and it decides how far my results reach.

Mini-applications were argued for early (Heroux et al., 2009) and have been weighed at length since then (Barrett et al., 2015). A proxy follows its parent fairly well on computation and on memory, less well on communication, and where the two part ways the gap is particular to that pair and not a pattern across all of them (Aaziz et al., 2018; 2019; Richards et al., 2020). Newer work says modern workloads need the whole proxy idea rethought (Matsuoka et al., 2022), and how alike a proxy and its parent are kernel by kernel is still open (McKinsey et al., 2026).

No result here assumes that a proxy behaves the way its parent does. LULESH (Karlin et al., 2013) and CoMD (Pearce et al., 2019), the two proxies I used, have papers of their own describing how they behave, and so do STREAM (McCalpin, 1995), HPL (Dongarra et al., 2003) and HPCG (Dongarra et al., 2015).

How this field validates itself

Hoefler and Belli (2015) shaped this project more than any paper about models did. They read 120 papers across three major HPC conferences, and much of the time they couldn’t tell whether an improvement a paper claimed was deterministic or just a fluctuation. What they asked for (show the distribution instead of a single point estimate, use non-parametric tests, and write the protocol down) is the shape of every result below, and each headline median comes with its bootstrap interval and its breakdown by application. The superseded results are still in my working repository too, with a note on each about what went wrong.

Where this work sits

Judged by model class, there’s nothing new here. The regressors are standard ones, the counters are the standard ones, and the efficiency factor is a CPI stack of the coarsest kind. What’s different is that the project measures itself. The studies above give one accuracy figure for the whole pipeline, and the ladder below asks how much of that figure each rung earned, the rungs costing more to measure as you climb.

One recent paper is close to this one in method. Orteu Aubach and colleagues (2026) take runs that each carried a different counter set and merge their traces, which lets them cover more counters than the hardware can read in one go, and they find the merged counters stay accurate enough, with how far that holds depending on the code. I reached the same tactic on my own, and I also measured what merging badly costs.

The four questions I set out to answer

My feasibility study wanted a lot from one fitted relationship. It had to hold for codes it hadn’t trained on, survive being carried to another machine, and reach configurations nobody had ever run. Those are three different demands, and a model can pass the first while failing the other two.

• RQ1 - Can hardware counters predict an application’s runtime well enough to be useful, when the model is tested on applications it never saw in training?
• RQ2 - What does each layer of measurement add, from job metadata that comes free, through one counter, to a full profiling campaign over many runs?
• RQ3 - Does a relationship found on one machine carry over to a second?
• RQ4 - Can the method predict a configuration nobody has run?

The proposal didn’t have RQ2. I put it in partway through, once I saw that the accuracy number I was about to report couldn’t say which part of the measuring had earned it.

The machines

I profiled everything on ARCHER2, and then once more on Cirrus.

Partway through the project Cirrus was swapped for a Turin system. PAPI runs on the new hardware, so what the feasibility study had found, that Cirrus had no usable counters, stopped being true, and that made the cross-platform work possible. A 288-core node and a 128-core node cannot be lined up configuration for configuration, though, and my first try at it was wrong.

Everything runs on single-node by design. The thing I wanted was the link between runtime and microarchitectural behaviour. Going multi-node would bring in the interconnect, and it would add nothing to a counter signature that’s gathered rank by rank in any case.

The eight applications

The five primary codes

STREAM is the synthetic kernel everyone uses for memory bandwidth. It runs four vector operations across three arrays, does almost no floating-point work and reuses nothing, and that puts it in the Roofline plane’s bandwidth-bound corner. I built it with the three arrays adding up to 1.9 GB, well past the last-level cache of the EPYC 7742, so nothing it touches can be served from that cache.

HPL uses LU factorisation to solve a dense linear system. It’s the compute-bound corner, the opposite of STREAM. Linking against Cray LibSci makes the inner kernel a DGEMM tuned by the vendor instead of the plain reference BLAS. On 128 cores it retires about 4.7 × 10¹¹ floating-point operations, against STREAM’s 5.3 × 10⁸, which is a ratio of nearly 900.

HPCG is the conjugate-gradient benchmark on sparse matrices, brought in on purpose to balance HPL. The solve is multigrid-preconditioned, and what fills its time is irregular access to memory, halo exchange and short vector lengths. It’s the only code here that is both memory-bound and communication-bound, and its runs, at 20.8 to 155.5 s, are the longest. It also supplies more rows than any other code.

GROMACS is a molecular-dynamics code used in production, and it’s the one full application here whose insides are heterogeneous. Its short-range kernels are vectorised by hand and compute-bound, its long-range part (PME) is held back by communication and by Fourier transforms, and every so often it rebuilds its neighbour list. At 145 to 215 s a run it’s the dearest code in the campaign, so it has the thinnest sampling, 26 rows. The input is a ribosome system of 2,136,412 atoms.

OpenFOAM I ran on the motorBike tutorial, under simpleFoam. It earns its place by being heavy on C++, heavy on indirection and unfriendly to the cache, in a way no other code here is. It’s also the only application I profiled dynamically.

The three proxies

Three Mantevo/ECP proxy applications came in to fill patterns the first five missed, and to lift the count of distinct applications from five to eight.

miniFE is a mini-application for implicit finite elements. It’s the cheapest code here (0.17 to 7.0 s), and its floating-point density is by far the lowest. Moving data and integer address arithmetic set its runtime, with arithmetic throughput barely in the picture.

LULESH stands in for shock hydrodynamics codes, and its mesh is Lagrangian. It’s the one code that needs $n^3$ ranks, so the sweep was 1, 8, 27 and 64 cores instead of the powers of two I used everywhere else.

CoMD stands in for classical molecular dynamics. It’s here to put a second MD signature beside GROMACS, far cheaper than GROMACS’s. It’s the same family of algorithms with two orders of magnitude less work in a run.

These eight codes are not eight independent draws. miniFE and HPCG are both iterative sparse solvers on regular grids. CoMD and GROMACS are both molecular dynamics. LULESH and OpenFOAM are both solvers on unstructured meshes, explicit or segregated. In effect there are nearer five distinct behavioural families here than eight.

The code I could not profile

CP2K I tried and gave up on. Internally it calls MPI_Init_thread in a way that clashes with the dynamic instrumentation in CrayPat, and the module the site installed can’t be rebuilt for static instrumentation instead. I keep it on the record as a limit of the method.

BabelStream I looked at and left out, for a different reason. Its memory-bandwidth signature is the same as STREAM’s, so it would have brought more rows and no new kernel.

Five counters at a time

Five hardware counters is all the EPYC 7742 will read at once. Request a sixth and the run stops with an error, so there’s no quiet switch to multiplexing.

Multiplexing was there to use and I chose not to. Every configuration got five runs, each carrying a different one of five counter sets, taken in turn.

The five sets aren’t fully separate. Four of them include the cycle count, since the other counters in that run are divided by it to put them on a common footing. Between them the sets reach twenty counters on ARCHER2. Cirrus has eleven of the twenty, and those eleven are the whole basis for every result that crosses from one machine to the other.

Energy and L3 cost nothing extra. Each has registers of its own, so they’re captured in all five runs. PAPI’s L3 presets don’t work on this processor, so the only L3 numbers I have come from the Zen uncore component.

Those five runs give me five wall-clock times per configuration too, but they aren’t replicates. Here each run had a different counter set switched on, so each may have slowed the code by a different amount, and the spread across the five is a mix of ordinary run-to-run noise and the drag from the instrumentation. Repeating the runs with no counters at all would have pulled the two apart. I never did that.

That spread caps how fine a difference any result here can make out. I took, for each of the 191 configurations, its slowest wall time divided by its fastest. That ratio has a median of 1.268, so for a typical configuration the slowest of its five runs was a little over a quarter longer than the fastest. Its ninetieth percentile is 4.576, and the worst is 32.9.

The bad ratios sit mostly in the sub-second runs. Every launch costs a few hundred milliseconds regardless, and for a run that short the launch is a big share of the whole, and by about ten seconds of runtime the ratio has come down close to 1. Everything I measure later moves the error factor by something of order 0.01, so the biggest uncertainty in the campaign is this spread.

The sweep

Three things change across the sweep. Core count goes 1, 2, 4, 8, 16, 32, 64 and 128, and on Cirrus there’s a 288 as well. Problem size takes up to four values for each application. The third is the MPI-to-OpenMP split, which for HPCG and miniFE I change while holding the core count fixed. Together those give 191 distinct configurations.

Problem size wasn’t in the sweep to begin with, and putting it in cured a real fault in the design. Originally every application had exactly one size, which meant the identity of the application and the size of its problem were one and the same thing, and a model had no way to tell them apart. It could have got a good score by learning to recognise eight fixed operating points, with nothing learnt about the hardware at all. GROMACS and OpenFOAM are still stuck at a single size each, so that fault is still partly present in the results.

The efficiency factor

CrayPat gives counters for rank 0 and no other rank, so each counter in this piece is what one MPI rank saw of the job, never a total across all of them. Rank 0’s cycle count and the wall time move together very tightly. Take logarithms of both across all 191 merged configurations and their correlation comes out at $r = 0.993$. Yet dividing cycles by the peak clock doesn’t give you the runtime. Work out the clock that $C/t$ implies and you get 1.98 GHz for GROMACS, a code that never lets rank 0 idle, and for STREAM on many cores it falls well under 1 GHz, since there rank 0 mostly sits waiting on memory.

So I write runtime as

$$t = \frac{C}{f\,\eta}, \qquad \eta = \frac{C/f}{t} \in (0,1]$$

Here $C$ is rank 0’s cycle count, $f$ is a fixed reference clock chosen by convention, and $\eta$ is an efficiency factor with no units that takes in stalls, load imbalance and communication. On ARCHER2 the median of $\eta$ is 0.807.

Physically, $\eta$ is the fraction of the elapsed time in which rank 0 was retiring cycles at the reference rate. $C/f$ is the time the run would have needed if the chip had ticked at $f$ from start to finish with rank 0 never once held up, and $t$ is the time it did need.

So $\eta = 0.8$ says that reference-rate cycles explain four fifths of the elapsed time, and the last fifth is somewhere the cycle count is blind to. That fifth might be rank 0 stuck inside an MPI call, or the clock dropping below its nominal speed, or the time it takes a process to start.

The equation is an identity. $\eta$ is defined as $(C/f)/t$, which makes $t = C/(f\eta)$ exact by construction.

It also divides the problem into a big easy piece and a small hard piece. $C$ alone carries the size of the runtime, whether that’s two hundred seconds or a tenth of one, and since $C$ is measured nothing has to guess it. What’s left to predict is $\eta$, a unitless number that stays within a narrow band. So I model runtime through $\eta$. The models get a target that’s bounded and free of scale, in place of one that spans four orders of magnitude.

The choice of $f$ needs to be exact, because $\eta$ inherits whatever scale it sets. On ARCHER2 the EPYC 7742 holds a fixed 2.25 GHz. The EPYC 9825 on Cirrus doesn’t hold one clock, so I take its 3.71 GHz maximum boost, and the working ratio between the two machines is $f_{\text{cir}}/f_{\text{a2}} = 1.649$.

That makes $f$ a reference chosen by convention and never a speed the chip was measured at, and $\eta$ becomes a relative efficiency whose absolute level depends on that convention. Had I picked another reference for Cirrus, every Cirrus $\eta$ would shift by one constant factor. Each comparison that follows is either inside one machine, where the convention drops out, or about the spread of $\log_{10}\eta$, and multiplying by a constant leaves a spread alone.

One further decision, and I hold to it throughout, is that the models predict $\log_{10}\eta$, never $\log_{10} t$. With $\eta$ as the target the magnitude drops out, and the cycle count gives the magnitude for nothing, so what the model adds has to come from the counters. Fitting runtime directly did worse than a constant would have, and one linear model answered 6 milliseconds for a run that took 6.1 seconds.

How I decided whether a model was any good

My validation is leave-one-application-out. I fit on seven applications, score on the one left out, and go round eight times so that each application takes its turn as the test.

The standard way to score a model holds back a random share of the rows, and here that random split is precisely the wrong choice. It lets STREAM at 32 cores into training while STREAM at 64 cores is in test, and those two rows are almost the same row. They’re one code on one problem, a single step apart on an axis where the behaviour drifts slowly and in a predictable way. Give a model STREAM at 16, 32 and 128 cores and it can land STREAM at 64 close to right just by picking a value in between. The score would call that a success, though the model knows nothing of the link between counters and efficiency.

Leaving out the whole application removes that shortcut. When STREAM is under test, training holds no STREAM row at all, so there’s nothing to fill in between, and the model must get to the held-out code from seven others whose algorithms, bottlenecks and counter signatures are all different.

I report error as a symmetric multiplicative factor.

$$\varepsilon = \max\!\left(\frac{\hat t}{t},\, \frac{t}{\hat t}\right) \ge 1$$

Taking the bigger of the two ratios makes an error in either direction show up as a figure above 1. Two other measures would have served less well. Runtimes go from 0.1 to 215 seconds, so an error counted in seconds would come almost entirely from the two slowest codes. A signed percentage, the other option, treats the two directions of error differently, because under-prediction can only reach −100% to 0, and over-prediction has no limit.

I give the median and the ninetieth percentile both, since they often point at different models. Dropping the sign of the error does throw information away, though keeping it wouldn’t change the comparisons, because one metric scores every model. A signed version would add one thing these results can’t say, whether the residuals lean towards optimism or towards pessimism. In principle $\eta$ can’t go above 1, so there’s a built-in reason to expect a lean.

Anyone repeating this work could add a signed variant cheaply. I can’t rebuild it from what I have, since forming the error factor throws the sign away.

The statistics

All the comparisons below are paired. Both methods predict every configuration, the difference between them is taken one configuration at a time, and so a configuration that’s hard for everyone doesn’t disturb the comparison. On those differences I use three statistical tools, and I picked each one to avoid leaning on an assumption the data can’t bear.

The Wilcoxon signed-rank test produces the $p$-value. I picked it because error factors can’t go below 1 and can go up without limit, and because one freak configuration shouldn’t decide the outcome.

The Holm correction handles the fact that I run many comparisons, and it holds up when the comparisons are dependent. Mine are, since every one shares the same baseline.

The Hodges-Lehmann pseudomedian gives me the direction and the size of an effect. It’s the location statistic that underlies the signed-rank test, and its direction and the $p$-value can never disagree. An earlier round of this analysis took its directions from the gap between two medians computed separately, and seven of the twenty-eight were the wrong way round.

Numbers in brackets are percentile bootstrap 95% confidence intervals. The effect size $r_{rb}$ is the matched-pairs rank-biserial correlation, and a negative value means the model beat its baseline.

The models

Three estimators appear below. There’s a random forest, a gradient-boosted ensemble and a multi-layer perceptron. Only one distinction between them matters, which is the shape of function each one can draw, and by that they form two families. The forest and the boosted ensemble both consist of trees that split along one axis at a time, so all either can draw is a staircase of flat steps. The perceptron draws a smooth curved surface.

So gradient boosting is a second tree model. When all three come in within 0.02 of one another, the perceptron agreeing counts for much more than the second tree model agreeing.

The job is the same narrow one for all three. In go twenty-one numbers describing a configuration, out comes $\log_{10}\eta$. The runtime is then computed as $\hat t = C/(f\hat\eta)$, a step with no model in it.

The forest here averages 400 trees, each grown on its own bootstrap resample. The feature that dominates at the top of these trees is the analytic time, with a question such as whether it’s under 0.45 seconds. A leaf’s value is an average of training targets, so a forest can never go outside the range of $\log_{10}\eta$ that it was trained on. Give it a configuration past anything it has seen and it returns the nearest step it knows. That makes it the first thing I suspect when, later, I ask the models about core counts beyond the largest in training.

The random forest. In the upper half, one feature vector goes to every one of the 400 trees, and because each tree grew on a bootstrap resample of its own, they don’t agree. The 400 answers are averaged, and C/f divided by the resulting η̂ is the predicted runtime. Below, a single tree, in which a configuration answers yes/no questions one after another until it lands in a shaded leaf. The thresholds and the leaf values are made up for the drawing. The root feature really is the one that dominates in practice, and its direction is physical, because a short run carries a fixed overhead and so a low efficiency factor.

The perceptron takes weighted sums of all the features at once, through two hidden layers, and outputs $\log_{10}\eta$. A network this size has more than a thousand fitted weights and only 191 rows to learn them from, so a penalty on its weights holds it back.

each line carries one fitted weight · information flows left to right

The multi-layer perceptron from input to output. The top row is the preprocessing that every model shares. Gaps are filled with the median, the values are compressed with a signed log, then rescaled to a mean of zero and a standard deviation of one. Only a handful of the 21, 32 and 16 units are drawn. For the cross-platform work the input layer shrinks to 14 features and nothing else changes.

The forest and the MLP are both in the project to contrast two functional forms. Suppose the real link from the counters to $\eta$ were smooth, and suppose the forest were quietly losing accuracy since a staircase is all it can build. Then the MLP would show that up by pulling clearly ahead. In practice every family comes in within 0.02 of the others, and that null result lets me claim that how you measure matters more than which model you fit.

---

Part II. Building it

Getting eight codes to run under instrumentation

Everything here was built with the Cray toolchain, then instrumented for CrayPat. This toolchain gives you two ways of doing that, and the code you’re building decides which one you get.

pat_build rewrites the binary on disk before it ever runs, and you get the richest data that way, but only from a binary that was compiled while the perftools module was loaded. The site’s GROMACS module wasn’t, so I built GROMACS from source. pat_run attaches to the binary at launch instead, so it needs no rebuild, and that’s how I handled OpenFOAM. It isn’t a fallback you can lean on for everything, since CP2K fails under it.

Each Slurm job I submit takes one configuration and runs it five times, and between runs it re-exports the counter selection and starts a fresh experiment directory. Two practical limits shaped the harness I built. Under the standard QoS, one user can have only 64 jobs waiting and 16 running, and bulk submission past that limit fails silently. So I wrote a drip-feeder, a script that works through a worklist, submits as slots free up and keeps a record of what it has sent, and the campaign can be stopped and picked up again.

The second limit is that you can only validate a counter set on a compute node, since the failures don’t show up on the login node. One of my five counter sets turned out to be invalid. The TLB-miss event in it is a derived event that quietly takes up more than one counter register, and the login node gave no hint of that.

Getting all eight codes running under instrumentation, at eight core counts each, took more work than the instrumenting itself. GROMACS refused to configure until I switched off three test targets in CMake that had nothing to do with it. For OpenFOAM I had to bring in scotch, replacing the hierarchical grid the tutorial uses. Without it the case runs only at six ranks, because the tutorial hard-codes that number, and after that it still needed two further corrections to the initial conditions it starts from. HPL arranges its ranks as a $P$ by $Q$ grid and insists that $P \times Q$ match the rank count exactly, so for every configuration I generated a grid as close to square as I could.

Nine misleading failures in the toolchain

A large share of this project went on making the measurement happen at all, and a much smaller share on modelling. The failures come with the method, so anyone who builds a counter dataset on this toolchain is going to hit most of them himself. I put a node-hour price on counter collection further down, but node-hours are only the part of the cost you can see.

Six of the nine messages point somewhere other than the fault, and three print nothing at all. A fourth case was a stray /tmp/inspect.py left behind by earlier testing, which shadowed the standard library’s module of that name. It threw the very error message an unrelated NumPy problem was throwing, and it outlived that problem, so my fix of the true cause appeared not to work. You edit, the output looks plausible, and you discover much later that it was wrong.

The same mechanism was behind the two analysis defects I caught while the project was running. One was a rigged baseline and the other a pseudo-replicated feature matrix, and both had produced numbers that looked entirely reasonable. What caught the silent failures was an assertion about the shape of the output.

Some things were never there to be had. Rome has fewer PAPI presets than the documentation suggests, and several cache-miss presets simply don’t exist, so the L3 numbers come from the Zen uncore component instead. ARCHER2 has its network and node-power counters disabled, so communication cost has to be found in the per-function MPI timings.

What counts as one observation

pat_report prints each counter value inside formatted text, so the values have to be parsed back out. The regular expression I wrote to match the value lines also matched the legend at the bottom of the report, which gave me an L3 miss count of three until I noticed.

At first every one of the five runs, one per counter set, got a row of its own, which gave 815 samples. That’s wrong in two ways.

The first is pseudo-replication. Those five runs are one configuration executed five times, and all that separates them is run-to-run noise plus which counters I asked for. Counting them as five observations claims five times as much evidence as the experiment had.

The second is that a run measures nothing beyond its own five counters, so most of each row is empty. Three of the features were present on 0% of the rows, IPC on 50%, and the table as a whole was 44.1% full.

Each gap was filled by median imputation, and a column that’s mostly missing becomes a column that’s mostly one number repeated. That changes how the forest hands out feature importance.

Merging to a single row per configuration, where the five sets’ counters are pooled and the runtime is the median of the five, gives 191 configurations at 97.4% populated. Nothing new was measured, and the coverage still went up. Three features that no unmerged row had ever observed end up with 0.078 of the importance mass after the merge.

Merging column by column hits $\eta$ directly. Counter set C has no cycle count in it, so the cycle count’s median is worked out over just the four runs with one, whereas the median runtime comes from all five. The two medians can be drawn from different runs, and $\eta$ becomes a ratio of two medians over different samples where the median of a ratio is what’s wanted. That shifts $\eta$ by more than 5% on 20 of the 191 configurations, and on one of them (HPL at size 2000 on 32 cores) the result is $\eta = 1.040$ when the paired computation gives 0.466.

The correct construction is to work out $\eta$ per run and merge afterwards, which removes the artefact. I didn’t do that.

The features

There are twenty-one features here, all derived, and a raw counter never goes in by itself. Every feature is a ratio of two counters, so instructions per cycle instead of instructions, and L2 misses per instruction instead of L2 misses.

Core, rank and thread counts go in as base-two logarithms, because from one configuration to the next they double. One magnitude term is kept on purpose, and that’s the logarithm of the analytic time $C/f$. The other raw magnitudes stay out, since several of them let a model pick up the size of the problem instead of its behaviour. $\eta$ depends heavily on run length, because a short run pays exactly the fixed startup cost a long run pays, and therefore looks less efficient. Deny the forest that input and it splits on something correlated with it instead.

Keeping it is consistent with the ladder. That feature is ranked first of the twenty-one by both importance measures, at 0.375 by impurity and at 0.0150 by held-out permutation, against 0.0059 for the next feature down. So the other twenty features are judged against a model with the cycle count already in it, inside the fit and outside it alike.

What the campaign cost

The ARCHER2 sweep came to 191 configurations, five runs each, plus the validation runs and the ones that failed. Altogether that was roughly 1000 profiled executions and about 210 node-hours, and Cirrus adds about 90 more from its campaigns.

Of those hours, the applications themselves used only about six. The rest went on job startup, the granularity of queue slots, post-processing, and the attempts that failed or had to be repeated. That’s one reason the cost argument in this piece is about how much wall-clock time you commit, which isn’t the same as compute. The twenty features beyond the cycle count cost four fifths of that budget.

Three hundred node-hours isn’t a lot on its own. What the cost argument depends on is how the total scales. Those hours bought eight applications on two machines, so the real unit is the cost per application-and-machine pair, and that multiplies with every axis you add. To cover the thirty-odd codes that a centre of middling size runs, on three machines, at this density, you’d be a full order of magnitude past this. Further down, the learning curve says that more data still helps the model along the application axis.

Model configuration

The random forests have 400 trees and no leaf smaller than two samples. Gradient boosting has 300 estimators at depth 3. Its learning rate is 0.05, and its quantile objective is matched to the error metric. The MLP tried hidden layers of {(16), (32,16), (64,32)} against an L2 penalty of {0.1, 1, 3, 10}, which makes twelve candidate settings in all.

I picked these hyperparameters by nested selection. Within each outer fold of leave-one-application-out, a second leave-one-application-out loop goes round the seven training applications by themselves. Every candidate setting is scored on the applications that inner loop holds back, and the winning setting is then refitted on all seven before it runs on the eighth. The application held out has no say in the choice.

A check before any modelling

Before trusting any of it, I checked that what the campaign measured was what I built it to measure. A plot of median runtime against core count, one panel per application, shows three things. The first is that each code scales as its algorithm says it should. HPCG sizes its problem per rank, so its ARCHER2 median goes up, from 31.6 s on one core to 46.8 on 128, and the extra time is what a growing halo exchange costs under weak scaling. OpenFOAM by contrast comes cleanly down from 132.7 s to 5.0.

The second is a shaded band marking the runs that took less than a second, and it shows what happened when Cirrus was run at ARCHER2’s problem sizes. For CoMD, HPL, LULESH and miniFE, somewhere between a quarter and two thirds of the Cirrus runs sit inside that band, and miniFE reaches 65%. HPCG never enters it, and HPCG is the one code sized per rank. The third is that package energy per rank falls monotonically with core count in every application.

One panel has no scaling signal in it. The STREAM runs from the expanded campaign sit at 3.3 seconds whatever the core count, because the launch was single-threaded. This checks the instrumentation and the parsing, and it passed.

Each code scales as its algorithm says it should. The shaded band marks the sub-second regime. The orange line that drops into it shows what happened when ARCHER2’s problem sizes were copied onto a 288-core node (65% of miniFE’s Cirrus configurations land there). HPCG is the one code sized per rank, and it never enters the band.

---

Part III. Results

How to read these numbers

Every result from here on is an error factor, the symmetric ratio from the section on scoring. It comes out as 1.00 when the prediction lands exactly, as 1.10 when it missed by ten per cent either way, and at 2.00 the prediction was double the truth or half of it. The ratio can’t go under 1.00. With every median I also report p90.

A pooled median treats the 191 configurations as one big set and reads the median off that. Each configuration gets one vote. An application with many configurations therefore gets many votes, and HPCG by itself holds 45 of them.

A macro-average reverses the order. First I find the median for each application on its own, which leaves me with eight numbers, and then I take the average of those eight. Each application gets one vote, whatever the number of configurations it brought.

Since some applications have far more configurations than others, the two answers can be different sizes, and once in a while they point different ways. Whenever that happens I show both.

• the constant - a single efficiency factor fitted once and used for every application. On ARCHER2 it comes out at 1.065. I also call it the zero-parameter baseline, since you get it by throwing away all twenty-one features and no coefficients are fitted to anything. It takes no information from the run it’s predicting, so any model that claims its profiling was worth the cost has to beat this first.
• the same-fold constant - the same idea, but computed on the same configurations that a transferred model gets tested on. For work across machines this is the fair comparison, and it comes to a different number.
• the trend fit - a straight line with two parameters, fitted to the cheap small-scale runs of the target application itself.
• the clock-ratio rescale - taking one machine’s runtime and scaling it to the other machine by the ratio of their clock speeds, with nothing else involved.
• the circular ceiling - the best that can be done when the model is allowed a counter read from the target run itself. That’s out of reach in practice, so it caps what the rest can do.

The resolution of these numbers

Two kinds of variation set a floor on how finely anything here can be read. The first kind is implementation. Three evaluation scripts, each one written from scratch without reference to the others, give the random forest with 21 features a score of 1.0496, 1.054 and 1.0554 on the same 191 rows. The code alone, then, moves the answer by 0.0030.

The second kind is the seed. Ten seeds on one implementation give scores from 1.0497 to 1.0655, which is a spread of 0.0158. That’s five times the implementation spread, so it’s the seed that sets the limit. Any difference under about 0.016 counts as noise.

That’s a higher bar than two of the differences I’ll report. The gap between the ladder’s top two rungs is 0.011, and the gap between the random forest and the constant is also 0.011. As point estimates, neither one clears it. As paired comparisons they still say something, because pairing fixes the seed and compares the error on each configuration in turn, instead of setting two independently fitted medians against each other. The conclusions here lean on the paired tests and on the macro-average, with the pooled point estimates as the weaker evidence.

Asking a better question

There was a stretch of the project where my pipeline produced one number and I was content with it. The median error factor was about 1.05, and a typical prediction came within roughly 5% of the measured runtime, on an application that was new to the model. It sounded like a respectable result and I’d started writing it up. Then I started to worry about what the number was telling me. My pipeline took in several kinds of measurement (free job metadata, a single counter, and five sets of counters taken over five separate runs), and they cost very different amounts. If the cheapest input was doing most of the work, the expensive campaign was a waste, and if the expensive input was doing it, the campaign earned its place. The 1.05 can’t tell you which, and someone deciding whether to run this profiling needs to know.

So the question changed, from how accurate the pipeline was to where in the measurement the accuracy was coming from. To answer it I built a measurement ladder, which is my name for a set of predictors stacked as rungs, where each rung uses more measurement, strictly more, than the rung beneath it. Each rung sees the identical 191 configurations, the identical folds and the identical metric. Between one rung and the next, the only change is the additional measurement the higher rung receives, so whatever improvement appears can be credited to that measurement and to nothing else.

The measurement ladder

• Rung 0 - the training median. This gives every configuration of the held-out application the same runtime, the median runtime across the seven applications in training, and it takes no measurement of the target at all. It sets the score for knowing nothing about the target, and every rung above is reported as a gain over it.
• Rung 1 - free job metadata. A random forest fed the three numbers that exist before a job is even submitted (core count, rank count and thread count). There’s no profiling and no counter, and the job never runs. Together with rung 0, it’s the only rung that predicts in the way the feasibility study intended, since it could be scored on a job that nobody has run.
• Rung 2 - the cycle count alone. This computes $C/f$ and goes no further, which is the same as assuming $\eta = 1$. It’s one counter, but that one counter is read off an instrumented run of the very configuration being predicted.
• Rung 3 - cycle count plus one fitted scalar. Take rung 2 and divide by the median $\eta$ across the training applications. That’s a single number, fitted once and used for everything.
• Rung 4 - the full campaign. Take rung 2 and divide by an $\eta$ that a random forest predicts out of the whole set of twenty-one counter-derived features. The price is that the target configuration has to be run five times under instrumentation.

The table drawn as a chart. The axis is logarithmic, so equal distances mean equal ratios. One cycle count improves on free metadata by a factor of six, and the remaining rungs together come to a factor of 1.17.

From rung 0 to rung 4 the error factor improves by 6.533 in total. Of that total, 6.588 comes from the cycle count on its own, a shade over 100%. A fitted constant $\eta$ contributes 2.6%, and the other twenty counter-derived features contribute 0.2%.

A share above 100% is possible because rung 1’s contribution is negative. Free job metadata does worse than nothing here. Under leave-one-application-out, all it gets is “128 cores, 128 ranks, 1 thread”, and there’s nothing in those three numbers to separate STREAM from GROMACS. Ask a model to put a code it hasn’t seen onto a scaling curve using only that, and it does worse than a model that refuses to guess.

Where the definitional part ends

Part of that result is true by definition and part of it is an empirical finding, and it’s easy to take the 100.8% for the project’s discovery when it’s nearer to an accounting identity. Rung 2 isn’t a model in any real sense. The equation $t = C/(f\eta)$ is the definition of $\eta$, so putting $\eta = 1$ is only the definition again, and predicting $C/f$ is a measurement with its terms rearranged. Runtime and cycle count are almost one quantity. The correlation between their logarithms is $r = 0.993$, and 98.6% of the variation in the log of runtime is already accounted for by the cycle count before any model touches it. Rung 3 is true by definition in the same way.

What the ladder measures that isn’t true by definition is how little is left to predict. A constant baseline of 1.065 says a typical configuration’s $\eta$ lies about 6.5% away from the median. That 6.5% band is the entire space a model has to work in. Within that band, the twenty features pick up a slice that’s big enough to detect.

Macro-averaging sharpens the counters’ case

With 191 rows pooled, HPCG’s 45 rows give it a quarter of the say. Give every application the same weight and rung 4 scores 1.058 against rung 3’s 1.111, a gap five times what the pooled numbers show.

The counters help most on the applications the constant handles worst, miniFE and LULESH. On HPCG, STREAM and OpenFOAM they do harm. Everything above rung 1 rests on one number, the cycle count, and that number only exists once you’ve run the very job whose runtime you’re predicting.

Green arrows are the applications where the counters help and red arrows the ones where they hurt. The two hardest cases hold most of the gain.

Do the counters justify their cost?

Twenty extra features buy 0.2% of typical accuracy. To see whether the five-set profiling campaign was worth running, I added the feature groups to a random forest one at a time and tested each step against the constant.

Only the last bar drops under the dashed baseline. The second bar goes in the wrong direction. Putting the analytic cycle term alongside the job configuration leaves the model a little worse off.

Taken at face value, only the full five-set rotation beats a zero-parameter constant. Free metadata is significantly worse than a guess at the training median. Adding one counter doesn’t save it, and one whole counter set is still no better than a guess.

There’s a second way to read the table that’s kinder to the counters. Measure against free metadata instead of the constant, and every counter rung adds signal that’s real. The full set of features wins 152 of the 191 paired comparisons, and the effect is large ($r_{rb} = -0.69$, $p = 6.3 \times 10^{-16}$). Even one counter, the analytic cycle count, is a significant step up on that reading.

So the campaign does pay for itself, but what it buys is small and none of it arrives until the end. Most of the gain is in the tail. The median goes from 1.065 to 1.050, about one percentage point. The p90 drops from 1.532 to 1.390. For the typical case a constant does the job. If you’re sizing a queue slot you size by the p90, and there the models take roughly 17% off the worst-case error.

The pipeline explains runtime but does not predict it

From rung 2 upwards, including the headline pipeline sitting at rung 4, runtime is put back together as $\hat t = C/(f\hat\eta)$, and what the models provide is $\hat\eta$. Nothing predicts $C$, which comes straight from a measurement.

Nor is it measured on something that merely resembles the target. The analytic term wants the cycle count of the exact configuration whose runtime it’s reporting, and by that I mean the same application, the same problem size, the same core count and the same machine. So the cycle count only exists once exactly the configuration being “predicted” has been run under instrumentation and finished.

From there the circularity is immediate. You can’t work out $C/(f\hat\eta)$ unless the job has been run, and a job that has run has already been timed, since one report from one execution prints both the wall time and the cycle count, so no practitioner ever holds $C$ without $t$. That makes the 1.054 figure depend on the thing being predicted having already been run, and nobody can take it as a prediction for a run that hasn’t happened yet.

How much must the profiled run resemble the run of interest?

The circularity matters most when the profiled run and the predicted run are one and the same, since there’s no way to get $C$ for a configuration no one has run, and $C$ holds essentially the whole of the accuracy. That’s one use of the method, and there are weaker ones.

A different configuration of the same code can be predicted. The held-out application is never shown to the models, and restricting the training to small core counts loses essentially nothing, so a profile taken at one scale does say something about another scale. The one thing that does it better is the cheap trend fit.

A different application, predicted from a profile of a different code on the same machine, is a bigger claim, and it’s exactly what leave-one-application-out puts to the test. On five of the eight applications the counters beat the zero-parameter constant, so some signal does carry across applications. It’s small, though, and on the other three it goes into reverse.

This does not make the result empty

It turns it into a different result. What comes out of the pipeline is a measurement broken into its parts. The gap from the ideal time to the measured wall time gets explained in terms of quantities measured inside the processor, and that explanation holds up on applications the models never saw.

What brings the circularity out is setting rung 1 beside rung 2 under one protocol and seeing that the entire gain lands at the step where measurement from the target run first comes in. If I’d reported only the headline 1.054, nobody reading the write-up would have seen it. The figure is computed the way you’d compute a prediction accuracy and validated on applications that were never in training, so nothing in how it’s presented gives the circularity away. I suspect the same thing goes unnoticed in some published work.

My p-values are too optimistic

Each $p$-value in this article is smaller than it should be. The configurations come from eight applications at eight core counts and up to four sizes, and they lie along smooth scaling curves. Once you know the forest’s result for miniFE at 32 cores, you know most of its result for miniFE at 64.

This is the same non-independence I relied on when I rejected a random train/test split. I can’t argue it both ways. If configurations that sit next to each other are too alike to be split between training and test, they’re too alike to count as separate pieces of evidence as well. So the effective sample size is closer to eight than to 191.

The fix is to run the comparison again at the unit that really is independent, the application, and to accept how much weaker eight observations are as evidence. At the level of applications the random forest wins against the constant in five cases out of eight. The right test for that is a sign test, and it gives $p \approx 0.36$. The bootstrap intervals say the same, since the forest’s [1.037, 1.074] and the constant’s [1.052, 1.081] overlap over most of their length.

Measurement choice dominates model choice

Impurity importance on an unmerged matrix built by counter rotation rises with the number of times a counter was recorded, and that number says nothing about how much the counter explains. I tested this with a control. The corrected data was given back the old pattern of which counter was present in which run, and importance was then compared with how often each counter had been seen. The two agree at $\rho = +0.798$. Merge the data so each configuration is a single row and the match disappears ($\rho = +0.080$).

All of them land inside a 0.02 band. The estimator you choose makes no difference that these numbers can see. Which one comes out on top even changes with the statistic, since the two tree models tie on the median while the ensemble does best on the tail, and every confidence interval overlaps every other. The random forest does beat the constant, winning 125 of 191 configurations at $p = 2.2 \times 10^{-4}$, and the MLP chosen by nested selection can’t be separated from the forest. None of these gaps is bigger than the 0.016 noise floor, so I don’t rank the learned models against each other.

The MLP row is in that table for its shape, and it doesn’t pull ahead. Two families of model with no structure in common finish within 0.02 of each other and within 0.02 of a constant.

Three rounds of neural networks

Three separate rounds went into neural networks, and at no point did one of them keep up. The first round used plain multi-layer perceptrons. The second repeated the tuning correctly, once it was clear the first round’s protocol had leaked. For the third I took the recent recipes the tabular literature recommends for making neural networks behave on tables of numbers. Those are PLR numerical embeddings, robust scaling with smooth clipping, RealMLP, and deep ensembling.

All of them finished behind the constant, which has no parameters at all. The one the literature pushes hardest, PLR embeddings, finished last, at $p = 3 \times 10^{-6}$.

The dataset has 191 rows. The round three methods are built and tested on datasets bigger by two or three orders of magnitude. At this size the added capacity has nothing to learn from, and 62% of the target sits inside a narrow band.

Would more data have helped?

A learning curve answers this without collecting a single extra run. I fitted on a quarter of the training data, then half, then three quarters, then all of it. The test fold is never touched, so every one of the four points is scored on the same configurations and they compare directly. There are two things “more data” could mean here, and thinning the training set the two ways gives opposite answers.

Left, extra runs of the eight codes already present. The curve goes flat, which means this axis is saturated. Right, extra codes. The curve is still falling at its right end and hasn’t begun to level out.

Thin the training rows at random and the forest goes from 1.0871 to 1.0585, with the final quarter adding nothing the campaign can resolve. Thin the training applications instead and the curve is still on its way down at seven codes, from 1.1333 with two to 1.0560 with seven, and the 75% and 100% intervals are disjoint.

The same axis sets the floor under the method. The constant also gains from coverage, only more slowly, and a forest trained on two applications does worse than the scalar. Give it fewer than roughly four codes to train on and the counters hurt the prediction.

So what limits the method is the count of applications, and my budget went the wrong way round. Every code was run at eight core counts and at four sizes, and that bought density the model no longer used, while the axis on which more data still improved the model got eight. The headline number therefore sits on the steep section of this curve.

The baseline to beat is a straight line

Up to now one axis has always stayed fixed. Leave-one-application-out swaps the application out while every core count stays in view of the model. The feasibility study promised something that moves both axes at once, where the model learns from the other applications on few cores and has to predict the held-out one on many.

The fair opponent in that setting is what an engineer with an unfamiliar code would do without any model. Run it cheaply on a few cores, then follow its own trend upward. Concretely, you plot the log of runtime against the log of core count for a handful of cheap runs, put a straight line through the points (two parameters), and read the line at the core count you want. That uses no counters, no training data, no model, and nothing from any other application, and it costs next to nothing.

The trend fit beats every learned model at each of the three thresholds I tried. A threshold is the biggest core count shown to the models during training.

How much this proves needs some care, since the thresholds overlap in the configurations they use and can’t be pooled as independent evidence. Separately, at $T = 8$ the forest trails by +0.023, which isn’t significant ($p = 0.094$). At $T = 16$ it trails by +0.032 and that is significant ($p = 0.047$). At $T = 32$ it trails by +0.052, again not significant ($p = 0.212$). The sign is the same at every threshold and for every learned arm, and significance is reached at one threshold out of three.

Reading the table naively, you’d say the forest at 1.113 loses to the constant at 1.102. The paired test disagrees, because the forest wins 140 of the 240 individual comparisons at $p = 1.5 \times 10^{-5}$. So a constant is beaten by the models. The trend fit beats them, and it’s the cheaper method.

I first assumed the models were losing because they can’t extrapolate in scale. That assumption was wrong. A forest can’t predict outside the range it was trained on, so predicting 128 cores after seeing only 8 is where that limit ought to hurt most. Removing the limit is the test. Let the forest train on every scale instead of only the small ones and its error changes by a factor of 0.993 at $T = 8$, 1.003 at $T = 16$ and 1.034 at $T = 32$. That’s under 1% in two of the three, and it goes the wrong way in the last.

Transfer between machines

Comparing the two machines needs matched configurations, where each row on one machine has a partner on the other, with application, problem size and core count all identical, so that the machine is the only thing that differs. An earlier draft of this work said each machine had 191 of them, and that wasn’t true. GROMACS and OpenFOAM had never been run on Cirrus, and its 288-core node goes to core counts ARCHER2 has no way to reach. A proper join gives 166 configurations that are truly paired. That both sweeps came to 191 was chance, a matter of how far each had happened to run. Comparing the two unmatched sets would’ve meant comparing two different mixes of work and putting the gap down to the hardware.

The error also leaned one way. The 25 Cirrus rows without a partner are its widest runs and its shortest, with a median runtime of 1.04 s against 1.20 for the paired rows, and those are exactly the runs where fixed overhead takes over and $\eta$ collapses. Keeping them in made Cirrus seem the harder machine to predict, for reasons that had nothing to do with its microarchitecture.

With the pairs fixed, a model that stays on its own machine scores 1.071 on ARCHER2 and 1.125 on Cirrus. One that crosses scores 1.203 going to Cirrus and 1.157 coming back. Crossing machines costs accuracy outright.

Measured against the correct baseline, though, the crossed model still wins. For a prediction going from ARCHER2 to Cirrus, the correct baseline is the constant fitted on the same folds, which scores 1.2203. The within-platform constant Cirrus has on its own, 1.129, is the wrong baseline, because it’s fitted on Cirrus data that the crossed model never sees. Against the right one, the random forest’s 1.2025 is a significant improvement (Holm $p = 0.006$). In the other direction no model does better than the same-fold constant, which is 1.1511 there. The lopsidedness between the two directions is real, in that the model that transfers worse is the one trained on the machine whose $\eta$ spread is wider.

Inside one machine the error stays near 1.05. Each crossing comes out between 1.14 and 1.19, which is about three times the leftover error above a perfect prediction.

An MLP fitted on nothing but rescaled Cirrus data, then applied as it stands to ARCHER2 with only the fourteen features both machines share. Its median error is 1.142, with 85% of configurations inside 1.25× and 93% inside 1.5×, over four orders of magnitude and eight applications, none of which it had met at these problem sizes. No application is much worse than another, the per-application medians running from 1.09 to 1.19.

Its one real miss is the labelled point, CoMD at 128 cores, a run of 103 ms that the model under-predicts by 5.8×. That run sits deep in the region where fixed overhead dominates, so the mechanism behind the sub-second problem in the next section is also behind the only outlier this figure has.

Ask for an unseen application and a new machine together and the signal runs out. Going from ARCHER2 to Cirrus, the forest reaches 1.254 while the same-fold constant reaches 1.251, and after correction the two can’t be separated. There’s a caveat attached to every number from this setting. Each one is averaged across the six applications that exist on both machines, and the two absent ones are the production MD and CFD codes, the pair that resemble the proxy benchmarks least. So the hardest cell of the matrix gets scored on the applications that were easier, which makes its number optimistic.

Sub-second runs, not Zen 5

Earlier I said Cirrus only seems hard to predict because such a large share of its runs are over inside a second, and that Zen 5 isn’t the cause. That’s a statement about cause, so I tested it with an experiment. If run length is the cause, then making the runs longer and changing nothing else has to make Cirrus easier to predict.

So the whole Cirrus campaign was run again with problem sizes chosen for the wider node instead of copied across from ARCHER2. The machine, the codes and the toolchain were the same as before, and only the sizes moved.

Left, reusing ARCHER2’s problem sizes on a wider node put close to half the runs under one second. Right, the effect of that on the quantity the models are asked to predict. After rescaling the spread is smaller than ARCHER2’s own.

On the distribution the answer is clear. The spread of $\eta$ drops from 0.285 to 0.072, lower even than ARCHER2’s 0.128. The share of runs under a second drops from 47.6% to 16.7%.

There’s nothing about Cirrus that makes it harder to model. What was wrong was the fixed-size campaign, since keeping the sizes the same on a node with twice the cores guarantees that the runs get short.

Above one second the efficiency factor barely moves and a single number describes it well. Under one second it breaks down, because a fixed cost per run that the counted cycles never include is being spread across a shrinking amount of work.

So $\eta$ is set by how much work a run does, and not by the hardware. Whatever the machine, a benchmark campaign whose runs drop under about one second will end up with an efficiency distribution that no model can fit. Sizing the problem per rank, as HPCG does, keeps the runs out of that region.

What this does for the models is a weaker result than what it does for the distribution. With the fixed sizes on Cirrus, the forest’s pooled median is 1.1863, above the constant’s 1.1663, and yet the paired test has the forest in front ($p = 0.0402$). With the rescaled sizes the forest’s 1.0603 is nominally in front of 1.0617 and the difference is not significant ($p = 0.366$). The experiment settles that run length is the cause, and leaves the model-against-constant question open in both datasets, because rescaling made the job easier for every method, the parameter-free one included.

Pricing the circularity

To put a number on what the circularity costs, I built the situation that a procurement panel faces. The application has been fully profiled on the machine you already have (ARCHER2). The machine you’re thinking of buying (Cirrus) is known only from its datasheet, which gives its clock, its memory bandwidth, its core count and its processor generation. Not one measurement from Cirrus reaches the predictor, so its counters, its runtimes and its $\eta$ are all absent from it. Validation stays leave-one-application-out, so the predictor has to cope with an application it hasn’t seen and a machine nobody has measured, both at once.

The previous section was a different test. There a model moved between machines, and the target run was still measured, since the cycle count from Cirrus was there at prediction time. Here that measurement is gone altogether. This section asks whether you can predict a machine you’ve never run anything on, the question the feasibility study asked in the first place.

The two rows in italics aren’t contenders. The circular row uses the identical method with a single difference, which is that it gets the Cirrus cycle count back. The distance from it to the best row without circularity is therefore precisely what that one counter from the target run is worth.

The oracle row fits one constant speedup using the Cirrus data of the very application under test, something nobody could do in practice. It still shows how close you would get if someone handed you the correct speedup for each application and nothing more.

The best predictor without circularity reaches 1.556, where the circular ceiling is 1.129. Measured as excess error, which is the distance a prediction sits above a perfect 1.000, taking away the target-run counter moves the excess from 0.129 to 0.556, a factor of 4.3. Put the way a user would put it, with a single instrumented run on the target machine a typical prediction lands within about 13%, and without that run it’s off by about 56%.

How the learned predictors compare with plain arithmetic

That depends on how much weight each application gets. Over the pool of 166 configurations, the strongest learned predictor, at 1.556, is nominally in front of the clock rescale at 1.608, and the lead is not significant (94 wins out of 166, $p = 0.99$). HPCG runs that pool. It contributes 44 rows, and it’s the one application on which the clock rescale wins outright, 1.395 against 2.810. Give the six applications equal weight and the order flips, 1.653 against 1.990, with the learned predictor in front on five applications out of six and behind only on HPCG.

So the no to RQ3 is narrower than it first sounds. Weighted by row, the learned predictor fails to beat plain arithmetic significantly, and that comes down to how much of the pool one application owns. Weighted by application it usually wins, and by a lot, and the single code it fails on badly is the one whose cycle count is least like the rest.

The uncomfortable row

One constant speedup per application, a single number fitted to the application under test (so not something you could do for real) and needing no counters at all, gets a median of 1.130. That equals the circular ceiling’s 1.129, and in the tail it does better, with a p90 of 1.669 against 3.601. Run the whole counter pipeline on the target machine and you gain nothing beyond what one number per application gives. I don’t like that result, and it’s the plainest summary of the project’s findings.

Whatever signal in these counters survives the move between machines is coarse and belongs to the application as a whole, and none of it reaches down to the microarchitecture. The cycle counts show why. Were the cycle count a property of the work alone, identical whichever machine ran it, its ratio between the two machines would be near constant and projecting would come down to correcting for clock speed. It isn’t constant. The ratio has a median of 0.651, and per application the medians run from 0.378 for LULESH up to 1.666 for HPCG, a 4.4× spread between applications, with identical source code doing identical work. Nothing in the ARCHER2 counters tells a model about that spread.

The whole experiment sits under one structural limit. There is only one target machine, so by construction the four static specification features hold the same value in every row. So the experiment can’t tell whether ARCHER2 counters fail to project onto Cirrus or whether a static description of a machine simply never got a chance. Adding a third platform would tell the two apart.

---

Part IV. What it means

The overfitting warning, revisited

My supervisor’s first warning, given early, was overfitting, and he was right. I’d guarded against the row-by-row version from the start, since every fold held out a whole application and the model was scored on a code it had never trained on. What went wrong instead was quieter, and all of it was one fault in different forms, which is information about the test data reaching a decision it should not have influenced.

The folds leaked into the tuning. I set mine by watching the leave-one-application-out scores, where I tried architecture after architecture and kept whichever scored highest on the applications being held out. Those applications were meant to have no influence on anything, so the reported score is too good by whatever the choice took from them. The odd part is that the model got a little worse, because one setting chosen for all the folds at once loses more than the leak gains.

The table’s structure leaked. For each counter I added a column of ones and zeros saying whether that counter was missing from the row, expecting it to help the model tell an absent value from an ordinary one. It emerged that each application had its own pattern of missing counters, so together those columns were a one-hot application label, the one fact leave-one-application-out exists to hide. Overall accuracy barely changed. The fold that held out STREAM went from 1.096 to 1.552.

The same run counted five times. Five runs of one configuration went in as five observations, the pseudo-replication from earlier. The row count went up and the feature matrix got emptier at the same time, since each row carries only the few counters its own run asked for. The per-application results caught all three.

Answering the four questions

RQ1 - from counters alone, can you predict the runtime of an application the model has never seen? As the question is worded, no. In a narrower sense, yes. Every figure here reconstructs the runtime of a run whose cycle count is already measured, and that isn’t the same as predicting a run that hasn’t happened.

Inside that narrower sense, the margin is known. The random forest scores an error factor of 1.054 on the pooled median, so a typical prediction is about five percent out, and 1.058 on the macro-average. The zero-parameter constant scores 1.065 and 1.111 on the same two measures. Five applications gain a lot and three lose. Per configuration the difference is significant ($p = 1.4 \times 10^{-4}$). Per application, and the validation protocol itself regards applications as the independent unit, it comes out at five wins against three losses and isn’t significant ($p \approx 0.36$).

What I can stand behind is narrower. The direction is consistent, the size is material on most applications, and at the coarser unit nothing has been established.

RQ2 - which measurement is the accuracy coming from? On the ladder of five rungs, 100.8% of the total improvement is the cycle count’s alone, and the other twenty features together contribute 0.2% pooled. Part of that is built in, since the efficiency factor is defined from the cycle count. What the ladder tells you past the definition is the small share of the residual that the counters get back. Job metadata that costs nothing does worse than the constant. The full five-run campaign is the only rung that beats the constant. And one speedup fitted per application does as well as the whole pipeline.

RQ3 - does what was learnt on one machine hold on another? On transfer, a yes with conditions. On projection, no.

A transferred model is, in absolute terms, worse than one that stays at home, but from ARCHER2 to Cirrus it still beats its own fold’s constant ($p = 0.006$), and in the other direction it doesn’t. Whether the transferred model beats the clock-ratio rescale depends on the weighting. Projection fails because the cycle count isn’t the same on both machines, with per-application ratios that differ by a factor of four.

Cirrus seemed the harder machine to predict, and the cause lay in the design of the experiment. Resizing the problems to suit the wider node brought the spread of $\log_{10}\eta$ down fourfold and cut the runs under a second by two thirds.

RQ4 - can it predict a configuration that was never run? No. A line fitted through the target application’s own cheap runs, two parameters and nothing more, sits ahead of every model I trained at each of the three scale thresholds. The trained models do beat the constant. A practitioner wouldn’t use the constant though, they’d use the cheap fit, and the models lose to that. It isn’t a failure to stretch to bigger scales either, since training on the small scales only costs about nothing.

What a practitioner should do

When you have the code and want the runtime of a configuration you haven’t tried, do two or three small runs at a low core count and put a straight line through them. Nothing I trained beat that line, it needs neither counters nor a training set, and the allocation it costs is a few minutes.

When the code is profiled on your machine and you want its runtime on someone else’s, fit one speedup per application. That one number kept pace with the entire counter apparatus. There’s one case left where counters repay their cost, and it is explaining, after a run exists, why it took as long as it did.

What the modelling buys over reading the profiles directly

Each configuration consumes five profiles, and the first thing the model adds to reading those five is reach. A profile speaks only for the one run behind it, while a fitted $\eta$ applies to configurations nobody profiled, and leave-one-application-out is the test of that reach. The second is compression, from twenty-one counters down to one number that has a physical meaning. The third is that the accuracy can be proved wrong, because the ladder states the counters’ share of it instead of leaving me to assert it.

Which counters point at hardware improvements

One obvious use of a fitted model is to read off the measured quantities that matter most, and from them work out which changes to the hardware would be worth paying for. The reading can be done, but only after the five runs are merged into one row per configuration, for the reasons above. And done properly, what survives is coarse. Most of the counters’ contribution comes from two kinds of feature, the stall fractions and arithmetic intensity. All twenty features past the cycle count, taken together, shift the pooled median by 0.2%.

The apparatus here lacks the sensitivity to say which improvement to the hardware would help. A study designed for the question would vary the hardware and keep the code fixed.

What the 4.3 factor means in practice

For the procurement scenario behind this work, the 4.3 separates a prediction that can put two machines in order from one that can only say the two lie within a factor of two. A panel choosing between two architectures on a 20% performance difference gets nothing from it, and a panel deciding an allocation’s size to the nearest doubling can use it.

What I got wrong

Some claims from earlier in the project failed to hold up. Every number quoted in this article is the corrected one. The superseded results are still in my working repository, none of them deleted, and each carries a note on its fault. At the bottom of this page you’ll find the harness, the measurements as parsed, and the code that produced the final numbers, so you can check my working or take the harness for your own study. My own analysis had five defects in it.

The retraction

For a while I was reporting one energy-derived feature as the model’s strongest predictor by a distance. Its random-forest importance was 0.509, more than all the other features put together. I had it ready as the clearest novelty claim the project would make, and the counter rotation had manufactured it.

Five counters fit at a time, so a configuration’s features come from five separate runs, which leaves gaps in the assembled matrix that follow the rotation schedule. RAPL energy sits on registers of its own, so every counter set recorded it, while every other counter appeared in one set only. Impurity-based importance can’t tell a feature that explains a lot from a feature that was seen a lot. Imputation makes the gappy columns carry no information, by construction, so the importance drifts to whichever counter was always there.

On the merged and corrected data that feature ranks 15th of 21, with an importance of 0.008. The leaders after correction are the analytic time (0.376), FLOPs per instruction (0.134), instructions per rank (0.112) and the FP stall fraction (0.077).

On the left, three defects one on top of another, each one taking a piece of the claim away. On the right, the mechanism, where importance before merging reflects how many times a counter was seen and says nothing about what it explains.

Three defects had stacked up. The original table of 40 rows had no run-length covariate. Each application had been run at one problem size, so its identity couldn’t be told apart from its counter signature. And the unmerged table of 815 rows had the structural gaps I’ve just described.

Impurity feature importance can’t be interpreted on a feature matrix put together by counter multiplexing. A campaign that rotates counters and reports importances before merging is describing its own sampling schedule.

The other four

I was beating a baseline that was too easy. One earlier version claimed a 20% gain against a baseline that guessed the mean of the target. The target is left-skewed, and the median minimises the error metric, so a mean baseline was easier to beat than a fair one. The median-constant baseline is the corrected one, and it sits behind every number in this article.

The two machines weren’t compared on the same configurations. Only 166 configurations exist on both, not 191. On that paired subset, within Cirrus, the constant loses to both tree models, which reverses my published claim that Zen 5 is the harder machine to predict.

Seven of the twenty-eight win/loss labels pointed the wrong way. Each script used a paired Wilcoxon test and then labelled the winner by comparing two medians computed separately. Those aren’t the same question, because median(a) − median(b) and median(a − b) are different numbers. The error distributions have a long right tail, so a model can improve most of the pairs while a handful of pairs with large errors keep its overall median where it was. All seven flips turned a “worse” into a “model better”, so the old labels were consistently too pessimistic about the models.

I quoted more decimals than the data allows, which supports two decimals and an interval. One and the same experiment appeared as 1.054, 1.055, 1.0548 and 1.0554. I reproduced all four exactly. Two incidental details of implementation account for the whole difference, the order of the feature columns and one script’s scaler saturating, and neither is a change of method or data. The seed matters more. Across ten random states the seed spread is 0.0158, five times the whole implementation spread.

Limitations

The circularity is fundamental. The predictor doing the work is measured on the run being predicted, and more data of the same kind changes nothing, so only a different predictor would remove it.

CrayPat collects counters on rank 0 and nowhere else. Imbalance between ranks never reaches the features and ends up in the residual. The blind spot isn’t a fixed size either. At one rank, rank 0 is the entire job, and at 128 ranks it’s 0.8% of the job. So the bias moves steadily with the core count, which is itself one of the modelled axes. A control run collecting counters from several ranks would have put a bound on it. No runtime was measured without CrayPat attached, so whatever CrayPat adds is part of the target variable.

Both machines belong to the AMD Cray EX family. Nothing in this article speaks to transfer onto an architecture that differs materially. The Zen 2 stall and L3 events have no equivalent on Zen 5c, so the cross-platform work has 14 of the 21 features. GROMACS and OpenFOAM were run at a single problem size, and neither was run on Cirrus, so both are left out of every cross-platform result. No run spans more than one node.

In the expanded campaign STREAM never ran threaded. STREAM is OpenMP-only, and the original harness did launch it that way. The expanded harness fixed the thread count at one for everything and launched each code as one task on every core. STREAM has no MPI in it, so what ran was $N$ separate single-threaded copies, and never one job with $N$ threads. The fingerprint is plain, since rank 0’s instruction count is identical to four significant figures at every one of the eight core counts, so the expanded STREAM configurations hold no scaling signal whatever.

On Cirrus, HPCG does 1.5 times the iterations it does on ARCHER2. The ARCHER2 harness sets 20 CG iterations, the Cirrus harness sets 30, and the cross-platform join pairs runs by problem size and by core count while ignoring the iteration count. HPCG supplies 44 of the 166 matched configurations, and on those Cirrus does half again as much work, which accounts for the sign and for much of the size of HPCG’s cycle ratio, 1.666 when every other application sits between 0.378 and 0.827.

Treat the pooled comparison across platforms as contaminated, then, and trust the macro-average instead. The qualitative conclusion holds, since the five clean applications still differ by a factor of 2.2 among themselves.

Sixteen configurations appear in both ARCHER2 sweeps, eight from HPCG and eight from STREAM. The merge key is the set of fields deciding which runs count as one configuration, and it doesn’t record the campaign a run belongs to, so those sixteen medians mix runs from two harnesses that differed in iteration count and in model of parallelism. Sixteen of 191 sounds small, but HPCG contributes 45 rows and carries the pooled medians. Adding the campaign to the merge key and recomputing would settle how much moves.

What I’d do differently

Profile more codes, and profile each one less. My learning curve left no doubt, though I drew it too late to change the campaign. Four sizes and eight core counts for each code bought a density of coverage the model had already stopped using.

Give Cirrus problems sized for its own node. I kept the same problem sizes on Cirrus as on ARCHER2 so both machines did identical work, as a fair comparison seems to demand. Cirrus’s node is over twice as wide, so nearly half the runs finished in under a second, and at that length the fixed overhead is most of the time and the efficiency factor collapses. I chose wrong without noticing a choice was there.

Repeat every configuration once with no instrumentation. The spread between a configuration’s five runs is the largest single uncertainty in the campaign, and one more run per configuration, with nothing attached, would have separated run-to-run noise from instrumentation overhead.

Start with the measurement ladder. The ladder took about a day to build, and I built it in the final months.

Where I would take it next

I’d start with the clock-ratio rescale, since none of the trained models beat it significantly and it costs almost nothing, and the natural next step is a direct model of the ratio of cycle counts between the two machines. The other line is recovering the instruction mix without running anything, by analysing the binary statically, which goes to the root of the circularity instead of pricing it.

After that, a third platform with a different architecture (Arm, Intel, or something built around high-bandwidth memory) would show whether $\eta$ means anything independent of the architecture, which a pair of AMD Cray EX systems can’t. Counters from ranks beyond rank 0 would move load imbalance out of the residual and into the features. And RAPL energy, which the campaign gathered throughout for free and never used except as a feature, gives the existing dataset a second target to predict.

---

Appendix. What was measured

Every counter collected

I collected twenty counters on ARCHER2, and eleven of them are also available on Cirrus. Those eleven carry every cross-platform result above. The rest exist on one machine and not the other. The Zen 2 uncore L3 events and the RAPL energy events have no Cirrus equivalent that CrayPat exposes. The branch, vector and FMA events go the other way, since ARCHER2’s Rome cores didn’t offer them at all.

The stall counters in the table are short forms, and their full names all begin with DISPATCH_RESOURCE_STALL_CYCLES_1. The two L2 traffic events are shortened the same way, from REQUESTS_TO_L2_GROUP1.

The rotation schedule

Only five counters can be read at once, so a wide set of features means running the same configuration more than once. The schedule below is the one that gave me the twenty.

The core has five counter registers. The RAPL energy and Zen L3 events in the last two rows have registers of their own, so they’re read on every run and take up none of the five slots. TOT_CYC turns up in four of the sets because everything else measured in a run gets divided by the cycle count to make sense of it. The sets overlap, so adding the rows up gives twenty-three names and only twenty distinct counters. I asked for BR_INS and BR_MSP in set E, and on the Rome cores they came back with no values at all, so they’re in the schedule above and missing from the ARCHER2 column of the table before it.

The derived features

No raw counter goes into a model as it is. Every feature the models see is a ratio. A ratio doesn’t care how big the run was, so the size divides out and only the character of the execution stays in. The table lists the features whose definition you can’t guess from the name.

In the table $C$ stands for cycles and $I$ for instructions. The 64 in the arithmetic-intensity row turns L3 misses into bytes, counting each miss as one cache line of 64 bytes.

The core energy fraction row is the feature I once reported with an importance of 0.509 and later retracted. The definition is sound, so I’ve kept it in the table.