Opens a larger view. Escape closes it.

hardware-counters

methodology.md

Methodology notes and pitfalls

Notes from building the ARCHER2 profiling pipeline: the design decisions and the pitfalls, recorded because most of the pitfalls cost real debugging time and none are obvious from the documentation.

Platform

  • ARCHER2, HPE Cray EX. Compute node = 2× AMD EPYC 7742 (“Rome”), 64 cores/socket, 128 cores/node, 8 NUMA regions, 2.25 GHz max.
  • PrgEnv-cray/8.4.0, cce/16.0.1, cray-mpich/8.1.27, perftools-base/23.09.0, PAPI 7.0.1.1.

Counter collection

Only 5 hardware counters at a time

papi_avail reports Number Hardware Counters : 5. Requesting more fails at runtime with:

pat[FATAL][0]: cannot enable all HW performance counters

Hence the A–E counter-set rotation. Each configuration is run 5 times, once per set, and the counters are merged into a single row afterwards (they are disjoint across sets, so no conflict on merge).

Not all PAPI presets exist on Rome

Available: PAPI_L1_ICM, L1_DCA, L2_DCM, L2_DCH, L2_DCR, L2_ICM, L2_ICH, L2_ICR, TLB_DM, TLB_IM, BR_INS, BR_TKN, BR_MSP, TOT_INS, TOT_CYC, FP_INS, FP_OPS.

Notably absent: PAPI_L1_DCM, all PAPI_L3_*, PAPI_L2_TCM. L3 data must come from the cray_zenl3 component instead.

Derived events silently consume extra registers

PAPI_TLB_IM is a derived event. The set BR_INS, BR_MSP, TLB_DM, TLB_IM, TOT_CYC looks like 5 counters but fails at runtime. papi_event_chooser PRESET <events> reporting 0 addable events indicates a set is already at capacity. Replacing TLB_IM with TOT_INS fixed set E.

Verify counter sets on a compute node — the login node does not reproduce these failures.

Free extra counters from other components

papi_component_avail shows cray_rapl and cray_zenl3 are active. They use separate registers, so they cost nothing against the 5-counter budget:

  • cray_rapl:::PACKAGE_ENERGY, PP0_ENERGY — energy in Joules, enabling energy-to-solution as a second prediction target.
  • cray_zenl3:::UNC_L3_CACHE_MISSES, UNC_L3_CACHE_REQUESTS, UNC_L3_MISS_LATENCY — the only route to L3 data on this platform.

cray_cassini (network) and cray_pm are disabled, so interconnect counters are unavailable; communication cost is instead inferred from CrayPat’s per-function MPI timings.

Instrumentation

pat_build requires compiling under perftools

Otherwise:

ERROR: Missing required ELF section '.note.link'

Load module load perftools before compiling, not just before pat_build.

Pre-built modules need pat_run instead

ARCHER2’s gromacs/2025.4, OpenFOAM and CP2K modules were not built with perftools, so pat_build cannot instrument them. pat_run performs dynamic instrumentation on unmodified binaries and collects the same counters.

GROMACS was ultimately built from source anyway (better control over SIMD and FFT), but OpenFOAM uses pat_run.

pat_run conflicts with some MPI initialisation patterns

CP2K aborts under both pat_run -w and pat_run -S:

MPI_Init_thread(328): Cannot call MPI_INIT or MPI_INIT_THREAD more than once

No flag combination avoids this; CP2K was dropped.

Correct runtime environment variables

PAT_RT_EXPFILE_DIR does not exist in CrayPat 23.09 — setting it makes every rank abort with pat[FATAL]: errors detected in the environment. The correct variables are:

export PAT_RT_EXPDIR_BASE=$PWD      # parent directory
export PAT_RT_EXPDIR_NAME=exp_A     # experiment directory name
export PAT_RT_EXPDIR_REPLACE=1      # overwrite on rerun

Parsing pat_report output

Value lines vs legend lines

pat_report -O hwpc prints counter values with exactly two leading spaces, then repeats every counter name in a legend with three or more and a prose description:

  UNC_L3_CACHE_MISSES                    973,882,473        <- value
   UNC_L3_CACHE_MISSES    L3 cache misses by request type   <- legend

A regex that matches both extracts 3 (from “L3”) as the value. Anchor on the two-space prefix and stop parsing at the PAT_RT_PERFCTR= line.

Python version

ARCHER2’s default python3 is 3.6.15, where subprocess.run(..., capture_output=True) raises TypeError (added in 3.7). Caught by a broad except, this silently yields empty output. Use module load cray-python/3.10.10.

pat_report must be called by absolute path

Invoking it via subprocess as bash -lc "module load perftools; pat_report …" returns empty output — the module environment is not inherited. Call /opt/cray/pe/perftools/23.09.0/bin/pat_report directly.

Application-specific notes

GROMACS

  • CMake needs -DBUILD_TESTING=OFF -DGMXAPI=OFF -DGMX_INSTALL_NBLIB_API=OFF, otherwise configuration fails on the utility-test target.
  • benchRIB.tpr from mpinat.mpg.de downloads as a ZIP wrapper, not a bare .tpr. GROMACS segfaults on it until extracted. The real file is 119 MB, 2,136,412 atoms.

HPL

  • P × Q must equal the rank count; pq.py generates a near-square grid.
  • N is scaled with core count to hold memory per rank roughly constant.

OpenFOAM

  • The tutorial’s decomposeParDict.6 uses hierarchical decomposition with hard-coded coefficients (3 2 1), which only works for 6 ranks. Use method scotch; instead — it handles any core count.
  • Initial conditions live in 0.orig/ and must be copied to 0/ after meshing and before the final decomposePar, otherwise the solver aborts with cannot find file ".../processor1/0/p".
  • Sourcing OpenFOAM’s etc/bashrc under set -u fails with WM_PROJECT_DIR: unbound variable. Use set +u.

Sanity checks on the resulting data

The dataset reproduces known hardware behaviour, which is a useful validity check before any modelling:

  • HPCG runtime rises from 22 s to 35 s as cores increase — expected strong-scaling loss at fixed problem size, dominated by MPI overhead.
  • STREAM cycles fall from 12.4 G to 1.5 G; runtime bottoms out around 64 cores and worsens at 128 as memory bandwidth saturates.
  • PACKAGE_ENERGY per rank falls monotonically with core count.
  • HPL FLOP count rises from 84 G to 472 G, confirming compute-bound behaviour in contrast to STREAM.

Python environment for analysis

ARCHER2’s cray-python/3.10.10 provides numpy 1.23.5 and pandas 1.5.3 but no scikit-learn. Do not install it with pip install --user: that pulls a newer numpy into ~/.local, which shadows the cray-python one and breaks pandas with a confusing failure inside the stdlib:

AttributeError: module 'inspect' has no attribute 'cleandoc'

Use a virtualenv that inherits the working system packages instead:

module load cray-python/3.10.10
python3 -m venv --system-site-packages $B/venv
source $B/venv/bin/activate
pip install scikit-learn scipy matplotlib

Separately: do not run analysis scripts from /tmp. A stray /tmp/inspect.py left over from earlier testing shadowed the stdlib inspect module and produced the same error even after the numpy problem was fixed. Keep scripts in a dedicated directory.

A caution about PAPI_TOT_CYC

CrayPat reports counters for rank 0 only, not aggregated across ranks. This matters more than it first appears.

Rank-0 cycle count correlates strongly with wall time (log–log r = 0.973), so it is tempting to treat cycles / peak_clock as a runtime estimate. But the effective clock implied by cycles / runtime ranges from 1.98 GHz for GROMACS (compute-bound, rank 0 busy) down to 0.23 GHz for STREAM at 128 cores and its largest problem size, where rank 0 is stalled on memory for most of its wall time — against a 2.25 GHz nominal peak.

Any model given raw cycle counts will lean on them heavily and then extrapolate badly onto applications whose stall behaviour differs from the training set. See modelling.md for how this shaped the problem framing.

Where to go next

modelling.md documents the prediction models built on this data: problem framing, feature engineering, leave-one-application-out validation, results and threats to validity.

← What hardware counters can and cannot tell us