Cross-platform prediction on the EXACTLY MATCHED configurations
================================================================================================
P1: THE MATCHING CLAIM IS FALSE
README states: '191 matched configurations each, identical codes
and problem sizes.' What the data actually contain:
ARCHER2 191 configurations, 8 applications: ['comd', 'gromacs', 'hpcg', 'hpl', 'lulesh', 'minife', 'openfoam', 'stream']
Cirrus 191 configurations, 6 applications: ['comd', 'hpcg', 'hpl', 'lulesh', 'minife', 'stream']
applications on ARCHER2 only : ['gromacs', 'openfoam']
applications on Cirrus only : []
core counts on Cirrus only : [125, 288] (288-core node; ARCHER2 has 128)
core counts on ARCHER2 only : []
inner join on ['app', 'size', 'ncore', 'nthread']
EXACTLY MATCHED configurations : 166
unmatched on ARCHER2 : 25
unmatched on Cirrus : 25
per application:
app ARCHER2 Cirrus matched
comd 25 28 25
gromacs 8 0 0 <- absent on Cirrus
hpcg 45 51 44
hpl 32 27 24
lulesh 13 16 13
minife 36 42 36
openfoam 8 0 0 <- absent on Cirrus
stream 24 27 24
The two totals being 191 each is a coincidence of how far each sweep
reached, not evidence of matching. Only 166 configurations exist on
both machines. Comparisons over the two full 191-row samples are
therefore UNPAIRED and confound platform with sample composition.
Why this biases the platform comparison in a specific direction:
Cirrus configurations that are matched : n=166 median runtime 1.203 s
Cirrus configurations with NO ARCHER2 twin : n= 25 median runtime 1.041 s
unmatched-only core counts: [125, 288]
The unmatched Cirrus rows are its widest and shortest runs, which
are exactly the ones where fixed overhead dominates and the
efficiency factor collapses. Including them makes Cirrus look
harder to predict for a reason that has nothing to do with Zen 5.
Paired universe built and verified: 166 ARCHER2 rows aligned
row-for-row with 166 Cirrus rows on ['app', 'size', 'ncore', 'nthread'].
unmatched (191 v 191, as published): 382 rows after the eta in (0,1.5] filter (0 dropped), 14 intersection features
matched (166 v 166, paired): 332 rows after the eta in (0,1.5] filter (0 dropped), 14 intersection features
================================================================================================
PLATFORM DIFFERENCE ON THE MATCHED SET (a paired comparison at last)
quantity ARCHER2 Cirrus
--------------------------------------------------------
n 166.0000 166.0000
median runtime (s) 3.3779 1.2030
fraction under 1 s 0.2831 0.4880
median eta 0.7937 0.9136
sd log10(eta) 0.1342 0.2377
the same on the UNMATCHED samples, for contrast:
median runtime (s) 5.1608 1.0467
fraction under 1 s 0.2461 0.4764
sd log10(eta) 0.1280 0.2844
================================================================================================
THE SIX SETTINGS, MATCHED VERSUS UNMATCHED, SIDE BY SIDE
intersection features only (the 14 computable on both machines).
Error factor = max(pred/actual, actual/pred); [.,.] is a percentile
bootstrap 95% CI on the median with 10000 resamples.
within-A2 rf unmatched 1.0549 [1.044,1.072] n=191 matched 1.0714 [1.057,1.104] n=166 delta +0.0165
within-A2 gbq unmatched 1.0547 [1.047,1.069] n=191 matched 1.0618 [1.048,1.082] n=166 delta +0.0071
within-A2 mlp unmatched 1.0760 [1.065,1.096] n=191 matched 1.0855 [1.071,1.110] n=166 delta +0.0096
within-A2 const unmatched 1.0654 [1.052,1.081] n=191 matched 1.0945 [1.084,1.102] n=166 delta +0.0291
within-CIR rf unmatched 1.1863 [1.131,1.236] n=191 matched 1.1248 [1.098,1.165] n=166 delta -0.0615
within-CIR gbq unmatched 1.2146 [1.165,1.333] n=191 matched 1.1241 [1.096,1.148] n=166 delta -0.0906
within-CIR mlp unmatched 1.2892 [1.163,1.400] n=191 matched 1.1845 [1.134,1.284] n=166 delta -0.1046
within-CIR const unmatched 1.1663 [1.154,1.200] n=191 matched 1.1294 [1.119,1.157] n=166 delta -0.0369
A2->CIR rf unmatched 1.1943 [1.187,1.206] n=191 matched 1.2025 [1.192,1.211] n=166 delta +0.0082
A2->CIR gbq unmatched 1.1956 [1.176,1.210] n=191 matched 1.2057 [1.192,1.222] n=166 delta +0.0100
A2->CIR mlp unmatched 1.2239 [1.213,1.235] n=191 matched 1.2342 [1.223,1.241] n=166 delta +0.0103
A2->CIR ridge unmatched 1.2082 [1.175,1.243] n=191 matched 1.1755 [1.129,1.213] n=166 delta -0.0327
A2->CIR const unmatched 1.2018 [1.197,1.216] n=191 matched 1.2203 [1.212,1.232] n=166 delta +0.0185
CIR->A2 rf unmatched 1.1514 [1.143,1.167] n=191 matched 1.1566 [1.143,1.174] n=166 delta +0.0053
CIR->A2 gbq unmatched 1.1603 [1.146,1.167] n=191 matched 1.1358 [1.129,1.149] n=166 delta -0.0244
CIR->A2 mlp unmatched 1.1625 [1.149,1.177] n=191 matched 1.1603 [1.146,1.174] n=166 delta -0.0022
CIR->A2 ridge unmatched 1.4412 [1.336,1.499] n=191 matched 1.3122 [1.261,1.399] n=166 delta -0.1289
CIR->A2 const unmatched 1.1184 [1.097,1.144] n=191 matched 1.1511 [1.129,1.204] n=166 delta +0.0328
A2->CIR LOAO rf unmatched 1.2277 [1.212,1.256] n=191 matched 1.2539 [1.233,1.272] n=166 delta +0.0262
A2->CIR LOAO mlp unmatched 1.2624 [1.230,1.292] n=191 matched 1.2485 [1.223,1.290] n=166 delta -0.0139
A2->CIR LOAO const unmatched 1.2170 [1.210,1.225] n=191 matched 1.2510 [1.236,1.264] n=166 delta +0.0341
CIR->A2 LOAO rf unmatched 1.1603 [1.138,1.174] n=191 matched 1.1732 [1.144,1.208] n=166 delta +0.0129
CIR->A2 LOAO mlp unmatched 1.2061 [1.174,1.242] n=191 matched 1.1873 [1.157,1.251] n=166 delta -0.0188
CIR->A2 LOAO const unmatched 1.1138 [1.074,1.139] n=191 matched 1.1307 [1.103,1.194] n=166 delta +0.0169
================================================================================================
P2: FOLD-BY-FOLD AUDIT OF THE TRANSFER-PLUS-UNSEEN-APPLICATION SETTINGS
crossplatform.py loops over the UNION of applications and skips any
fold with an empty test set via a bare `continue`. It then prints
n=191 (the mask size, not the number of finite predictions) and
never computes the constant reference for these settings, which is
where the unexplained `const=nan` in crossplatform_summary.txt came
from. Every fold is now listed with its disposition.
A2->CIR LOAO, unmatched (191 v 191):
app n_test n_train disposition
comd 28 166 evaluated
gromacs 0 183 NOT EVALUABLE: application absent on the TARGET platform
hpcg 51 146 evaluated
hpl 27 159 evaluated
lulesh 16 178 evaluated
minife 42 155 evaluated
openfoam 0 183 NOT EVALUABLE: application absent on the TARGET platform
stream 27 167 evaluated
-> 6 of 8 folds evaluated, 2 not evaluable
A2->CIR LOAO, matched (166 v 166):
app n_test n_train disposition
comd 25 141 evaluated
hpcg 44 122 evaluated
hpl 24 142 evaluated
lulesh 13 153 evaluated
minife 36 130 evaluated
stream 24 142 evaluated
-> 6 of 6 folds evaluated, 0 not evaluable
CIR->A2 LOAO, unmatched (191 v 191):
app n_test n_train disposition
comd 25 163 evaluated
gromacs 8 191 evaluated
hpcg 45 140 evaluated
hpl 32 164 evaluated
lulesh 13 175 evaluated
minife 36 149 evaluated
openfoam 8 191 evaluated
stream 24 164 evaluated
-> 8 of 8 folds evaluated, 0 not evaluable
CIR->A2 LOAO, matched (166 v 166):
app n_test n_train disposition
comd 25 141 evaluated
hpcg 44 122 evaluated
hpl 24 142 evaluated
lulesh 13 153 evaluated
minife 36 130 evaluated
stream 24 142 evaluated
-> 6 of 6 folds evaluated, 0 not evaluable
DISPOSITION OF THE TWO MISSING APPLICATIONS
gromacs and openfoam are present on ARCHER2 and absent on Cirrus.
For A2->CIR LOAO their test set is empty, so the fold cannot be
scored: this is reported as NOT EVALUABLE (application absent on
the target platform), not as a silent skip and not as a zero.
For CIR->A2 LOAO their ARCHER2 test set is non-empty, but no Cirrus
training rows for them exist either, so the fold reduces to
predicting an application the source platform has never seen in any
form. That is still a legitimate (and harder) test and it IS
scored here; the earlier code scored it too but did not say so.
The consequence for the headline claim: the A2->CIR LOAO figure of
1.214 is an average over SIX applications, not eight, and the two
that are missing (a production MD code and a production CFD code)
are the two least like the proxy benchmarks. The hardest setting
in the experiment matrix is evaluated only on the easy half of the
application space. This must be stated wherever 1.214 is quoted.
The constant reference for these settings is now computed on the
SAME folds as the models, so the 'BEATS/loses to constant' verdict
is meaningful instead of a comparison against NaN.
================================================================================================
PAIRED SIGNIFICANCE ON THE MATCHED SET
Now that rows are aligned one-to-one, the model-versus-constant
comparisons are genuinely paired. Reported through stats_rigour.
### within-A2 (matched) LOAO vs constant, matched 166 (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
rf 166 1.0714 [1.057,1.104] 1.0945 [1.084,1.102] -0.0269 [-0.046,-0.010] 102/64 -0.297 0.000888 0.00153 BETTER (Holm)
gbq 166 1.0618 [1.048,1.081] 1.0945 [1.084,1.103] -0.0354 [-0.050,-0.022] 114/52 -0.396 9.78e-06 2.93e-05 BETTER (Holm)
mlp 166 1.0855 [1.071,1.110] 1.0945 [1.084,1.103] -0.0278 [-0.045,-0.011] 105/61 -0.301 0.000767 0.00153 BETTER (Holm)
### within-CIR (matched) LOAO vs constant, matched 166 (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
rf 166 1.1248 [1.098,1.165] 1.1294 [1.119,1.156] -0.0452 [-0.091,-0.008] 104/62 -0.203 0.0234 0.0469 BETTER (Holm)
gbq 166 1.1241 [1.096,1.148] 1.1294 [1.119,1.157] -0.0578 [-0.116,-0.021] 102/64 -0.278 0.00187 0.00562 BETTER (Holm)
mlp 166 1.1845 [1.134,1.288] 1.1294 [1.118,1.157] -0.0082 [-0.058,+0.042] 83/83 -0.032 0.722 0.722 n.s. (model better) [!direction conflict with naive median comparison]
### Cross-platform transfer vs constant, matched 166 (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
A2->CIR rf 166 1.2025 [1.192,1.211] 1.2203 [1.212,1.232] -0.0398 [-0.056,-0.019] 109/57 -0.294 0.001 0.00601 BETTER (Holm)
A2->CIR gbq 166 1.2057 [1.192,1.222] 1.2203 [1.212,1.232] -0.0244 [-0.047,-0.007] 96/70 -0.244 0.00638 0.0319 BETTER (Holm)
A2->CIR mlp 166 1.2342 [1.223,1.241] 1.2203 [1.212,1.232] -0.0053 [-0.019,+0.008] 87/79 -0.066 0.459 0.918 n.s. (model better) [!direction conflict with naive median comparison]
CIR->A2 rf 166 1.1566 [1.142,1.174] 1.1511 [1.129,1.202] +0.0220 [-0.007,+0.044] 63/103 +0.139 0.12 0.481 n.s. (model worse)
CIR->A2 gbq 166 1.1358 [1.129,1.149] 1.1511 [1.129,1.202] -0.0184 [-0.046,+0.014] 73/93 -0.106 0.238 0.713 n.s. (model better)
CIR->A2 mlp 166 1.1603 [1.146,1.176] 1.1511 [1.128,1.202] -0.0066 [-0.031,+0.018] 73/93 -0.046 0.604 0.918 n.s. (model better) [!direction conflict with naive median comparison]
### Transfer + unseen application vs constant, matched 166, six evaluable folds (family size 4, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
A2->CIR LOAO rf 166 1.2539 [1.233,1.272] 1.2510 [1.235,1.265] -0.0116 [-0.047,+0.013] 83/83 -0.078 0.383 1 n.s. (model better) [!direction conflict with naive median comparison]
A2->CIR LOAO mlp 166 1.2485 [1.223,1.290] 1.2510 [1.235,1.264] -0.0150 [-0.045,+0.022] 86/80 -0.070 0.435 1 n.s. (model better)
CIR->A2 LOAO rf 166 1.1732 [1.144,1.206] 1.1307 [1.103,1.189] +0.0123 [-0.045,+0.059] 81/85 +0.045 0.612 1 n.s. (model worse)
CIR->A2 LOAO mlp 166 1.1873 [1.157,1.249] 1.1307 [1.103,1.189] +0.0283 [-0.016,+0.071] 74/92 +0.112 0.21 0.84 n.s. (model worse)
================================================================================================
WHAT THE RESTRICTION TO MATCHED CONFIGURATIONS COSTS AND CHANGES
setting model unmatched matched delta
--------------------------------------------------------
within-A2 rf 1.0549 1.0714 +0.0165
within-A2 gbq 1.0547 1.0618 +0.0071
within-A2 mlp 1.0760 1.0855 +0.0096
within-A2 const 1.0654 1.0945 +0.0291
within-CIR rf 1.1863 1.1248 -0.0615
within-CIR gbq 1.2146 1.1241 -0.0906
within-CIR mlp 1.2892 1.1845 -0.1046
within-CIR const 1.1663 1.1294 -0.0369
A2->CIR rf 1.1943 1.2025 +0.0082
A2->CIR gbq 1.1956 1.2057 +0.0100
A2->CIR mlp 1.2239 1.2342 +0.0103
A2->CIR ridge 1.2082 1.1755 -0.0327
A2->CIR const 1.2018 1.2203 +0.0185
CIR->A2 rf 1.1514 1.1566 +0.0053
CIR->A2 gbq 1.1603 1.1358 -0.0244
CIR->A2 mlp 1.1625 1.1603 -0.0022
CIR->A2 ridge 1.4412 1.3122 -0.1289
CIR->A2 const 1.1184 1.1511 +0.0328
A2->CIR LOAO rf 1.2277 1.2539 +0.0262
A2->CIR LOAO mlp 1.2624 1.2485 -0.0139
A2->CIR LOAO const 1.2170 1.2510 +0.0341
CIR->A2 LOAO rf 1.1603 1.1732 +0.0129
CIR->A2 LOAO mlp 1.2061 1.1873 -0.0188
CIR->A2 LOAO const 1.1138 1.1307 +0.0169
Sample size lost: 25 configurations per platform (13.1%), and two of eight applications lost entirely from any
setting that requires them on both machines.
Reading the deltas: where a delta is large and negative the
unmatched result was pessimistic because of the unmatched Cirrus
rows (its 125- and 288-core short runs), not because of the
hardware. Where it is near zero the published conclusion survives
the correction. Both are reportable, and only the matched column
supports a statement of the form 'platform A is harder than
platform B', because only it holds the workload fixed.