Corrected statistical reporting for all key comparisons
====================================================================================================
Direction is the SIGN OF THE HODGES-LEHMANN PSEUDOMEDIAN of the paired
differences (model - baseline), the location estimate the Wilcoxon
signed-rank test actually targets; NOT a comparison of two independent
medians. Intervals are percentile bootstrap 95% CIs (10000 resamples
for arm medians, 2000 for the O(n^2) Hodges-Lehmann). p_holm is Holm-
Bonferroni adjusted within the family shown in each block heading.
r_rb is the matched-pairs rank-biserial effect size; w/l counts pairs.
Error factor = max(pred/actual, actual/pred); 1.0 is perfect.
### ARCHER2 (fixed size, LOAO) vs constant baseline, intersection features (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
Random Forest 191 1.0549 [1.044,1.072] 1.0654 [1.052,1.081] -0.0254 [-0.042,-0.012] 121/70 -0.313 0.000178 0.000357 BETTER (Holm)
GBM quantile(0.5) 191 1.0547 [1.047,1.069] 1.0654 [1.052,1.081] -0.0269 [-0.043,-0.013] 118/73 -0.321 0.000118 0.000355 BETTER (Holm)
MLP (32,16) alpha=1 191 1.0760 [1.065,1.096] 1.0654 [1.052,1.080] -0.0166 [-0.028,-0.005] 119/72 -0.225 0.007 0.007 BETTER (Holm) [!direction conflict with naive median comparison]
### Cirrus fixed-size (LOAO) vs constant baseline, intersection features (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
Random Forest 191 1.1863 [1.131,1.236] 1.1663 [1.154,1.200] -0.0612 [-0.154,-0.014] 113/78 -0.206 0.0134 0.0402 BETTER (Holm) [!direction conflict with naive median comparison]
GBM quantile(0.5) 191 1.2146 [1.164,1.340] 1.1663 [1.154,1.200] -0.0591 [-0.141,-0.007] 105/86 -0.183 0.0279 0.0558 n.s. after Holm (model better, raw p<0.05) [!direction conflict with naive median comparison]
MLP (32,16) alpha=1 191 1.2892 [1.163,1.400] 1.1663 [1.154,1.200] -0.0356 [-0.099,+0.032] 98/93 -0.088 0.291 0.291 n.s. (model better) [!direction conflict with naive median comparison]
### Cirrus weak-scaled (LOAO) vs constant baseline, intersection features (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
Random Forest 162 1.0603 [1.047,1.079] 1.0617 [1.056,1.068] -0.0074 [-0.015,+0.003] 101/61 -0.140 0.122 0.366 n.s. (model better)
GBM quantile(0.5) 162 1.0565 [1.046,1.072] 1.0617 [1.057,1.069] -0.0061 [-0.018,+0.007] 91/71 -0.089 0.327 0.367 n.s. (model better)
MLP (32,16) alpha=1 162 1.0807 [1.069,1.104] 1.0617 [1.056,1.068] +0.0111 [-0.003,+0.029] 71/91 +0.121 0.183 0.367 n.s. (model worse)
### Counter-group ablation on ARCHER2, RF, vs CONFIG-ONLY (family size 3, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
+ analytic cycles (1 counter) 191 1.1052 [1.074,1.154] 1.1040 [1.094,1.121] -0.0375 [-0.060,-0.016] 118/73 -0.307 0.00024 0.00024 BETTER (Holm) [!direction conflict with naive median comparison]
+ instruction ratios (1 set) 191 1.0795 [1.064,1.110] 1.1040 [1.094,1.121] -0.0514 [-0.081,-0.032] 131/60 -0.490 4.31e-09 8.62e-09 BETTER (Holm)
+ all counters (5 sets) 191 1.0496 [1.037,1.073] 1.1040 [1.093,1.121] -0.0698 [-0.098,-0.052] 152/39 -0.686 2.11e-16 6.33e-16 BETTER (Holm)
### Counter-group ablation on ARCHER2, RF, vs CONSTANT baseline (family size 4, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
config only (free) 191 1.1040 [1.094,1.121] 1.0654 [1.052,1.081] +0.0174 [+0.008,+0.028] 64/127 +0.266 0.00145 0.00435 WORSE (Holm)
+ analytic cycles (1 counter) 191 1.1052 [1.076,1.154] 1.0654 [1.052,1.081] +0.0007 [-0.018,+0.019] 95/96 +0.007 0.931 0.931 n.s. (model worse)
+ instruction ratios (1 set) 191 1.0795 [1.064,1.110] 1.0654 [1.052,1.080] -0.0119 [-0.029,+0.003] 101/90 -0.126 0.132 0.265 n.s. (model better) [!direction conflict with naive median comparison]
+ all counters (5 sets) 191 1.0496 [1.037,1.074] 1.0654 [1.052,1.080] -0.0292 [-0.044,-0.016] 130/61 -0.345 3.56e-05 0.000142 BETTER (Holm)
### Cross-platform transfer vs constant, intersection features (family size 12, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
ARCHER2 -> Cirrus-fixed: RF 191 1.1943 [1.187,1.206] 1.2018 [1.196,1.216] -0.0357 [-0.117,-0.019] 124/67 -0.317 0.000149 0.000908 BETTER (Holm)
ARCHER2 -> Cirrus-fixed: GBQ 191 1.1956 [1.176,1.210] 1.2018 [1.197,1.216] -0.0482 [-0.144,-0.024] 122/69 -0.354 2.24e-05 0.000179 BETTER (Holm)
ARCHER2 -> Cirrus-fixed: MLP 191 1.2239 [1.213,1.235] 1.2018 [1.197,1.216] -0.0164 [-0.095,+0.006] 98/93 -0.116 0.163 0.326 n.s. (model better) [!direction conflict with naive median comparison]
Cirrus-fixed -> ARCHER2: RF 191 1.1514 [1.143,1.168] 1.1184 [1.097,1.146] +0.0450 [+0.022,+0.063] 59/132 +0.296 0.000396 0.00198 WORSE (Holm)
Cirrus-fixed -> ARCHER2: GBQ 191 1.1603 [1.146,1.167] 1.1184 [1.100,1.144] +0.0122 [-0.023,+0.043] 67/124 +0.062 0.455 0.455 n.s. (model worse)
Cirrus-fixed -> ARCHER2: MLP 191 1.1625 [1.149,1.177] 1.1184 [1.097,1.144] +0.0324 [+0.011,+0.054] 73/118 +0.218 0.00887 0.0332 WORSE (Holm)
ARCHER2 -> Cirrus-weak: RF 162 1.1822 [1.172,1.189] 1.1902 [1.174,1.202] -0.0130 [-0.021,-0.004] 105/57 -0.239 0.0083 0.0332 BETTER (Holm)
ARCHER2 -> Cirrus-weak: GBQ 162 1.1764 [1.167,1.185] 1.1902 [1.174,1.202] -0.0158 [-0.023,-0.009] 109/53 -0.347 0.00013 0.000908 BETTER (Holm)
ARCHER2 -> Cirrus-weak: MLP 162 1.2107 [1.200,1.218] 1.1902 [1.174,1.202] +0.0177 [+0.010,+0.025] 61/101 +0.401 9.46e-06 8.52e-05 WORSE (Holm)
Cirrus-weak -> ARCHER2: RF 191 1.1433 [1.137,1.153] 1.1719 [1.154,1.201] -0.0594 [-0.100,-0.034] 117/74 -0.518 5.46e-10 5.46e-09 BETTER (Holm)
Cirrus-weak -> ARCHER2: GBQ 191 1.1473 [1.135,1.162] 1.1719 [1.155,1.201] -0.0630 [-0.112,-0.037] 122/69 -0.539 1.08e-10 1.19e-09 BETTER (Holm)
Cirrus-weak -> ARCHER2: MLP 191 1.1392 [1.126,1.152] 1.1719 [1.154,1.201] -0.1107 [-0.159,-0.080] 122/69 -0.612 2.31e-13 2.77e-12 BETTER (Holm)
====================================================================================================
WHAT THE CORRECTION COSTS
comparisons run : 28
significant at raw alpha=0.05 : 20
still significant after Holm-Bonferroni: 19
LOST to the correction : 1
Cirrus fixed-size (LOAO) vs constant baseline, intersection features | GBM quantile(0.5) raw p=0.0279 -> Holm p=0.0558
comparisons where the OLD naive direction label (median(a) vs
median(b)) disagrees with the paired direction: 7
ARCHER2 (fixed size, LOAO) vs constant baseline, intersection features | MLP (32,16) alpha=1: paired HL=-0.0166 says 'model better', naive median said 'model worse'
Cirrus fixed-size (LOAO) vs constant baseline, intersection features | Random Forest: paired HL=-0.0612 says 'model better', naive median said 'model worse'
Cirrus fixed-size (LOAO) vs constant baseline, intersection features | GBM quantile(0.5): paired HL=-0.0591 says 'model better', naive median said 'model worse'
Cirrus fixed-size (LOAO) vs constant baseline, intersection features | MLP (32,16) alpha=1: paired HL=-0.0356 says 'model better', naive median said 'model worse'
Counter-group ablation on ARCHER2, RF, vs CONFIG-ONLY | + analytic cycles (1 counter): paired HL=-0.0375 says 'model better', naive median said 'model worse'
Counter-group ablation on ARCHER2, RF, vs CONSTANT baseline | + instruction ratios (1 set): paired HL=-0.0119 says 'model better', naive median said 'model worse'
Cross-platform transfer vs constant, intersection features | ARCHER2 -> Cirrus-fixed: MLP: paired HL=-0.0164 says 'model better', naive median said 'model worse'
Comparisons that survive Holm AND favour the model:
ARCHER2 (fixed size, LOAO) vs constant baseline, intersection features | Random Forest: 1.0549 vs 1.0654, HL diff -0.0254 [-0.0422,-0.0118], r_rb=-0.313, Holm p=0.000357
ARCHER2 (fixed size, LOAO) vs constant baseline, intersection features | GBM quantile(0.5): 1.0547 vs 1.0654, HL diff -0.0269 [-0.0425,-0.0130], r_rb=-0.321, Holm p=0.000355
ARCHER2 (fixed size, LOAO) vs constant baseline, intersection features | MLP (32,16) alpha=1: 1.0760 vs 1.0654, HL diff -0.0166 [-0.0282,-0.0050], r_rb=-0.225, Holm p=0.007
Cirrus fixed-size (LOAO) vs constant baseline, intersection features | Random Forest: 1.1863 vs 1.1663, HL diff -0.0612 [-0.1544,-0.0142], r_rb=-0.206, Holm p=0.0402
Counter-group ablation on ARCHER2, RF, vs CONFIG-ONLY | + analytic cycles (1 counter): 1.1052 vs 1.1040, HL diff -0.0375 [-0.0599,-0.0164], r_rb=-0.307, Holm p=0.00024
Counter-group ablation on ARCHER2, RF, vs CONFIG-ONLY | + instruction ratios (1 set): 1.0795 vs 1.1040, HL diff -0.0514 [-0.0809,-0.0322], r_rb=-0.490, Holm p=8.62e-09
Counter-group ablation on ARCHER2, RF, vs CONFIG-ONLY | + all counters (5 sets): 1.0496 vs 1.1040, HL diff -0.0698 [-0.0979,-0.0517], r_rb=-0.686, Holm p=6.33e-16
Counter-group ablation on ARCHER2, RF, vs CONSTANT baseline | + all counters (5 sets): 1.0496 vs 1.0654, HL diff -0.0292 [-0.0438,-0.0157], r_rb=-0.345, Holm p=0.000142
Cross-platform transfer vs constant, intersection features | ARCHER2 -> Cirrus-fixed: RF: 1.1943 vs 1.2018, HL diff -0.0357 [-0.1166,-0.0186], r_rb=-0.317, Holm p=0.000908
Cross-platform transfer vs constant, intersection features | ARCHER2 -> Cirrus-fixed: GBQ: 1.1956 vs 1.2018, HL diff -0.0482 [-0.1444,-0.0243], r_rb=-0.354, Holm p=0.000179
Cross-platform transfer vs constant, intersection features | ARCHER2 -> Cirrus-weak: RF: 1.1822 vs 1.1902, HL diff -0.0130 [-0.0208,-0.0035], r_rb=-0.239, Holm p=0.0332
Cross-platform transfer vs constant, intersection features | ARCHER2 -> Cirrus-weak: GBQ: 1.1764 vs 1.1902, HL diff -0.0158 [-0.0231,-0.0086], r_rb=-0.347, Holm p=0.000908
Cross-platform transfer vs constant, intersection features | Cirrus-weak -> ARCHER2: RF: 1.1433 vs 1.1719, HL diff -0.0594 [-0.1000,-0.0338], r_rb=-0.518, Holm p=5.46e-09
Cross-platform transfer vs constant, intersection features | Cirrus-weak -> ARCHER2: GBQ: 1.1473 vs 1.1719, HL diff -0.0630 [-0.1123,-0.0375], r_rb=-0.539, Holm p=1.19e-09
Cross-platform transfer vs constant, intersection features | Cirrus-weak -> ARCHER2: MLP: 1.1392 vs 1.1719, HL diff -0.1107 [-0.1588,-0.0798], r_rb=-0.612, Holm p=2.77e-12