TASK A -- combined generalisation: unseen APPLICATION x unseen SCALE
====================================================================================================
TASK A: combined generalisation -- unseen APPLICATION x unseen SCALE.
Every earlier result in this project holds one axis fixed: LOAO varies the
application with all scales visible, and the weak/strong-scaling studies vary
the scale within known applications. This script crosses both axes.
Protocol, for each held-out application H and scale threshold T in (8,16,32):
train : all applications EXCEPT H, at ncore <= T
predict : application H, at ncore > T
The point of the exercise is NOT to beat a constant. Beating a constant is a
weak bar. The honest competitor is what a practitioner would actually do with
a new code: run it cheaply at small scale and extrapolate its own trend. So
the baseline ladder is
const constant eta = median of log10(eta) on the small-scale
training set (other applications only). Zero-parameter,
no knowledge of H at all.
same_app H's OWN small-scale median eta, assumed flat in scale.
Uses cheap runs of the target code but no trend.
same_app_trend OLS fit of log10(eta) ~ log2(ncore) on H's own small runs,
extrapolated past T. THE HARD BAR.
loao_allscale standard LOAO (train on 7 apps, ALL scales) evaluated on
the same test rows. Upper bound: isolates the cost of the
scale restriction from the cost of the app restriction.
Learned models: RandomForest and GradientBoosting quantile(0.5), the two
carried forward from retrain2.py, trained on the restricted training set.
Significance is paired against same_app_trend specifically, with direction
taken from the Hodges-Lehmann pseudomedian of the paired differences (never
from comparing independent medians) and Holm-Bonferroni within each family.
191 usable configurations (0 dropped for eta outside (0,1.5]), 21 features, 8 applications
Error factor = max(pred/actual, actual/pred); 1.000 is perfect.
T= 8 H=comd train= 72 rows/7 apps own small= 13 (slope -0.0060 dex/doubling) test= 12 rows
T= 8 H=gromacs train= 81 rows/7 apps own small= 4 (slope -0.0057 dex/doubling) test= 4 rows
T= 8 H=hpcg train= 68 rows/7 apps own small= 17 (slope +0.0051 dex/doubling) test= 28 rows
T= 8 H=hpl train= 69 rows/7 apps own small= 16 (slope -0.0181 dex/doubling) test= 16 rows
T= 8 H=lulesh train= 78 rows/7 apps own small= 7 (slope +0.0144 dex/doubling) test= 6 rows
T= 8 H=minife train= 73 rows/7 apps own small= 12 (slope -0.0175 dex/doubling) test= 24 rows
T= 8 H=openfoam train= 81 rows/7 apps own small= 4 (slope -0.0027 dex/doubling) test= 4 rows
T= 8 H=stream train= 73 rows/7 apps own small= 12 (slope -0.0024 dex/doubling) test= 12 rows
T= 16 H=comd train= 94 rows/7 apps own small= 16 (slope -0.0070 dex/doubling) test= 9 rows
T= 16 H=gromacs train= 105 rows/7 apps own small= 5 (slope -0.0047 dex/doubling) test= 3 rows
T= 16 H=hpcg train= 86 rows/7 apps own small= 24 (slope +0.0018 dex/doubling) test= 21 rows
T= 16 H=hpl train= 90 rows/7 apps own small= 20 (slope -0.0194 dex/doubling) test= 12 rows
T= 16 H=lulesh train= 103 rows/7 apps own small= 7 (slope +0.0144 dex/doubling) test= 6 rows
T= 16 H=minife train= 92 rows/7 apps own small= 18 (slope -0.0394 dex/doubling) test= 18 rows
T= 16 H=openfoam train= 105 rows/7 apps own small= 5 (slope -0.0027 dex/doubling) test= 3 rows
T= 16 H=stream train= 95 rows/7 apps own small= 15 (slope -0.0024 dex/doubling) test= 9 rows
T= 32 H=comd train= 119 rows/7 apps own small= 19 (slope -0.0087 dex/doubling) test= 6 rows
T= 32 H=gromacs train= 132 rows/7 apps own small= 6 (slope -0.0030 dex/doubling) test= 2 rows
T= 32 H=hpcg train= 107 rows/7 apps own small= 31 (slope +0.0026 dex/doubling) test= 14 rows
T= 32 H=hpl train= 114 rows/7 apps own small= 24 (slope -0.0056 dex/doubling) test= 8 rows
T= 32 H=lulesh train= 128 rows/7 apps own small= 10 (slope +0.0166 dex/doubling) test= 3 rows
T= 32 H=minife train= 114 rows/7 apps own small= 24 (slope -0.0366 dex/doubling) test= 12 rows
T= 32 H=openfoam train= 132 rows/7 apps own small= 6 (slope -0.0040 dex/doubling) test= 2 rows
T= 32 H=stream train= 120 rows/7 apps own small= 18 (slope -0.0036 dex/doubling) test= 6 rows
====================================================================================================
POOLED MEDIAN ERROR FACTOR, by scale threshold
method T=8 T=16 T=32 ALL T
----------------------------------------------------------------------
const 1.096 1.107 1.115 1.102
same_app 1.111 1.139 1.121 1.125
same_app_trend 1.072 1.091 1.084 1.083
RF 1.093 1.114 1.163 1.113
GBM_q50 1.081 1.148 1.134 1.106
loao_allscale_RF 1.101 1.110 1.125 1.110
loao_allscale_GBM 1.083 1.093 1.117 1.093
(p90)
const 1.791 1.874 2.360 1.875
same_app 1.724 1.768 2.032 1.773
same_app_trend 1.471 1.454 1.598 1.514
RF 1.612 1.646 1.804 1.646
GBM_q50 1.634 1.633 1.770 1.653
loao_allscale_RF 1.585 1.598 1.643 1.602
loao_allscale_GBM 1.576 1.624 1.630 1.624
====================================================================================================
PER-APPLICATION MEDIAN ERROR FACTOR
--- T = 8 (train on other apps at ncore<=8, test H at ncore>8) ---
n const same_app same_app_trend RF GBM_q50 loao_allscale_RF loao_allscale_GBM
app
comd 12 1.072 1.100 1.061 1.093 1.101 1.036 1.062
gromacs 4 1.024 1.006 1.043 1.029 1.042 1.004 1.004
hpcg 28 1.029 1.042 1.031 1.082 1.034 1.124 1.066
hpl 16 1.196 1.202 1.211 1.103 1.116 1.081 1.098
lulesh 6 1.096 1.147 1.043 1.028 1.037 1.027 1.041
minife 24 1.611 1.499 1.253 1.362 1.338 1.171 1.159
openfoam 4 1.051 1.083 1.056 1.206 1.122 1.137 1.117
stream 12 1.031 1.021 1.044 1.055 1.064 1.161 1.120
--- T = 16 (train on other apps at ncore<=16, test H at ncore>16) ---
n const same_app same_app_trend RF GBM_q50 loao_allscale_RF loao_allscale_GBM
app
comd 9 1.113 1.150 1.067 1.066 1.105 1.039 1.069
gromacs 3 1.037 1.003 1.037 1.030 1.049 1.007 1.003
hpcg 21 1.021 1.048 1.063 1.106 1.189 1.150 1.089
hpl 12 1.264 1.264 1.280 1.141 1.176 1.144 1.135
lulesh 6 1.091 1.147 1.043 1.030 1.023 1.027 1.041
minife 18 1.606 1.454 1.168 1.316 1.343 1.127 1.159
openfoam 3 1.058 1.118 1.091 1.217 1.128 1.117 1.111
stream 9 1.045 1.022 1.045 1.131 1.105 1.199 1.133
--- T = 32 (train on other apps at ncore<=32, test H at ncore>32) ---
n const same_app same_app_trend RF GBM_q50 loao_allscale_RF loao_allscale_GBM
app
comd 6 1.201 1.246 1.165 1.062 1.092 1.048 1.089
gromacs 2 1.059 1.022 1.044 1.030 1.033 1.020 1.021
hpcg 14 1.027 1.045 1.054 1.171 1.099 1.187 1.107
hpl 8 1.369 1.374 1.258 1.149 1.162 1.148 1.139
lulesh 3 1.086 1.097 1.026 1.038 1.057 1.035 1.046
minife 12 1.749 1.445 1.125 1.539 1.476 1.245 1.248
openfoam 2 1.155 1.218 1.171 1.076 1.082 1.070 1.083
stream 6 1.057 1.024 1.067 1.121 1.081 1.205 1.220
====================================================================================================
SIGNIFICANCE -- paired against same_app_trend (the honest competitor)
A learned model only earns its place if it beats trend extrapolation
from cheap small-scale runs of the target code itself.
### T=8: unseen app x unseen scale, vs same_app_trend (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
const 106 1.0956 [1.049,1.158] 1.0723 [1.047,1.113] +0.0645 [+0.018,+0.147] 39/67 +0.463 3.52e-05 0.000176 WORSE (Holm)
same_app 106 1.1105 [1.059,1.149] 1.0723 [1.047,1.113] +0.0643 [+0.023,+0.108] 34/72 +0.506 6.05e-06 3.63e-05 WORSE (Holm)
RF 106 1.0927 [1.065,1.118] 1.0723 [1.048,1.113] +0.0227 [+0.003,+0.042] 41/65 +0.253 0.0235 0.0941 n.s. after Holm (model worse, raw p<0.05)
GBM_q50 106 1.0811 [1.065,1.099] 1.0723 [1.047,1.113] +0.0116 [-0.006,+0.028] 43/63 +0.153 0.171 0.513 n.s. (model worse)
loao_allscale_RF 106 1.1007 [1.080,1.130] 1.0723 [1.047,1.113] +0.0083 [-0.023,+0.042] 51/55 +0.052 0.644 1 n.s. (model worse)
loao_allscale_GBM 106 1.0829 [1.066,1.106] 1.0723 [1.047,1.113] +0.0013 [-0.030,+0.026] 45/61 +0.011 0.923 1 n.s. (model worse)
### T=16: unseen app x unseen scale, vs same_app_trend (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
const 81 1.1074 [1.058,1.230] 1.0911 [1.063,1.114] +0.0573 [+0.009,+0.165] 37/44 +0.365 0.00436 0.0174 WORSE (Holm)
same_app 81 1.1392 [1.066,1.195] 1.0911 [1.063,1.114] +0.0536 [+0.010,+0.138] 37/44 +0.376 0.00328 0.0164 WORSE (Holm)
RF 81 1.1135 [1.093,1.171] 1.0911 [1.063,1.112] +0.0321 [+0.008,+0.068] 29/52 +0.309 0.0156 0.0469 WORSE (Holm)
GBM_q50 81 1.1479 [1.110,1.191] 1.0911 [1.063,1.114] +0.0536 [+0.021,+0.093] 27/54 +0.392 0.00219 0.0132 WORSE (Holm)
loao_allscale_RF 81 1.1101 [1.087,1.152] 1.0911 [1.063,1.112] +0.0236 [-0.012,+0.066] 36/45 +0.159 0.213 0.426 n.s. (model worse)
loao_allscale_GBM 81 1.0931 [1.073,1.133] 1.0911 [1.063,1.114] +0.0167 [-0.013,+0.042] 30/51 +0.144 0.261 0.426 n.s. (model worse)
### T=32: unseen app x unseen scale, vs same_app_trend (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
const 53 1.1151 [1.075,1.358] 1.0836 [1.058,1.141] +0.0743 [+0.023,+0.229] 21/32 +0.557 0.000419 0.00251 WORSE (Holm)
same_app 53 1.1213 [1.080,1.270] 1.0836 [1.058,1.141] +0.0667 [+0.031,+0.140] 23/30 +0.511 0.00121 0.00607 WORSE (Holm)
RF 53 1.1633 [1.115,1.221] 1.0836 [1.058,1.141] +0.0517 [+0.002,+0.124] 21/32 +0.305 0.0531 0.212 n.s. (model worse)
GBM_q50 53 1.1341 [1.080,1.237] 1.0836 [1.058,1.141] +0.0313 [-0.005,+0.112] 21/32 +0.284 0.0716 0.215 n.s. (model worse)
loao_allscale_RF 53 1.1255 [1.068,1.199] 1.0836 [1.058,1.141] +0.0182 [-0.028,+0.085] 26/27 +0.124 0.433 0.433 n.s. (model worse)
loao_allscale_GBM 53 1.1166 [1.084,1.209] 1.0836 [1.058,1.141] +0.0249 [-0.019,+0.074] 20/33 +0.215 0.174 0.348 n.s. (model worse)
--- pooled over all three thresholds (rows are not independent across
T, since the same configuration can appear at more than one T;
treat this block as descriptive, the per-T blocks as primary) ---
### pooled over T, vs same_app_trend (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
const 240 1.1024 [1.082,1.144] 1.0832 [1.064,1.098] +0.0671 [+0.032,+0.135] 97/143 +0.439 3.64e-09 1.82e-08 WORSE (Holm)
same_app 240 1.1251 [1.080,1.147] 1.0832 [1.064,1.099] +0.0625 [+0.035,+0.092] 94/146 +0.460 6.75e-10 4.05e-09 WORSE (Holm)
RF 240 1.1129 [1.096,1.134] 1.0832 [1.064,1.098] +0.0304 [+0.015,+0.046] 91/149 +0.283 0.000143 0.000573 WORSE (Holm)
GBM_q50 240 1.1056 [1.092,1.136] 1.0832 [1.064,1.099] +0.0275 [+0.013,+0.044] 91/149 +0.265 0.000376 0.00113 WORSE (Holm)
loao_allscale_RF 240 1.1101 [1.095,1.125] 1.0832 [1.064,1.098] +0.0162 [-0.005,+0.040] 113/127 +0.107 0.151 0.302 n.s. (model worse)
loao_allscale_GBM 240 1.0931 [1.078,1.111] 1.0832 [1.063,1.099] +0.0130 [-0.006,+0.026] 95/145 +0.102 0.171 0.302 n.s. (model worse)
--- secondary: same comparisons against the CONSTANT baseline, to show
how much easier that bar is ---
### pooled over T, vs const (weak bar) (family size 6, Holm-Bonferroni, alpha=0.05)
comparison n median [95% CI] baseline [95%CI] HodgesLehmann diff w/l r_rb p_raw p_holm verdict
--------------------------------------------------------------------------------------------------------------------------------------------------------
same_app 240 1.1251 [1.081,1.147] 1.1024 [1.082,1.144] -0.0125 [-0.027,-0.003] 135/105 -0.215 0.00386 0.00386 BETTER (Holm) [!direction conflict with naive median comparison]
same_app_trend 240 1.0832 [1.064,1.098] 1.1024 [1.082,1.143] -0.0671 [-0.134,-0.033] 143/97 -0.439 3.64e-09 2.19e-08 BETTER (Holm)
RF 240 1.1129 [1.096,1.134] 1.1024 [1.082,1.143] -0.0656 [-0.099,-0.033] 140/100 -0.344 3.76e-06 1.5e-05 BETTER (Holm) [!direction conflict with naive median comparison]
GBM_q50 240 1.1056 [1.092,1.136] 1.1024 [1.082,1.144] -0.0708 [-0.108,-0.037] 139/101 -0.366 9.04e-07 4.52e-06 BETTER (Holm) [!direction conflict with naive median comparison]
loao_allscale_RF 240 1.1101 [1.095,1.125] 1.1024 [1.082,1.144] -0.0593 [-0.094,-0.029] 151/89 -0.296 7.17e-05 0.000143 BETTER (Holm) [!direction conflict with naive median comparison]
loao_allscale_GBM 240 1.0931 [1.078,1.111] 1.1024 [1.082,1.144] -0.0801 [-0.119,-0.037] 146/94 -0.335 6.95e-06 2.08e-05 BETTER (Holm)
====================================================================================================
DIAGNOSIS
(a) Does error grow with how far past T we extrapolate?
log2(ncore/T) n same_app_trend RF loao_allscale_RF
1 81 1.057 1.087 1.090
2 81 1.077 1.113 1.110
3 53 1.098 1.132 1.125
4 25 1.178 1.167 1.183
(b) Tree models cannot extrapolate log_ncore: every test point has
log_ncore beyond the training range, so RF/GBM see the largest
training value clamped. Range check per T:
T= 8: training log2(ncore) max = 3.0; test log2(ncore) range 4.0 .. 7.0
T= 16: training log2(ncore) max = 4.0; test log2(ncore) range 4.8 .. 7.0
T= 32: training log2(ncore) max = 5.0; test log2(ncore) range 6.0 .. 7.0
(c) Fitted own-application eta slope (dex per core-count doubling),
which is what same_app_trend extrapolates:
T 8 16 32
app
comd -0.0060 -0.0070 -0.0087
gromacs -0.0057 -0.0047 -0.0030
hpcg 0.0051 0.0018 0.0026
hpl -0.0181 -0.0194 -0.0056
lulesh 0.0144 0.0144 0.0166
minife -0.0175 -0.0394 -0.0366
openfoam -0.0027 -0.0027 -0.0040
stream -0.0024 -0.0024 -0.0036
(d) Cost of the scale restriction (RF): restricted vs all-scale LOAO
T= 8: RF restricted 1.093 vs all-scale LOAO 1.101 (ratio 0.993x)
T= 16: RF restricted 1.114 vs all-scale LOAO 1.110 (ratio 1.003x)
T= 32: RF restricted 1.163 vs all-scale LOAO 1.125 (ratio 1.034x)