FLAMEFALLAnalytics

Validation · grouped cross-validation, seed 42

Model Lab

Three small logistic models, each tested on groups it never saw. Droplet and solid data are never pooled. One fails, and is shown failing.

Model

Positive class: test ended in Extinction (visible flame went out before the fuel was gone).

Headline AUC · leave-one-fuel-out

0.74

is the worse of grouped cross-validation (0.88) and the leave-one-fuel-out stress test (0.74). 0.5 is a coin flip; 1 is perfect.

Source: scripts/train_models.py → data/models/droplet_extinction.json; fold membership in data/cv_splits.json

0.88

grouped-CV ROC-AUC

95% interval 0.81 to 0.94 · cluster bootstrap over 46 groups

18%

false-positive rate at the working threshold

16 of 90 negatives called positive at threshold 0.68 (Youden's J)

275

rows after exclusions

185 positive (67%) · 46 groups · class weights not needed

Can it tell the two outcomes apart, and what does it get wrong?

Every point was scored by a model that never saw its group.

ROC curve, out-of-fold predictions only

  • Model
  • Coin flip
  • Working threshold
PR-AUC
0.94
Brier score
0.126
Majority-class accuracy
67% (the bar to beat)
Sensitivity for “the visible flame did not go out” at 90% specificity
64%
Row-wise CV AUC
0.89 (leaky; shown only for contrast)

Confusion matrices

Threshold 0.50

58

actual 0, predicted 0

32

actual 0, predicted 1

19

actual 1, predicted 0

166

actual 1, predicted 1

Threshold 0.68 (Youden's J, used by the console)

74

actual 0, predicted 0

16

actual 0, predicted 1

39

actual 1, predicted 0

146

actual 1, predicted 1

The costly error is top right: 16 tests where the flame did not go out but the model said it would. A model like this must not be relied on by itself for a fire-protection decision.

Source: Pooled out-of-fold predictions, outer GroupKFold with 5 folds; inner GroupKFold with 3 folds chooses C from [0.01, 0.1, 1, 10] (chosen for the final fit: 1).

Are the probabilities honest, and does it hold up group by group?

On the dashed line means well calibrated. A blank AUC means the group has only one outcome.

Calibration, in ten bins

  • Model, one dot per bin (bigger = more tests)
  • Perfect calibration

Calibration slope 0.90 (1 is ideal; below 1 means over-confident), intercept 0.09. Bin sizes are small; read the shape, not single points.

Per group, and the leave-one-fuel-out stress test

Out-of-fold results per group
O2 0.12 / CO2 0.00 / He 0.00 @ 1 atm88—0.981.00
O2 0.13 / CO2 0.00 / He 0.00 @ 1 atm88—0.961.00
O2 0.13 / CO2 0.00 / He 0.40 @ 1 atm11—1.001.00
O2 0.14 / CO2 0.00 / He 0.00 @ 1 atm761.000.880.86
O2 0.14 / CO2 0.00 / He 0.35 @ 1 atm22—0.981.00
O2 0.14 / CO2 0.36 / He 0.00 @ 1 atm22—0.991.00
O2 0.15 / CO2 0.00 / He 0.00 @ 1 atm1181.000.830.73
O2 0.15 / CO2 0.00 / He 0.30 @ 1 atm22—1.001.00
O2 0.15 / CO2 0.32 / He 0.00 @ 1 atm44—0.981.00
O2 0.15 / CO2 0.50 / He 0.00 @ 0.7 atm44—0.971.00
O2 0.16 / CO2 0.00 / He 0.00 @ 1 atm33—0.831.00
O2 0.16 / CO2 0.00 / He 0.25 @ 1 atm33—0.901.00
O2 0.16 / CO2 0.25 / He 0.00 @ 1 atm44—0.981.00
O2 0.17 / CO2 0.00 / He 0.20 @ 1 atm640.750.880.67
O2 0.17 / CO2 0.00 / He 0.45 @ 0.7 atm431.000.930.75
O2 0.17 / CO2 0.20 / He 0.00 @ 1 atm44—0.931.00
O2 0.17 / CO2 0.45 / He 0.00 @ 0.7 atm77—0.941.00
O2 0.18 / CO2 0.00 / He 0.00 @ 1 atm761.000.820.86
O2 0.18 / CO2 0.00 / He 0.15 @ 1 atm531.000.820.60
O2 0.18 / CO2 0.00 / He 0.40 @ 0.7 atm55—0.911.00
O2 0.18 / CO2 0.15 / He 0.00 @ 1 atm13100.670.720.77
O2 0.18 / CO2 0.40 / He 0.00 @ 0.7 atm66—0.911.00
O2 0.19 / CO2 0.00 / He 0.10 @ 1 atm211.000.740.50
O2 0.19 / CO2 0.10 / He 0.00 @ 1 atm541.000.660.80
O2 0.20 / CO2 0.00 / He 0.00 @ 0.7 atm530.330.670.60
O2 0.20 / CO2 0.00 / He 0.05 @ 1 atm421.000.640.50
O2 0.20 / CO2 0.00 / He 0.35 @ 0.7 atm44—0.901.00
O2 0.20 / CO2 0.05 / He 0.00 @ 1 atm1291.000.650.75
O2 0.20 / CO2 0.35 / He 0.00 @ 0.7 atm55—0.851.00
O2 0.21 / CO2 0.00 / He 0.00 @ 1 atm29140.660.510.48
O2 0.21 / CO2 0.00 / He 0.15 @ 1 atm22—0.671.00
O2 0.21 / CO2 0.00 / He 0.30 @ 0.7 atm55—0.781.00
O2 0.21 / CO2 0.15 / He 0.00 @ 1 atm321.000.810.67
O2 0.21 / CO2 0.30 / He 0.00 @ 0.7 atm640.130.860.67
O2 0.21 / CO2 0.70 / He 0.00 @ 2 atm210.000.860.50
O2 0.21 / CO2 0.70 / He 0.00 @ 3 atm761.000.620.86
O2 0.23 / CO2 0.00 / He 0.25 @ 0.7 atm310.500.640.33
O2 0.23 / CO2 0.20 / He 0.00 @ 0.7 atm410.330.840.25
O2 0.23 / CO2 0.25 / He 0.00 @ 0.7 atm1290.520.530.75
O2 0.24 / CO2 0.00 / He 0.20 @ 0.7 atm320.500.490.67
O2 0.24 / CO2 0.20 / He 0.00 @ 0.7 atm741.000.350.57
O2 0.24 / CO2 0.20 / He 0.00 @ 1 atm911.000.140.11
O2 0.25 / CO2 0.15 / He 0.00 @ 0.7 atm60—0.450.00
O2 0.30 / CO2 0.00 / He 0.00 @ 0.7 atm80—0.090.00
O2 0.30 / CO2 0.00 / He 0.00 @ 1 atm720.700.010.29
O2 0.34 / CO2 0.00 / He 0.00 @ 0.7 atm90—0.020.00

Stress test: train on the others, test on the one left out

  • Heptane60 of 119 positiveAUC 0.66
  • Methanol125 of 156 positiveAUC 0.81

Source: Out-of-fold predictions per group; the stress test is reported separately and never averaged into the grouped-CV figure.

What moved the prediction?

Each bar is a change in log-odds for a one-standard-deviation increase, with a 95% bootstrap interval.

  • raises the odds of the positive class
  • lowers them
  • 95% bootstrap interval

How to read this chart

These are associations in the tests that were run, not causes. Gas fractions sum to one, and oxygen fraction, pressure and oxygen partial pressure are tied together, so their effects cannot be separated cleanly. Helium appears in only 51 tests, run at their own oxygen levels and pressures: its bar is not evidence that helium suppresses or sustains a flame.

Source: Final model refitted on all 275 rows; intervals from 1000 cluster-bootstrap refits.

Where does the model stop?

The ranges it was trained on, and a live check of how far any query sits from that data.

Training-data envelope

  • Oxygen, %12.00 to 34.00 %
  • Pressure70.17 to 309.29 kPa
  • O2 partial pressure10.67 to 64.95 kPa
  • CO2, %0.00 to 70.00 %
  • Helium, %0.00 to 45.00 %
  • Droplet diameter1.10 to 4.67 mm

Nearest-neighbour distance between training rows: median 0.13, 95th percentile 0.70 (standardised units).

Try a query

It starts outside the envelope on purpose.

Distance-to-data warning: this query is an extrapolation

Oxygen, % 45 is outside the tested 12 to 34; Pressure 56.5 is outside the tested 70.2 to 309; distance to the nearest training row is 2.47, beyond the 0.70 that covers 95% of training rows. Any output is a DEMO value, not an estimate to act on.

Model output: 0% (95% interval 0% to 0%)DEMO

Nearest training rows: FLEX-039 (2.47), FLEX-023 (2.48), FLEX-024 (2.48), FLEX-037 (2.49)

How it was built

The protocol, the features left out, and the features added with their written reason.

Sensitivity check: leave the disrupted tests out

Refitted on Extinction against Completion only (213 rows), grouped-CV AUC is 0.76 (95% interval 0.59 to 0.89). Disruption is never counted as extinction in either version; the main model simply keeps those rows as “did not extinguish”.

Source: Research specification §7.3, adapted where the data forced it; every adaptation is listed in DECISIONS.md in the repository.