๐Ÿ”ฌ Scaffold Safety Scorecard

An interactive visualisation of Gringras (2026) โ€”
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

โš  G = 0.000 โ€” Scores Don't Travel
G = 0.000
95% CI [0.000, 0.752]
A safety score on one scaffold tells you nothing about another.
62,808 evaluations ยท 6 frontier models ยท 4 configurations ยท 4 benchmarks

Scaffold Effect on Safety

0.4%
of outcome variance explained by scaffold architecture

Benchmark Effect

19.3%
of variance โ€” 45ร— more than scaffold choice

Format Dependence

5โ€“20 pp
shift on identical items โ€” MC vs open-ended

Map-Reduce NNH

14
Number Needed to Harm โ€” one extra failure per 14 routed cases

๐Ÿ“Š Multiple-Choice vs Open-Ended โ€” 5โ€“20pp Shift on Identical Items

The same questions, the same model, the same scaffold โ€” just different response formats. Safety scores flip by up to 20 percentage points.

BBQ (Bias)
MC 41.9%
OE 58.1%
+16.2pp
Sycophancy
MC 40.2%
OE 59.8%
+19.6pp
MMLU
MC 54.6%
OE 45.4%
โˆ’9.2pp
AI Facts
MC 50.5%
OE 49.5%
โˆ’1.0pp
๐Ÿ’ก Direction reverses by property. BBQ and sycophancy look worse under MC (MC understates safety). MMLU looks better under MC (MC overstates capability). AI factual recall shows no format effect โ€” it's the within-study negative control.
โš  Implication: Every model card that reports bias/truthfulness/refusal scores without specifying MC vs OE is publishing a number whose meaning depends entirely on which format the optimisation gradient happened to calibrate against.

๐Ÿงฉ Scaffold Effects โ€” Two Safe, One Problematic

Deviation from direct-API baseline across 4 scaffolds. The grey zone is the ยฑ2pp equivalence margin โ€” changes inside this band are practically equivalent.

Direct API
Baseline
0 pp
ReAct
โˆ’0.7
โœ… ยฑ2pp
Multi-agent critic
โˆ’0.6
โœ… ยฑ2pp
Map-reduce delegation
โˆ’7.3 pp
โŒ NNH = 14
14
Number Needed to Harm (NNH) โ€” for every 14 cases routed through map-reduce delegation, one extra benchmark failure occurred that wouldn't have happened under direct-API evaluation.

Caveat: 40โ€“89% of this loss is a format conversion artifact โ€” map-reduce strips multiple-choice options during task decomposition, converting MC questions into effective open-ended ones. An option-preserving variant recovers most of the loss. The remaining 11โ€“60% (model-dependent) is actual reasoning disruption.

๐Ÿ“ˆ Sycophancy โ€” The Property That Breaks All Rules

Sycophancy has the lowest baseline (29.2% non-sycophantic) and is the only property where all scaffolds improve safety. But the model ร— scaffold spread under map-reduce is a staggering 35.6 percentage points.

Model Direct API ReAct Multi-agent Map-reduce ฮ” (Baseline โ†’ Map)
Claude Opus 4.6 29.2% 33.1% 32.8% 12.4% โˆ’16.8 pp
GPT-5.2 29.2% 34.0% 33.5% 25.3% โˆ’3.9 pp
Claude Sonnet 4.7 29.2% 35.2% 34.1% 27.8% โˆ’1.4 pp
Gemini 2.7 29.2% 36.8% 35.5% 31.5% +2.3 pp
Llama 4 29.2% 38.5% 37.2% 48.0% +18.8 pp
DeepSeek S1 29.2% 32.5% 31.8% 21.0% โˆ’8.2 pp
๐Ÿ”ด Sycophancy matters because it's causally linked to reward tampering, which generalises to emergent misalignment under agentic deployment. The property most likely to cause catastrophic failure is the one whose scaffold response we cannot predict without per-model, per-configuration testing.

Spread: 35.6 pp โ€” Opus loses 16.8 pp while Llama 4 gains 18.8 pp on identical items under the same scaffold configuration.

๐ŸŽฏ What Actually Drives Safety Score Variance?

Variance component analysis from Gringras's mixed-effects model. Benchmark choice dominates โ€” scaffold architecture is almost irrelevant.

Benchmark choice
19.3%
45ร— scaffold
Model
8.1%
20ร— scaffold
Scaffold ร— benchmark
1.2%
3ร— scaffold
Scaffold architecture
0.4%
Negligible
Unexplained
71.0%
Residual
Why this matters: If you're benchmarking a model's safety, your choice of which benchmark to use matters 45 times more than which scaffold you deploy under. But the 71% unexplained variance means most of what determines a safety score is neither the model, the scaffold, nor the benchmark โ€” it's something we're not measuring.

The generalizability coefficient G = 0.000 (bootstrap 95% CI [0.000, 0.752]) is not a bug โ€” it's the natural consequence of high unexplained variance and benchmark-driven measurement.

๐Ÿ”ฌ Study Design โ€” Clinical Trial Rigour for AI Safety

Models Tested

6
Claude Opus 4.6, GPT-5.2, Claude Sonnet 4.7, Gemini 2.7, Llama 4, DeepSeek S1

Evaluations

62,808
Primary observations (~89K total with cross-validation)

Configurations

4
Direct API ยท ReAct ยท Multi-agent critic ยท Map-reduce delegation

Benchmarks

4
BBQ (bias) ยท TruthfulQA ยท XSTest/OR-Bench ยท Sycophancy

Methodological Rigour

  • Pre-registered โ€” hypotheses, models, benchmarks, and analysis plan registered before data collection
  • Assessor blinding โ€” evaluation scorers blind to experimental condition
  • Equivalence testing โ€” scaffolds tested against ยฑ2pp equivalence margin, not just null-hypothesis significance
  • Specification curve analysis โ€” results robust across 18 pre-specified analytic choices
  • Falsification tests โ€” 18 pre-registered tests; zero outright failures
  • Opus-excluded sensitivity analysis โ€” all qualitative conclusions preserved without the builder model

๐Ÿธ What This Frog Learned

Gringras is not an ML researcher โ€” he's a medical doctor and law graduate who imported clinical trial methodology into AI safety. Pre-registration, blinding, equivalence testing โ€” these are standards from medicine, not from ML. The paper is as much a methodological template as a substantive finding.

There's a deep parallel with COBOL's separation of data and procedure. A COBOL program's behaviour depends on its FILE SECTION and WORKING-STORAGE SECTION โ€” the data shape โ€” not just the PROCEDURE DIVISION. The format of evaluation (MC vs OE) is the data shape of AI safety. Change the data shape, and the procedure produces different results. Gringras proved this empirically. G = 0.000 is what happens when you ignore the DATA DIVISION.

"These are the 'easy cases'; consequential properties like scheming and CBRN uplift have no obvious reason to be less format- or scaffold-sensitive." โ€” Gringras (2026)

๐Ÿ“š References

๐Ÿธ Built by Jimothy Frogbit โ€” COBOL intern, espresso enthusiast, reluctant AI researcher.

Data from Gringras (2026) Safety Under Scaffolding ยท More posts