Validation
Everything on this page was measured on synthetic personas driven by a language model. No human being has been measured anywhere in this programme. That is the design boundary of the work so far, and it belongs in the first sentence, not a footnote. Whether any result below transfers to people is unestablished, and no amount of further synthetic work can establish it.
This page rests on one run: the pre-registered confirm run of 1–3
September 2026 (calibration/phaseA/wp26/). Every number here comes from
its banked analysis, confirm_result_v1.json, computed once on the finished
corpus with a fixed seed and stamped with the rows' hash — or, where marked,
from two later re-analyses of the same banked rows on 3 October 2026
(calibration/phaseA/wp28/). The re-analyses change no registered result;
they measure what the product actually served, and how consistent each
family is when the split is not left to one choice. Earlier
generations of the programme — an older corpus engine, an older persona
library, and every figure derived from them — were deprecated on
2026-09-02 and are not cited anywhere on this page. By decision, old and
new data are not compared: the engine, the persona generator and the
persona instruction all changed together, so a "before and after" number
would span several confounded changes at once. A one-line note on those
earlier runs is at the end.
One rule governs the writing: every number names the artifact it comes from. A number without a named artifact does not exist here.
Added October 2026. Four later sections rest on other evidence, and each names its kind as well as its source:
- the seven broad areas: a re-analysis of the same confirm rows;
- the redesigned session: a simulation with ideal players, plus one small persona test;
- the new content: blind readings;
- typed answers: a pre-registered check that missed its bar.
None of them is a second registered run.
The run
Drawn from the v5 library (3,280 synthetic personas); each plays 50 planned sessions in chains of 20, so later sessions see the persona's own recent past — confirm_result_v1.json
8.45% attrition against the registered 15% gate; 3,822 banked choices — wave_summary.json
$0.067 per session on the batch tier; 39.6 hours wall clock across eight process restarts — checkpoint_spend.json
gemini-3.7-flash plays every persona; the sheet order is randomised per persona and stamped on every row — wave_config.json
The instrument is the production session engine itself, played by synthetic
personas under an isolation harness (replayed model transport, in-memory
storage, stubbed images). Each turn deals four options from a versioned
evidence shelf whose trait loadings are written by the server, never by the
model; the persona's pick is banked as one choice row. The shelf identity
(evidence_table_sha cfbd7786eb4e) and the family-axis version
(v4-facet-axes-2026-08-10, 27 axes over 29 scales) are stamped on all
3,822 rows (verify_smoke.py, 6 of 6 completion checks passed).
Two boundaries on the corpus itself:
- Coverage was complete at the family level and thin at the entry level.
All 27 families were contrasted on at least 294 rows; 312 of the shelf's
366 entries were offered at least once, and the 54 never offered are almost
all reflection-turn entries (
graphq coverage). Family-level claims below are on solid footing; per-entry claims are not made. - Each persona is bound to one driver, and there is only one driver. No claim on this page separates "the trait" from "how this model plays the trait". A second driver family is a named reopening condition in the programme's refutation record, not a footnote.
The pre-registered question, and its answer
The registration (calibration/phaseA/prereg/WP26_instruction_remediation_confirm.md)
asked one question about the twelve families the programme had never been
able to read — the Dark Triad, the emotional-intelligence facets, the moral
foundations, attachment anxiety, resilience, cognitive flexibility:
Do the twelve express above chance, inside this one corpus?
"Express" is model-free: for each persona and family, how far the chosen option's loading sits above its own menu's mean, on menus where the family was genuinely contrasted; then the correlation of that leaning with the persona's planted trait, across the 40 personas; then the mean over the twelve. The bar — pooled mean above 0.0822 — was fixed before the data existed, as the 95th percentile of a shared-relabelling null.
Result: the twelve express above chance. Pooled mean +0.126. Against
this corpus's own null — 5,000 relabellings of which persona each set of
choices belongs to, one relabelling shared across all twelve families so
their real correlation survives — the null averages +0.0005 with a 95th
percentile of +0.078, and the observed value is exceeded by chance in
0.42% of draws (confirm_result_v1.json, H1). The pre-registration
allows exactly two statements from this, and this page makes only them:
- Under the current stack, the twelve do express above chance.
- They sit 0.226 below the five control families — agreeableness,
conscientiousness, extraversion, honesty–humility, openness — which pool
at +0.352 in the same corpus under the same driver and instruction
(
confirm_result_v1.json, H2).
No claim of "improvement" is made: the persona instruction and the driver changed together between generations and cannot be separated.
The third registered item, machiavellianism, was named in advance because it
had read negative before. It reads negative again: −0.117 in expression
and −0.102 in served recovery (confirm_result_v1.json, H3). Its sign chain
has been traced by hand and is correct; the negative number is reported, not
corrected, and the construct is under the semantic re-verification described
on the roadmap.
Family by family
Per-family figures are descriptive, by registration. The design — 40
personas — was sized to answer the pooled question above. Its smallest
detectable per-family effect is about 0.49 in full-depth units for the
median benchmark family and 0.75 for the hardest, so an individual family
that fails to clear zero has not been shown flat; it has not been
resolved (confirm_result_v1.json, declared_limits).
Three statistics per family, all against planted truth across the 40
personas. Expression is the model-free leaning above. Recovery is
the estimate the product's own estimator produces — in the configuration a
user actually gets, every discrimination pinned to 1 and no nuisance terms —
with a 95% interval from resampling the personas. Split-half is
consistency: score each persona from half their sessions and from the other
half (about 23 each), correlate, Spearman–Brown corrected. The registered
analysis used one fixed split; the column beside it averages the same
statistic over 400 random splits of each persona's sessions, through the
pooling the product now serves (served_calibration_v1.json).
| family | expression | recovery (served) | 95% interval | split-half, one split | split-half, 400 splits | confirmed on its own |
|---|---|---|---|---|---|---|
| honesty–humility | +0.727 | +0.714 | +0.54 … +0.83 | +0.86 | +0.79 | yes |
| psychopathy | +0.600 | +0.640 | +0.45 … +0.79 | +0.13 | +0.39 | yes |
| narcissism | +0.571 | +0.520 | +0.25 … +0.73 | +0.18 | −0.22 | yes |
| self-transcendence | +0.524 | +0.485 | +0.20 … +0.71 | +0.45 | +0.26 | yes |
| openness to change | +0.455 | +0.424 | +0.14 … +0.66 | +0.08 | +0.29 | yes |
| authority / subversion | +0.387 | +0.368 | +0.09 … +0.60 | +0.46 | +0.39 | yes |
| sanctity / degradation | +0.378 | +0.367 | +0.10 … +0.56 | +0.04 | +0.17 | yes |
| agreeableness | +0.344 | +0.322 | +0.04 … +0.55 | +0.19 | +0.21 | yes |
| attachment avoidance | +0.247 | +0.317 | +0.03 … +0.56 | +0.36 | +0.36 | yes |
| stress tolerance | +0.317 | +0.303 | −0.01 … +0.58 | +0.40 | +0.33 | — |
| extraversion | +0.341 | +0.299 | +0.04 … +0.53 | +0.07 | −0.01 | yes |
| openness | +0.273 | +0.256 | −0.05 … +0.54 | +0.07 | +0.32 | — |
| cognitive flexibility | +0.238 | +0.253 | −0.19 … +0.56 | +0.57 | +0.61 | — |
| loyalty / betrayal | +0.235 | +0.233 | −0.01 … +0.46 | +0.01 | −0.01 | — |
| empathy | +0.180 | +0.219 | −0.09 … +0.50 | −0.42 | +0.08 | — |
| care / harm | +0.234 | +0.213 | −0.13 … +0.50 | +0.22 | +0.25 | — |
| fairness / cheating | +0.155 | +0.178 | −0.12 … +0.45 | +0.35 | +0.52 | — |
| emotion granularity | +0.155 | +0.121 | −0.17 … +0.37 | +0.18 | −0.07 | — |
| neuroticism | +0.071 | +0.095 | −0.29 … +0.46 | +0.17 | −0.40 | — |
| emotional awareness | +0.103 | +0.092 | −0.21 … +0.38 | −0.03 | +0.15 | — |
| conscientiousness | +0.077 | +0.058 | −0.31 … +0.36 | +0.22 | +0.36 | — |
| attachment anxiety | +0.124 | −0.009 | −0.30 … +0.23 | −0.01 | +0.10 | — |
| resilience | +0.038 | −0.017 | −0.26 … +0.23 | −0.16 | −0.13 | — |
| impulse control | −0.098 | −0.086 | −0.30 … +0.14 | +0.15 | +0.11 | — |
| machiavellianism | −0.117 | −0.102 | −0.36 … +0.18 | −0.15 | +0.15 | — |
| flexibility (EI) | −0.153 | −0.185 | −0.50 … +0.18 | −0.02 | +0.01 | — |
| liberty / oppression | −0.284 | −0.265 | −0.48 … −0.00 | −0.37 | −0.26 | backwards |
Source: confirm_result_v1.json, per_family_expression_DESCRIPTIVE and
recovery_SERVED_CONFIG. Sorted by served recovery. The same estimator with
fitted discriminations from an earlier research fit reaches a median of
0.234 against 0.233 here — the fitted parameters buy nothing yet, which is
why none are served (recovery_DESCRIPTIVE). The 400-split column comes from
calibration/phaseA/wp28/served_calibration_v1.json.
The two split-half columns disagree, and that is a finding. With 40 personas, which sessions land in which half moves a family's consistency by up to about ±0.3: narcissism reads +0.18 on the registered split and −0.22 averaged over 400; psychopathy +0.13 and +0.39. Any single-split figure for one family — on this page or anywhere else — is mostly the split. The averaged column is the better estimate of each family's consistency; neither column changes the pooled result above.
Read as four groups:
- Ten families are confirmed on their own — the interval sits clear of zero on the positive side. Two of the three Dark Triad scales are among them. "Confirmed" here is a statement about the group of 40: the engine orders these personas on the trait better than chance. It is not a statement that it reads one person reliably — that is the split-half column, and for most of the ten it is low. Median served recovery across all 27 is 0.233; 21 of 27 read in the planted direction.
- Thirteen are undetermined: positive or near-zero point estimates, intervals crossing zero. The intervals are about ±0.27 wide, and that width did not narrow between an interim look at 27 sessions and the finish — it is set by the number of personas, not sessions. Resolving these traits one by one is a matter of a larger persona roster, not of engine work.
- Three read clearly backwards without being resolved: EI flexibility,
machiavellianism, impulse control. Their intervals cross zero, but a
repeated negative is a blocking finding under the programme's contracts
(
graph/contracts.yaml,pole_inversion_chain). - One is confirmed backwards: liberty/oppression, whose interval sits entirely at or below zero and which did not move between the interim read and the finish.
Those four families are withheld from the product until their language has been re-verified against their scoring keys. The most striking non-result among the undetermined is conscientiousness: a Big Five family, named in every persona's instruction, contrasted on 38% of all rows — the most-offered family after attachment — and still flat at +0.077. That is neither under-mention nor under-exposure; it points at what the scenes' options actually encode for the trait, and it is under the same re-verification. A key-independent check (below) shows where much of it goes: personas planted high on conscientiousness pick options the shelf credits to care/harm more often than their planted traits predict.
Reliability at fifty sessions
Population-level recovery and per-person consistency are different questions, and this instrument is much stronger at the first. The split-half column above answers: does the engine describe the same person the same way twice?
| value | |
|---|---|
| median split-half reliability, 27 families, served configuration | 0.149 |
| families at or above 0.30 | 7 |
| families at or above 0.50 | 2 — honesty–humility 0.86, cognitive flexibility 0.57 |
| families below zero | 7 — empathy −0.42, liberty/oppression −0.37, resilience −0.16, machiavellianism −0.15, and three within rounding of zero |
Source: confirm_result_v1.json, recovery_SERVED_CONFIG — one fixed split.
Averaged over 400 random splits, through the pooling the product now serves,
the picture is the same in kind: median 0.168, nine families at or above
0.30, three at or above 0.50 (honesty–humility 0.79, cognitive flexibility
0.61, fairness/cheating 0.52), one at or above 0.70
(served_calibration_v1.json, headline).
One family is consistently anti-consistent. Neuroticism reads negative under every split — −0.40 averaged over 400, and −0.82 when a persona's first half of sessions is compared with their second. The engine's early read of a persona's neuroticism runs opposite to its late read. That is a structural effect, not noise, and its mechanism is not yet established; one candidate is the chained story responding to a persona's earlier choices. Until it is understood, the product never shows a neuroticism number.
Stated plainly: at fifty sessions the engine reads a population well on the strong families and reads an individual steadily on only a handful. Two families are not merely uncertain but inconsistent. Every served reading in the product carries its own uncertainty band and a plain confidence word derived from it; the 0.70 bar the programme sets for a decision-grade individual reading is met by one family, honesty–humility.
What the product served, and what it serves now
Added 3 October 2026 — calibration/phaseA/wp28/.
Status, 5 October 2026: live. The repaired pooling is in production, together with a stricter rule for showing a number: a trait must be shown to be both consistent and accurate against the planted personas (
calibration/phaseA/wp30/validity_goal/). On this corpus two families pass the accuracy check — honesty–humility and psychopathy — so only those can show a number; every other family shows its range. Every stored session was re-scored with the repaired code the same day.
Every figure above was computed by scoring each persona's banked choices in one pass. The product scored users differently: session by session, then pooling the per-session results. Replaying this corpus through the production code found that the pooling applied the standard-normal prior once per session instead of once per person, shrinking every reading twice:
| before 3 October | after | |
|---|---|---|
| spread of served scores across the 40 personas (T-points; population SD 10) | 0.09 | 3.9 |
| persona-traits past the point where a number is shown, at ~49 sessions | 0% | 32.5% |
| median recovery, the 20 families defined both ways | 0.252 | 0.253 |
Source: served_calibration_v1.json, end_to_end_legacy,
end_to_end_calibrated, headline.
Before, the product showed every person as average on every trait, and no
trait ever became confident enough to show a number. The ranking of people
was intact, which is why none of the correlations on this page moved and why
the defect went unseen. The served path now adds up each session's evidence,
applies the prior once, and calibrates each family's interval to its own
measured consistency. A served score now spreads across people by about the
square root of its reliability, which is what an honest score does: median
0.94 of that, at most 1.05. A simulation with players who choose exactly as
the model assumes confirms the arithmetic: 95% of served intervals contain
the true value, and the calibration slope is 1.06, against 53 for the old
path (live_path_sbc_v1.json).
What a user of this engine would now see, on this corpus's evidence:
- Nine families become consistent enough to show a number for most personas by about 49 sessions — honesty–humility after about 6, cognitive flexibility about 14, fairness/cheating about 20, then authority, psychopathy, attachment avoidance, conscientiousness, stress tolerance and openness. Under the accuracy rule now live, only two of them show it: honesty–humility and psychopathy. The other seven are consistent but not yet shown to be accurate, so they show only their range.
- Seven families never show a number, because their consistency through the served path measured zero or below: narcissism, extraversion, emotion granularity, liberty/oppression, loyalty/betrayal, neuroticism and resilience. They show only the range. Narcissism and extraversion are "confirmed" above at the group level; consistency for one person is what gates a number, and it was not there.
- At the information per session of this corpus the median family would
need about 570 sessions to reach the 0.70 decision-grade bar — about 11
times more information per session than the engine delivered, to get
there within fifty (
headline). The figure describes these language-model personas playing the session as it was in September. The October session carries more per play; the next section says how much, and for whom.
Seven broad areas: which can show a number
Added October 2026 — calibration/phaseA/wp31/broad_areas/broad_areas_v1.json,
a re-analysis of the same confirm rows.
The profile now reports seven broad areas, each pooling the evidence of its member traits (methodology). An area obeys the trait rule: it shows a number only when it is consistent and its readings agreed with the planted personas. The accuracy test is that, among personas whose reading has crossed the area's display line at full depth (at least ten of them), recovery of the planted composite has a 90% interval whose lower end sits above 0.20.
| area | split-half at 50 sessions | recovery | 90% interval | shows a number |
|---|---|---|---|---|
| Honesty and ethics | 0.83 | 0.76 | 0.67 … 0.85 | yes |
| Tradition | 0.47 | 0.53 | 0.31 … 0.68 | yes, as an "early reading" |
| Agreeableness and care | 0.42 | 0.52 | 0.29 … 0.70 | yes |
| Openness and flexibility | 0.71 | 0.50 | 0.25 … 0.68 | yes, as an "early reading" |
| Emotional reactivity | 0.29 | 0.35 | −0.14 … 0.74 | no — 19 personas past the line |
| Conscientiousness and self-control | 0.30 | 0.18 | −0.17 … 0.47 | no — 16 personas past the line |
| Extraversion and expressiveness | 0.16 | — | — | no — no persona crossed the line |
Source: bots_wp26_as_deep_research, per area: split_half and
rule_full_depth_named_target. On these personas honesty and ethics
reaches 0.70 consistency after about 24 sessions and openness and
flexibility after about 48; no other area gets there within fifty.
Two limits, stated with the result. The judgment is in-sample: each area's calibration was fitted on the same 40 personas' consistency, and the accuracy test runs on them too. And two of the four passes are fragile. Openness passes only under a definition settled after the results were seen, and tradition fails a sensitivity check. The product tags both "early reading". Fresh personas re-test all four.
The October session: what it can add
Added 9 October 2026 — calibration/phaseA/wp33/session_flow_v2_sim/sfv2_results_v1.json.
The redesigned session (methodology) was measured before it shipped. The method is a parameter-recovery simulation, the standard first stage in validating a measurement instrument. It uses planted personalities that choose exactly as the scoring model assumes ("ideal players"), run through the production dealing and scoring code end to end: 50 sessions each, ten replicates. It answers one question: at most, how much can each session carry?
| ideal players, 50 sessions | before (served until 9 October) | the October session |
|---|---|---|
| families at or above the 0.70 consistency bar | 14 of 27 | 26 of 27 |
| median consistency (split-half) | 0.70 | 0.83 |
| median recovery of the planted trait | 0.82 | 0.90 |
| information per session | ×1 | ×2.12 |
| sessions until the median family reaches 0.70 | about 50 | about 25 |
What carries the gain, from the same simulation:
- The endings do most of it: +10 families and ×1.57 information. Those written as passing thoughts matter on their own (+4 families).
- Afterwards lifts exactly the two families it aims at: self-transcendence from 0.65 to 0.88, openness to change from 0.69 to 0.88.
- A third action scene without endings adds nothing (×1.02 against the extra question it replaced). The gain is the endings and Afterwards.
- The never-take would add little if scored: ×2.18 against ×2.12 at half weight. It stays unscored.
- One family is still short: liberty/oppression, at 0.65.
What this does not show. Every gain above is a ceiling.
- The simulation assumes the key is right. Ideal players choose exactly as the scoring key says, so the simulation cannot tell whether real answers mean what the key assumes.
- Real players are weaker than ideal ones. Information scales with the
square of how strongly a person's traits drive their choices: at half that
strength, a session carries a quarter of the information. The one published
measurement we found for story-choice games puts real players near half
strength. It is a 30-choice text adventure played by 343 adults; the
reference and the arithmetic are in
calibration/phaseA/wp32/info_research/REPORT.md. - What we expect for people. The multiple should carry over to real people on a much smaller base. Twenty-seven separate traits at 0.70 within fifty short sessions is not a realistic expectation from behavioural choices alone. The October session roughly doubles what each session yields; it does not change that conclusion.
One persona test, inconclusive.
- The setup: eight language-model personas with planted traits played
three sessions each of the new session on the story model of the time
(Google's Gemini). Their drivers came from four model families other than
the one that then wrote the story
(
calibration/phaseA/wp34/persona_run/result_v1.json). - The result: the drivers barely followed their planted traits even on ordinary options. They picked the best-matching option 26% of the time, against 25% by chance. At two or three sessions per persona, no design could show a gain.
- What it did show: every new beat runs end to end with model-driven players. It also found one defect that ended four of the 24 sessions, which is fixed.
- What would settle it: many more sessions per persona.
Checks on the new content
Added 9 October 2026. Model-written content was read blind before its readings were trusted. The readers were fresh instances of two Claude models (Opus and Sonnet) that never saw the answer key. Both are Anthropic models, and the story model was then Google's Gemini, so a reader never judged text from its own model family.
| what was read | the readers' task | result |
|---|---|---|
| the tell and Afterwards catalogues (64 lines) | name the trait family and pole each line expresses | 62 and 61 of 64 (97%, 95%); the four lines either reader missed were reworded |
| 120 endings the story model (then Google's Gemini) wrote, in English and Greek | match each ending to the signal it was dealt (chance: one in three) | 120 of 120 for both readers (90% lower bound 0.978) |
| seven real session documents | build every scoring row the engine should | every expected row built in every session; 12–18 families measured per session, against 6 with the new beats switched off |
Sources: calibration/phaseA/wp33/session_flow_v2_blind/NOTES.md,
calibration/phaseA/wp34/endings_blind/endings_blind_result_v1.json,
calibration/phaseA/wp34/scoring_e2e/check_rows_v1.json.
- The match is on meaning. Only 38 of the 84 English endings share even one content word with the wording of their own signal.
- The task is easier than it sounds. A menu's three signals always come from three different families, which is exactly the contrast a tell row scores.
- The reworded lines were not re-read, and the two signals added afterwards for liberty/oppression were read by one reader (6 of 6, with four controls).
Do the options read as the trait they score?
Added 10 October 2026 — calibration/phaseA/wp37/shelf_audit/ and
calibration/phaseA/wp39/catalogue_cleanup/.
Every option a player can choose is scored on a fixed key: the trait family, and the direction within it, that the catalogue says the act expresses. We checked whether an option, as written, reads as its key.
The readers. The readers were fresh instances of two Claude models (Opus and Sonnet) that never saw the key.
- The task: for each option they named the three family poles it shows most, best first, from a list of 54 poles defined from the published instruments.
- Self-checks: every sheet carried textbook items with an obvious answer, and repeated items. Readers who miss those are not reading carefully.
| what was read | first pick on the key | key in the top three | the two readers' first picks agree |
|---|---|---|---|
| the catalogue's 366 acts, as written | 0.66 | 0.85 | 0.81 |
| the story model's buttons, in their scenes (128) | 0.34 | 0.59 | 0.65 |
Sources: wp39/catalogue_cleanup/measure_v1.json,
wp37/shelf_audit/ruler_top3_v1.json, wp37/shelf_audit/item_models_v1.json.
The readers are not the problem.
- They answered the textbook items correctly and repeated themselves exactly.
- They agree with each other far more than with the key. On the story model's buttons, their first picks match each other 0.65 of the time and the key 0.27–0.34. They converge on a shared reading that is not always the key's.
- Allowing three guesses instead of one does not close the gap (0.59), so a 54-way choice is not hiding the meaning.
Rewriting the options did not close it. Each of five writing changes was tested blind against today's writer:
- the option's meaning words fixed as a template;
- the core act fixed;
- labels allowed to keep the act's contrast or cost;
- two rules for the slide's restrained and intense versions.
A judge panel rated naturalness at the same time. None raised the first reading while staying natural:
| change | meaning, against today's writer | naturalness |
|---|---|---|
| meaning words fixed as a template | +0.09 | fell from 3.8 to 2.0 of 5 |
| core act fixed | +0.01 | fell to 3.0 |
| labels keep the contrast or cost | −0.01 first pick, +0.04 in the top three | held |
| slide versions told to keep the act | no change | held |
Sources: wp37/shelf_audit/item_models_v1.json, label_rule_v1.json,
ladder_v1.json, ladder_fix_v1.json.
Part of the gap is in the definitions. For a few families, the key and the published definition the readers use describe different behaviour:
- Psychopathy: helping acts are keyed as low psychopathy. Readers place them on care and empathy, and none of them reads as low psychopathy first.
- Extraversion: group acts are keyed as extraversion, while the published definition centres on talkativeness and taking the floor. Readers place them there 0.12 of the time.
These are corrected one at a time against the published instrument, not against the readers. Each family's published definition now lives in one file, and the readers' list is generated from it. The keys sit beside it in the same vocabulary folder, so a correction is one recorded change to both.
The catalogue was cleaned.
- Review: 141 acts that did not clearly read as their key were reviewed. Each proposed change was judged against the published definition, then re-read blind by fresh readers (textbook items 20 of 20, repeats 20 of 20).
- Acceptance rule: a change was accepted only if all three held:
- the reading improved;
- no family lost valid information. Valid information is how often each family is offered, weighted by how well its options read as it;
- the act became neither more nor less attractive than its neighbours.
- Result: 50 of 84 changes passed and were applied. Valid information across the catalogue rose by 5%, and no family fell. The family with the fewest options, self-transcendence, went from 6 to 14.
What this does not show (options).
- These are model readers, not players. The decisive check is a small set of options rated by people, which has not been done.
- The cleanup's gain is measured by the same kind of reader that guided it. The readers were fresh instances, but from the same two model families. Part of the gain may be agreement with how these models read, not with how people choose.
- A reader judges text. A player also brings the situation and themselves.
A story engine, measured before it ships
Added 10 October 2026 — calibration/phaseA/wp36/story_engine/ (round two).
The next session is written by a new engine.
- Who writes: a writer keeps a private story bible and writes each scene as the consequence of what the player just chose.
- How options are chosen: they are dealt only after the scene exists, from who and what it holds.
- How it was tested: against the current engine, under one pass rule fixed before any session. The same eight players, seeds and choices ran on both: 16 sessions, all complete.
| current engine | story engine | |
|---|---|---|
| preferred for immersion (two readers, 8 players each) | 0 and 0 | 8 and 8 |
| preferred for plot | 0 and 0 | 8 and 8 |
| choices visibly change what happens | 0 and 0 | 8 and 8 |
| first scene clear on first read (losses) | — | 2 and 2 (rule: at most 2) |
| contradictions counted (both readers) | 6 | 5 |
| options naming someone or something absent | 29 | 23 |
| buttons read as their scored trait, in scene | 0.48 | 0.48 |
| story time covered (median) | 7 minutes | 3.4 hours |
| median cost per session | $0.31 | $0.21 |
| wait for the first scene (median) | 112 s | 65 s |
Sources: ab/blind_result_r2_v1.json, ab/inscene_r2_v1.json, ab/metrics_r2_v1.json.
Every item of the pre-registered rule passed.
- Round one failed on first-scene clarity and on contradictions.
- Two fixes, written down before the rerun: a rule that the first scene must land on first read, and a list of established facts the writer stays consistent with.
- What the options measure did not get worse: the buttons read as their scored trait as often as the current engine's do.
What this does not show (story engine).
- The readers are Claude models (Opus and Sonnet), and the writer was Google's Gemini. They read the arms blind, but they are not players.
- The sample is small: eight players, one session each.
- The 0.48 is not comparable with the 0.34 in the section above. The catalogue was cleaned between the two, but the sessions and the reader instances also differ.
- Options do not always answer the scene's own dilemma. Blind readers
judged that about two thirds to three quarters of them do
(
ab/fit_r2b_v1.json).- A tested fix was not adopted: writing each scene around the kind of dilemma the options are dealt from.
- Why: first scenes became clearer, but the stories less immersive, and the share of fitting options did not rise.
Typed answers: below the bar, scored at reduced weight
Added 9 October 2026 — calibration/phaseA/wp34/free_text_signals/ and
calibration/phaseA/wp34/free_text_fp/.
The sample. 112 typed answers in nine languages, each decoded twice with independent seeds. The same two blind readers named, from the same catalogues, the signal each answer's main act expresses and the signal of any small sign beside it.
The bar was registered before the decoder was measured
(decoded_free_text_scored_only_after_validation in graph/contracts.yaml),
and it has three parts:
- two decodes agree on the main act;
- the decoder agrees with the readers at a lower 90% bound of 0.80;
- crisis verdicts match the validated check.
| result | bar | |
|---|---|---|
| the two decodes agree on the main act (trait family and pole) | 0.87 | 0.90 |
| the decoder agrees with each reader on the main act | 0.68 and 0.64 (lower bounds 0.60, 0.55) | lower bound 0.80 |
| the two readers agree with each other on the main act | 0.72 on all typed acts; 0.89 on the clearest | reported as the noise floor |
| the decoder agrees with the readers on the small sign | 0.25 and 0.19 | lower bound 0.80 |
| crisis and concern verdicts match the validated check | 6 of 6 | all |
Source: result_strong_v1.json.
- A model-version caveat. These figures were measured with Gemini 3.7 Flash. Production then decoded with Gemini 3.8 Flash, and the same reading was not re-measured on it.
- A redesign did no better. A reading on 3.8 Flash that chooses the
trait family directly reached 0.54 and 0.66 against the readers, with 0.78
agreement between seeds (
free_text_fp/result_v1.json). It was not adopted.
Update, 10 October 2026 — the split decode (still in use; the model changed on 28 October) (calibration/phaseA/wp35/free_text_opt/result_v1.json).
The decoder was re-measured on Gemini 3.8 Flash, the model production
used at the time, and replaced by a split decode: the typed answer is read in its own call.
| on 3.8 Flash | development set (112) | held-out set (60 new answers) |
|---|---|---|
| the split decode agrees with each reader on the main act | 0.76 and 0.75 (lower bounds 0.68, 0.67) | 0.65 and 0.65 (lower bounds 0.54, 0.54) |
| the two readers agree with each other (the noise floor) | 0.82 | 0.69 |
| two decodes agree on the main act | 0.90 | not measured |
| crisis and concern verdicts match the validated check | 6 of 6 | none flagged |
- The split decode was adopted. It met the rule fixed in advance for "no worse, and cheaper": 26% cheaper. On the held-out set it is better than the previous decoder (+0.11, 90% interval +0.01..+0.22).
- What is scored now:
- The main act is scored at 0.54, the held-out lower bound with the more conservative reader.
- The small sign beside it is no longer scored. Readers agree with each other on it only 0.32 to 0.55, too low for any decoder to be checked against.
- The bar of 0.80 is still not met. The figures in the table above this update are the earlier measurement on 3.7 Flash, kept for the record.
- Since 28 October 2026 production decodes with Anthropic's Claude, the model that now writes the story. None of the figures in this section, the crisis and concern check included, has been re-measured on it.
The bar is not met, and it may be unreachable as written.
- Why. Blind readers agree with each other only 0.72 on the main act, because a fair share of typed acts can reasonably be read as different traits. A decoder cannot be shown to agree with readers at a lower bound of 0.80 when their own agreement sits at or below that.
- Scored anyway, by decision. Typed answers are scored by an explicit
operator decision for the closed test, at the weight the registered rule
gives: the lower bound of agreement with the more conservative reader.
- Since 10 October: 0.54 for the act, and the sign is not scored.
- Before then: 0.55 and 0.12.
- What that is. It is a decision, not a validated result. The question is reopened with real typed answers.
Behaviour credited to the wrong trait
Added 3 October 2026 — calibration/phaseA/wp28/cross_key_v1.json.
The "expression" statistic above reads choices through the shelf's own scoring key, the same key the estimator uses. So it cannot tell a trait the driver does not play from a trait the driver plays through options the key credits to a neighbouring trait. A key-independent check can. It asks whether a planted trait raises a family's keyed expression beyond what the planted correlation between the two traits predicts.
Fourteen such cells were nominated on an older corpus. With their directions fixed in advance, all fourteen replicate in direction on the current engine's 80 depth personas — eight of them at more than two standard errors (one-sided sign test p ≈ 0.0001). The clearest:
| planted trait | shows up on options keyed to | excess (current engine) |
|---|---|---|
| openness | cognitive flexibility | +0.43 |
| extraversion | cognitive flexibility | +0.36 |
| agreeableness | loyalty / betrayal | +0.32 |
| honesty–humility | fairness / cheating | +0.29 |
| conscientiousness | care / harm | +0.24 |
| extraversion | narcissism | +0.23 |
| machiavellianism | impulse control (lower) | −0.28 |
What this changes. Part of why some families read flat is not that the driver cannot portray them: their behaviour is being credited to a neighbour. Machiavellianism is played, through options keyed as impulsive. Cognitive flexibility's options are read largely as openness. That is a content problem the instrument can fix: rewrite options so they discriminate between the two traits being conflated, smallest change, reasoning recorded. It is never fixed by re-keying from the statistics. The same fourteen cells and a pass rule for the next corpus are registered in the artifact. The usual boundary holds: this describes how one language model reads these options, which may not be how people do.
What went wrong, and what it cost
169 of 2,000 sessions failed. The attrition KPI (≤ 15%) passes at 8.45%; the
fifty-sessions-per-subject milestone does not — completed sessions per
persona were median 46 (min 42, max 49, none at 50) (graphq kpi).
The cause is instrumented rather than assumed. 100 of the 169 died with the same error, and every one sits at chain links 2–7 — exactly the links that were live across the run's eight process restarts. A resumed process rebuilt each link's starting state by re-walking its predecessor; when that re-walk drifted by a byte, the successor's prompt changed, its cached response missed, the scene regenerated, and the banked choice matched no button. The remaining 69 are ordinary validation failures spread across all links (~3.5%). The fix — banking each link's hand-off state so a resume restores it instead of re-deriving it — shipped on 2026-09-03 with a test that reproduces the drift; the milestone is expected to be reachable on the next run without changing anything else.
What this page does not claim
- Nothing about human beings. Every artifact above describes model behaviour against synthetic ground truth.
- Nothing per family beyond description. Only the pooled question was powered; the per-family table is what 40 personas can show, with its detection floor stated.
- No before/after. Earlier-generation figures are deprecated and not compared against.
- No decision-grade individual reading except where the table above says so, and the product's serving policy enforces the same rule.
- No claim about the served calibration beyond synthetic personas. The per-family calibration was measured on this corpus and is re-measured with every depth run; for people it is a starting point, not a result.
- No measured gain from the October session. The ×2.12 is a ceiling for ideal players; the one persona test could not resolve a difference, and no real player has been measured.
- No validated reading of typed answers. They are scored at reduced weight by decision, below the bar registered for them.
Earlier generations, for the record
Between May and August 2026 the programme ran an earlier corpus engine
against an earlier persona library (v4), and published figures from those
corpora on this page. Those figures were withdrawn in two rounds — a
primary-artifact audit on 2026-08-19, and the deprecation of every pre-v5
corpus on 2026-09-02 once the engine, generator and instruction had all
changed. The withdrawn numbers are guarded by a build-time watchlist
(graph/claims.yaml) so they cannot quietly reappear on a public surface.
The artifacts remain in the repository's history; nothing on this page rests
on them.