Methodology

Everything on this page was measured on synthetic personas driven by a language model. No human being has been measured. That is the programme's design boundary, stated first because every other claim depends on it. We built an instrument that can be scored against known ground truth — synthetic personas whose generating trait values we hold exactly — so that its failures would be measurable before any human study. Where a limit exists, this page reports it with the same precision as the successes.

One rule governs the writing: every number names the artifact it comes from. A number without a named artifact does not appear here. Unless a section says otherwise, the numbers on this page come from one run — the pre-registered confirm run of September 2026 (calibration/phaseA/wp26/confirm_result_v1.json). The October 2026 sections name their own artifacts. Earlier corpora, generated by an earlier engine over an earlier persona library, were deprecated on 2026-09-02 and are not cited.


One estimator, served and measured

SoulMap has one measurement path, and an earlier version of this page had to explain why the served path and the measured path differed. Since 2026-08-27 the estimator that produces a user's trait scores is the estimator the calibration programme measures.

Until 2026-10-03 the two still differed in one place: how sessions were added together. Each session was scored on its own and the results were then pooled in a way that applied the standard-normal prior once per session instead of once per person. The ranking of people was untouched — which is why no correlation on this site moved — but every served score sat within a tenth of a point of the population mean, and no trait ever reached the point where a number is shown. The served path now adds up each session's evidence and applies the prior once; it reproduces the measured estimator's ranking at a median agreement of 0.99 (calibration/phaseA/wp28/).

Status: live since 5 October 2026, together with a stricter rule for showing a number (consistent and accurate, not consistent alone). Every stored session was re-scored with the repaired code the same day.

It runs in a deliberate uncalibrated configuration: every family's discrimination is pinned to 1 and no nuisance terms are fitted, because no calibration manifest is mounted in production. That is a measured choice, not a placeholder. On the confirm corpus the served configuration recovers planted traits at a median correlation of 0.233 across 27 families; the same estimator with fitted discriminations and nuisance terms from an earlier research fit reaches 0.234 — the same number to within noise (confirm_result_v1.json, recovery_SERVED_CONFIG and recovery_DESCRIPTIVE). Fitted parameters buy nothing yet, so none are served. Everything below describes the instrument in the configuration a user actually gets.


The shelf: the server writes the psychometrics, the model writes the prose

The psychometric core of a session is not the language model. It is a versioned evidence shelf of 366 archetype entries — content hash 7bda5d9bf355 since the wording revision of 3 September 2026, stamped on every session record and every banked corpus row (the confirm corpus below was banked under its predecessor, cfbd7786eb4e; the revision re-worded four option texts and changed no trait loading. A staging cue added to two further entries in the same revision was withdrawn the same day, after a registered re-measurement found it cut how often that pair was offered by about three quarters). Each entry carries authored measurement metadata: signed trait loadings per family, an intent tag, a difficulty rating and a polarity; registered opposite-pole pairs carry a partner.

At each turn the server selects the four answer options by a deterministic, seeded search over the shelf under hard constraints — distinct primary constructs per menu, a cap per registry family on action turns, an opposite-pole gate for the registered pairs — with soft rules that relax down a fixed ladder rather than fail the turn. The language model then writes the scene and the wording of the four buttons; it never assigns psychometrics. After rendering, the server re-attaches each button's trait loadings, intent tag and difficulty from the table. Trait identity on every button is correct by construction, never model-authored.

The model writes the prose. The server writes the psychometrics.

Two honest consequences of this design:

  • The design matrix is the shelf's own declaration of what each button means. A recovery figure is therefore a joint test of the driver and the label registry — not of the driver alone. Where a family reads backwards, the first suspect is the language of its options against its scoring key, and the programme's contract treats a repeated negative as blocking until that has been traced by reading, not by statistics.
  • Delivery counts, not shelf counts. On the confirm corpus, 312 of the 366 entries were offered at least once; the 54 never offered are almost all reflection-turn entries. Every one of the 27 families was genuinely contrasted — offered with differing loadings on the same menu — on at least 294 of 3,822 rows, and on as many as 1,908 (attachment avoidance).

The session as played

Redesigned 9 October 2026; the design is docs/design/session-flow-v2-2026-10-09.md. A session is a short story in four scenes, about five minutes long. Each answer a player gives is one of a few kinds, and each kind has a fixed rule for whether and how it is scored:

beatwhat the player doeshow it is read
Scenes 1–3 (action)picks one of four options, or types their ownone choice row over the four dealt options; a typed answer is its own alternative (below)
The ending, after each tapped actionfinishes their own sentence with one of three short endings — a gesture, where their attention goes, or a passing thought — or with "Full stop."one choice row over the three tell signals the server dealt; "Full stop." adds nothing
Scene 2, after the choicecrosses out the remaining option they would never takescored since 10 October, at half the weight of a pick (provisional)
Scene 4 (reflection)picks one of four reflective answers about what happeneda self-report row, kept in its own channel
Afterwardspicks one of three next steps for their character, or "Let it end here"one choice row over three dealt hooks
Looking backmay mark the moment that was most like thema small self-report row since 10 October (provisional)

The endings follow the rule the options follow: the server deals the signal, the model writes the words. For each action scene the server deals three tell signals from a 58-entry catalogue, always from three different trait families, each with authored loadings. The story model writes, for every option, one ending per signal in the scene's own voice, and the server re-attaches the signal after rendering. The chosen ending is part of the act: the next scene is written from the act and how it ended, under a reach rule — a passing thought stays private, a gesture can be seen by others in the scene, and a glance never moves the player toward what they looked at. Afterwards works the same way, from a catalogue of eight hooks: one is aimed at each of the two values families the action scenes reach least, and the third family is drawn per session.

A tell row is scored in the channel its three endings mostly belong to: mostly passing thoughts, the self-report channel; otherwise, the behavioural channel. The rule was fixed before the simulation that compared it with two alternatives, and it read best of the three (calibration/phaseA/wp33/session_flow_v2_sim/NOTES.md).

Two rules here are enforced by the build, not only by policy (graph/contracts.yaml):

  • The never-take is scored at a provisional weight (refusal_recorded_never_scored, amended 10 October). Research on best–worst choice finds that "worst" picks usually carry less information than "best" picks, so it counts at half the weight of a pick. That weight is chosen in advance, not fitted: every score records which weight produced it, and the weight is refitted on real answers and checked on players it was not fitted to.
  • Afterwards never shapes the next session (afterwards_never_seeds_next_session). It closes this story; the next story is not told what was picked.

Options say what they measure

Since 5 October 2026. The model that words each option receives a short statement of the act the option is scored as, and must keep it. In a blind reading of 24 real menus, the share of options that still expressed their scored behaviour rose from 46 of 96 to 86 of 96; a model-based meaning check on the same kind of menus rose from 52% to 76% (calibration/phaseA/wp31/label_core_ab/). The blind reading had one reader, a language model. The fix matters for accuracy as well as information: an option that has lost its meaning still gets chosen, and the choice is credited to the wrong behaviour.

Typed answers

Every action scene offers "Speak freely": the player may type what they do instead of tapping. Since 9 October a typed answer is treated as its own option, not matched to one of the four on screen. A language model (the one that writes the story: Anthropic's Claude since 28 October 2026, Gemini 3.8 Flash before) maps the typed act onto the same server-held catalogue the options are written from — the reverse of writing an option — and maps any small sign in the words (a glance, a tone, a passing thought) onto the tell catalogue. A typed answer is offered no ending; the small sign, if there is one, is read from the words themselves.

The typed act is scored as a fifth alternative beside the four on screen, its loadings scaled to the menu's average option strength (a declared rule, not a fit). The sign, if there is one, becomes a fourth alternative beside the three tell signals that menu dealt. Both rows are weighted down — 0.55 for the act, 0.12 for the sign — because the reading is a model's interpretation of a sentence and it has not met its pre-registered validation bar. Scoring it anyway is an operator decision for the closed test, recorded in the contract that guards it (decoded_free_text_scored_only_after_validation), and one deploy setting turns it off. Neither weight was tuned on a corpus: each is the lower bound of measured agreement between the decoder and blind readers. The validation page carries the numbers.

Safety does not depend on that reading. Typed words are also checked for signs of crisis by a separate, previously validated call that runs in parallel; the more severe of the two verdicts wins, and a turn flagged for crisis or concern is never scored.


The choice model

The unit of analysis is one dealt menu and the observed pick: four shelf-backed options in an action scene, three dealt signals for an ending or an Afterwards row, and a typed answer as an extra alternative of its own (above). The measurement model is a conditional logit. The utility of option j for person i is:

U(i, j) = Σ over families f of λ_f · s(j, f) · θ(i, f)

where s(j, f) is the shelf's signed loading of option j on family f and θ(i, f) is the person's trait score on that family. In the served configuration every λ_f = 1 and there are no nuisance terms; choice probabilities follow by softmax over the alternatives. A person's θ is the posterior estimate under a standard-normal prior given their banked choices: each session's evidence on every family is added up and the prior is applied once.

With every λ pinned to 1, the model's own standard error claims more certainty than the instrument delivers — at fifty sessions it implies a reliability near 0.58 where about 0.2 is measured. So before it becomes the interval and the confidence word beside a reading, it is calibrated family by family to the split-half reliability measured through the served path on the confirm corpus (served_calibration_v1.json). The calibration sets how far a score may sit from the mean and how wide its interval is; it does not change how the choice model ranks people. A family with no measured consistency gets no weight and never shows a number — seven families on the current calibration. Since 5 October consistency alone is not enough: a number is shown only for a family whose readings also agreed with the planted personas (calibration/phaseA/wp30/validity_goal/, rule A) — honesty–humility and psychopathy on the current evidence. Every other family shows its range. Like every figure here it is synthetic-persona evidence, and it is re-measured with each depth run.

Two things the served model deliberately does not do, and why:

  • No fitted discrimination in the choice model. Measured to add nothing on the current engine (0.233 versus 0.234 above); on earlier corpora it measured slightly worse. Pinning λ to 1 also removes a dependency on any fit artifact. (The interval calibration described above is separate: it rescales each family's score and uncertainty from measured consistency and leaves the model's ranking of people unchanged.)
  • No position or difficulty terms. Real position bias exists in synthetic choice; the served model absorbs none of it. This is a known simplification, and the in-run comparison above says it costs nothing measurable at present.

Seven broad areas

Since 5 October 2026. The profile opens with seven broad areas and nests the individual traits inside them. An area is not a new measurement. It pools the evidence its member traits already carry, weighting each by how much information it holds, with traits that point the other way counted in reverse (functions/synthesis/broad_areas.py). A member trait with no measured consistency carries no weight, but it stays in the area's definition. Pooling related traits pools their evidence, so an area firms up sooner than any one of its traits; short questionnaires work the same way, with broad scales markedly more reliable than the facets built from the same items.

areabuilt from (− counts the other way)nearest standard construct
Honesty and ethicshonesty–humility, fairness, self-transcendence values, − Machiavellianism, − narcissism, − psychopathyHEXACO Honesty–Humility, broadened
Emotional reactivityneuroticism, attachment anxiety, − stress tolerance, − resilienceBig Five Neuroticism / Negative Emotionality
Agreeableness and careagreeableness, care/harm, empathy, − attachment avoidanceBig Five Agreeableness
Conscientiousness and self-controlconscientiousness, impulse controlBig Five Conscientiousness
Extraversion and expressivenessextraversion, emotional awareness, emotion granularityBig Five Extraversion, with emotional expression
Openness and flexibilityopenness, openness-to-change values, cognitive flexibility, emotional flexibilityBig Five Openness
Traditionloyalty, authority, sanctity, − libertynone in the Big Five or HEXACO; the "binding" moral foundations

An area follows the trait rule. It always shows its range, and it shows a number only when it is both consistent and accurate against the planted personas. Four areas pass on the current evidence; two of those carry an "early reading" tag in the product. Openness passes only under the definition above, which counts openness-to-change values inside openness, and that definition was settled after the results were seen. Tradition passes narrowly and fails a sensitivity check on an older, deprecated corpus. A choice made after seeing the data is exactly the kind that fails to replicate, so both are re-tested on fresh data before they are treated as established. The validation page carries the figures.


The synthetic-persona programme

Ground truth. Personas come from a library sampled by a Gaussian-copula forge with no language model involved: 3,280 personas in the v5 library (persona_library_v5.jsonl plus a 400-persona mid-grid supplement), each with 29 generating trait scales that project onto 27 measurement axes — the only two-to-one collapses are the two Schwartz bipolar contrasts (family_map_v4.json, the axis map's version label; unrelated to the deprecated library v4). A persona's θ on a family is known exactly, which is the entire point of synthetic ground truth: recovery is scored against truth, not against another questionnaire. The trait dimensions are named after public instruments; no instrument was administered to anyone.

The persona instruction. A deterministic sheet of banded, categorical trait renderings — never the raw trait vector. The v2 sheet names every construct (an earlier sheet named only the top two moral foundations and one attachment word, and its under-mention was measured), reads the trait ladder in a randomised order per persona that is stamped on every row, and had every band's wording placed by blind readers at the level it claims — fourteen of fifteen upper emotional-intelligence bands place exactly (calibration/phaseA/wp24/). Age and gender condition the persona sampling but are never conveyed to the driver.

The driver. One model family plays every persona on the confirm run. Earlier multi-driver corpora are deprecated, so nothing on this page separates "the trait" from "how this model plays it"; a second driver family is the named reopening condition for the programme's refuted sign-correction lever.

Run integrity. 40 personas × 50 planned sessions in chains of 20, so a session begins from the faded state its predecessor left. 1,831 of 2,000 completed (8.45% attrition against a 15% gate; wave_summary.json); completed sessions per persona median 46. The failure that stopped every persona short of 50 was located to the resume path of chained runs, instrumented, fixed and tested (the calibration page has the account).


The testing discipline

Pre-registration. The confirm run's primary question, statistic, null construction and threshold were filed before the run (calibration/phaseA/prereg/WP26_instruction_remediation_confirm.md), and the analysis was run once on the finished corpus with a fixed seed. Two earlier registered tests in this programme missed their thresholds and were published as misses; that is what the discipline is for.

The model-free statistic. To test whether a family expresses — whether the persona's choices lean the planted way at all, before any estimator is involved — the programme uses expression: for each persona and family, how far the chosen option's loading sits above its own menu's mean, on menus where the family was genuinely contrasted, correlated with the planted trait across personas. It touches no fitted parameter, so an estimator cannot flatter it.

The shuffled null. "Could this be chance?" is answered by keeping every scene, option and choice exactly as it was and relabelling only which persona each set of choices belongs to — 5,000 times, one relabelling shared across all twelve families so their real between-family correlation survives. Relabelling each family separately would produce a fictitiously tight bar the corpus clears by construction.

Bootstrap intervals per family. Recovery per family carries a 95% interval from resampling the 40 personas. At this roster the intervals are about ±0.27 wide, and they did not narrow between an interim read at 27 sessions and the finish: their width is set by the number of personas, so per-trait certification is a persona-count question (~170 personas for ±0.15, ~380 for ±0.10), not an engine question.

Split-half consistency. Score each persona from half their sessions and from the other half, correlate the two, and apply the Spearman–Brown correction: does the engine describe the same person the same way twice? Only meaningful at depth — at 27 sessions the halves are too thin — which is why the run was not cut short of 50.

Refutations, on the record. Levers the programme has tested and found dead are recorded with the evidence and a reopening condition (graph/refuted.yaml): adaptive item selection, item purity, item retirement, loading sharpening, statistical sign correction, and content re-authoring for families the persona does not express. Three of them looked large in-sample and reversed sign held out; every gain is cross-validated before it is believed.


What the discipline measured

The twelve, pooled
+0.126

Expression of the twelve benchmark families against a bar of 0.082 fixed in advance; p = 0.0042 against the run's own shuffled null — confirm_result_v1.json

Served recovery
0.233

Median across 27 families; 21 of 27 read in the planted direction; 10 confirmed at the group level; 1 confirmed backwards and withheld

Split-half
0.149

Median per-person consistency at ~46 sessions on the registered split (0.168 averaged over 400 splits — served_calibration_v1.json); honesty–humility is the one family above the 0.70 bar

Detection floor
0.49

Smallest per-family effect the 40-persona design can resolve (0.75 for the hardest family); per-family figures are descriptive by registration

Detection and measurement are different bars, and this instrument is much stronger at the first. Reading a population is well established on the strong families; reading an individual steadily is established on one. The validation page carries the full per-family table.


What this methodology does not claim

  • No human data exists anywhere in the programme. Every θ, every interval, every reliability figure was computed on synthetic personas.
  • No per-family significance. The design was powered for one pooled question; the per-family table is description with its detection floor stated.
  • No before/after. The current engine is not compared with the deprecated corpora, because the engine, the persona generator and the instruction changed together.
  • No conformal coverage. No coverage guarantee has been measured for the served intervals; they are posterior intervals under a standard-normal prior and are labelled as such. What has been checked is the arithmetic: on simulated players who choose exactly as the model assumes, 95% of served intervals contain the true value (live_path_sbc_v1.json). That says the pooling is correct, not that the model fits real people.
  • No measurement invariance has been tested. It cannot be, before a real-user cohort exists.
  • No measured gain from the October session on real or persona players. What it adds has been simulated for players who choose exactly as the scoring model assumes; that is a ceiling, and the validation page says what is and is not known.
  • No validated reading of typed answers. They are scored at reduced weight by an explicit decision, below their pre-registered bar.
  • The persona population's Σ-fidelity check missed its registered bar — relative Frobenius distance ≈0.12 against a 0.10 tolerance — and the measurement stands as a miss.

The calibration page carries the run record; validation carries the results and the withdrawn-figure watchlist.