Methods & evidence

1. Variance comes from dice, not the model

Each cohort member — one of the fifteen simulated people who react to what you paste — is dealt a substrate: mood, money pressure, expertise, attention, age, place, and more, sampled by externally seeded randomness across ~48 dimensions (individual traits, not just those six headline categories) before the model is ever called. The model embodies the hand it’s dealt; it never invents its own variance.

Receipt: ablation against a conventional persona-only prompt, same stimulus, same model: 9 of 15 baseline responses opened with the identical sentence (5/15 distinct openers); the substrate cohort produced 15/15 distinct openers and ~3× the length variance (coefficient of variation 0.23 → 0.60).

2. Reproducible cohorts

Randomness enters once, at study creation, as a recorded seed. The same seed re-deals the identical cohort — same substrates, ages, locations — so duplicating a study and changing one variable is a controlled comparison.

Receipt: paired same-seed studies verified byte-identical substrates across a model swap, a template swap, and a demographic change — the basis for every robustness claim below.

3. Anti-smoothing: splits and verbatim dissent, or regenerate

A deterministic check — a fixed rule enforced in code after every study, not a judgment call — rejects any collation (the write-up a study produces) that generalises without a numeric split (“9 of 15”) or whose dissenting voice isn’t verbatim: a character-exact quote of a real cohort response, never paraphrased. Failures trigger targeted regeneration; persistent failure ships with a visible quality warning, never silently.

Repeated statements are a stronger signal than sporadic ones — but sporadic isn’t the same as unimportant. Counts convey convergence, not consequence: a theme nine of fifteen members raise is common; the one member who raises it alone isn’t noise to be averaged out, because outliers can be exactly the signal — one confused reading of a pricing table is still a reading someone will have. That’s the reasoning behind the verbatim-dissent rule above, and it’s why the tool’s own replication rules (§6) treat spontaneous-theme counts as the least trustworthy number it produces: read them as “this objection exists,” not “this many hold it.”

Is that dissenting voice tested against real people? Yes — that’s the job of calibration: paired studies where a researcher codes a synthetic split, dissent included, against what a real cohort actually said.

4. Intent tallies are stripped — even from legitimate studies

The tool refuses purchase-intent studies outright (until calibrated). But the guard runs deeper: a deterministic check inside every collation strips intent tallies that try to sneak into legitimate work.

Receipt:in production use, a collation reported “11 of 15 say they will subscribe”; the guard was built, and the same study re-run reported the reaction and its reasons instead. A later evasion via spelled-out numbers (“one member states they will not subscribe”) was caught and closed the same day.

5. Hypotheses must cite their evidence

Every study ends with 2–4 falsifiable hypotheses about real users. Each must cite the numeric cohort split that motivates it — deterministically checked — and propose one concrete, cheap next test in the real world. Hypotheses are questions raised by synthetic data, never findings.

6. Known noise, measured and disclosed

Same-seed replication testing produced the tool’s own reading rules, shown with every study: strong consensus (12+ of 15) replicates reliably; split magnitudes wobble by ±2 at N=15; counts of spontaneously-mentioned themes are the noisiest number produced and should be read as “this objection exists,” not “this many hold it.”

Receipt: headline findings on the test stimulus replicated across two models, two prompt templates, and within-persona demographic variance — four internal robustness checks, all on identical seeds.

7. Uncalibrated means uncalibrated

Every output is labelled synthetic and exploratory-only — a lead worth chasing, not a result to cite — until a use case has been validated against real research; trust is earned per use case, not asserted per model. The internal checks above make the tool consistent; only calibration against real users can make it credible. That work is the roadmap’s centre.

Published work does show synthetic responses correlating with real ones — in specific, carefully calibrated settings (see References below). “Correlates in that setting” and “transfers to yours” are different claims, and treating them as the same one is the industry’s most common mistake. Closing that gap, study by study, is what the calibration workflow below is for.

The comparison screen where a researcher codes a synthetic study against its real-research pairing has an AI co-pilot: it can propose a directional match, split difference, and reasons-overlap verdict from both texts, with evidence that quotes each side. It never grades its own homework unsupervised — a proposal is only ever a starting point the researcher must review, correct, and submit themselves, and every record carries a provenance flag (not a trust discount) marking whether a draft was requested for it.

8. From hypotheses to decisions

A synthetic study is not a finding — it’s a triage tool. Every hypothesis a cohort raises is a typed claim, not a verdict: it names the numeric split that motivated it and lands on the project’s hypothesis rollup (in the app, under each project), where a team works through it like a backlog instead of filing it away. The cheap ones get resolved cheaply — if a hypothesis is really just a comprehension problem (“9 of 15 misread the pricing tier as annual”) and that’s obviously true on inspection, fix the copy and move on. The ones that actually matter — money, positioning, a launch decision riding on them — go to real users, because that’s what calibration intake exists for: bring in a real study on the same question, code the two against each other, and that comparison is what builds the trust map. Trust, once earned per use case, is what unlocks stronger claims from the next synthetic study on that same question — the loop tightens with use, it doesn’t stay flat.

1. A hypothesis is raised, citing the cohort split that motivated it.
2. It’s triaged on the project’s hypothesis rollup — open, testing, supported, refuted, or actioned.
3. Cheap confirmations get fixed immediately: obvious copy or comprehension issues don’t need a real study to resolve.
4. The ones that matter go to real users — that’s what calibration intake is for.
5. Coded pairings — synthetic split against real one, side by side — build the trust map, use case by use case.
6. Earned trust unlocks stronger claims from the next synthetic study on that same use case.

SpareBrain narrows what you take to real users; it never replaces them — not even once a use case reads green, because trust decays. It goes stale after 90 days and demotes on failure, so the loop never fully closes.

9. What we measure and why

SpareBrain’s own product-improvement measurement follows the same discipline as everything above: it’s labelled honestly, kept to what it needs, and checkable against what’s actually in the code — not asserted in a policy doc no one enforces.

Metadata and aggregates only.Product-improvement measurement reads operational columns — which use case or persona a study picked, its status, its cohort size, whether a collation carried a quality warning, which programmatic tool or route was called and how it went — never the content those columns describe. Your stimulus, your cohort’s responses, your hypotheses, and your comments are never read for analytics. Not summarised, not sampled, not peeked at “just this once” to debug something: the aggregate queries behind this measurement are written so that reading research content isn’t something they can do, not just something they’re asked not to do.

Purpose-limited.This measurement exists to answer product questions — which features get used, where studies fail, whether the honesty mechanisms above are actually engaging in practice — and nothing else. It is never used to build behavioural profiles of individual users, never sold or shared with anyone, and never optimised toward engagement or time-on-site. There is no metric anywhere in SpareBrain that rewards you for using it more; the ones that exist ask whether it was useful.

What’s measured, concretely:counts and rates — how many organizations are active, study outcomes by status, which use cases and personas get picked, cohort sizes, how often the anti-smoothing and exploratory-only labels actually fire, and programmatic (API/MCP) call volume by outcome. What’s deliberately not measured:the text of any stimulus, any cohort member’s response, any hypothesis or comment, or any profile of an individual user’s behaviour across sessions.

Your own usage is yours to see.An organization’s API/MCP call volume is visible to its own owners on the account page— the same metadata this measurement aggregates across organizations, scoped to just your own. Nothing here is collected quietly for later; it’s the same data, at two different zoom levels.

Receipt:this isn’t a separate analytics pipeline — it runs off the same tables the product already writes (study records, collation flags, API/MCP request logs), gated so only the person operating SpareBrain can view the cross-organization rollup, and every query in it is a plain aggregate: a count, a rate, or a distribution, never a row of someone’s research. That’s the same standard this page holds the synthetic-research labelling to — say what’s measured, say what isn’t, and make both true in the code, not just the copy.

References

The claim above isn’t ours to assert without receipts. Read the methodology sections, not just the abstracts — some of this literature comes from teams with a commercial stake in the answer:

• Argyle, Busby, Fulda, Gubler, Rytting & Wingate, “Out of One, Many: Using Language Models to Simulate Human Samples” — Political Analysis 31(3), 2023. LLMs conditioned on real survey demographics reproduced population-level opinion distributions.
• Horton, “Large Language Models as Simulated Economic Agents” — NBER Working Paper 31122, 2023. LLM agents replicated classic economic experiments qualitatively.
• Aher, Arriaga & Kalai, “Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies” — ICML 2023 (PMLR 202). LLM populations reproduced results from classic psychology and economics experiments.
• Park, O’Brien, Cai, Morris, Liang & Bernstein, “Generative Agents: Interactive Simulacra of Human Behavior” — UIST 2023. Believable simulated behaviour from LLM agents with memory and reflection — the architecture behind several commercial digital-twin products today.

Read all four narrowly. Each shows correlation in one specific, carefully calibrated setting — a demographic-conditioned opinion survey, a set of economic games, a handful of classic psychology experiments, a sandbox of daily routines — not a general licence for synthetic prediction. None shows a result transferring automatically to a new product, a new audience, or a question the original study didn’t ask, and the wider literature is genuinely contested — replication attempts on this class of finding are mixed. Population-level correlation also says nothing about whether any one dissenting respondent’s reasoning was captured faithfully. That gap is exactly why the calibration layer above exists, rather than a citation.