The numbers, and how we got them.

Most AI detectors publish one accuracy figure and no method. Ghost Act publishes how often it catches AI text, how often it wrongly flags people, on which models and genres it struggles, and how it does on independent 2026 datasets it never saw. Every figure on this page is generated by the same scripts that fit the model.

The ensemble is refitted whenever a model or a genre is added to its corpus: 8 fits between 24 September and 25 September 2026. The instrument alongside replays the last one, stage by stage, from that fit’s own files.

Refit pipeline2026-09-25 · replay 38

Replay of the fit of 25 Sept 2026; live telemetry to follow. Every figure shown is from that fit’s own files.

Corpus
LM statistics
Text signals
Fit
Held-out
Leave one model out
Leave one genre out
Bootstrap
Publish

epoch: 2,421 / 4,000
loss: 0.137
held-out AUROC: 0.974

loss 0.137

AUROC 0.974

46 signals per text · split 70/30, stratified by label × source · equal total weight per label × genre cell
epoch 500 / 4,000 · loss 0.150 · held-out AUROC 0.973
epoch 1,000 / 4,000 · loss 0.140 · held-out AUROC 0.974
epoch 1,500 / 4,000 · loss 0.138 · held-out AUROC 0.974
epoch 2,000 / 4,000 · loss 0.137 · held-out AUROC 0.974

81%
of held-out AI texts caught: 3%
of held-out human texts wrongly flagged: 82%
caught on models the weights never saw: 0.974
held-out AUROC

In plain words: of every 100 AI-written texts set aside for testing, Ghost Act caught 81. Of every 100 human-written texts, it wrongly flagged 3 (somewhere between 1 and 6). On 4 independent 2026 datasets it never trained on, it caught 33%–52% of the AI text and wrongly flagged 0%–9% of the human text; those are the figures to quote to a sceptic.

Calibration corpus: 660 human texts in 10 genres and 1,674 AI texts from 24 models (2023–26), most written on the same prompts as the human texts. Fitted 25 September 2026. Text you paste into Ghost Act is never used to fit the model.

The same recipe on 20 other 70/30 splits gives a held-out AUROC of 0.974 to 0.985; the intervals on this page are bootstraps within the shipped split (seed 42, ringed).

Fig. 1

8 fits in two days: the test got harder, then the signals caught up

The ensemble is refitted whenever a model or genre is added, and each point is scored on that fit’s own held-out split, so this is a history of the fit, not a learning curve.

held-out AUROC leave-one-model-out AUROC texts in the corpus▲ corpus grew · ◆ signals changed · tuned = tuning

Hover or tab to a fit to read its date, note, corpus, models, signals, AUROC, share caught and false flags.

Figure 1. Held-out AUROC of that fit on its own 30% split; the split, corpus and model list change between fits, so points are not on one benchmark. Filled dots: held-out AUROC; hollow dots: leave-one-model-out AUROC for the 3 fits that ran it. Bars: texts in the corpus at each fit, with the number of models and signals beneath; the word under each date says what changed since the fit before. Read from the git history of the calibration file: 8 fits between 24 Sep 2026, 17:13 UTC and 25 Sep 2026, 12:24 UTC. No interval is drawn, because each point is one split; the two annotations quote the fit’s own note.

Fig. 2

Reading a flag

How often a flag is right depends on how much of the text you check is actually AI-written.

Share of AI-written text among what you check

Your assumption, not a measurement: the base rate is the one thing Ghost Act cannot know about what you check.

10%

Measured on the held-out split at the shipped threshold: 81% of AI texts caught (396 of 491), 3.0% of human texts wrongly flagged (6 of 198).

Of the texts flagged,75% are AI-written95% interval 61–90%8 of 11 flagged in the hundred

Of the texts passed,98% are human-written95% interval 97–98%87 of 89 passed in the hundred

In a classroom: of thirty essays the students wrote themselves, about 1 would be wrongly flagged at 3.0%, and up to 2 at the upper end of the interval (5.6%). A flag is a reason to look at the essay with its author, never a verdict.

Hover or tap a text, or focus a legend entry, for its count and where it comes from.

Figure 2. Positive and negative predictive value at the shipped threshold (p = 0.78) against an assumed share of AI-written text, from the held-out split (689 texts: 198 human, 491 AI): 81% of AI texts caught (396 of 491), 3.0% of human texts wrongly flagged (6 of 198). Each interval bounds the predictive value with the endpoints of the two rates’ 95% bootstrap intervals (2,000 resamples; caught 77–84%, wrongly flagged 1.0–5.6%); it is descriptive, not a joint interval. The hundred texts are rounded to whole texts; the printed percentages are exact. The share of AI text is the reader’s assumption, not a measurement, and a flag is a reason to look, not a verdict.

Fig. 3

Where it struggles: by model, and by genre

Each model's texts are scored by an ensemble refitted without that model (leave-one-model-out), and each genre's human texts by a fit that never saw the genre (leave-one-genre-out). "Never saw" is true of the logistic weights; the vocabulary table was mined once on an earlier training split and stays fixed throughout.

By model: share of each model's texts caught

24 models · 1,674 AI texts · leave-one-model-out · flag threshold p = 0.78

2025–26 · 15 models

Qwen 3.8 Flash67%4 of 6
Claude Opus 5.572%52 of 72
Grok 4.772%52 of 72
DeepSeek V4.1 Flash75%38 of 51
GPT-6 Luna80%146 of 182
Claude Sonnet 580%147 of 183
GPT-6 Sol85%61 of 72
Kimi K385%61 of 72
Mistral Large100%12 of 12
GLM 5.386%62 of 72
Gemini 3.8 Flash91%148 of 162
Gemini 3.1 Pro97%61 of 63
GPT-5.6 Sol97%70 of 72
GPT-5.6 Luna99%71 of 72
Llama 4 Maverick100%72 of 72

2023–24 · 9 models

ChatGPT (2023)51%109 of 213
GPT-481%21 of 26
Mistral 7B Chat100%8 of 8
GPT-4o90%45 of 50
Qwen 2 72B100%15 of 15
Yi Large100%15 of 15
Llama 2 Chat100%22 of 22
Llama 3.3 70B98%44 of 45
GPT-4o mini100%45 of 45

By genre: share of human texts wrongly flagged

10 genres · 660 human texts · leave-one-genre-out · same threshold

Encyclopaedic prose14%11 of 78
Medical answers8%3 of 40
Scientific abstracts4%10 of 262
University essays (BAWE)7%2 of 30
News2%1 of 55
Book excerpts0%0 of 22
School essays (PERSUADE)0%0 of 30
Finance answers0%0 of 40
Film reviews0%0 of 40
Reddit answers0%0 of 63

Hover, tap or tab to a row for its counts, 95% interval, median probability and quartiles.

Figure 3. Left: for each of the 24 models in the corpus, the share of its texts scored above the flag threshold (p = 0.78) by an ensemble refitted without that model (1,674 AI texts in all). Right: for each of the 10 human genres, the share of its texts wrongly flagged by a fit that never saw the genre (660 human texts). A dot is the observed share, printed as k of n; the whisker is its 95% Wilson interval; hollow dots mark rows with fewer than 25 texts. Rows are ordered by the lower end of the interval, the hardest first, the models within two eras. The dashed rule is the threshold's own target, that in the training split no genre's 90th percentile lies above the threshold, so at most 10% of its humans are flagged; on this out-of-genre check Encyclopaedic prose sits above it. The solid tick is the pooled rate: 82% of all AI texts caught under leave-one-model-out, 3.0% of the 198 held-out human texts wrongly flagged. Intervals are per estimate and descriptive, not simultaneous, so a few of the 34 rows will miss their true rate by chance. Hover or focus a row for its interval, median probability and quartiles. "Never saw" applies to the logistic weights, refitted for every check; the vocabulary table was mined once on an earlier training split of 957 rows (25 Sept 2026) and is fixed throughout.

What the numbers do not say

A probability is evidence, not proof. Never use a score alone to accuse anyone.
Paraphrasing or rewriting by another model lowers every detector’s recall, including this one’s.
Human text rewritten by a model was caught 33% of the time on the ARB dataset, at 9% false flags: rewriting evades, as the literature says.
Lightly AI-edited human text usually reads as human: on OpAI-Bench, 2% of half-edited drafts were caught. The mixed-authorship check shows which parts differ.
Recall on a model the weights never saw varies widely: the hardest with a sizeable sample, ChatGPT (2023), was caught 51% of the time (109 of 213).
Keyed provider watermarks (Claude, Gemini) cannot be verified by anyone but the provider.
Tuned for English; other languages get no calibrated verdict. Zero-shot detectors are also known to be biased against non-native English writers.

A flag is a reason to look at the evidence with the person who wrote the text. It is never a verdict on them.

Fig. 4

How far it travels: from our held-out split to data we never touched

The further the data is from ours, the lower the numbers; these are the ones to quote to a sceptic.

a The generalisation ladder

Each rung is one evaluation, further from our own data than the last: the held-out split, leave-one-model-out, both genre and model unseen, then the 4 independent 2026 datasets that have both human and AI texts, scored at the shipped threshold. Chance is 0.5.

Hover or focus a rung for its AUROC, interval and the texts it was measured on.

b The ledger: 5 datasets the fit never touched

Every text scored at the shipped threshold. A violet dot is the share of an AI-written group caught; a teal dot is the share of a human-written group wrongly flagged.

DatasetTextsAUROC

share above the threshold (%)

ARBhuman text rewritten by an LLMarXiv:2607.29539

n = 120 human/ 120 AI

AUROC0.812

Rewriting evades: 33% of rewritten texts caught, 9% of human texts flagged.

AITDNAreal co-written documents with full edit historiesarXiv:2606.04906

n = 148 human/ 214 AI

AUROC0.866

Mostly-AI co-written documents are caught about half the time (52%; 34% to 82% by model); human-only 2% flagged, mostly-human 8%.

OpAI-Benchhuman drafts progressively edited by AIarXiv:2606.06481

n = 120 human/ 240 AI

AUROC0.706

Fully AI drafts caught about half the time (63%); half-edited drafts read as human (2% flagged, against 1% of the human drafts).

MELD-evalfresh 2026 generators across eight genresarXiv:2605.06903

n = 60 human/ 240 AI

AUROC0.827

Varies widely by generator: qwen-3.6-plus 82% caught, gpt-5.4-mini 10%; humans 0% flagged.

PERSUADE 2.0essays by English-language learners and native writers

n = 240 human

AUROCn/a

English-language learners 1% vs native writers 2% flagged: no gap detectable at n = 120 per group.

Human writers on these sets are flagged between 0% and 9%; English-language learners and native writers show no gap detectable at n = 120 per group.

Hover or focus a group for k of n, its rate, the 95% Wilson interval and the median probability.

Figure 4. (a) AUROC of the ensemble on evaluations at increasing distance from the calibration data: the held-out split (n = 689; whisker: 95% bootstrap interval, 2,000 resamples), leave-one-model-out (n = 2,334: human texts scored out of fold, AI texts by fits that never saw their model), the pooled out-of-fold set (n = 2,334: humans scored out of genre, AI out of model) and 4 independent 2026 datasets scored at the shipped threshold, p = 0.78. A hollow marker would mean fewer than 200 texts. (b) For each set, the share of each group above the threshold, with a 95% Wilson interval and k of n printed beside it; AI groups are the share caught, human groups the share wrongly flagged (false flags). The vocabulary table was mined once on an earlier training split, so “never touched” is true of the fitted weights and of these texts, not of that table. The sets are datasets, not products; their AUROCs are not comparable with one another or with ours (different genre mixes and n), and no single set says much about any one model.

Fig. 5

One threshold, three views

The flag threshold is a choice: lower it and more AI text is caught, but more people are wrongly flagged. Here is the whole trade-off, and where the shipped threshold sits in each view.

Where the scores land

Every text scored by a fit that never saw its model or its genre, and the band and threshold they meet.

At the shipped threshold p = 0.78

73% caught · 4.1% wrongly flagged

Of 1,674 AI texts scored by a fit that never saw their model and 660 human texts scored by a fit that never saw their genre. A what-if for reading the trade-off: it does not change what Ghost Act flags.

Share of AI texts caught, reading each curve at a fixed false-flag rate
false flags	held-out split	pooled out-of-fold
0.1%	n/aone false flag in 198 is already 0.5%	n/aone false flag in 660 is already 0.15%
1%	45%	52%
5%	85%	76%
shipped p = 0.78	81%at 3.0% false flags	73%at 4.1% false flags

The whole trade-off

Every possible threshold at once; the ringed point is the one that ships.

Is the probability honest?

Texts given a probability near p should turn out to be AI about p of the time.

Classes weighted equally, as in the fit. Bar width follows the number of texts in the bin; the shading is the Uncertain band.

Hover or focus a bin, a bar or the curve to print its numbers here.

Figure 5. Panel A: calibrated probabilities of 1,674 AI texts, each scored by a fit that never saw its model (violet, upward), and 660 human texts, each scored by a fit that never saw its genre (teal, downward), pooled out-of-fold, in 25 equal bins; the shaded range is the Uncertain band (0.29–0.78) and the dashed rule the shipped threshold, p = 0.78. The explorer reads both histograms (mass taken as uniform inside a bin) and snaps to the nearest point of the pooled out-of-fold ROC; it is a what-if, not a setting. Panel B: ROC curve of the held-out split (198 human, 491 AI texts) with a 95% bootstrap band of the caught rate at each false-flag rate, resampled within the split; the headline AUROC interval uses 2,000 resamples. The thin line is the pooled out-of-fold curve, not the ROC of the shipped model. The ringed point is the shipped operating point on the held-out split: 81% caught (396 of 491) at 3.0% false flags (6 of 198); the table reads each curve at a fixed false-flag rate. Panel C: reliability diagram of the held-out split in 10 bins of predicted probability; the bar top is the share of texts in the bin that were AI with the two classes weighted equally, as in the fit (71% of this split is AI); bar width is proportional to the bin's n; the whisker is the Wilson 95% interval of the raw share (AI texts of n), so it is centred on the raw value, not the balanced one.

Fig. 6

How a text becomes a probability

Every signal is one number measured from the text. Alone, most separate AI from human writing only weakly; the logistic fit reads all 46 together and gives each a weight, so the figure shows strength alone beside weight in this fit.

texttokens46 signalslogistic fusionprobability

flag above p = 0.78; Uncertain 0.29–0.78

AUROC alone; weight toward AIweight toward humanchance zone

more of it reads as AI less of it reads as AI

Language model4

0.69

Clipped log-likelihood
Clipped Fast-DetectGPT
Bottom-20% log-prob
Bottom-20% Rényi entropy

Zero-shot, for reference: Clipped Fast-DetectGPT alone 0.53; Binoculars 0.57; the 4 language-model statistics together 0.69; the full ensemble 0.97. All held-out AUROC under our browser scorer (distilGPT-2).

Vocabulary profile1

0.76

Vocabulary profile+1.06

Style detectors9

0.81

LLM vocabulary
Stock phrases
Transition stacking
Uniform rhythm
Rule of three
Repeated openers
Em-dashes
Lexical diversity
Paragraph uniformity

Surface statistics18

0.94

Sentence length
Sentence length spread
Sentence length skew
Word length
Commas
Semicolons
Colons
Dashes
Parentheses0.79−0.86
Quotation marks
Exclamations and questions
Contractions
First person
Second person
Digits
Capitalised words
Lexical variety
Paragraph breaks

Stylometry14

0.75

Hapax ratio
Lexical density
That-clauses
Nominalisations
Participial openers
Wh-relatives
Concessives
Discourse particles
Hedges
Intensifiers
Headings and lists
Sentence length p10
Sentence length p90
Adjacent-sentence overlap

Hover or focus a signal to read its AUROC alone, its direction and its coefficient in this fit.

Figure 6. Left: held-out AUROC of each signal alone on 689 texts (198 human, 491 AI), on a 0.5–1.0 scale; the shaded zone runs from chance to about two standard errors above it (0.5–0.55). The bar under each group name is the held-out AUROC of a logistic fit on that group alone. Arrows give the direction that reads as AI-like when the signal is taken alone. Right: standardised coefficients of the shipped fit (each signal scaled to unit variance on the training split), violet toward AI, teal toward human. Correlated signals share weight, so a coefficient can point the other way from the signal's own direction (11 of 46 do); coefficients are in this fit and are refitted when a model or genre is added. Flag threshold p = 0.78; Uncertain band 0.29–0.78.

Appendix

Protocol, provenance and sources

Everything a reader needs to check the figures above: the recipe in order, what each fitted part was allowed to see, where the texts came from, how to run it again, and the papers behind the signals.

Protocol

The pipeline in order. Every parameter is read from the fit’s own files.

Corpus. 2,334 texts: 660 human across 10 genres and 1,674 AI from 24 models, read from 53 labelled source folders (each a public corpus, or one model crossed with one prompt set).
Language-model statistics. The scorer is distilGPT-2: Xenova/distilgpt2 at revision a41c104, file decoder_model_merged, 85 MB, run in the browser. It yields the 4 language-model signals below. The performer GPT-2 (bf2c7f0, 128 MB) computes Binoculars for reference only; it is not an ensemble input.
Signals. 46 signals in 5 groups: language model 4, vocabulary profile 1, style detectors 9, surface statistics 18, stylometry 14. Each is standardised by the training mean and spread before the fit.
Split. 70/30, stratified by label × source, seed 42: 1,645 training texts and 689 held-out texts (198 human, 491 AI).
Fit. L2-regularised logistic regression, batch gradient descent on standardised signals: 4,000 epochs, learning rate 0.1, L2 penalty 0.01. Balancing: equal total weight per label × genre cell.
Threshold rule. 95th percentile of training humans scored by fits that never saw their genre (leave-one-domain-out): the threshold is p = 0.78. The Uncertain band runs from the out-of-genre humans’ 80th percentile, 0.29, up to the threshold.
Evaluations. The held-out split (689 texts) with bootstrap intervals from 2,000 resamples; leave-one-model-out, one refit per model over 24 models, humans scored out of fold in 5 folds and the threshold recomputed each time; leave-one-genre-out over 10 human genres.
External evaluation. 5 independent 2026 datasets scored by the shipped fit at the same threshold, p = 0.78, on 25 Sept 2026. Nothing from them feeds back into the fit.

What each fitted part saw

Which parts are refitted in the evaluations and which stay fixed, so the word “unseen” is precise.

Part	Fitted or mined on	In the evaluations
Logistic weights	The logistic regression is fitted on the 1,645 training texts.	Refitted without the held-out model or genre in every evaluation, so “unseen” is true of the weights.
Vocabulary table	Mined once on 957 training rows of an earlier split, 25 Sept 2026.	Fixed during leave-one-model-out and leave-one-genre-out, so “unseen” is true of the weights, not of this table.
Style-detector thresholds	Checked against the whole corpus of 2,334 texts.	Fixed.
Style-score bands	Whole corpus: mixed above 18, strong above 28 on the 0 to 100 style score.	Fixed.
Flag threshold	Training humans scored leave-one-genre-out: p = 0.78.	Recomputed from the remaining texts in every leave-one-model-out refit.
Uncertain band	Out-of-genre humans, 80th percentile up to the threshold: 0.29 to 0.78.	Fixed; the same band on every figure.
Scorer	distilGPT-2, pinned at revision `a41c104`.	Never fitted.

Sources and licences

The human rows are third-party text used unmodified for calibration; the committed files hold fitted numbers only, no source text.

Source	Rows	Licence
RAID	Dugan et al., ACL 2024: abstracts, news and books; human text plus GPT-4, ChatGPT, Llama 2 Chat and Mistral 7B Chat generations	MIT
HC3	Guo et al., 2023: Reddit ELI5, finance and medical answers and Wikipedia-style paragraphs, human against ChatGPT	CC BY-SA 4.0
GEDE	Gehring et al., 2025: student essays from BAWE and PERSUADE, plus essays on the same tasks by GPT-4o mini and Llama 3.3 70B	CC BY-NC-SA 4.0
Defactify	Roy et al., 2025: New York Times articles and GPT-4o, Yi Large and Qwen 2 72B versions of them (clean train split only)	CC BY 4.0
Wikipedia	wikimedia/wikipedia, the 20231101.en dump: article leads	CC BY-SA 4.0
IMDB	stanfordnlp/imdb: film reviews	Research use (Maas et al., 2011)
arXiv	ccdv/arxiv-summarization: abstracts	arXiv terms of use
Project samples	Generated for this project in September 2026 with scripts/generate_samples.ts: each model answering the GEDE assignments and writing to the Defactify headlines at the matching human lengths. Claude via the Anthropic API; the rest via OpenRouter	Model output; the prompts derive from GEDE (CC BY-NC-SA 4.0) and Defactify (CC BY 4.0)

Text you paste into Ghost Act is never used to fit the model. Analysis runs on your device; the corpus above is all the text the fit has seen.

Reproduce it

The commands that make every figure on this page, and the files they wrote, as they are in the repository.

$ npm run calibrate# fits the ensemble; writes calibration.json and curves.json$ npx tsx scripts/external-eval.ts# scores the independent sets; writes external-eval.json$ npx tsx scripts/per-source.ts# prints the probability by source, for the struggles figure$ bash scripts/fit-history.sh# reads every past fit from git, for fit-history.json

calibration.jsongenerated 25 Sept 2026, 12:24 UTC13 KBweights, thresholds, held-out and per-model figures, style bands
curves.jsongenerated 25 Sept 2026, 12:24 UTC55 KBthe recipe, fit trace, ROC and reliability curves, per-model and per-genre score summaries
external-eval.jsongenerated 25 Sept 2026, 12:48 UTC3 KBper-set and per-group results on the independent datasets
fit-history.jsonlatest fit 25 Sept 2026, 12:24 UTC3 KBthe 8 past fits read from git

Hover or focus a file for what it holds and when it was generated.

Each link is the imported JSON itself, compact, as a data URL; the size is the download’s.

References

The papers named in the methods document, in the order they appear there.

Gehrmann, Strobelt and RushGLTR: statistical detection and visualization of generated text.ACL 2019 demoarXiv 1906.04043
Kirchenbauer, Geiping, Wen, Katz, Miers and GoldsteinA watermark for large language models (KGW).ICML 2023arXiv 2301.10226
Dathathri et al.Scalable watermarking for identifying large language model outputs (SynthID-Text).Nature 634, 2024doi 10.1038/s41586-024-08025-4
Bao, Zhao, Teng, Yang and ZhangFast-DetectGPT: efficient zero-shot detection of machine-generated text via conditional probability curvature.ICLR 2024arXiv 2310.05130
Hans et al.Spotting LLMs with Binoculars: zero-shot detection of machine-generated text.ICML 2024arXiv 2401.12070
Monroe, Colaresi and QuinnFightin’ words: lexical feature selection and evaluation for identifying the content of political conflict (weighted log-odds).Political Analysis 16, 2008doi 10.1093/pan/mpn018
Liang, Yuksekgonul, Mao, Wu and ZouGPT detectors are biased against non-native English writers.Patterns 4, 2023arXiv 2304.02819
Dugan et al.RAID: a shared benchmark for robust evaluation of machine-generated text detectors.ACL 2024arXiv 2405.07940
Omidi et al.On SynthID-Text detection: the mean score’s power is unimodal in the number of layers; a Bayesian score is monotone.arXiv preprint, 2026arXiv 2603.03410
Clipped likelihood statisticsFloor-clipped log-probability and Fast-DetectGPT curvature, so partial human edits stop dominating the average.arXiv preprint, 2026arXiv 2609.29935
Least-likely-token statisticsThe mean log-probability and Rényi-2 entropy of a text’s least likely tokens.ICML 2026arXiv 2606.02158
Robust stylometryHapax ratio and lexical density hold across 27 models and 10 domains.arXiv preprint, 2026arXiv 2606.04177
Lexicogrammatical ratesBiber-style rates of that-clauses, nominalisations, hedges and intensifiers in machine text.arXiv preprint, 2026arXiv 2604.14111
SlopShapeHeading and list density as a signal of machine text.arXiv preprint, 2026arXiv 2609.15369
MELD-evalFresh 2026 generators across eight genres.arXiv preprint, 2026arXiv 2605.06903
AITDNAReal co-written documents with full edit histories.arXiv preprint, 2026arXiv 2606.04906
OpAI-BenchHuman drafts progressively edited by AI.arXiv preprint, 2026arXiv 2606.06481
ARBHuman text rewritten by an LLM.arXiv preprint, 2026arXiv 2607.29539

Terms used on this page

The 19 defined terms, in one place, readable without a pointer.

Show the 19 definitions

AUROC: Area under the ROC curve: the chance that a randomly chosen AI text scores higher than a randomly chosen human text. 0.5 is a coin flip, 1.0 is perfect separation.
held-out: Texts set aside before fitting (30% of the corpus) and scored only afterwards, so the fit could not have learned them.
caught: The share of AI-written texts scored above the flag threshold (the recall).
false flag: A human-written text scored above the flag threshold.
threshold: The probability above which a text is flagged, chosen so that about 5% of human texts from unfamiliar genres would be flagged.
Uncertain band: Probabilities between the band’s lower edge and the threshold get an Uncertain verdict rather than a flag: a range a fifth of unfamiliar human writing also reaches.
leave-one-model-out: Each model’s texts are scored by a version of the ensemble fitted without that model, so the figure estimates performance on a model never seen.
leave-one-genre-out: Each genre’s human texts are scored by a fit that never saw that genre.
95% confidence interval: The range that would contain the true rate in 95 of 100 repeats of the experiment; wider when there are fewer texts.
bootstrap: Re-sampling the held-out texts with replacement, 2,000 times, to see how far the figure moves.
Wilson interval: A confidence interval for a proportion that behaves sensibly near 0% and 100% and with small counts.
reliability diagram: For texts given a probability near p, the share that really were AI; on a well-calibrated detector the points sit on the diagonal.
ROC curve: Every possible threshold at once: the share of AI texts caught (up) against the share of human texts wrongly flagged (across).
logistic regression: A weighted sum of the signals squashed into a probability; the weights are what the fit learns.
operating point: The one threshold the product ships at: a single point on the ROC curve.
base rate: How much of the text you actually check is AI-written; it decides how often a flag is right.
epoch: One pass of gradient descent over the training texts; the fit runs 4,000 of them.
loss: How far the fit’s probabilities are from the labels on the training texts (weighted log-loss); it falls as the fit improves.
quartiles: The 25th, 50th and 75th percentile: the middle half of the values lies between the outer two.