Language models infer writers' emotion appraisals better than one reader but worse than five combined
Terms used on this page
- Appraisal
- How a person evaluated an event on one dimension, such as how sudden, how pleasant or how controllable it was. Here, 21 statements each rated from 1 to 5.
- Writer
- The person who lived the event, wrote the text and rated their own appraisals of it.
- Reader
- A crowd worker who read a text, with emotion words masked, and guessed the writer's emotion and ratings. Each text had five readers.
- Five-reader average
- For each text and statement, the mean of its five readers' ratings.
- Panel
- A group of raters, readers or models, whose ratings of a text are averaged.
- No-text baseline
- The average rating on each statement over 3,144 other texts of the corpus by other writers, used as the guess for every text.
- Mean squared error
- The squared difference between a rating and the writer's rating, averaged over the 21 statements and the texts. Lower is better.
- Skill score
- Where a system's mean squared error falls between the no-text baseline (0) and the five-reader average (1). Below 0 is worse than the baseline; above 1 would beat the five readers.
- Emotion accuracy
- The share of texts where the emotion chosen is the one the writer was asked to write about.
- Macro-F1
- For each of the 13 emotion options, the balance (harmonic mean) of how often a choice of that emotion was right and how often that emotion was found, averaged over the options so that each counts equally.
- 95% interval
- The range that contains the result in 95% of 10,000 resamples of the texts.
- Exploratory
- An analysis chosen after the results were known, not in the design fixed before the runs.
- Default effort
- The amount of reasoning a model uses when the request does not set one.
- Standard prompt
- The readers' questions as the readers were asked them: one whole number from 1 to 5 for each statement, as the guess for this writer.
- Average prompt
- The standard prompt with one added instruction: for each statement, give the average rating that many writers of the same description would give, to one decimal place.
- Rule against one whole number
- The first sentence of the average prompt's instruction: "do not choose one whole number for this writer".
In the crowd-enVENT corpus (Troiano, Oberländer and Klinger, 2023), people described an event from their own life and rated how they had appraised it on 21 statements. For 1,200 of these texts, five other people each read the text, with emotion words masked, and tried to give the writer's ratings. Ten current language models answered those readers' questions at default reasoning effort, under a design fixed before any request on these texts, and scored from 0.22 to 0.68 on a scale where 0 is a guess that ignores the text and 1 is the five-reader average. In a condition added afterwards, nine of the models were asked for the average rating that many writers of the same description would give, a question the readers were not asked, and scored from 0.66 to 1.15.
Findings
- Every model was more accurate than the no-text baseline and less accurate than the five-reader average. Skill scores ranged from 0.22 for Claude Haiku 5.5 to 0.68 for Claude Opus 5.5 and 0.68 for Gemini 3.8 Flash. All 20 comparisons with the two benchmarks were statistically significant after correction for multiple tests (each p ≤ 0.002). Appraisal skill
- Every model was more accurate than one reader on average, and eight of the ten matched an average of two to three readers (exploratory). One reader scored −0.35, worse than the no-text baseline. Claude Opus 5.5's error equalled that of an average of 2.7 readers and Claude Haiku 5.5's that of 1.7. The corpus authors' model, fine-tuned on the rest of the corpus, scored 0.42, below eight of the ten models. Appraisal skill
- Asked for an average rating in place of one guess, every model tested had lower error, and five of nine then had lower error than the five-reader average (exploratory). The added instruction asks for the average rating that many writers of the same description would give, to one decimal place. Error fell by 0.250 to 0.413, and skill scores ranged from 0.66 to 1.15. The readers were not asked for an average, and against their average after rescaling only Gemini 4 Argon had lower error. The condition was added after the results above were known. Asking for an average
- Every model's error was lower under the request prompt than under the decimal prompt, and under the average prompt than under the strict prompt (exploratory). A control on the 300 repeated texts compared five prompts: the standard prompt; the decimal prompt, which only changes the answer format to one decimal place; the strict prompt, which adds the rule against one whole number; the request prompt, which adds the request for the average instead; and the average prompt, which has both. Averaged over the nine models, the error was 1.928, 1.812, 1.698, 1.621 and 1.607, and both differences in the heading hold for this average too. The decimal prompt's error was lower than the standard prompt's for the average of the models (36% of the average prompt's difference from the standard prompt) and for six models, and higher for none; for three there is no credible evidence of a difference. Parts of the average prompt
- Every model's ratings followed the average of four readers more closely than a fifth reader's did (exploratory). Their correlation with that average was 0.67 to 0.75 for the models and 0.56 for a fifth reader. Two models' ratings correlated 0.86 on average, two readers' 0.43. Models beside readers
- Replacing one reader of a panel with one model's answer raised the panel's correlation with the writers' ratings, for every model and every panel size from two to five (exploratory). With five raters, the correlation was 0.543 to 0.557, against 0.534 for five readers. The panel's error fell as well, but part of that gain came without the text: replacing the reader with the no-text baseline lowered it too. Models beside readers
- Nine models chose the emotion the writer was asked to write about more often than the readers' majority vote, seven of them with a 95% interval of the difference above zero (exploratory). Accuracy ranged from 53.5% to 70.2%, against 57.9% for the majority vote and 50.4% for one reader. Gemini 4 Argon was the most accurate on emotion but eighth of ten on appraisals. Emotion
- Most of the gap to the five-reader average came from systematic offsets and too wide a spread of ratings (exploratory). The ratings of seven models correlated more strongly with the writers' ratings than the five-reader average did. After a per-statement rescaling, the same seven models had lower error than the five-reader average rescaled in the same way. Where the gap comes from
- The largest offsets were shared across models. All ten rated events as less urgent (−0.82 on the 1 to 5 scale, averaged over models), less effortful (−0.49) and less sudden (−0.43) than the writers did, and nine of ten agreed more than the writers that the consequences could be lived with (mean +0.83), a statement worded differently for writers. Shared offsets
- The prespecified checks did not change these results. With integer benchmarks the skill scores ranged from 0.35 to 0.81; on events the writers confirmed, every model scored higher; and on a repeat of 300 texts each model's error changed by at most 0.023. The planned check that resampled readers instead of texts could not give a usable interval. Checks
Task
Each writer recalled an event in which they had felt a given emotion (one of 12, or no particular emotion), completed the sentence "I felt [emotion] when/because…" and rated 21 statements about how they had seen the event, such as "The event was sudden or abrupt." Readers saw the text with words naming the emotion replaced by three dots, chose the emotion they thought the writer felt and rated the 21 statements as they thought the writer had. Readers were paid crowd workers and were offered a bonus if their ratings were among the 5% closest to the writers'. Each model received the readers' instructions and questions for one text per request, without the questions about the reader, the attention checks and the text about consent and payment, and answered in a fixed format without explanation.
Writers rated first-person statements from 1 ("Not at all") to 5 ("Extremely"). Readers and models rated third-person versions and were told that 1 means "I don't agree at all" and 5 "I completely agree". Three statements differed in more than person. Writers rated "The event was familiar", "I anticipated that I would easily live with the unavoidable consequences of the event" and "I tried to shut the situation out of my mind"; the readers' and models' versions are A2, A16 and A20 in the example below.
The example below is the first text in the random order in which texts were sent. Its writer was asked to write about surprise.
One text with the writer's, the readers' and two models' answers
“I was a child and my parents ... my sister and I with a golden retriever puppy”
| Question | Writer | Five readers | Claude Opus 5.5 | Claude Haiku 5.5 |
|---|---|---|---|---|
| Emotion | surprise (asked to write about) | joy, joy, surprise, surprise, surprise | surprise | joy |
| A1. The event was sudden or abrupt. | 2 | 4 2 5 3 5 (mean 3.8) | 5 | 4 |
| A2. The event was familiar to its experiencer. | 1 | 3 4 2 1 1 (mean 2.2) | 1 | 1 |
| A3. The experiencer could have predicted the occurrence of the event. | 1 | 3 4 3 1 2 (mean 2.6) | 1 | 1 |
| A4. The event was pleasant for the experiencer. | 5 | 2 3 5 4 5 (mean 3.8) | 5 | 5 |
| A5. The event was unpleasant for the experiencer. | 1 | 4 3 1 1 1 (mean 2.0) | 1 | 1 |
| A6. The experiencer expected the event to have important consequences for him/herself. | 4 | 2 2 4 1 3 (mean 2.4) | 4 | 3 |
| A7. The event was caused by chance, special circumstances, or natural forces. | 1 | 3 3 1 1 1 (mean 1.8) | 1 | 3 |
| A8. The event was caused by the experiencer’s own behavior. | 1 | 2 4 1 1 3 (mean 2.2) | 1 | 1 |
| A9. The event was caused by somebody else’s behavior. | 5 | 2 3 5 5 5 (mean 4.0) | 5 | 4 |
| A10. The experiencer anticipated the consequences of the event. | 1 | 2 3 2 1 2 (mean 2.0) | 2 | 2 |
| A11. The experiencer expected positive consequences for her/himself. | 5 | 2 3 5 3 5 (mean 3.6) | 5 | 4 |
| A12. The event required an immediate response. | 5 | 2 3 5 2 4 (mean 3.2) | 2 | 2 |
| A13. The experiencer was able to influence what was going on during the event. | 2 | 1 2 1 1 3 (mean 1.6) | 1 | 1 |
| A14. Someone other than the experiencer was influencing what was going on. | 5 | 2 2 5 5 5 (mean 3.8) | 5 | 4 |
| A15. The situation was the result of outside influences of which nobody had control. | 1 | 2 4 1 1 1 (mean 1.8) | 1 | 1 |
| A16. The experiencer anticipated that he/she could live with the unavoidable consequences of the event. | 5 | 2 3 1 4 4 (mean 2.8) | 5 | 4 |
| A17. The event clashed with her/his standards and ideals. | 1 | 3 3 1 1 1 (mean 1.8) | 1 | 1 |
| A18. The actions that produced the event violated laws or socially accepted norms. | 1 | 3 2 1 1 1 (mean 1.6) | 1 | 1 |
| A19. The experiencer had to pay attention to the situation. | 5 | 2 4 3 2 2 (mean 2.6) | 4 | 3 |
| A20. The experiencer wanted to shut the situation out of her/his mind. | 1 | 3 3 1 1 1 (mean 1.8) | 1 | 1 |
| A21. The situation required her/him a great deal of energy to deal with it. | 3 | 3 3 1 2 2 (mean 2.2) | 2 | 1 |
| Mean squared error on this text | 1.74 (of their mean ratings) | 1.05 | 1.52 |
Appraisal skill
The skill scores and their 95% intervals are in the chart at the top of the page. The table gives each system's mean squared error on the 21 ratings, from which the skill scores are computed, and its emotion results. "Readers with the same error" is exploratory: averaging more readers lowers the error, from 2.475 for one reader to 1.923, 1.739, 1.647 and 1.592 for two to five, and each model is placed on that curve. The models' error exceeded the five-reader average's by 0.207 to 0.513. In an exploratory comparison, every model's error was below one reader's average error, by 0.370 for Claude Haiku 5.5 to 0.676 for Claude Opus 5.5, with 95% intervals well clear of zero.
| System | Mean squared error | Readers with the same error | Emotion accuracy | Emotion macro-F1 |
|---|---|---|---|---|
| No text (average ratings of other texts) | 2.247 | |||
| One reader (average over readings) | 2.475 | 1 | 50.4% | 0.492 |
| Five-reader average (emotion: the readers' majority vote) | 1.592 | 5 | 57.9% | 0.563 |
| RoBERTa fine-tuned on the rest of the corpus | 1.975 | 1.9 | 60.5% | 0.594 |
| Claude Opus 5.5 | 1.799 | 2.7 | 65.6% | 0.646 |
| Gemini 3.8 Flash | 1.801 | 2.7 | 67.3% | 0.663 |
| GPT-6.1 Terra | 1.820 | 2.6 | 66.2% | 0.652 |
| Claude Sonnet 5.5 | 1.846 | 2.4 | 59.7% | 0.590 |
| GPT-6 Astra | 1.865 | 2.3 | 67.8% | 0.670 |
| GPT-6.1 Sol | 1.867 | 2.3 | 67.6% | 0.669 |
| GPT-6 Luna | 1.897 | 2.1 | 58.1% | 0.563 |
| Gemini 4 Argon | 1.907 | 2.1 | 70.2% | 0.699 |
| DeepSeek V4.1 Flash | 2.018 | 1.8 | 63.4% | 0.616 |
| Claude Haiku 5.5 | 2.105 | 1.7 | 53.5% | 0.525 |
Earlier studies of this corpus used root mean squared error per statement, averaged over statements. On that measure the models scored 1.33 to 1.44, against 1.57 for one reader, 1.46 for the readers' majority vote, 1.26 for the five-reader average, 1.40 for the fine-tuned model and 1.50 for the no-text baseline. Ruder, Uusberg and Sirts (2025) reported 1.45 for one GPT-4 answer per text and 1.57 for one reader.
Models beside readers
This section is exploratory. One reader's ratings are spread so widely that the no-text baseline, which ignores the text, has a lower error than one reader. Each comparison below therefore also shows the no-text baseline, and is repeated as a correlation, which a guess that is the same for every text cannot improve.
Against the readers. For each text, one reader was set aside and the other four were averaged, in all five ways. A rater's distance from that average of four readers is the mean squared difference between the two sets of ratings, and lower is closer. The reader set aside, here called the fifth reader, was at 1.379, the no-text baseline at 1.199 and the writer's own ratings at 1.647 (the error of an average of four readers, seen from the readers' side). Every model was closer than the baseline, by 0.156 to 0.400. Every model's ratings also correlated more strongly with the average of four readers (0.67 to 0.75, averaged over statements) than the fifth reader's did (0.56). Each 95% interval of these differences excludes zero.
Each rater's distance from the average of four readers, and its correlation with it
| Rater | Distance from four readers | Difference from the no-text baseline (95% interval) | Correlation with four readers |
|---|---|---|---|
| A fifth reader | 1.379 | 0.56 | |
| No-text baseline | 1.199 | ||
| The writer | 1.647 | 0.52 | |
| Claude Opus 5.5 | 0.798 | −0.400 (−0.438 to −0.364) | 0.75 |
| Gemini 3.8 Flash | 0.807 | −0.392 (−0.429 to −0.354) | 0.73 |
| GPT-6.1 Terra | 0.824 | −0.374 (−0.412 to −0.337) | 0.74 |
| Claude Sonnet 5.5 | 0.829 | −0.370 (−0.405 to −0.334) | 0.73 |
| GPT-6 Astra | 0.851 | −0.348 (−0.387 to −0.309) | 0.75 |
| GPT-6.1 Sol | 0.865 | −0.334 (−0.372 to −0.294) | 0.75 |
| GPT-6 Luna | 0.845 | −0.354 (−0.392 to −0.316) | 0.71 |
| Gemini 4 Argon | 0.911 | −0.288 (−0.327 to −0.248) | 0.75 |
| DeepSeek V4.1 Flash | 0.975 | −0.224 (−0.263 to −0.184) | 0.71 |
| Claude Haiku 5.5 | 1.043 | −0.156 (−0.193 to −0.120) | 0.67 |
| Mean of the ten models | 0.630 | −0.569 (−0.603 to −0.535) | 0.78 |
Two models' ratings correlated 0.86 across texts, averaged over statements and pairs of models (from 0.75 for DeepSeek V4.1 Flash and Claude Haiku 5.5 to 0.98 for GPT-6 Astra and GPT-6.1 Sol). Two readers of the same text correlated 0.43 (difference 0.430, 95% interval 0.422 to 0.439), and a model with a reader 0.56.
In a panel. The table gives panels of two to five raters against the writers' ratings. Replacing one reader with one model's answer lowered the panel's error for every model at every size. Replacing the reader with the no-text baseline lowered it too, so part of that gain comes without the text. With five raters, eight models lowered the error by more than the baseline did, and for DeepSeek V4.1 Flash and Claude Haiku 5.5 there is no credible evidence of a difference from the baseline. On correlation with the writers' ratings, which the baseline cannot raise, the panel with a model's answer was ahead of the panel of readers for every model at every size. Every difference stated in this paragraph has a 95% interval that excludes zero.
| Panel | Error, readers only | Error, one reader replaced by the no-text baseline | Error, one reader replaced by a model's answer (ten models) | Correlation, readers only | Correlation, one reader replaced by a model's answer (ten models) |
|---|---|---|---|---|---|
| Two raters | 1.923 | 1.854 | 1.729 to 1.823 | 0.470 | 0.503 to 0.542 |
| Three raters | 1.739 | 1.703 | 1.642 to 1.691 | 0.502 | 0.521 to 0.546 |
| Four raters | 1.647 | 1.624 | 1.586 to 1.618 | 0.521 | 0.534 to 0.552 |
| Five raters | 1.592 | 1.575 | 1.549 to 1.572 | 0.534 | 0.543 to 0.557 |
Rescaled together. The rescaling of Where the gap comes from can be fitted to the five-reader average and one model's ratings at once. The error was then 1.403 to 1.467 (skill scores 1.19 to 1.29). For every model this was lower than the rescaled five-reader average's 1.522 and lower than the model's own rescaled error, with each 95% interval excluding zero, so the readers and the model each carried information that the other lacked. With the mean of the ten models in place of one model, the error was 1.389 (skill score 1.31, 95% interval 1.27 to 1.36).
Error of each model rescaled alone and together with the five-reader average
| Model | Error, rescaled alone | Error, rescaled with the five-reader average | Difference from the rescaled five-reader average (95% interval) | Difference from the model rescaled alone (95% interval) |
|---|---|---|---|---|
| Claude Opus 5.5 | 1.445 | 1.403 | −0.119 (−0.135 to −0.105) | −0.043 (−0.051 to −0.034) |
| Gemini 3.8 Flash | 1.470 | 1.414 | −0.108 (−0.123 to −0.094) | −0.056 (−0.067 to −0.046) |
| GPT-6.1 Terra | 1.462 | 1.411 | −0.111 (−0.125 to −0.096) | −0.051 (−0.060 to −0.041) |
| Claude Sonnet 5.5 | 1.493 | 1.426 | −0.097 (−0.110 to −0.083) | −0.067 (−0.079 to −0.056) |
| GPT-6 Astra | 1.463 | 1.413 | −0.109 (−0.123 to −0.095) | −0.050 (−0.060 to −0.040) |
| GPT-6.1 Sol | 1.457 | 1.409 | −0.113 (−0.128 to −0.099) | −0.048 (−0.058 to −0.038) |
| GPT-6 Luna | 1.541 | 1.447 | −0.075 (−0.087 to −0.063) | −0.094 (−0.107 to −0.081) |
| Gemini 4 Argon | 1.447 | 1.404 | −0.119 (−0.134 to −0.103) | −0.043 (−0.053 to −0.034) |
| DeepSeek V4.1 Flash | 1.534 | 1.444 | −0.078 (−0.090 to −0.066) | −0.089 (−0.102 to −0.076) |
| Claude Haiku 5.5 | 1.606 | 1.467 | −0.055 (−0.064 to −0.045) | −0.139 (−0.154 to −0.123) |
| Mean of the ten models | 1.412 | 1.389 | −0.134 (−0.149 to −0.118) | −0.023 (−0.030 to −0.016) |
The five-reader average is a reference point, not a ceiling. From how much a text's five readers differ from one another, an unlimited number of such readers, drawn independently, would have an error of 1.371 without rescaling (95% interval 1.328 to 1.415; skill score 1.34).
Emotion
The emotion each writer was asked to write about is the criterion here, so this measures recognising the prompted emotion, not a free report. In exploratory paired comparisons with the readers' majority vote, the 95% interval of the difference in accuracy lies above zero for seven models (Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra, GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon and DeepSeek V4.1 Flash) and below zero for Claude Haiku 5.5. For Claude Sonnet 5.5 and GPT-6 Luna it includes zero, so there is no credible evidence of a difference. Every model was more accurate than one reader. Emotion accuracy and appraisal skill ranked the models differently: the rank correlation across the ten models was 0.25.
The models' confidence ratings correlated 0.21 to 0.36 with whether their emotion choice was right, against 0.21 for one reader. The models' intensity ratings had a mean squared error of 0.885 to 1.356 against the writers', against 1.494 for one reader and 0.959 for the five-reader average. Their event durations matched the writer's for 47.1% to 57.3% of texts, against 45.3% for one reader.
Where the gap comes from
The split of the error and the rescaling in this section are exploratory. A set of ratings can miss the writers' ratings in three ways. It can be offset, too high or too low on average. Its ratings can be spread too widely for how well they track the writers', which enlarges every mistake. The rest of the error depends on how well it orders the texts: it is the error that would remain after the best straight-line rescaling of its ratings. The chart splits each system's mean squared error into these three parts, averaged over the 21 statements.
The five-reader average is barely offset (0.011) and its spread costs 0.063, because averaging narrows the spread of individual readers. One reader has the same offset part (0.011) but is spread far too widely (0.669). Claude Opus 5.5's error of 1.799 is 1.441 left after the best rescaling fitted on all texts, less than the five-reader average's 1.518, plus 0.192 from spread and 0.166 from offset. Every model's ratings were spread too widely on all 21 statements. The table compares the models with the five-reader average directly, with 95% intervals. The correlation is the mean over statements of each system's correlation with the writers' ratings (the models' rank correlations averaged 0.49 to 0.56). The rescaling is a straight line for each statement fitted on 9 of 10 parts of the texts and applied to the tenth, for the models and for the five-reader average alike.
| System | Correlation with writers | Difference from the five-reader average (95% interval) | Error after rescaling | Difference from the five-reader average, both rescaled (95% interval) |
|---|---|---|---|---|
| Five-reader average | 0.53 | 1.522 | ||
| Claude Opus 5.5 | 0.57 | +0.035 (+0.026 to +0.044) | 1.445 | −0.077 (−0.099 to −0.055) |
| Gemini 3.8 Flash | 0.56 | +0.026 (+0.016 to +0.036) | 1.470 | −0.052 (−0.075 to −0.029) |
| GPT-6.1 Terra | 0.56 | +0.029 (+0.020 to +0.039) | 1.462 | −0.060 (−0.082 to −0.038) |
| Claude Sonnet 5.5 | 0.55 | +0.017 (+0.008 to +0.027) | 1.493 | −0.029 (−0.052 to −0.007) |
| GPT-6 Astra | 0.57 | +0.031 (+0.022 to +0.041) | 1.463 | −0.059 (−0.081 to −0.037) |
| GPT-6.1 Sol | 0.57 | +0.033 (+0.024 to +0.042) | 1.457 | −0.065 (−0.088 to −0.043) |
| GPT-6 Luna | 0.53 | −0.004 (−0.014 to +0.006) | 1.541 | +0.019 (−0.003 to +0.042) |
| Gemini 4 Argon | 0.57 | +0.036 (+0.027 to +0.046) | 1.447 | −0.075 (−0.098 to −0.052) |
| DeepSeek V4.1 Flash | 0.53 | −0.003 (−0.013 to +0.007) | 1.534 | +0.011 (−0.012 to +0.034) |
| Claude Haiku 5.5 | 0.50 | −0.034 (−0.045 to −0.024) | 1.606 | +0.084 (+0.061 to +0.107) |
Seven models correlated more strongly than the five-reader average and, after rescaling, had lower error than the rescaled five-reader average. Claude Haiku 5.5 was worse on both measures. For GPT-6 Luna and DeepSeek V4.1 Flash, the 95% intervals of both differences include zero, so there is no credible evidence that they differ from the five-reader average on either measure. After rescaling, nine models, all but Claude Haiku 5.5, had lower error than the five-reader average without rescaling. The rescaled errors show what calibration on texts with writers' ratings could achieve, not what a model achieves unaided. Within emotions, after subtracting each emotion's average ratings from other texts, the models' mean correlations were 0.36 to 0.43, against 0.41 for the five-reader average and 0.28 for one reader, a comparison with no interval.
The design also compared each model with the average ratings, from other texts, of the emotion the model chose for the text. For Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra and Claude Sonnet 5.5, there is no credible evidence that their error differs from that of these averages (the 95% intervals of the difference include zero). Each of the other six models had a higher error than the averages of the emotions it chose, by 0.066 to 0.260. In the exploratory counterpart for the readers, the five-reader average's error was 0.259 lower than that of the average ratings of the readers' majority emotion.
Shared offsets
Each bar is the models' mean signed error on one statement, averaged over the ten models: negative where the models rated lower than the writers. The line under each statement says how many models share the sign and gives the five-reader average's offset. The statement with the largest error, on living with the consequences, said "would easily live with" for writers and "could live with" for readers and models. The five readers rated the same wording as the models.
All 21 statements: correlations and signed errors for the models, one reader and the five-reader average
| Statement | Writers' mean | Correlation, five-reader average | Correlation, one reader | Correlation, models | Signed error, models | Signed error, five-reader average |
|---|---|---|---|---|---|---|
| A1. Suddenness: The event was sudden or abrupt. | 3.11 | 0.58 | 0.45 | 0.53 to 0.64 | −0.56 to −0.24 | −0.20 |
| A2. Familiarity: The event was familiar to its experiencer. | 2.66 | 0.51 | 0.37 | 0.48 to 0.56 | −0.50 to +0.19 | +0.01 |
| A3. Event predictability: The experiencer could have predicted the occurrence of the event. | 2.76 | 0.50 | 0.35 | 0.48 to 0.51 | −0.57 to +0.27 | +0.09 |
| A4. Pleasantness: The event was pleasant for the experiencer. | 2.52 | 0.82 | 0.73 | 0.80 to 0.82 | +0.03 to +0.20 | +0.05 |
| A5. Unpleasantness: The event was unpleasant for the experiencer. | 3.17 | 0.78 | 0.68 | 0.76 to 0.80 | −0.25 to −0.04 | +0.01 |
| A6. Goal relevance: The experiencer expected the event to have important consequences for him/herself. | 3.05 | 0.53 | 0.37 | 0.52 to 0.58 | −0.22 to +0.40 | +0.14 |
| A7. Caused by chance: The event was caused by chance, special circumstances, or natural forces. | 2.30 | 0.48 | 0.34 | 0.41 to 0.50 | −0.42 to −0.01 | −0.14 |
| A8. Caused by self: The event was caused by the experiencer’s own behavior. | 2.75 | 0.71 | 0.58 | 0.61 to 0.72 | −0.57 to +0.11 | +0.06 |
| A9. Caused by others: The event was caused by somebody else’s behavior. | 3.15 | 0.63 | 0.50 | 0.61 to 0.65 | −0.49 to +0.32 | +0.03 |
| A10. Consequences anticipated: The experiencer anticipated the consequences of the event. | 2.94 | 0.35 | 0.21 | 0.23 to 0.36 | −0.30 to +0.50 | +0.04 |
| A11. Positive consequences: The experiencer expected positive consequences for her/himself. | 2.65 | 0.64 | 0.52 | 0.60 to 0.66 | −0.44 to +0.13 | −0.07 |
| A12. Urgency: The event required an immediate response. | 3.25 | 0.38 | 0.24 | 0.36 to 0.44 | −1.22 to −0.49 | −0.23 |
| A13. Own control: The experiencer was able to influence what was going on during the event. | 2.62 | 0.51 | 0.37 | 0.46 to 0.58 | −0.29 to +0.28 | +0.02 |
| A14. Others' control: Someone other than the experiencer was influencing what was going on. | 3.43 | 0.52 | 0.38 | 0.45 to 0.58 | −1.24 to +0.24 | −0.01 |
| A15. Nobody's control: The situation was the result of outside influences of which nobody had control. | 2.25 | 0.45 | 0.31 | 0.45 to 0.52 | −0.57 to −0.03 | −0.04 |
| A16. Could live with consequences: The experiencer anticipated that he/she could live with the unavoidable consequences of the event. | 3.10 | 0.15 | 0.08 | 0.15 to 0.29 | −0.32 to +1.24 | +0.05 |
| A17. Clashed with own standards: The event clashed with her/his standards and ideals. | 2.45 | 0.61 | 0.47 | 0.59 to 0.69 | −0.55 to −0.07 | −0.05 |
| A18. Violated norms: The actions that produced the event violated laws or socially accepted norms. | 1.79 | 0.64 | 0.51 | 0.60 to 0.68 | −0.36 to +0.07 | +0.10 |
| A19. Had to pay attention: The experiencer had to pay attention to the situation. | 3.71 | 0.41 | 0.25 | 0.34 to 0.46 | −0.44 to +0.42 | −0.10 |
| A20. Wanted to shut out: The experiencer wanted to shut the situation out of her/his mind. | 2.40 | 0.52 | 0.40 | 0.46 to 0.57 | −0.72 to +0.19 | +0.17 |
| A21. Required effort: The situation required her/him a great deal of energy to deal with it. | 3.20 | 0.49 | 0.34 | 0.50 to 0.55 | −0.85 to −0.17 | −0.16 |
Asking for an average
This section is exploratory and was run after the results above. The standard prompt asks, as the readers were asked, for the one rating this writer gave. An answer of that kind has no reason to stay away from the ends of the scale, and the models' ratings were spread too widely (Where the gap comes from). The average prompt adds one instruction: "For the statements A1 to A21, do not choose one whole number for this writer. Imagine many different people who each wrote this exact description of an event in their own life. For each statement, give the average of the ratings those writers would give, as a number from 1.0 to 5.0 with one decimal place." The 21 answer lines ask for a number from 1.0 to 5.0 in place of 1 to 5. Nine models answered all 1,200 texts once more with it. GPT-6 Astra, the most expensive model of the main run, was left out to limit cost.
Every model's error was lower under the average prompt, by 0.250 for GPT-6 Luna to 0.413 for Gemini 4 Argon. For scale, a model's error on the 300 repeated texts differed by at most 0.023 between its two runs of the standard prompt. Five models then had a lower error than the five-reader average: Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra, GPT-6.1 Sol and Gemini 4 Argon, by 0.062 to 0.098. For Claude Sonnet 5.5 there is no credible evidence of a difference (−0.015, 95% interval −0.045 to +0.014). GPT-6 Luna, DeepSeek V4.1 Flash and Claude Haiku 5.5 stayed less accurate than the five-reader average, by 0.055 to 0.225. The mean of the nine models' average-prompt ratings had an error of 1.461 (skill score 1.20, 95% interval 1.16 to 1.25).
The comparison with the five-reader average is not like for like. Each reader gave one whole-number guess for one writer, and the five-reader average gains from averaging what a model was here asked to estimate directly. The readers' average is also too widely spread. Against the five-reader average after the rescaling of Where the gap comes from (error 1.522), only Gemini 4 Argon had a lower error (−0.028, 95% interval −0.056 to −0.001). For four models there is no credible evidence of a difference, and four had a higher error. Whether readers asked for an average would also come closer to the writers was not tested.
| Model | Error, standard prompt | Error, average prompt | Average prompt minus standard prompt (95% interval) | Average prompt minus the five-reader average (95% interval) | Average prompt minus the rescaled five-reader average (95% interval) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 1.799 | 1.509 | −0.290 (−0.309 to −0.272) | −0.084 (−0.112 to −0.056) | −0.014 (−0.041 to +0.013) |
| Gemini 3.8 Flash | 1.801 | 1.522 | −0.279 (−0.300 to −0.258) | −0.070 (−0.098 to −0.042) | 0.000 (−0.027 to +0.028) |
| GPT-6.1 Terra | 1.820 | 1.501 | −0.319 (−0.339 to −0.299) | −0.091 (−0.120 to −0.064) | −0.021 (−0.048 to +0.005) |
| Claude Sonnet 5.5 | 1.846 | 1.577 | −0.269 (−0.289 to −0.250) | −0.015 (−0.045 to +0.014) | +0.055 (+0.028 to +0.082) |
| GPT-6.1 Sol | 1.867 | 1.530 | −0.337 (−0.356 to −0.317) | −0.062 (−0.092 to −0.033) | +0.008 (−0.022 to +0.037) |
| GPT-6 Luna | 1.897 | 1.648 | −0.250 (−0.271 to −0.229) | +0.055 (+0.026 to +0.085) | +0.125 (+0.096 to +0.155) |
| Gemini 4 Argon | 1.907 | 1.494 | −0.413 (−0.437 to −0.389) | −0.098 (−0.127 to −0.071) | −0.028 (−0.056 to −0.001) |
| DeepSeek V4.1 Flash | 2.018 | 1.677 | −0.341 (−0.369 to −0.314) | +0.085 (+0.053 to +0.116) | +0.155 (+0.123 to +0.186) |
| Claude Haiku 5.5 | 2.105 | 1.817 | −0.289 (−0.313 to −0.264) | +0.225 (+0.189 to +0.260) | +0.295 (+0.261 to +0.328) |
| Mean of the nine models | 1.642 | 1.461 | −0.181 (−0.195 to −0.166) | −0.131 (−0.157 to −0.106) | −0.061 (−0.086 to −0.037) |
Rounded to whole numbers, the average-prompt ratings still had a lower error than the standard prompt's for every model, by 0.161 to 0.305 with halves rounded up and by 0.094 to 0.324 with halves rounded down. The ratings were less spread, with a standard deviation across texts of 0.90 to 1.12 against 1.14 to 1.42 under the standard prompt. Every model's correlation with the writers' ratings rose as well, by 0.015 to 0.023. For the ratings rounded to whole numbers, there is no credible evidence that any model's correlation was higher than under the standard prompt. The gain was 58% to 90% as large as the gain from the rescaling of Where the gap comes from on the same model's standard-prompt ratings, and unlike the rescaling the prompt contains no writers' ratings. The rescaled standard-prompt ratings still had the lower error for every model, by 0.039 to 0.210, and rescaling the average-prompt ratings lowered their error by a further 0.087 to 0.249. The differences in error and the rise in correlation in this paragraph each have a 95% interval that excludes zero.
Average-prompt ratings rounded and rescaled, and spread and correlation under both prompts
| Model | Error, rounded to whole numbers | Error, rescaled | Spread of ratings, standard prompt | Spread of ratings, average prompt | Correlation, standard prompt | Correlation, average prompt | Correlation, average prompt rounded |
|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 1.569 | 1.406 | 1.29 | 0.98 | 0.569 | 0.585 | 0.562 |
| Gemini 3.8 Flash | 1.623 | 1.421 | 1.31 | 1.05 | 0.560 | 0.578 | 0.556 |
| GPT-6.1 Terra | 1.594 | 1.414 | 1.33 | 0.98 | 0.563 | 0.583 | 0.558 |
| Claude Sonnet 5.5 | 1.635 | 1.452 | 1.21 | 0.90 | 0.551 | 0.566 | 0.542 |
| GPT-6.1 Sol | 1.646 | 1.409 | 1.37 | 1.07 | 0.567 | 0.585 | 0.563 |
| GPT-6 Luna | 1.736 | 1.500 | 1.27 | 1.06 | 0.530 | 0.547 | 0.528 |
| Gemini 4 Argon | 1.602 | 1.387 | 1.42 | 1.08 | 0.570 | 0.593 | 0.574 |
| DeepSeek V4.1 Flash | 1.809 | 1.496 | 1.40 | 1.12 | 0.531 | 0.546 | 0.532 |
| Claude Haiku 5.5 | 1.838 | 1.567 | 1.14 | 0.90 | 0.500 | 0.517 | 0.497 |
| Mean of the nine models | 1.542 | 1.392 | 1.22 | 0.97 | 0.585 | 0.592 | 0.567 |
The instruction and the decimal answer format were changed together, and Parts of the average prompt compares them. Pulling each standard-prompt rating a quarter of the way to the middle of the scale, a correction chosen after the results were seen that needs no new request, gave 78% to 100% of the gain. The average-prompt ratings had a lower error than the pulled ratings for six models, and for the other three there is no credible evidence of a difference. A request for an average could also lower the error by removing the chance variation of a single answer. Measured on the 300 texts that every model answered twice, that variation was 6% of the gain for GPT-6.1 Sol and 45% for DeepSeek V4.1 Flash, so it does not account for the whole gain of any model.
The instruction names only the 21 statements, but other answers differed as well. Every model's rating of the emotion's intensity was lower under the average prompt, by 0.07 to 0.11 on the scale from 1 to 5. Emotion accuracy was 1.9 percentage points lower for Claude Haiku 5.5, and for each of the other eight models there is no credible evidence of a difference; over the nine models together it was 0.6 points lower (95% interval −1.0 to −0.2). No appraisal rating was invalid.
Checks added after the answers had been seen: rounding with halves down, ratings pulled to the middle of the scale, the variation of one answer, and intensity
| Model | Error, rounded with halves down | Error, standard-prompt ratings pulled to the middle of the scale | Average prompt minus pulled ratings (95% interval) | Variation of one answer, share of the gain | Intensity rating, average prompt minus standard prompt (95% interval) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 1.575 | 1.563 | −0.054 (−0.065 to −0.043) | 13% | −0.08 (−0.10 to −0.06) |
| Gemini 3.8 Flash | 1.602 | 1.564 | −0.042 (−0.055 to −0.028) | 26% | −0.10 (−0.13 to −0.08) |
| GPT-6.1 Terra | 1.562 | 1.566 | −0.065 (−0.076 to −0.055) | 14% | −0.09 (−0.11 to −0.08) |
| Claude Sonnet 5.5 | 1.686 | 1.621 | −0.044 (−0.055 to −0.033) | 13% | −0.07 (−0.09 to −0.06) |
| GPT-6.1 Sol | 1.623 | 1.574 | −0.044 (−0.054 to −0.034) | 6% | −0.11 (−0.13 to −0.09) |
| GPT-6 Luna | 1.777 | 1.653 | −0.005 (−0.020 to +0.010) | 43% | −0.11 (−0.13 to −0.08) |
| Gemini 4 Argon | 1.583 | 1.583 | −0.089 (−0.103 to −0.076) | 14% | −0.11 (−0.13 to −0.09) |
| DeepSeek V4.1 Flash | 1.765 | 1.682 | −0.005 (−0.025 to +0.014) | 45% | −0.08 (−0.10 to −0.05) |
| Claude Haiku 5.5 | 2.011 | 1.817 | −0.001 (−0.018 to +0.017) | 22% | −0.10 (−0.12 to −0.08) |
Parts of the average prompt
This section is exploratory and was run after the average prompt. The average prompt differs from the standard prompt in three ways: it asks for ratings with one decimal place, it rules out one whole number for this writer, and it asks for the average rating of many writers. A control compared these parts. Each of the nine models answered five prompts for each of the 300 repeated texts, one after the other in an order drawn at random: the standard prompt, and four prompts with the decimal format in which the rule and the request were each present or absent. The prompts, the comparisons, the tests and the rules for wording the results were written down before any request, after five rounds of review by GPT-6 Astra and two by a second instance of Claude Fable 5.2. Errors in this section are averaged over the nine models unless a model is named.
| Prompt | Answer format | Rules out one whole number | Asks for the average | Error, mean of the nine models |
|---|---|---|---|---|
| Standard | Whole numbers | No | No | 1.928 |
| Decimal | One decimal place | No | No | 1.812 |
| Strict | One decimal place | Yes | No | 1.698 |
| Request | One decimal place | No | Yes | 1.621 |
| Average | One decimal place | Yes | Yes | 1.607 |
With the decimal format alone, the error was 0.115 (95% interval 0.102 to 0.129) lower than under the standard prompt. It was lower for six models and higher for none. For GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, which gave whole numbers for more than 96% of their ratings under the decimal prompt, there is no credible evidence of a difference.
The request prompt's error was 0.192 (95% interval 0.172 to 0.211) below the decimal prompt's, and the average prompt's was 0.091 (95% interval 0.079 to 0.103) below the strict prompt's. Each of these two differences was lower for every model, so the request for the average lowered the error with and without the rule. It lowered the error by less with the rule than without it (difference 0.101, 95% interval 0.086 to 0.116). This held for six models and was reversed for Claude Haiku 5.5; for two there is no credible evidence of a difference.
Without the request, the strict prompt's error was 0.114 (95% interval 0.101 to 0.129) below the decimal prompt's: lower for seven models and higher for Claude Haiku 5.5, with no credible evidence of a difference for Claude Sonnet 5.5. With the request, the average prompt's error was 0.014 (95% interval 0.006 to 0.021) below the request prompt's; among the models, only DeepSeek V4.1 Flash's error was lower. No model's error was higher, and for the other eight there is no credible evidence of a difference.
The average prompt's error was 0.321 (95% interval 0.293 to 0.349) below the standard prompt's, and lower for every model. When the two prompts had been answered on consecutive days on the same texts, the difference was 0.322, and there is no credible evidence that the two differences differ (by 0.001, p = 0.86). For Claude Sonnet 5.5 alone, the average prompt lowered the error by 0.043 (95% interval 0.016 to 0.071) more in the control than in the earlier answers (significant after Holm correction over the nine models), a difference between sessions of unknown cause. Measured against the standard prompt, the decimal prompt gave 36% of the average prompt's difference, the strict prompt 72% and the request prompt 96%. Each of the seven differences between prompts given with an interval for the average of the nine models is statistically significant (p < 0.001; the five between prompts that differ in one part were corrected for multiple tests with the Holm method).
Each model's error under the five prompts
| Model | Error, standard prompt | Error, decimal prompt | Error, strict prompt | Error, request prompt | Error, average prompt | Ratings that were not whole numbers, decimal prompt |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 1.806 | 1.603 | 1.570 | 1.529 | 1.528 | 64% |
| Gemini 3.8 Flash | 1.830 | 1.735 | 1.614 | 1.554 | 1.542 | 11% |
| GPT-6.1 Terra | 1.837 | 1.841 | 1.609 | 1.515 | 1.513 | 0% |
| Claude Sonnet 5.5 | 1.892 | 1.615 | 1.614 | 1.598 | 1.590 | 74% |
| GPT-6.1 Sol | 1.918 | 1.901 | 1.650 | 1.575 | 1.566 | 0% |
| GPT-6 Luna | 1.944 | 1.890 | 1.777 | 1.670 | 1.657 | 4% |
| Gemini 4 Argon | 1.927 | 1.773 | 1.581 | 1.525 | 1.527 | 22% |
| DeepSeek V4.1 Flash | 2.071 | 2.058 | 1.919 | 1.774 | 1.702 | 3% |
| Claude Haiku 5.5 | 2.125 | 1.896 | 1.947 | 1.847 | 1.839 | 48% |
| Mean of the nine models | 1.928 | 1.812 | 1.698 | 1.621 | 1.607 |
Differences between prompts that differ in one part, for each model
| Model | Decimal minus standard (95% interval) | Request minus decimal (95% interval) | Average minus strict (95% interval) | Strict minus decimal (95% interval) | Average minus request (95% interval) |
|---|---|---|---|---|---|
| Claude Opus 5.5 | −0.203 (−0.238 to −0.166) * | −0.074 (−0.093 to −0.054) * | −0.042 (−0.058 to −0.026) * | −0.033 (−0.052 to −0.016) * | −0.002 (−0.014 to +0.010) |
| Gemini 3.8 Flash | −0.096 (−0.131 to −0.062) * | −0.181 (−0.218 to −0.144) * | −0.072 (−0.100 to −0.045) * | −0.121 (−0.155 to −0.085) * | −0.012 (−0.034 to +0.007) |
| GPT-6.1 Terra | +0.005 (−0.018 to +0.028) | −0.326 (−0.367 to −0.286) * | −0.096 (−0.122 to −0.073) * | −0.232 (−0.265 to −0.199) * | −0.003 (−0.014 to +0.008) |
| Claude Sonnet 5.5 | −0.277 (−0.309 to −0.244) * | −0.017 (−0.031 to −0.004) * | −0.024 (−0.038 to −0.010) * | −0.001 (−0.013 to +0.012) | −0.007 (−0.018 to +0.004) |
| GPT-6.1 Sol | −0.017 (−0.035 to +0.001) | −0.326 (−0.362 to −0.290) * | −0.084 (−0.101 to −0.068) * | −0.251 (−0.280 to −0.221) * | −0.009 (−0.018 to −0.001) |
| GPT-6 Luna | −0.054 (−0.090 to −0.019) * | −0.220 (−0.261 to −0.179) * | −0.121 (−0.153 to −0.089) * | −0.113 (−0.148 to −0.078) * | −0.014 (−0.043 to +0.016) |
| Gemini 4 Argon | −0.154 (−0.194 to −0.115) * | −0.248 (−0.287 to −0.211) * | −0.054 (−0.078 to −0.032) * | −0.192 (−0.231 to −0.153) * | +0.002 (−0.013 to +0.016) |
| DeepSeek V4.1 Flash | −0.013 (−0.074 to +0.050) | −0.284 (−0.331 to −0.236) * | −0.218 (−0.261 to −0.176) * | −0.139 (−0.185 to −0.091) * | −0.072 (−0.110 to −0.035) * |
| Claude Haiku 5.5 | −0.229 (−0.278 to −0.180) * | −0.049 (−0.084 to −0.014) * | −0.108 (−0.143 to −0.074) * | +0.051 (+0.014 to +0.089) * | −0.008 (−0.039 to +0.022) |
| Mean of the nine models | −0.115 (−0.129 to −0.102) * | −0.192 (−0.211 to −0.172) * | −0.091 (−0.103 to −0.079) * | −0.114 (−0.129 to −0.101) * | −0.014 (−0.021 to −0.006) * |
* The difference is statistically significant after correction for multiple tests with the Holm method (p < 0.05): over the nine models within a column, and for the mean of the models over the five columns.
How the control was run: 13,500 requests, none failed
The control sent 13,500 requests over 49 minutes. A model's five requests for a text were sent by one worker, and the median time from the first to the start of the last was 15 seconds. No request failed, no answer was cut off and no appraisal rating was invalid. 16 answers had another field invalid: in 12 the intensity was written with a decimal point, and in four Claude Haiku 5.5 named an emotion or an event duration that is not among the options. A pilot of the three new prompts on the 26 other texts (702 requests) checked the answer format first. The control was sent in two parts, first the three Claude models with at most 10 requests at a time, to stay within Anthropic's limit on simultaneous requests, and then the other six.
Checks
The design included six checks. Two of them, scoring valid answers only and charging the largest possible loss for each invalid rating, changed nothing, because no appraisal rating was invalid. Models answer in whole numbers, as readers did, while the no-text baseline and the five-reader average are fractional, which favours the benchmarks. Rounding both benchmarks to whole numbers raised every skill score. On the 653 texts whose writers confirmed the event happened to them, every model scored higher than on all texts, a difference that was not tested. Each model answered 300 texts a second time: it chose the same emotion for 90.3% to 99.0% of them, and its error changed by at most 0.023.
| Model | Skill score, integer benchmarks | Skill score, confirmed events | Same emotion in both answers | Error on the 300 texts, first and second answer |
|---|---|---|---|---|
| Claude Opus 5.5 | 0.81 | 0.72 | 97.7% | 1.805, 1.808 |
| Gemini 3.8 Flash | 0.81 | 0.74 | 94.3% | 1.823, 1.808 |
| GPT-6.1 Terra | 0.78 | 0.70 | 95.0% | 1.841, 1.830 |
| Claude Sonnet 5.5 | 0.74 | 0.68 | 97.3% | 1.860, 1.883 |
| GPT-6 Astra | 0.71 | 0.63 | 97.7% | 1.916, 1.910 |
| GPT-6.1 Sol | 0.71 | 0.64 | 99.0% | 1.919, 1.925 |
| GPT-6 Luna | 0.66 | 0.60 | 90.3% | 1.912, 1.896 |
| Gemini 4 Argon | 0.65 | 0.56 | 94.7% | 1.945, 1.931 |
| DeepSeek V4.1 Flash | 0.48 | 0.38 | 93.7% | 2.097, 2.104 |
| Claude Haiku 5.5 | 0.35 | 0.28 | 96.7% | 2.152, 2.146 |
The sixth check resampled readers instead of texts and recomputed the five-reader average. It raised every model's skill score (median 0.33 to 1.05), because a resampled text keeps on average 3.2 distinct readers of its five, which makes the benchmark less accurate. It therefore does not give a usable interval. No answer was cut off. Claude Haiku 5.5 gave 8 invalid answers to other questions in the main run and 3 in the repeat: emotions outside the options, such as "anxiety", and event durations of "months". Invalid emotions counted as wrong.
One further check is exploratory and concerns the corpus, which the models may have seen in training. A model reciting a writer's published ratings would match all 21 of them. No answer did. The most that any answer matched exactly was 19, the same as for the closest single reading, and 19 of the 12,000 answers matched 18 or more ratings, against 4 of the 6,000 readings.
Cost and skill
This section is exploratory. Each point is a model's skill score against what its answers cost per 1,000 texts at the providers' list prices. A model is on the cost-score frontier when no other model costs no more and has at least as high a skill score. The line through those models gives the highest skill score available at each cost. Hover over a point for its 95% interval.
Five models were on the cost-score frontier: Claude Haiku 5.5, GPT-6 Luna, GPT-6.1 Terra, Gemini 3.8 Flash and Claude Opus 5.5. In a paired comparison with Claude Opus 5.5, the model with the highest skill score (0.684, against 0.681 for Gemini 3.8 Flash), there is no credible evidence that the error of Gemini 3.8 Flash (difference +0.002, 95% interval −0.022 to +0.026) or of GPT-6.1 Terra (+0.021, −0.004 to +0.046) differs from Claude Opus 5.5's. Their answers cost 60% and 21% as much. Every other model's error was higher than Claude Opus 5.5's. At the list price announced for Gemini 3.8 Flash from 1 January 2027, its answers would cost $11.58 per 1,000 texts, more than Claude Opus 5.5's, and it would no longer be on the cost-score frontier. Gemini 4 Argon's list price is announced to double on 1 December 2026, and it is not on the cost-score frontier at either price.
| Model | Output tokens per text | Cost per 1,000 texts | Skill score | Error minus Claude Opus 5.5's (95% interval) |
|---|---|---|---|---|
| Claude Opus 5.5 cost-score frontier | 194 | $9.61 | 0.68 | reference |
| Gemini 3.8 Flash cost-score frontier | 1,321 | $5.79 | 0.68 | +0.002 (−0.022 to +0.026) |
| GPT-6.1 Terra cost-score frontier | 197 | $2.02 | 0.65 | +0.021 (−0.004 to +0.046) |
| Claude Sonnet 5.5 | 167 | $4.53 | 0.61 | +0.047 (+0.020 to +0.075) |
| GPT-6 Astra | 176 | $19.09 | 0.58 | +0.066 (+0.040 to +0.092) |
| GPT-6.1 Sol | 172 | $3.78 | 0.58 | +0.068 (+0.042 to +0.094) |
| GPT-6 Luna cost-score frontier | 393 | $0.30 | 0.53 | +0.098 (+0.068 to +0.128) |
| Gemini 4 Argon | 175 | $4.03 | 0.52 | +0.108 (+0.081 to +0.136) |
| DeepSeek V4.1 Flash | 5,893 | $1.84 | 0.35 | +0.219 (+0.186 to +0.254) |
| Claude Haiku 5.5 cost-score frontier | 215 | $0.25 | 0.22 | +0.307 (+0.265 to +0.348) |
Limitations
- Only default reasoning effort was tested. Maximum effort was planned and dropped before any request on these texts, because the pilot cost $0.258 per text for the ten models at maximum effort against $0.050 at default. The default differs between models: an answer averaged 167 output tokens for Claude Sonnet 5.5 and 5,893, reasoning included, for DeepSeek V4.1 Flash (Cost and skill).
- The average prompt was added after the main results were known. Its wording was written down before any request with it, and no other wording was tried. Claude Opus 5.5 was added while the other eight models were running, after their results on part of the texts had been seen. The prompt changed the instruction and the answer format together, and each model answered it once, on the day after the standard prompt (Asking for an average).
- The control of the average prompt's parts compares prompts as worded. It was designed with the average prompt's results known, tried each part in one wording, and covers 300 of the 1,200 texts and nine of the ten models. It has no prompt that asks for the average in whole numbers, and it does not show why a prompt lowers the error (Parts of the average prompt).
- Writers and readers did not rate identical wording. The statements differed in person, in the labels of the scale ends and, for three statements, in wording (Task). Models rated the readers' wording, which may account for part of how closely they followed the readers.
- The writers' ratings are retrospective self-reports. Nine writers said they had imagined the event, and 538 were not asked.
- The readers were paid crowd workers who did not know the writers, and the corpus is in English, written by people from six English-speaking countries. Each reader read about five texts, too few to identify the most accurate readers, so the comparisons are with average readers of this kind.
- The models may have seen the corpus. It has been public, with the writers' ratings, since at least 2023. The check for recited ratings found none (Checks), which does not exclude a weaker influence.
- The exploratory analyses were chosen after the results were known, their intervals are not corrected for multiple comparisons, and the rescaling was fitted within the evaluation texts. Comparisons given without an interval were not tested. Texts share readers, so the intervals of comparisons between readers are somewhat too narrow.
- Emotion durations were not analysed, because the released reader answers lack the questionnaire's "this event did not cause any emotion" option.
Future research
The exploratory results suggest that a model's error can be reduced without changing the model. Each idea below can be tested on crowd-enVENT with every choice fixed in advance. Two exploratory checks on the answers already collected give a first indication.
- Confirm the average prompt. A confirmatory test would fix the instruction in advance, include all ten models and ask readers the same question, so that models and readers are compared on the same task. It would also repeat the comparisons of Parts of the average prompt with other wordings of each part and with a request for the average in whole numbers. The models can first be tested on the 3,144 other corpus texts, which have writers' ratings but no readers.
- Calibrate on writers' ratings. Offsets remained under the average prompt, and rescaling still lowered every model's error (Asking for an average). A confirmatory test would fit the rescaling (Where the gap comes from) on the models' ratings of the 3,144 other corpus texts, which have writers' ratings but no readers, and apply it unchanged to the evaluation texts. The same information could instead go in the prompt, as the writers' wording of the statements or as a few example texts with their writers' ratings.
- Average repeated answers. Repeated answers from one model may average out as readers' ratings do. On the 300 texts every model answered twice, the mean of a model's two answers had a higher skill score than its first answer for all ten models, by 0.009 to 0.137, a gain with no interval. Both are scored against the baseline and five-reader average of those 300 texts. More answers per text would show how far this goes.
- Combine models. The mean of the ten models' ratings had a skill score of 0.92, as accurate as an average of 4.0 readers. Its error was still higher than the five-reader average's, by 0.055 (95% interval 0.023 to 0.088). After the same rescaling, its error was 1.412, lower than the rescaled five-reader average's 1.522 (difference −0.111, 95% interval −0.132 to −0.090). One model's answer also improved panels of readers (Models beside readers). Which mix of readers and models gives the lowest error at a given cost has not been tested.
- Try maximum reasoning effort, further prompts and fine-tuning. None of these has been tested on these texts. A prompt could ask the model to describe the event from the writer's point of view before rating. Tak and Gratch (2024) found GPT-4's ratings closer to the readers' average than to the writers'. Where a provider offers fine-tuning, the 3,144 other corpus texts with their writers' ratings could serve as training data.
Skill score of each model's first answer and of the mean of its two answers, on the 300 repeated texts
| Model | First answer | Mean of two answers | Gain |
|---|---|---|---|
| Claude Opus 5.5 | 0.704 | 0.731 | +0.027 |
| Gemini 3.8 Flash | 0.676 | 0.749 | +0.073 |
| GPT-6.1 Terra | 0.646 | 0.690 | +0.044 |
| Claude Sonnet 5.5 | 0.615 | 0.624 | +0.009 |
| GPT-6 Astra | 0.525 | 0.548 | +0.024 |
| GPT-6.1 Sol | 0.520 | 0.533 | +0.013 |
| GPT-6 Luna | 0.531 | 0.632 | +0.101 |
| Gemini 4 Argon | 0.477 | 0.536 | +0.059 |
| DeepSeek V4.1 Flash | 0.231 | 0.369 | +0.137 |
| Claude Haiku 5.5 | 0.143 | 0.205 | +0.062 |
Two recent datasets would allow the comparison with human readers to be repeated on new texts, and the ideas above to be tested on data that did not suggest them.
- Mult2EMo (Bagdon, Silberer and Klinger), co-authored by one of the authors of crowd-enVENT. Texts: 7,200 social media posts with images from 1,101 authors, who labelled their own emotions and intensities and rated 21 appraisal dimensions for the event. Readers: three per post for 1,440 posts, each trying to reconstruct the author's answers. Fit: submitted to arXiv on 16 September 2026, after the latest training cutoff the providers publish for the models tested here (June 2026, for GPT-6.1 Terra and the three Claude models). DeepSeek publishes no cutoff. The readers saw the images, so a comparison with them needs models that read images as well as text. The data are available on request for research, under terms that ask users not to let closed-source models train on them.
- CAREBench (Sun and colleagues). Texts: 1,000 first-person narratives with the writers' ratings on 22 appraisal items and their emotion intensities and labels. The writers also answered open questions on five core appraisal dimensions, so a model's explanations could be compared with the writer's, not only its ratings. Readers: five per narrative for 500 narratives, on the same scales. Fit: submitted to arXiv on 16 May 2026 and, by arXiv's schedule, announced on 18 May, the date of GPT-6 Luna's published training cutoff and before those of GPT-6.1 Terra and the three Claude models. Those four models may have seen it, and GPT-6 Luna cannot be ruled out. GPT-4o assembled each narrative, and each answer to the open questions, from the participant's replies with minimal rewording, and the participant then reviewed the narrative. Models may read such texts differently from texts written entirely by people. Comparing the gap between models and readers across the two datasets would show whether this matters.
Method
Data, models, scoring and statistics
Data. crowd-enVENT (licence CC BY 4.0) has 6,600 texts by 2,379 writers recruited on Prolific. The 1,200 evaluation texts are those the corpus authors had read by others: 100 for each of ten emotions and for no emotion, and 50 each for guilt and shame, each by a different writer, read by 1,217 readers (6,000 readings). The corpus authors fine-tuned RoBERTa-large models on the 5,400 other texts, with these 1,200 as the test set, and released the predictions of five training runs. The no-text baseline and the emotion averages come from the 3,144 other texts left after removing every text by a writer of an evaluation text and every text with the same wording as one.
Models. Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5 (Anthropic); GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna (OpenAI); Gemini 4 Argon and Gemini 3.8 Flash (Google); DeepSeek V4.1 Flash (DeepSeek, through OpenRouter on a single host, DeepInfra, with temperature and top-p set to 1.0 as its model card recommends). Requests set no reasoning effort and no system prompt. Each request was a new conversation with one text. Each model answered all 1,200 texts once and 300 of them, chosen at random within each emotion, a second time. Nine of the models then answered all 1,200 texts once more under the average prompt (Asking for an average), and later five prompts for each of the 300 repeated texts (Parts of the average prompt).
Scoring. Answers were read with fixed rules. An invalid rating would have been replaced by the no-text baseline's rating for that statement, and an invalid emotion counted as wrong. A request that ended in an error before any output would have been repeated; none did. The scoring code was checked against the readers' predictions released with the corpus, which reproduce the corpus paper's published human results to within 0.01, except three rows of its error table (statements A16 to A18) that the paper prints rotated by one row, in the readers' column and in the fine-tuned model's column alike.
Statistics. Intervals come from 10,000 resamples of the texts, drawn within each emotion and shared by all systems, with every benchmark recomputed in each resample. Differences from the two benchmarks were tested two-sided and corrected for the 20 comparisons with the Holm method. Exploratory differences are given with intervals only, except in Parts of the average prompt, whose differences between prompts were tested in the same way and corrected with the Holm method.
Design. The outcome, benchmarks, tests, prompt, parsing rules and subsets were fixed in a design document before any request on the evaluation texts, after a critique of the design by GPT-6 Astra. It was changed three times. Maximum effort was dropped before any request on the evaluation texts, and the average prompt was added after the main results were known, with its wording and eight comparisons written down before any request with it. The control of the average prompt's parts was added after the average prompt's results were known; its prompts, comparisons, tests and rules for wording the results were written down and reviewed before any request with them. The other comparisons in Asking for an average were added after its answers had been seen, among them the rescaled five-reader average, the ratings pulled to the middle of the scale and the variation of one answer. An optional condition in the design, predicting the emotion from the writers' own ratings without the text, needed separate approval and was not run.
API cost: $186.94 in total
Costs are from the run records at the providers' list prices, standard tier; DeepSeek V4.1 Flash at DeepInfra's price on OpenRouter.
| Stage | Requests | Cost |
|---|---|---|
| Pilot on 26 other texts (both effort levels) and a batching check | 540 | $10.27 |
| Main run: every text, ten models | 12,000 | $61.49 |
| Repeat on 300 texts, ten models | 3,000 | $15.26 |
| Average prompt: pilot on the 26 other texts, nine models | 234 | $1.00 |
| Average prompt: every text, nine models | 10,800 | $43.67 |
| Parts of the average prompt: pilot of the three new prompts on the 26 other texts, nine models | 702 | $2.84 |
| Parts of the average prompt: five prompts on the 300 repeated texts, nine models | 13,500 | $52.42 |
| Total | 40,776 | $186.94 |