Language models infer writers' emotion appraisals better than one reader but worse than five combined

Alejandro Legarda, a.legarda.j@gmail.com

Terms used on this page
Appraisal
How a person evaluated an event on one dimension, such as how sudden, how pleasant or how controllable it was. Here, 21 statements each rated from 1 to 5.
Writer
The person who lived the event, wrote the text and rated their own appraisals of it.
Reader
A crowd worker who read a text, with emotion words masked, and guessed the writer's emotion and ratings. Each text had five readers.
Five-reader average
For each text and statement, the mean of its five readers' ratings.
Panel
A group of raters, readers or models, whose ratings of a text are averaged.
No-text baseline
The average rating on each statement over 3,144 other texts of the corpus by other writers, used as the guess for every text.
Mean squared error
The squared difference between a rating and the writer's rating, averaged over the 21 statements and the texts. Lower is better.
Skill score
Where a system's mean squared error falls between the no-text baseline (0) and the five-reader average (1). Below 0 is worse than the baseline; above 1 would beat the five readers.
Emotion accuracy
The share of texts where the emotion chosen is the one the writer was asked to write about.
Macro-F1
For each of the 13 emotion options, the balance (harmonic mean) of how often a choice of that emotion was right and how often that emotion was found, averaged over the options so that each counts equally.
95% interval
The range that contains the result in 95% of 10,000 resamples of the texts.
Exploratory
An analysis chosen after the results were known, not in the design fixed before the runs.
Default effort
The amount of reasoning a model uses when the request does not set one.
Standard prompt
The readers' questions as the readers were asked them: one whole number from 1 to 5 for each statement, as the guess for this writer.
Average prompt
The standard prompt with one added instruction: for each statement, give the average rating that many writers of the same description would give, to one decimal place.
Rule against one whole number
The first sentence of the average prompt's instruction: "do not choose one whole number for this writer".

In the crowd-enVENT corpus (Troiano, Oberländer and Klinger, 2023), people described an event from their own life and rated how they had appraised it on 21 statements. For 1,200 of these texts, five other people each read the text, with emotion words masked, and tried to give the writer's ratings. Ten current language models answered those readers' questions at default reasoning effort, under a design fixed before any request on these texts, and scored from 0.22 to 0.68 on a scale where 0 is a guess that ignores the text and 1 is the five-reader average. In a condition added afterwards, nine of the models were asked for the average rating that many writers of the same description would give, a question the readers were not asked, and scored from 0.66 to 1.15.

Language model, default effortHuman readers and the no-text baselineModel fine-tuned on the rest of the corpus
Skill score on the 21 appraisal ratings of 1,200 texts (0 = no text, 1 = five-reader average; higher is better)
Five-reader average
1.00
the mean of each text's five readers
Average of 4 readers
0.92
every combination of the five readers
Average of 3 readers
0.78
every combination of the five readers
Claude Opus 5.5
0.68
95% interval 0.62 to 0.74
Gemini 3.8 Flash
0.68
95% interval 0.62 to 0.74
GPT-6.1 Terra
0.65
95% interval 0.59 to 0.71
Claude Sonnet 5.5
0.61
95% interval 0.55 to 0.67
GPT-6 Astra
0.58
95% interval 0.51 to 0.65
GPT-6.1 Sol
0.58
95% interval 0.51 to 0.65
GPT-6 Luna
0.53
95% interval 0.46 to 0.60
Gemini 4 Argon
0.52
95% interval 0.44 to 0.59
Average of 2 readers
0.49
every combination of the five readers
RoBERTa fine-tuned on the rest of the corpus
0.42
the corpus authors' model, mean of 5 training runs (0.23 to 0.56)
DeepSeek V4.1 Flash
0.35
95% interval 0.27 to 0.43
Claude Haiku 5.5
0.22
95% interval 0.13 to 0.29
No text
0.00
average ratings of 3,144 other texts
One reader
−0.35
average over all 6,000 readings

Findings

Task

Each writer recalled an event in which they had felt a given emotion (one of 12, or no particular emotion), completed the sentence "I felt [emotion] when/because…" and rated 21 statements about how they had seen the event, such as "The event was sudden or abrupt." Readers saw the text with words naming the emotion replaced by three dots, chose the emotion they thought the writer felt and rated the 21 statements as they thought the writer had. Readers were paid crowd workers and were offered a bonus if their ratings were among the 5% closest to the writers'. Each model received the readers' instructions and questions for one text per request, without the questions about the reader, the attention checks and the text about consent and payment, and answered in a fixed format without explanation.

Writers rated first-person statements from 1 ("Not at all") to 5 ("Extremely"). Readers and models rated third-person versions and were told that 1 means "I don't agree at all" and 5 "I completely agree". Three statements differed in more than person. Writers rated "The event was familiar", "I anticipated that I would easily live with the unavoidable consequences of the event" and "I tried to shut the situation out of my mind"; the readers' and models' versions are A2, A16 and A20 in the example below.

The example below is the first text in the random order in which texts were sent. Its writer was asked to write about surprise.

One text with the writer's, the readers' and two models' answers

“I was a child and my parents ... my sister and I with a golden retriever puppy”

QuestionWriterFive readersClaude Opus 5.5Claude Haiku 5.5
Emotionsurprise (asked to write about)joy, joy, surprise, surprise, surprisesurprisejoy
A1. The event was sudden or abrupt.24 2 5 3 5 (mean 3.8)54
A2. The event was familiar to its experiencer.13 4 2 1 1 (mean 2.2)11
A3. The experiencer could have predicted the occurrence of the event.13 4 3 1 2 (mean 2.6)11
A4. The event was pleasant for the experiencer.52 3 5 4 5 (mean 3.8)55
A5. The event was unpleasant for the experiencer.14 3 1 1 1 (mean 2.0)11
A6. The experiencer expected the event to have important consequences for him/herself.42 2 4 1 3 (mean 2.4)43
A7. The event was caused by chance, special circumstances, or natural forces.13 3 1 1 1 (mean 1.8)13
A8. The event was caused by the experiencer’s own behavior.12 4 1 1 3 (mean 2.2)11
A9. The event was caused by somebody else’s behavior.52 3 5 5 5 (mean 4.0)54
A10. The experiencer anticipated the consequences of the event.12 3 2 1 2 (mean 2.0)22
A11. The experiencer expected positive consequences for her/himself.52 3 5 3 5 (mean 3.6)54
A12. The event required an immediate response.52 3 5 2 4 (mean 3.2)22
A13. The experiencer was able to influence what was going on during the event.21 2 1 1 3 (mean 1.6)11
A14. Someone other than the experiencer was influencing what was going on.52 2 5 5 5 (mean 3.8)54
A15. The situation was the result of outside influences of which nobody had control.12 4 1 1 1 (mean 1.8)11
A16. The experiencer anticipated that he/she could live with the unavoidable consequences of the event.52 3 1 4 4 (mean 2.8)54
A17. The event clashed with her/his standards and ideals.13 3 1 1 1 (mean 1.8)11
A18. The actions that produced the event violated laws or socially accepted norms.13 2 1 1 1 (mean 1.6)11
A19. The experiencer had to pay attention to the situation.52 4 3 2 2 (mean 2.6)43
A20. The experiencer wanted to shut the situation out of her/his mind.13 3 1 1 1 (mean 1.8)11
A21. The situation required her/him a great deal of energy to deal with it.33 3 1 2 2 (mean 2.2)21
Mean squared error on this text1.74 (of their mean ratings)1.051.52

Appraisal skill

The skill scores and their 95% intervals are in the chart at the top of the page. The table gives each system's mean squared error on the 21 ratings, from which the skill scores are computed, and its emotion results. "Readers with the same error" is exploratory: averaging more readers lowers the error, from 2.475 for one reader to 1.923, 1.739, 1.647 and 1.592 for two to five, and each model is placed on that curve. The models' error exceeded the five-reader average's by 0.207 to 0.513. In an exploratory comparison, every model's error was below one reader's average error, by 0.370 for Claude Haiku 5.5 to 0.676 for Claude Opus 5.5, with 95% intervals well clear of zero.

SystemMean squared errorReaders with the same errorEmotion accuracyEmotion macro-F1
No text (average ratings of other texts)2.247
One reader (average over readings)2.475150.4%0.492
Five-reader average (emotion: the readers' majority vote)1.592557.9%0.563
RoBERTa fine-tuned on the rest of the corpus1.9751.960.5%0.594
Claude Opus 5.51.7992.765.6%0.646
Gemini 3.8 Flash1.8012.767.3%0.663
GPT-6.1 Terra1.8202.666.2%0.652
Claude Sonnet 5.51.8462.459.7%0.590
GPT-6 Astra1.8652.367.8%0.670
GPT-6.1 Sol1.8672.367.6%0.669
GPT-6 Luna1.8972.158.1%0.563
Gemini 4 Argon1.9072.170.2%0.699
DeepSeek V4.1 Flash2.0181.863.4%0.616
Claude Haiku 5.52.1051.753.5%0.525

Earlier studies of this corpus used root mean squared error per statement, averaged over statements. On that measure the models scored 1.33 to 1.44, against 1.57 for one reader, 1.46 for the readers' majority vote, 1.26 for the five-reader average, 1.40 for the fine-tuned model and 1.50 for the no-text baseline. Ruder, Uusberg and Sirts (2025) reported 1.45 for one GPT-4 answer per text and 1.57 for one reader.

Models beside readers

This section is exploratory. One reader's ratings are spread so widely that the no-text baseline, which ignores the text, has a lower error than one reader. Each comparison below therefore also shows the no-text baseline, and is repeated as a correlation, which a guess that is the same for every text cannot improve.

Against the readers. For each text, one reader was set aside and the other four were averaged, in all five ways. A rater's distance from that average of four readers is the mean squared difference between the two sets of ratings, and lower is closer. The reader set aside, here called the fifth reader, was at 1.379, the no-text baseline at 1.199 and the writer's own ratings at 1.647 (the error of an average of four readers, seen from the readers' side). Every model was closer than the baseline, by 0.156 to 0.400. Every model's ratings also correlated more strongly with the average of four readers (0.67 to 0.75, averaged over statements) than the fifth reader's did (0.56). Each 95% interval of these differences excludes zero.

Each rater's distance from the average of four readers, and its correlation with it
RaterDistance from four readersDifference from the no-text baseline (95% interval)Correlation with four readers
A fifth reader1.3790.56
No-text baseline1.199
The writer1.6470.52
Claude Opus 5.50.798−0.400 (−0.438 to −0.364)0.75
Gemini 3.8 Flash0.807−0.392 (−0.429 to −0.354)0.73
GPT-6.1 Terra0.824−0.374 (−0.412 to −0.337)0.74
Claude Sonnet 5.50.829−0.370 (−0.405 to −0.334)0.73
GPT-6 Astra0.851−0.348 (−0.387 to −0.309)0.75
GPT-6.1 Sol0.865−0.334 (−0.372 to −0.294)0.75
GPT-6 Luna0.845−0.354 (−0.392 to −0.316)0.71
Gemini 4 Argon0.911−0.288 (−0.327 to −0.248)0.75
DeepSeek V4.1 Flash0.975−0.224 (−0.263 to −0.184)0.71
Claude Haiku 5.51.043−0.156 (−0.193 to −0.120)0.67
Mean of the ten models0.630−0.569 (−0.603 to −0.535)0.78

Two models' ratings correlated 0.86 across texts, averaged over statements and pairs of models (from 0.75 for DeepSeek V4.1 Flash and Claude Haiku 5.5 to 0.98 for GPT-6 Astra and GPT-6.1 Sol). Two readers of the same text correlated 0.43 (difference 0.430, 95% interval 0.422 to 0.439), and a model with a reader 0.56.

In a panel. The table gives panels of two to five raters against the writers' ratings. Replacing one reader with one model's answer lowered the panel's error for every model at every size. Replacing the reader with the no-text baseline lowered it too, so part of that gain comes without the text. With five raters, eight models lowered the error by more than the baseline did, and for DeepSeek V4.1 Flash and Claude Haiku 5.5 there is no credible evidence of a difference from the baseline. On correlation with the writers' ratings, which the baseline cannot raise, the panel with a model's answer was ahead of the panel of readers for every model at every size. Every difference stated in this paragraph has a 95% interval that excludes zero.

PanelError, readers onlyError, one reader replaced by the no-text baselineError, one reader replaced by a model's answer (ten models)Correlation, readers onlyCorrelation, one reader replaced by a model's answer (ten models)
Two raters1.9231.8541.729 to 1.8230.4700.503 to 0.542
Three raters1.7391.7031.642 to 1.6910.5020.521 to 0.546
Four raters1.6471.6241.586 to 1.6180.5210.534 to 0.552
Five raters1.5921.5751.549 to 1.5720.5340.543 to 0.557

Rescaled together. The rescaling of Where the gap comes from can be fitted to the five-reader average and one model's ratings at once. The error was then 1.403 to 1.467 (skill scores 1.19 to 1.29). For every model this was lower than the rescaled five-reader average's 1.522 and lower than the model's own rescaled error, with each 95% interval excluding zero, so the readers and the model each carried information that the other lacked. With the mean of the ten models in place of one model, the error was 1.389 (skill score 1.31, 95% interval 1.27 to 1.36).

Error of each model rescaled alone and together with the five-reader average
ModelError, rescaled aloneError, rescaled with the five-reader averageDifference from the rescaled five-reader average (95% interval)Difference from the model rescaled alone (95% interval)
Claude Opus 5.51.4451.403−0.119 (−0.135 to −0.105)−0.043 (−0.051 to −0.034)
Gemini 3.8 Flash1.4701.414−0.108 (−0.123 to −0.094)−0.056 (−0.067 to −0.046)
GPT-6.1 Terra1.4621.411−0.111 (−0.125 to −0.096)−0.051 (−0.060 to −0.041)
Claude Sonnet 5.51.4931.426−0.097 (−0.110 to −0.083)−0.067 (−0.079 to −0.056)
GPT-6 Astra1.4631.413−0.109 (−0.123 to −0.095)−0.050 (−0.060 to −0.040)
GPT-6.1 Sol1.4571.409−0.113 (−0.128 to −0.099)−0.048 (−0.058 to −0.038)
GPT-6 Luna1.5411.447−0.075 (−0.087 to −0.063)−0.094 (−0.107 to −0.081)
Gemini 4 Argon1.4471.404−0.119 (−0.134 to −0.103)−0.043 (−0.053 to −0.034)
DeepSeek V4.1 Flash1.5341.444−0.078 (−0.090 to −0.066)−0.089 (−0.102 to −0.076)
Claude Haiku 5.51.6061.467−0.055 (−0.064 to −0.045)−0.139 (−0.154 to −0.123)
Mean of the ten models1.4121.389−0.134 (−0.149 to −0.118)−0.023 (−0.030 to −0.016)

The five-reader average is a reference point, not a ceiling. From how much a text's five readers differ from one another, an unlimited number of such readers, drawn independently, would have an error of 1.371 without rescaling (95% interval 1.328 to 1.415; skill score 1.34).

Emotion

The emotion each writer was asked to write about is the criterion here, so this measures recognising the prompted emotion, not a free report. In exploratory paired comparisons with the readers' majority vote, the 95% interval of the difference in accuracy lies above zero for seven models (Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra, GPT-6 Astra, GPT-6.1 Sol, Gemini 4 Argon and DeepSeek V4.1 Flash) and below zero for Claude Haiku 5.5. For Claude Sonnet 5.5 and GPT-6 Luna it includes zero, so there is no credible evidence of a difference. Every model was more accurate than one reader. Emotion accuracy and appraisal skill ranked the models differently: the rank correlation across the ten models was 0.25.

Share of the 1,200 texts where the emotion chosen was the one the writer was asked to write about
Gemini 4 Argon
70.2%
GPT-6 Astra
67.8%
GPT-6.1 Sol
67.6%
Gemini 3.8 Flash
67.3%
GPT-6.1 Terra
66.2%
Claude Opus 5.5
65.6%
DeepSeek V4.1 Flash
63.4%
RoBERTa fine-tuned on the rest of the corpus
60.5%
Claude Sonnet 5.5
59.7%
GPT-6 Luna
58.1%
Readers' majority vote
57.9%
Claude Haiku 5.5
53.5%
One reader
50.4%

The models' confidence ratings correlated 0.21 to 0.36 with whether their emotion choice was right, against 0.21 for one reader. The models' intensity ratings had a mean squared error of 0.885 to 1.356 against the writers', against 1.494 for one reader and 0.959 for the five-reader average. Their event durations matched the writer's for 47.1% to 57.3% of texts, against 45.3% for one reader.

Where the gap comes from

The split of the error and the rescaling in this section are exploratory. A set of ratings can miss the writers' ratings in three ways. It can be offset, too high or too low on average. Its ratings can be spread too widely for how well they track the writers', which enlarges every mistake. The rest of the error depends on how well it orders the texts: it is the error that would remain after the best straight-line rescaling of its ratings. The chart splits each system's mean squared error into these three parts, averaged over the 21 statements.

Left after the best rescalingSpreadOffsetNo-text baseline
Mean squared error split into three parts (mean over the 21 statements; lower is better)
Five-reader average
1.592
Claude Opus 5.5
1.799
Gemini 3.8 Flash
1.801
GPT-6.1 Terra
1.820
Claude Sonnet 5.5
1.846
GPT-6 Astra
1.865
GPT-6.1 Sol
1.867
GPT-6 Luna
1.897
Gemini 4 Argon
1.907
DeepSeek V4.1 Flash
2.018
Claude Haiku 5.5
2.105
One reader
2.475

The five-reader average is barely offset (0.011) and its spread costs 0.063, because averaging narrows the spread of individual readers. One reader has the same offset part (0.011) but is spread far too widely (0.669). Claude Opus 5.5's error of 1.799 is 1.441 left after the best rescaling fitted on all texts, less than the five-reader average's 1.518, plus 0.192 from spread and 0.166 from offset. Every model's ratings were spread too widely on all 21 statements. The table compares the models with the five-reader average directly, with 95% intervals. The correlation is the mean over statements of each system's correlation with the writers' ratings (the models' rank correlations averaged 0.49 to 0.56). The rescaling is a straight line for each statement fitted on 9 of 10 parts of the texts and applied to the tenth, for the models and for the five-reader average alike.

SystemCorrelation with writersDifference from the five-reader average (95% interval)Error after rescalingDifference from the five-reader average, both rescaled (95% interval)
Five-reader average0.531.522
Claude Opus 5.50.57+0.035 (+0.026 to +0.044)1.445−0.077 (−0.099 to −0.055)
Gemini 3.8 Flash0.56+0.026 (+0.016 to +0.036)1.470−0.052 (−0.075 to −0.029)
GPT-6.1 Terra0.56+0.029 (+0.020 to +0.039)1.462−0.060 (−0.082 to −0.038)
Claude Sonnet 5.50.55+0.017 (+0.008 to +0.027)1.493−0.029 (−0.052 to −0.007)
GPT-6 Astra0.57+0.031 (+0.022 to +0.041)1.463−0.059 (−0.081 to −0.037)
GPT-6.1 Sol0.57+0.033 (+0.024 to +0.042)1.457−0.065 (−0.088 to −0.043)
GPT-6 Luna0.53−0.004 (−0.014 to +0.006)1.541+0.019 (−0.003 to +0.042)
Gemini 4 Argon0.57+0.036 (+0.027 to +0.046)1.447−0.075 (−0.098 to −0.052)
DeepSeek V4.1 Flash0.53−0.003 (−0.013 to +0.007)1.534+0.011 (−0.012 to +0.034)
Claude Haiku 5.50.50−0.034 (−0.045 to −0.024)1.606+0.084 (+0.061 to +0.107)

Seven models correlated more strongly than the five-reader average and, after rescaling, had lower error than the rescaled five-reader average. Claude Haiku 5.5 was worse on both measures. For GPT-6 Luna and DeepSeek V4.1 Flash, the 95% intervals of both differences include zero, so there is no credible evidence that they differ from the five-reader average on either measure. After rescaling, nine models, all but Claude Haiku 5.5, had lower error than the five-reader average without rescaling. The rescaled errors show what calibration on texts with writers' ratings could achieve, not what a model achieves unaided. Within emotions, after subtracting each emotion's average ratings from other texts, the models' mean correlations were 0.36 to 0.43, against 0.41 for the five-reader average and 0.28 for one reader, a comparison with no interval.

The design also compared each model with the average ratings, from other texts, of the emotion the model chose for the text. For Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra and Claude Sonnet 5.5, there is no credible evidence that their error differs from that of these averages (the 95% intervals of the difference include zero). Each of the other six models had a higher error than the averages of the emotions it chose, by 0.066 to 0.260. In the exploratory counterpart for the readers, the five-reader average's error was 0.259 lower than that of the average ratings of the readers' majority emotion.

Shared offsets

Each bar is the models' mean signed error on one statement, averaged over the ten models: negative where the models rated lower than the writers. The line under each statement says how many models share the sign and gives the five-reader average's offset. The statement with the largest error, on living with the consequences, said "would easily live with" for writers and "could live with" for readers and models. The five readers rated the same wording as the models.

Mean signed error by statement, averaged over the ten models (negative: rated lower than the writer; scale −1 to +1)
Urgency
−0.82
all ten models below; five-reader average −0.23
Required effort
−0.49
all ten models below; five-reader average −0.16
Suddenness
−0.43
all ten models below; five-reader average −0.20
Nobody's control
−0.38
all ten models below; five-reader average −0.04
Clashed with own standards
−0.31
all ten models below; five-reader average −0.05
Caused by chance
−0.25
all ten models below; five-reader average −0.14
Others' control
−0.20
five of ten models below; five-reader average −0.01
Unpleasantness
−0.14
all ten models below; five-reader average +0.01
Violated norms
−0.12
nine of ten models below; five-reader average +0.10
Caused by self
−0.09
six of ten models below; five-reader average +0.06
Familiarity
−0.07
seven of ten models below; five-reader average +0.01
Wanted to shut out
−0.06
four of ten models below; five-reader average +0.17
Positive consequences
−0.02
four of ten models below; five-reader average −0.07
Event predictability
+0.01
seven of ten models above; five-reader average +0.09
Caused by others
+0.01
six of ten models above; five-reader average +0.03
Own control
+0.04
seven of ten models above; five-reader average +0.02
Had to pay attention
+0.10
seven of ten models above; five-reader average −0.10
Pleasantness
+0.15
all ten models above; five-reader average +0.05
Consequences anticipated
+0.20
seven of ten models above; five-reader average +0.04
Goal relevance
+0.22
nine of ten models above; five-reader average +0.14
Could live with consequences
+0.83
nine of ten models above; five-reader average +0.05
All 21 statements: correlations and signed errors for the models, one reader and the five-reader average
StatementWriters' meanCorrelation, five-reader averageCorrelation, one readerCorrelation, modelsSigned error, modelsSigned error, five-reader average
A1. Suddenness: The event was sudden or abrupt.3.110.580.450.53 to 0.64−0.56 to −0.24−0.20
A2. Familiarity: The event was familiar to its experiencer.2.660.510.370.48 to 0.56−0.50 to +0.19+0.01
A3. Event predictability: The experiencer could have predicted the occurrence of the event.2.760.500.350.48 to 0.51−0.57 to +0.27+0.09
A4. Pleasantness: The event was pleasant for the experiencer.2.520.820.730.80 to 0.82+0.03 to +0.20+0.05
A5. Unpleasantness: The event was unpleasant for the experiencer.3.170.780.680.76 to 0.80−0.25 to −0.04+0.01
A6. Goal relevance: The experiencer expected the event to have important consequences for him/herself.3.050.530.370.52 to 0.58−0.22 to +0.40+0.14
A7. Caused by chance: The event was caused by chance, special circumstances, or natural forces.2.300.480.340.41 to 0.50−0.42 to −0.01−0.14
A8. Caused by self: The event was caused by the experiencer’s own behavior.2.750.710.580.61 to 0.72−0.57 to +0.11+0.06
A9. Caused by others: The event was caused by somebody else’s behavior.3.150.630.500.61 to 0.65−0.49 to +0.32+0.03
A10. Consequences anticipated: The experiencer anticipated the consequences of the event.2.940.350.210.23 to 0.36−0.30 to +0.50+0.04
A11. Positive consequences: The experiencer expected positive consequences for her/himself.2.650.640.520.60 to 0.66−0.44 to +0.13−0.07
A12. Urgency: The event required an immediate response.3.250.380.240.36 to 0.44−1.22 to −0.49−0.23
A13. Own control: The experiencer was able to influence what was going on during the event.2.620.510.370.46 to 0.58−0.29 to +0.28+0.02
A14. Others' control: Someone other than the experiencer was influencing what was going on.3.430.520.380.45 to 0.58−1.24 to +0.24−0.01
A15. Nobody's control: The situation was the result of outside influences of which nobody had control.2.250.450.310.45 to 0.52−0.57 to −0.03−0.04
A16. Could live with consequences: The experiencer anticipated that he/she could live with the unavoidable consequences of the event.3.100.150.080.15 to 0.29−0.32 to +1.24+0.05
A17. Clashed with own standards: The event clashed with her/his standards and ideals.2.450.610.470.59 to 0.69−0.55 to −0.07−0.05
A18. Violated norms: The actions that produced the event violated laws or socially accepted norms.1.790.640.510.60 to 0.68−0.36 to +0.07+0.10
A19. Had to pay attention: The experiencer had to pay attention to the situation.3.710.410.250.34 to 0.46−0.44 to +0.42−0.10
A20. Wanted to shut out: The experiencer wanted to shut the situation out of her/his mind.2.400.520.400.46 to 0.57−0.72 to +0.19+0.17
A21. Required effort: The situation required her/him a great deal of energy to deal with it.3.200.490.340.50 to 0.55−0.85 to −0.17−0.16

Asking for an average

This section is exploratory and was run after the results above. The standard prompt asks, as the readers were asked, for the one rating this writer gave. An answer of that kind has no reason to stay away from the ends of the scale, and the models' ratings were spread too widely (Where the gap comes from). The average prompt adds one instruction: "For the statements A1 to A21, do not choose one whole number for this writer. Imagine many different people who each wrote this exact description of an event in their own life. For each statement, give the average of the ratings those writers would give, as a number from 1.0 to 5.0 with one decimal place." The 21 answer lines ask for a number from 1.0 to 5.0 in place of 1 to 5. Nine models answered all 1,200 texts once more with it. GPT-6 Astra, the most expensive model of the main run, was left out to limit cost.

Average promptStandard promptFive-reader average (1)
Skill score under the two prompts (0 = no text, 1 = five-reader average; higher is better)
Gemini 4 Argon, average prompt
1.15
95% interval 1.11 to 1.20
Gemini 4 Argon, standard prompt
0.52
95% interval 0.44 to 0.59
GPT-6.1 Terra, average prompt
1.14
95% interval 1.09 to 1.19
GPT-6.1 Terra, standard prompt
0.65
95% interval 0.59 to 0.71
Claude Opus 5.5, average prompt
1.13
95% interval 1.08 to 1.18
Claude Opus 5.5, standard prompt
0.68
95% interval 0.62 to 0.74
Gemini 3.8 Flash, average prompt
1.11
95% interval 1.06 to 1.15
Gemini 3.8 Flash, standard prompt
0.68
95% interval 0.62 to 0.74
GPT-6.1 Sol, average prompt
1.10
95% interval 1.05 to 1.14
GPT-6.1 Sol, standard prompt
0.58
95% interval 0.51 to 0.65
Claude Sonnet 5.5, average prompt
1.02
95% interval 0.98 to 1.07
Claude Sonnet 5.5, standard prompt
0.61
95% interval 0.55 to 0.67
GPT-6 Luna, average prompt
0.92
95% interval 0.87 to 0.96
GPT-6 Luna, standard prompt
0.53
95% interval 0.46 to 0.60
DeepSeek V4.1 Flash, average prompt
0.87
95% interval 0.82 to 0.92
DeepSeek V4.1 Flash, standard prompt
0.35
95% interval 0.27 to 0.43
Claude Haiku 5.5, average prompt
0.66
95% interval 0.61 to 0.71
Claude Haiku 5.5, standard prompt
0.22
95% interval 0.13 to 0.29

Every model's error was lower under the average prompt, by 0.250 for GPT-6 Luna to 0.413 for Gemini 4 Argon. For scale, a model's error on the 300 repeated texts differed by at most 0.023 between its two runs of the standard prompt. Five models then had a lower error than the five-reader average: Claude Opus 5.5, Gemini 3.8 Flash, GPT-6.1 Terra, GPT-6.1 Sol and Gemini 4 Argon, by 0.062 to 0.098. For Claude Sonnet 5.5 there is no credible evidence of a difference (−0.015, 95% interval −0.045 to +0.014). GPT-6 Luna, DeepSeek V4.1 Flash and Claude Haiku 5.5 stayed less accurate than the five-reader average, by 0.055 to 0.225. The mean of the nine models' average-prompt ratings had an error of 1.461 (skill score 1.20, 95% interval 1.16 to 1.25).

The comparison with the five-reader average is not like for like. Each reader gave one whole-number guess for one writer, and the five-reader average gains from averaging what a model was here asked to estimate directly. The readers' average is also too widely spread. Against the five-reader average after the rescaling of Where the gap comes from (error 1.522), only Gemini 4 Argon had a lower error (−0.028, 95% interval −0.056 to −0.001). For four models there is no credible evidence of a difference, and four had a higher error. Whether readers asked for an average would also come closer to the writers was not tested.

ModelError, standard promptError, average promptAverage prompt minus standard prompt (95% interval)Average prompt minus the five-reader average (95% interval)Average prompt minus the rescaled five-reader average (95% interval)
Claude Opus 5.51.7991.509−0.290 (−0.309 to −0.272)−0.084 (−0.112 to −0.056)−0.014 (−0.041 to +0.013)
Gemini 3.8 Flash1.8011.522−0.279 (−0.300 to −0.258)−0.070 (−0.098 to −0.042)0.000 (−0.027 to +0.028)
GPT-6.1 Terra1.8201.501−0.319 (−0.339 to −0.299)−0.091 (−0.120 to −0.064)−0.021 (−0.048 to +0.005)
Claude Sonnet 5.51.8461.577−0.269 (−0.289 to −0.250)−0.015 (−0.045 to +0.014)+0.055 (+0.028 to +0.082)
GPT-6.1 Sol1.8671.530−0.337 (−0.356 to −0.317)−0.062 (−0.092 to −0.033)+0.008 (−0.022 to +0.037)
GPT-6 Luna1.8971.648−0.250 (−0.271 to −0.229)+0.055 (+0.026 to +0.085)+0.125 (+0.096 to +0.155)
Gemini 4 Argon1.9071.494−0.413 (−0.437 to −0.389)−0.098 (−0.127 to −0.071)−0.028 (−0.056 to −0.001)
DeepSeek V4.1 Flash2.0181.677−0.341 (−0.369 to −0.314)+0.085 (+0.053 to +0.116)+0.155 (+0.123 to +0.186)
Claude Haiku 5.52.1051.817−0.289 (−0.313 to −0.264)+0.225 (+0.189 to +0.260)+0.295 (+0.261 to +0.328)
Mean of the nine models1.6421.461−0.181 (−0.195 to −0.166)−0.131 (−0.157 to −0.106)−0.061 (−0.086 to −0.037)

Rounded to whole numbers, the average-prompt ratings still had a lower error than the standard prompt's for every model, by 0.161 to 0.305 with halves rounded up and by 0.094 to 0.324 with halves rounded down. The ratings were less spread, with a standard deviation across texts of 0.90 to 1.12 against 1.14 to 1.42 under the standard prompt. Every model's correlation with the writers' ratings rose as well, by 0.015 to 0.023. For the ratings rounded to whole numbers, there is no credible evidence that any model's correlation was higher than under the standard prompt. The gain was 58% to 90% as large as the gain from the rescaling of Where the gap comes from on the same model's standard-prompt ratings, and unlike the rescaling the prompt contains no writers' ratings. The rescaled standard-prompt ratings still had the lower error for every model, by 0.039 to 0.210, and rescaling the average-prompt ratings lowered their error by a further 0.087 to 0.249. The differences in error and the rise in correlation in this paragraph each have a 95% interval that excludes zero.

Average-prompt ratings rounded and rescaled, and spread and correlation under both prompts
ModelError, rounded to whole numbersError, rescaledSpread of ratings, standard promptSpread of ratings, average promptCorrelation, standard promptCorrelation, average promptCorrelation, average prompt rounded
Claude Opus 5.51.5691.4061.290.980.5690.5850.562
Gemini 3.8 Flash1.6231.4211.311.050.5600.5780.556
GPT-6.1 Terra1.5941.4141.330.980.5630.5830.558
Claude Sonnet 5.51.6351.4521.210.900.5510.5660.542
GPT-6.1 Sol1.6461.4091.371.070.5670.5850.563
GPT-6 Luna1.7361.5001.271.060.5300.5470.528
Gemini 4 Argon1.6021.3871.421.080.5700.5930.574
DeepSeek V4.1 Flash1.8091.4961.401.120.5310.5460.532
Claude Haiku 5.51.8381.5671.140.900.5000.5170.497
Mean of the nine models1.5421.3921.220.970.5850.5920.567

The instruction and the decimal answer format were changed together, and Parts of the average prompt compares them. Pulling each standard-prompt rating a quarter of the way to the middle of the scale, a correction chosen after the results were seen that needs no new request, gave 78% to 100% of the gain. The average-prompt ratings had a lower error than the pulled ratings for six models, and for the other three there is no credible evidence of a difference. A request for an average could also lower the error by removing the chance variation of a single answer. Measured on the 300 texts that every model answered twice, that variation was 6% of the gain for GPT-6.1 Sol and 45% for DeepSeek V4.1 Flash, so it does not account for the whole gain of any model.

The instruction names only the 21 statements, but other answers differed as well. Every model's rating of the emotion's intensity was lower under the average prompt, by 0.07 to 0.11 on the scale from 1 to 5. Emotion accuracy was 1.9 percentage points lower for Claude Haiku 5.5, and for each of the other eight models there is no credible evidence of a difference; over the nine models together it was 0.6 points lower (95% interval −1.0 to −0.2). No appraisal rating was invalid.

Checks added after the answers had been seen: rounding with halves down, ratings pulled to the middle of the scale, the variation of one answer, and intensity
ModelError, rounded with halves downError, standard-prompt ratings pulled to the middle of the scaleAverage prompt minus pulled ratings (95% interval)Variation of one answer, share of the gainIntensity rating, average prompt minus standard prompt (95% interval)
Claude Opus 5.51.5751.563−0.054 (−0.065 to −0.043)13%−0.08 (−0.10 to −0.06)
Gemini 3.8 Flash1.6021.564−0.042 (−0.055 to −0.028)26%−0.10 (−0.13 to −0.08)
GPT-6.1 Terra1.5621.566−0.065 (−0.076 to −0.055)14%−0.09 (−0.11 to −0.08)
Claude Sonnet 5.51.6861.621−0.044 (−0.055 to −0.033)13%−0.07 (−0.09 to −0.06)
GPT-6.1 Sol1.6231.574−0.044 (−0.054 to −0.034)6%−0.11 (−0.13 to −0.09)
GPT-6 Luna1.7771.653−0.005 (−0.020 to +0.010)43%−0.11 (−0.13 to −0.08)
Gemini 4 Argon1.5831.583−0.089 (−0.103 to −0.076)14%−0.11 (−0.13 to −0.09)
DeepSeek V4.1 Flash1.7651.682−0.005 (−0.025 to +0.014)45%−0.08 (−0.10 to −0.05)
Claude Haiku 5.52.0111.817−0.001 (−0.018 to +0.017)22%−0.10 (−0.12 to −0.08)

Parts of the average prompt

This section is exploratory and was run after the average prompt. The average prompt differs from the standard prompt in three ways: it asks for ratings with one decimal place, it rules out one whole number for this writer, and it asks for the average rating of many writers. A control compared these parts. Each of the nine models answered five prompts for each of the 300 repeated texts, one after the other in an order drawn at random: the standard prompt, and four prompts with the decimal format in which the rule and the request were each present or absent. The prompts, the comparisons, the tests and the rules for wording the results were written down before any request, after five rounds of review by GPT-6 Astra and two by a second instance of Claude Fable 5.2. Errors in this section are averaged over the nine models unless a model is named.

PromptAnswer formatRules out one whole numberAsks for the averageError, mean of the nine models
StandardWhole numbersNoNo1.928
DecimalOne decimal placeNoNo1.812
StrictOne decimal placeYesNo1.698
RequestOne decimal placeNoYes1.621
AverageOne decimal placeYesYes1.607

With the decimal format alone, the error was 0.115 (95% interval 0.102 to 0.129) lower than under the standard prompt. It was lower for six models and higher for none. For GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, which gave whole numbers for more than 96% of their ratings under the decimal prompt, there is no credible evidence of a difference.

The request prompt's error was 0.192 (95% interval 0.172 to 0.211) below the decimal prompt's, and the average prompt's was 0.091 (95% interval 0.079 to 0.103) below the strict prompt's. Each of these two differences was lower for every model, so the request for the average lowered the error with and without the rule. It lowered the error by less with the rule than without it (difference 0.101, 95% interval 0.086 to 0.116). This held for six models and was reversed for Claude Haiku 5.5; for two there is no credible evidence of a difference.

Without the request, the strict prompt's error was 0.114 (95% interval 0.101 to 0.129) below the decimal prompt's: lower for seven models and higher for Claude Haiku 5.5, with no credible evidence of a difference for Claude Sonnet 5.5. With the request, the average prompt's error was 0.014 (95% interval 0.006 to 0.021) below the request prompt's; among the models, only DeepSeek V4.1 Flash's error was lower. No model's error was higher, and for the other eight there is no credible evidence of a difference.

The average prompt's error was 0.321 (95% interval 0.293 to 0.349) below the standard prompt's, and lower for every model. When the two prompts had been answered on consecutive days on the same texts, the difference was 0.322, and there is no credible evidence that the two differences differ (by 0.001, p = 0.86). For Claude Sonnet 5.5 alone, the average prompt lowered the error by 0.043 (95% interval 0.016 to 0.071) more in the control than in the earlier answers (significant after Holm correction over the nine models), a difference between sessions of unknown cause. Measured against the standard prompt, the decimal prompt gave 36% of the average prompt's difference, the strict prompt 72% and the request prompt 96%. Each of the seven differences between prompts given with an interval for the average of the nine models is statistically significant (p < 0.001; the five between prompts that differ in one part were corrected for multiple tests with the Holm method).

Each model's error under the five prompts
ModelError, standard promptError, decimal promptError, strict promptError, request promptError, average promptRatings that were not whole numbers, decimal prompt
Claude Opus 5.51.8061.6031.5701.5291.52864%
Gemini 3.8 Flash1.8301.7351.6141.5541.54211%
GPT-6.1 Terra1.8371.8411.6091.5151.5130%
Claude Sonnet 5.51.8921.6151.6141.5981.59074%
GPT-6.1 Sol1.9181.9011.6501.5751.5660%
GPT-6 Luna1.9441.8901.7771.6701.6574%
Gemini 4 Argon1.9271.7731.5811.5251.52722%
DeepSeek V4.1 Flash2.0712.0581.9191.7741.7023%
Claude Haiku 5.52.1251.8961.9471.8471.83948%
Mean of the nine models1.9281.8121.6981.6211.607
Differences between prompts that differ in one part, for each model
ModelDecimal minus standard (95% interval)Request minus decimal (95% interval)Average minus strict (95% interval)Strict minus decimal (95% interval)Average minus request (95% interval)
Claude Opus 5.5−0.203 (−0.238 to −0.166) *−0.074 (−0.093 to −0.054) *−0.042 (−0.058 to −0.026) *−0.033 (−0.052 to −0.016) *−0.002 (−0.014 to +0.010)
Gemini 3.8 Flash−0.096 (−0.131 to −0.062) *−0.181 (−0.218 to −0.144) *−0.072 (−0.100 to −0.045) *−0.121 (−0.155 to −0.085) *−0.012 (−0.034 to +0.007)
GPT-6.1 Terra+0.005 (−0.018 to +0.028)−0.326 (−0.367 to −0.286) *−0.096 (−0.122 to −0.073) *−0.232 (−0.265 to −0.199) *−0.003 (−0.014 to +0.008)
Claude Sonnet 5.5−0.277 (−0.309 to −0.244) *−0.017 (−0.031 to −0.004) *−0.024 (−0.038 to −0.010) *−0.001 (−0.013 to +0.012)−0.007 (−0.018 to +0.004)
GPT-6.1 Sol−0.017 (−0.035 to +0.001)−0.326 (−0.362 to −0.290) *−0.084 (−0.101 to −0.068) *−0.251 (−0.280 to −0.221) *−0.009 (−0.018 to −0.001)
GPT-6 Luna−0.054 (−0.090 to −0.019) *−0.220 (−0.261 to −0.179) *−0.121 (−0.153 to −0.089) *−0.113 (−0.148 to −0.078) *−0.014 (−0.043 to +0.016)
Gemini 4 Argon−0.154 (−0.194 to −0.115) *−0.248 (−0.287 to −0.211) *−0.054 (−0.078 to −0.032) *−0.192 (−0.231 to −0.153) *+0.002 (−0.013 to +0.016)
DeepSeek V4.1 Flash−0.013 (−0.074 to +0.050)−0.284 (−0.331 to −0.236) *−0.218 (−0.261 to −0.176) *−0.139 (−0.185 to −0.091) *−0.072 (−0.110 to −0.035) *
Claude Haiku 5.5−0.229 (−0.278 to −0.180) *−0.049 (−0.084 to −0.014) *−0.108 (−0.143 to −0.074) *+0.051 (+0.014 to +0.089) *−0.008 (−0.039 to +0.022)
Mean of the nine models−0.115 (−0.129 to −0.102) *−0.192 (−0.211 to −0.172) *−0.091 (−0.103 to −0.079) *−0.114 (−0.129 to −0.101) *−0.014 (−0.021 to −0.006) *

* The difference is statistically significant after correction for multiple tests with the Holm method (p < 0.05): over the nine models within a column, and for the mean of the models over the five columns.

How the control was run: 13,500 requests, none failed

The control sent 13,500 requests over 49 minutes. A model's five requests for a text were sent by one worker, and the median time from the first to the start of the last was 15 seconds. No request failed, no answer was cut off and no appraisal rating was invalid. 16 answers had another field invalid: in 12 the intensity was written with a decimal point, and in four Claude Haiku 5.5 named an emotion or an event duration that is not among the options. A pilot of the three new prompts on the 26 other texts (702 requests) checked the answer format first. The control was sent in two parts, first the three Claude models with at most 10 requests at a time, to stay within Anthropic's limit on simultaneous requests, and then the other six.

Checks

The design included six checks. Two of them, scoring valid answers only and charging the largest possible loss for each invalid rating, changed nothing, because no appraisal rating was invalid. Models answer in whole numbers, as readers did, while the no-text baseline and the five-reader average are fractional, which favours the benchmarks. Rounding both benchmarks to whole numbers raised every skill score. On the 653 texts whose writers confirmed the event happened to them, every model scored higher than on all texts, a difference that was not tested. Each model answered 300 texts a second time: it chose the same emotion for 90.3% to 99.0% of them, and its error changed by at most 0.023.

ModelSkill score, integer benchmarksSkill score, confirmed eventsSame emotion in both answersError on the 300 texts, first and second answer
Claude Opus 5.50.810.7297.7%1.805, 1.808
Gemini 3.8 Flash0.810.7494.3%1.823, 1.808
GPT-6.1 Terra0.780.7095.0%1.841, 1.830
Claude Sonnet 5.50.740.6897.3%1.860, 1.883
GPT-6 Astra0.710.6397.7%1.916, 1.910
GPT-6.1 Sol0.710.6499.0%1.919, 1.925
GPT-6 Luna0.660.6090.3%1.912, 1.896
Gemini 4 Argon0.650.5694.7%1.945, 1.931
DeepSeek V4.1 Flash0.480.3893.7%2.097, 2.104
Claude Haiku 5.50.350.2896.7%2.152, 2.146

The sixth check resampled readers instead of texts and recomputed the five-reader average. It raised every model's skill score (median 0.33 to 1.05), because a resampled text keeps on average 3.2 distinct readers of its five, which makes the benchmark less accurate. It therefore does not give a usable interval. No answer was cut off. Claude Haiku 5.5 gave 8 invalid answers to other questions in the main run and 3 in the repeat: emotions outside the options, such as "anxiety", and event durations of "months". Invalid emotions counted as wrong.

One further check is exploratory and concerns the corpus, which the models may have seen in training. A model reciting a writer's published ratings would match all 21 of them. No answer did. The most that any answer matched exactly was 19, the same as for the closest single reading, and 19 of the 12,000 answers matched 18 or more ratings, against 4 of the 6,000 readings.

Cost and skill

This section is exploratory. Each point is a model's skill score against what its answers cost per 1,000 texts at the providers' list prices. A model is on the cost-score frontier when no other model costs no more and has at least as high a skill score. The line through those models gives the highest skill score available at each cost. Hover over a point for its 95% interval.

On the cost-score frontierNot on the cost-score frontierCost-score frontier: the highest skill score available at each cost
Skill score against API cost per 1,000 texts (one text per request, default effort)
$0.3
$1
$3
$10
$30
0.20
0.40
0.60Claude Opus 5.5Claude Sonnet 5.5Claude Haiku 5.5GPT-6 AstraGPT-6.1 SolGPT-6.1 TerraGPT-6 LunaGemini 4 ArgonGemini 3.8 FlashDeepSeek V4.1 Flash
API cost per 1,000 texts (USD, log scale)

Five models were on the cost-score frontier: Claude Haiku 5.5, GPT-6 Luna, GPT-6.1 Terra, Gemini 3.8 Flash and Claude Opus 5.5. In a paired comparison with Claude Opus 5.5, the model with the highest skill score (0.684, against 0.681 for Gemini 3.8 Flash), there is no credible evidence that the error of Gemini 3.8 Flash (difference +0.002, 95% interval −0.022 to +0.026) or of GPT-6.1 Terra (+0.021, −0.004 to +0.046) differs from Claude Opus 5.5's. Their answers cost 60% and 21% as much. Every other model's error was higher than Claude Opus 5.5's. At the list price announced for Gemini 3.8 Flash from 1 January 2027, its answers would cost $11.58 per 1,000 texts, more than Claude Opus 5.5's, and it would no longer be on the cost-score frontier. Gemini 4 Argon's list price is announced to double on 1 December 2026, and it is not on the cost-score frontier at either price.

ModelOutput tokens per textCost per 1,000 textsSkill scoreError minus Claude Opus 5.5's (95% interval)
Claude Opus 5.5 cost-score frontier194$9.610.68reference
Gemini 3.8 Flash cost-score frontier1,321$5.790.68+0.002 (−0.022 to +0.026)
GPT-6.1 Terra cost-score frontier197$2.020.65+0.021 (−0.004 to +0.046)
Claude Sonnet 5.5167$4.530.61+0.047 (+0.020 to +0.075)
GPT-6 Astra176$19.090.58+0.066 (+0.040 to +0.092)
GPT-6.1 Sol172$3.780.58+0.068 (+0.042 to +0.094)
GPT-6 Luna cost-score frontier393$0.300.53+0.098 (+0.068 to +0.128)
Gemini 4 Argon175$4.030.52+0.108 (+0.081 to +0.136)
DeepSeek V4.1 Flash5,893$1.840.35+0.219 (+0.186 to +0.254)
Claude Haiku 5.5 cost-score frontier215$0.250.22+0.307 (+0.265 to +0.348)

Limitations

Future research

The exploratory results suggest that a model's error can be reduced without changing the model. Each idea below can be tested on crowd-enVENT with every choice fixed in advance. Two exploratory checks on the answers already collected give a first indication.

Skill score of each model's first answer and of the mean of its two answers, on the 300 repeated texts
ModelFirst answerMean of two answersGain
Claude Opus 5.50.7040.731+0.027
Gemini 3.8 Flash0.6760.749+0.073
GPT-6.1 Terra0.6460.690+0.044
Claude Sonnet 5.50.6150.624+0.009
GPT-6 Astra0.5250.548+0.024
GPT-6.1 Sol0.5200.533+0.013
GPT-6 Luna0.5310.632+0.101
Gemini 4 Argon0.4770.536+0.059
DeepSeek V4.1 Flash0.2310.369+0.137
Claude Haiku 5.50.1430.205+0.062

Two recent datasets would allow the comparison with human readers to be repeated on new texts, and the ideas above to be tested on data that did not suggest them.

Method

Data, models, scoring and statistics

Data. crowd-enVENT (licence CC BY 4.0) has 6,600 texts by 2,379 writers recruited on Prolific. The 1,200 evaluation texts are those the corpus authors had read by others: 100 for each of ten emotions and for no emotion, and 50 each for guilt and shame, each by a different writer, read by 1,217 readers (6,000 readings). The corpus authors fine-tuned RoBERTa-large models on the 5,400 other texts, with these 1,200 as the test set, and released the predictions of five training runs. The no-text baseline and the emotion averages come from the 3,144 other texts left after removing every text by a writer of an evaluation text and every text with the same wording as one.

Models. Claude Opus 5.5, Claude Sonnet 5.5 and Claude Haiku 5.5 (Anthropic); GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna (OpenAI); Gemini 4 Argon and Gemini 3.8 Flash (Google); DeepSeek V4.1 Flash (DeepSeek, through OpenRouter on a single host, DeepInfra, with temperature and top-p set to 1.0 as its model card recommends). Requests set no reasoning effort and no system prompt. Each request was a new conversation with one text. Each model answered all 1,200 texts once and 300 of them, chosen at random within each emotion, a second time. Nine of the models then answered all 1,200 texts once more under the average prompt (Asking for an average), and later five prompts for each of the 300 repeated texts (Parts of the average prompt).

Scoring. Answers were read with fixed rules. An invalid rating would have been replaced by the no-text baseline's rating for that statement, and an invalid emotion counted as wrong. A request that ended in an error before any output would have been repeated; none did. The scoring code was checked against the readers' predictions released with the corpus, which reproduce the corpus paper's published human results to within 0.01, except three rows of its error table (statements A16 to A18) that the paper prints rotated by one row, in the readers' column and in the fine-tuned model's column alike.

Statistics. Intervals come from 10,000 resamples of the texts, drawn within each emotion and shared by all systems, with every benchmark recomputed in each resample. Differences from the two benchmarks were tested two-sided and corrected for the 20 comparisons with the Holm method. Exploratory differences are given with intervals only, except in Parts of the average prompt, whose differences between prompts were tested in the same way and corrected with the Holm method.

Design. The outcome, benchmarks, tests, prompt, parsing rules and subsets were fixed in a design document before any request on the evaluation texts, after a critique of the design by GPT-6 Astra. It was changed three times. Maximum effort was dropped before any request on the evaluation texts, and the average prompt was added after the main results were known, with its wording and eight comparisons written down before any request with it. The control of the average prompt's parts was added after the average prompt's results were known; its prompts, comparisons, tests and rules for wording the results were written down and reviewed before any request with them. The other comparisons in Asking for an average were added after its answers had been seen, among them the rescaled five-reader average, the ratings pulled to the middle of the scale and the variation of one answer. An optional condition in the design, predicting the emotion from the writers' own ratings without the text, needed separate approval and was not run.

API cost: $186.94 in total

Costs are from the run records at the providers' list prices, standard tier; DeepSeek V4.1 Flash at DeepInfra's price on OpenRouter.

StageRequestsCost
Pilot on 26 other texts (both effort levels) and a batching check540$10.27
Main run: every text, ten models12,000$61.49
Repeat on 300 texts, ten models3,000$15.26
Average prompt: pilot on the 26 other texts, nine models234$1.00
Average prompt: every text, nine models10,800$43.67
Parts of the average prompt: pilot of the three new prompts on the 26 other texts, nine models702$2.84
Parts of the average prompt: five prompts on the 300 repeated texts, nine models13,500$52.42
Total40,776$186.94