Frontier language models outscore human norms on emotional intelligence tests: a replication with newer models

Alejandro Legarda, a.legarda.j@gmail.com

Terms used on this page
STEM
Situational Test of Emotion Management: 44 short situations, each with four possible actions. The keyed answer, the one the scoring key marks as correct, is the action most experts chose.
STEU
Situational Test of Emotion Understanding: 42 situations. According to the original study, the keyed answer is the emotion or cause that appraisal theory predicts.
GEMOK-Blends
Geneva Emotion Knowledge Test, blends subtest: 20 short stories in which a person feels two emotions, described through their thoughts, feelings, urges, expressions and bodily signs. The keyed answer is the pair of emotion words whose signs the test's authors wrote into the story.
Proportion correct
Share of a test's items answered with the keyed option.
Run
One new conversation in which a model received the whole test and answered every item.
Effort
How much a model reasons before answering. At default effort the request sets no effort level, so each model uses its provider's default. Where documented, this is medium for most models and high for Claude Sonnet 5.5 and, on DeepSeek's own service, DeepSeek V4.1 Flash. At maximum effort (max in tables and charts) the request asks for the provider's highest level.
Original six models
ChatGPT-4, ChatGPT-o1, Gemini 1.5 flash, Copilot 365, Claude 3.5 Haiku and DeepSeek V3, tested by the original study in December 2024 and January 2025.
Human norm
The mean score of the human sample used when each test was first published. For the STEU this was 200 psychology undergraduates in Australia, and for the STEM 112 of them. For GEMOK-Blends it was 180 English-speaking workers on the online work platform Amazon Mechanical Turk.
Reliable
Statistically significant at 0.05, two-sided, after correction for the number of comparisons tested together.
Scenario
An item's text without its answer options.
Model judge
A language model asked to rate how similar two scenarios are, on the 1 to 7 scale the original study's human raters used.
Flagged
A new scenario is flagged when its highest rated similarity to any original scenario is above 5 of 7. The original study counted possible paraphrases by this rule.
Realism and clarity
The two questions the original study's participants answered after each item, on a 0 to 100 slider: how plausible or realistic, and how clear, the situation and its answer options are.
Model rater
A frontier model asked to rate how realistic and how clear an item is, on the original study's 0 to 100 sliders.
Panel
The frontier models taking a set of items as a test. The panel disputes an item's key when, counting each model equally, it preferred another option to the key.

Schlegel, Sommer and Mortillaro (2025, Communications Psychology) reported that six language models outscored people on five tests of emotional intelligence, and that ChatGPT-4 could write new test items of similar difficulty. This replication repeats three of those tests with ten models released since, called the frontier models below: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna, Gemini 4 Argon, Gemini 3.8 Flash and DeepSeek V4.1 Flash. Each took the full Situational Test of Emotion Management (STEM), the full Situational Test of Emotion Understanding (STEU) and the blends subtest of the Geneva Emotion Knowledge Test (GEMOK-Blends), 10 times at its default reasoning effort (how much the model reasons before it answers) and 10 times at maximum effort. Their scores are compared with the published human norms and with the six models the original study tested. Each frontier model also wrote new STEM and STEU items, as ChatGPT-4 did in the original study, and two of the frontier models, acting as judges, rated how closely those items resemble the originals. The nine models that did not write an item set then rated its realism and clarity and answered it as a test. Each of these model checks was first tested against the original study's human data.

Emotion management (STEM), 44 items: mean proportion correct
Human norm
0.52
112 students, 2008
Original six models
0.78
60 runs (6 models × 10)
Frontier models, default effort
0.82
100 runs (10 models × 10)
Frontier models, max effort
0.83
100 runs (10 models × 10)
Emotion understanding (STEU), 42 items: mean proportion correct
Human norm
0.60
200 students, 2008
Original six models
0.78
60 runs (6 models × 10)
Frontier models, default effort
0.93
100 runs (10 models × 10)
Frontier models, max effort
0.98
100 runs (10 models × 10)
Emotion blends (GEMOK-Blends), 20 items: mean proportion correct
Human norm
0.67
180 online workers, 2018
Original six models
0.85
60 runs (6 models × 10)
Frontier models, default effort
0.96
100 runs (10 models × 10)
Frontier models, max effort
0.97
100 runs (10 models × 10)

Findings

Original six models against the frontier models

Each bar is one model's mean proportion correct. For the original six models it is the mean of 10 runs, as reported to two decimals in the original study. Each frontier model has one bar per effort level, each the mean of 10 runs. All 600 frontier runs returned a complete set of answers that could be scored.

Original six modelsFrontier models, default effortFrontier models, maximum effortHuman norm
Emotion management (STEM), 44 items: mean per model
Anthropic
Claude 3.5 Haiku
0.75
Claude Haiku 5.5
0.8000.820
Claude Sonnet 5.5
0.7750.800
Claude Opus 5.5
0.8340.823
OpenAI
ChatGPT-4
0.75
ChatGPT-o1
0.83
GPT-6 Luna
0.7750.802
GPT-6.1 Terra
0.8230.841
GPT-6.1 Sol
0.8180.848
GPT-6 Astra
0.8250.855
Google
Gemini 1.5 flash
0.76
Gemini 3.8 Flash
0.8680.873
Gemini 4 Argon
0.8590.848
DeepSeek
DeepSeek V3
0.80
DeepSeek V4.1 Flash
0.7930.791
Microsoft (no newer model tested)
Copilot 365
0.79
Emotion understanding (STEU), 42 items: mean per model
Anthropic
Claude 3.5 Haiku
0.77
Claude Haiku 5.5
0.7861.000
Claude Sonnet 5.5
0.8671.000
Claude Opus 5.5
0.9931.000
OpenAI
ChatGPT-4
0.72
ChatGPT-o1
0.85
GPT-6 Luna
0.8830.869
GPT-6.1 Terra
0.9121.000
GPT-6.1 Sol
0.9671.000
GPT-6 Astra
1.0001.000
Google
Gemini 1.5 flash
0.77
Gemini 3.8 Flash
0.9381.000
Gemini 4 Argon
1.0001.000
DeepSeek
DeepSeek V3
0.81
DeepSeek V4.1 Flash
0.9120.952
Microsoft (no newer model tested)
Copilot 365
0.78
Emotion blends (GEMOK-Blends), 20 items: mean per model
Anthropic
Claude 3.5 Haiku
0.87
Claude Haiku 5.5
0.9050.920
Claude Sonnet 5.5
0.9001.000
Claude Opus 5.5
0.9951.000
OpenAI
ChatGPT-4
0.80
ChatGPT-o1
0.87
GPT-6 Luna
0.9500.930
GPT-6.1 Terra
0.9500.980
GPT-6.1 Sol
1.0001.000
GPT-6 Astra
1.0001.000
Google
Gemini 1.5 flash
0.83
Gemini 3.8 Flash
0.9500.950
Gemini 4 Argon
1.0001.000
DeepSeek
DeepSeek V3
0.90
DeepSeek V4.1 Flash
0.9200.945
Microsoft (no newer model tested)
Copilot 365
0.85

On the STEU and GEMOK-Blends, every frontier model scored above the best original model from its developer at both effort levels. On the STEM, 13 of the 20 model and effort combinations scored above the best original model from the same developer. At default effort, none of OpenAI's four frontier models scored above ChatGPT-o1 (0.83), and GPT-6 Astra's 0.825 was level with it at two decimals. GPT-6 Luna (0.802) stayed below ChatGPT-o1 at maximum effort. DeepSeek V4.1 Flash (0.793 and 0.791) stayed below DeepSeek V3 (0.80) at both effort levels.

Statistical comparison

The table below compares the frontier models with the original six models, counting each model's mean once (10 frontier against 6 original models). Each difference was tested with an exact permutation test and corrected for the four STEM and STEU comparisons or, separately, the two GEMOK-Blends comparisons. The original study compared its models with the human sample in a t-test that treated each run as a separate person. Repeating that t-test on these data gives very large effects against the human norm. Cohen's d, the difference between the two means in units of their pooled standard deviation, is 5.22 on the STEM at default effort, against 4.08 for the original six models. The t-test overstates certainty, because runs of one model are not independent.

TestEffortFrontier models, mean (range)Original six models, mean (range)Difference95% intervalp (corrected)
STEMdefault0.817 (0.775 to 0.868)0.780 (0.75 to 0.83)+0.037+0.000 to +0.074p = 0.044reliable
STEMmax0.830 (0.791 to 0.873)0.780 (0.75 to 0.83)+0.050+0.015 to +0.085p = 0.011reliable
STEUdefault0.926 (0.786 to 1.000)0.783 (0.72 to 0.85)+0.142+0.082 to +0.203p = 0.003reliable
STEUmax0.982 (0.869 to 1.000)0.783 (0.72 to 0.85)+0.199+0.149 to +0.248p < 0.001reliable
GEMOK-Blendsdefault0.957 (0.900 to 1.000)0.853 (0.80 to 0.90)+0.104+0.062 to +0.145p < 0.001reliable
GEMOK-Blendsmax0.973 (0.920 to 1.000)0.853 (0.80 to 0.90)+0.119+0.080 to +0.158p < 0.001reliable

No STEM run was perfect, and the best scored 0.909. The STEM key is the option most experts chose, and the experts often disagreed. Averaged over items, the options a model picked had been chosen by 63% to 69% of the experts, depending on the model and effort level.

GEMOK-Blends items with wrong answers

Each row is a GEMOK-Blends item that at least one frontier run answered wrongly. The keyed pair is the pair of emotions whose signs the test's authors wrote into the story. The authors note that it need not be the answer most people choose. The original six models' proportions correct come from the original study's item-level data, and the people are the 91 participants in the original study's GEMOK-Blends study.

ItemKeyed pairMost common wrong answerFrontier runs wrongOriginal six models, proportion correctPeople, proportion correct
12happiness and pleasurejoy and happiness57 of 2000.130.58
5surprise and jealousyjealousy and anger50 of 2000.030.21
7happiness and jealousypride and irritation16 of 2000.780.46
1pride and anxietypride and fear10 of 2000.920.76
11anger and sadnesscontempt and sadness5 of 2001.000.78
19shame and anxietyguilt and anxiety2 of 2000.950.75
3anxiety and guiltfear and shame1 of 2000.930.54

The original study's supplement (Supplementary Notes 5) says 76% of people answered item 5 correctly. That figure comes from a data file of the study in which the GEMOK-Blends human accuracies are listed against the wrong items (76% is the accuracy on item 10). The study's own item table (its Supplementary Table 11b) and its raw responses give 21%.

All 60 model, effort and test combinations: spread, perfect runs, output length and human percentile

Human percentile is estimated from the published human mean and standard deviation, assuming normally distributed scores. The standard deviations are 0.07 on the STEM, 0.13 on the STEU and 0.18 on GEMOK-Blends.

TestModelEffortMeans.d.Lowest to highest runPerfect runsOutput tokens per runHuman percentile
STEMClaude Opus 5.5default0.8340.0240.795 to 0.8640694>99.9
STEMClaude Opus 5.5max0.8230.0210.795 to 0.841047,225>99.9
STEMClaude Sonnet 5.5default0.7750.0070.773 to 0.7950177>99.9
STEMClaude Sonnet 5.5max0.8000.0180.773 to 0.818097,356>99.9
STEMClaude Haiku 5.5default0.8000.0350.750 to 0.84101,142>99.9
STEMClaude Haiku 5.5max0.8200.0350.750 to 0.864094,471>99.9
STEMGPT-6 Astradefault0.8250.0260.773 to 0.8640726>99.9
STEMGPT-6 Astramax0.8550.0240.818 to 0.886010,114>99.9
STEMGPT-6.1 Soldefault0.8180.0210.795 to 0.8640486>99.9
STEMGPT-6.1 Solmax0.8480.0220.818 to 0.88608,908>99.9
STEMGPT-6.1 Terradefault0.8230.0210.795 to 0.8640620>99.9
STEMGPT-6.1 Terramax0.8410.0280.795 to 0.886028,790>99.9
STEMGPT-6 Lunadefault0.7750.0170.750 to 0.79501,383>99.9
STEMGPT-6 Lunamax0.8020.0370.750 to 0.841018,067>99.9
STEMGemini 4 Argondefault0.8590.0140.841 to 0.8860214>99.9
STEMGemini 4 Argonmax0.8480.0220.818 to 0.88608,748>99.9
STEMGemini 3.8 Flashdefault0.8680.0210.841 to 0.90903,900>99.9
STEMGemini 3.8 Flashmax0.8730.0120.864 to 0.886039,236>99.9
STEMDeepSeek V4.1 Flashdefault0.7930.0230.750 to 0.818021,131>99.9
STEMDeepSeek V4.1 Flashmax0.7910.0510.682 to 0.841025,128>99.9
STEUClaude Opus 5.5default0.9930.0120.976 to 1.00071,18799.9
STEUClaude Opus 5.5max1.0000.0001.000 to 1.000107,93399.9
STEUClaude Sonnet 5.5default0.8670.0170.833 to 0.88101,53498.0
STEUClaude Sonnet 5.5max1.0000.0001.000 to 1.0001037,75499.9
STEUClaude Haiku 5.5default0.7860.0320.738 to 0.83302,28992.3
STEUClaude Haiku 5.5max1.0000.0001.000 to 1.00010130,55499.9
STEUGPT-6 Astradefault1.0000.0001.000 to 1.0001027899.9
STEUGPT-6 Astramax1.0000.0001.000 to 1.000101,34999.9
STEUGPT-6.1 Soldefault0.9670.0230.952 to 1.000336799.8
STEUGPT-6.1 Solmax1.0000.0001.000 to 1.000101,62699.9
STEUGPT-6.1 Terradefault0.9120.0250.881 to 0.952049099.2
STEUGPT-6.1 Terramax1.0000.0001.000 to 1.000105,62799.9
STEUGPT-6 Lunadefault0.8830.0330.833 to 0.92901,92598.5
STEUGPT-6 Lunamax0.8690.0170.857 to 0.905019,06898.1
STEUGemini 4 Argondefault1.0000.0001.000 to 1.0001039099.9
STEUGemini 4 Argonmax1.0000.0001.000 to 1.000104,71099.9
STEUGemini 3.8 Flashdefault0.9380.0550.881 to 1.00041,85999.5
STEUGemini 3.8 Flashmax1.0000.0001.000 to 1.0001013,36199.9
STEUDeepSeek V4.1 Flashdefault0.9120.0550.857 to 1.000225,59199.2
STEUDeepSeek V4.1 Flashmax0.9520.0660.833 to 1.000635,24499.7
GEMOK-BlendsClaude Opus 5.5default0.9950.0160.950 to 1.00091,06796.5
GEMOK-BlendsClaude Opus 5.5max1.0000.0001.000 to 1.0001037,56896.7
GEMOK-BlendsClaude Sonnet 5.5default0.9000.0000.900 to 0.90008189.9
GEMOK-BlendsClaude Sonnet 5.5max1.0000.0001.000 to 1.0001097,43096.7
GEMOK-BlendsClaude Haiku 5.5default0.9050.0280.850 to 0.95002,30390.4
GEMOK-BlendsClaude Haiku 5.5max0.9200.0260.900 to 0.9500103,23391.8
GEMOK-BlendsGPT-6 Astradefault1.0000.0001.000 to 1.000109896.7
GEMOK-BlendsGPT-6 Astramax1.0000.0001.000 to 1.0001085096.7
GEMOK-BlendsGPT-6.1 Soldefault1.0000.0001.000 to 1.0001012096.7
GEMOK-BlendsGPT-6.1 Solmax1.0000.0001.000 to 1.0001091396.7
GEMOK-BlendsGPT-6.1 Terradefault0.9500.0330.900 to 1.000231994.0
GEMOK-BlendsGPT-6.1 Terramax0.9800.0350.900 to 1.000717,56695.7
GEMOK-BlendsGPT-6 Lunadefault0.9500.0000.950 to 0.950047994.0
GEMOK-BlendsGPT-6 Lunamax0.9300.0260.900 to 0.950010,44992.6
GEMOK-BlendsGemini 4 Argondefault1.0000.0001.000 to 1.000109496.7
GEMOK-BlendsGemini 4 Argonmax1.0000.0001.000 to 1.000106,43896.7
GEMOK-BlendsGemini 3.8 Flashdefault0.9500.0000.950 to 0.95002,82894.0
GEMOK-BlendsGemini 3.8 Flashmax0.9500.0000.950 to 0.950011,18994.0
GEMOK-BlendsDeepSeek V4.1 Flashdefault0.9200.0350.850 to 0.950019,80891.8
GEMOK-BlendsDeepSeek V4.1 Flashmax0.9450.0440.900 to 1.000332,58393.7

Effect of maximum reasoning effort

The table below gives each frontier model's mean score and output tokens per run at default and at maximum effort. Output tokens include reasoning tokens. Each change in score was tested with an exact permutation test on the 10 runs at each level. The label "reliable" marks a change that survives correction for the 20 STEM and STEU comparisons or, separately, for the 10 GEMOK-Blends comparisons.

ModelSTEM score, default → maxSTEM output tokens per runSTEU score, default → maxSTEU output tokens per runGEMOK-Blends score, default → maxGEMOK-Blends output tokens per run
Claude Opus 5.50.834 → 0.823 (−0.011)694 → 47,2250.993 → 1.000 (+0.007)1,187 → 7,9330.995 → 1.000 (+0.005)1,067 → 37,568
Claude Sonnet 5.50.775 → 0.800 (+0.025)177 → 97,3560.867 → 1.000 (+0.133)reliable1,534 → 37,7540.900 → 1.000 (+0.100)reliable81 → 97,430
Claude Haiku 5.50.800 → 0.820 (+0.020)1,142 → 94,4710.786 → 1.000 (+0.214)reliable2,289 → 130,5540.905 → 0.920 (+0.015)2,303 → 103,233
GPT-6 Astra0.825 → 0.855 (+0.030)726 → 10,1141.000 → 1.000 (+0.000)278 → 1,3491.000 → 1.000 (+0.000)98 → 850
GPT-6.1 Sol0.818 → 0.848 (+0.030)486 → 8,9080.967 → 1.000 (+0.033)367 → 1,6261.000 → 1.000 (+0.000)120 → 913
GPT-6.1 Terra0.823 → 0.841 (+0.018)620 → 28,7900.912 → 1.000 (+0.088)reliable490 → 5,6270.950 → 0.980 (+0.030)319 → 17,566
GPT-6 Luna0.775 → 0.802 (+0.027)1,383 → 18,0670.883 → 0.869 (−0.014)1,925 → 19,0680.950 → 0.930 (−0.020)479 → 10,449
Gemini 4 Argon0.859 → 0.848 (−0.011)214 → 8,7481.000 → 1.000 (+0.000)390 → 4,7101.000 → 1.000 (+0.000)94 → 6,438
Gemini 3.8 Flash0.868 → 0.873 (+0.005)3,900 → 39,2360.938 → 1.000 (+0.062)1,859 → 13,3610.950 → 0.950 (+0.000)2,828 → 11,189
DeepSeek V4.1 Flash0.793 → 0.791 (−0.002)21,131 → 25,1280.912 → 0.952 (+0.040)25,591 → 35,2440.920 → 0.945 (+0.025)19,808 → 32,583

Item difficulty compared with humans

The original study's item-level data cover short forms of the STEM (18 items, 187 people) and the STEU (19 items, 185 people), each pooled from two of the original study's samples. Across these 37 items, the share of the original six models' answers that were correct correlated 0.71 with the share of people who answered correctly. For the frontier models the correlation was 0.61 at default and 0.51 at maximum effort. At maximum effort, 27 of the 37 items were answered correctly in every frontier run, so the correlation rests on the other 10 items, where it was 0.41. These data do not show whether the frontier models and people find different items hard.

Model judges against human similarity ratings

The paraphrase check below needs a rater of how similar two scenarios are. A scenario is an item's text without its answer options. To test language models as judges, each of the ten frontier models rated the 1,590 scenario pairs that human raters scored in the original study. These pair each of ChatGPT-4's 37 new scenarios with every original scenario of the same test (18 × 44 STEM and 19 × 42 STEU pairs). GPT-6.1 Sol and Claude Opus 5.5 were chosen as the two judges for the new items from these ratings. GPT-6.1 Sol had the second-highest correlation with the mean human rating, at about a fifth of the cost of the highest, GPT-6 Astra. Claude Opus 5.5 was paired with it so that the judges came from different developers. The pair's averaged rating correlated 0.75 with the mean human rating and flagged 6 of the 7 ChatGPT-4 scenarios that the human raters flagged, plus 2 that they did not.

Correlation with the mean human similarity rating (1,590 scenario pairs)
Ceiling set by human rater agreement
0.83
highest correlation a judge can be expected to reach, given how much the human raters disagree
GPT-6.1 Sol and Claude Opus 5.5, averaged
0.75
the two judges used to rate the new items
GPT-6 Astra
0.75
95% interval 0.73 to 0.77, and its 3 repeat ratings correlate 0.91
GPT-6.1 Sol
0.74
95% interval 0.72 to 0.76, and its 3 repeat ratings correlate 0.90
GPT-6.1 Terra
0.73
95% interval 0.70 to 0.75, and its 3 repeat ratings correlate 0.85
Claude Opus 5.5
0.72
95% interval 0.70 to 0.75, and its 3 repeat ratings correlate 0.84
GPT-6 Luna
0.72
95% interval 0.69 to 0.74, and its 3 repeat ratings correlate 0.81
Gemini 3.8 Flash
0.71
95% interval 0.69 to 0.74, and its 3 repeat ratings correlate 0.85
Gemini 4 Argon
0.71
95% interval 0.68 to 0.73, and its 3 repeat ratings correlate 0.85
DeepSeek V4.1 Flash
0.70
95% interval 0.68 to 0.73, and its 3 repeat ratings correlate 0.77
Claude Sonnet 5.5
0.67
95% interval 0.64 to 0.70, and its 3 repeat ratings correlate 0.63
Claude Haiku 5.5
0.60
95% interval 0.57 to 0.63, and its 3 repeat ratings correlate 0.40
Median human rater
0.52
the median rater's ratings against the mean of the other raters
All human ratings pooled
0.41
every rating against the mean of the other raters on that pair
Each pair was rated by 6 to 8 human raters. Each model judge rated every pair 3 times, and its bar shows the correlation of its mean rating with the mean human rating.

Paraphrase check of the model-written items

Each frontier model wrote 18 new STEM and 19 new STEU items with the original study's item-writing procedure (see Method). The two judges rated every new scenario against every original scenario of the same test, 15,900 pairs in all. A new scenario is flagged when its highest rated similarity to any original is above 5 of 7, the original study's rule for a possible paraphrase.

Each flagged scenario was put in the first of these four categories that fits its most similar original. Near-verbatim copy: the text matches that original almost word for word (a character-level match of at least 75%). Concrete version of an abstract original: the original is one of the 14 STEU items that name no person. Close to a prompt example: the original is one of the five STEU items that the original study's third STEU item-writing prompt shows as examples. Same theme: every other flagged scenario. The concrete-version category is assigned from the original alone, and six of the 97 scenarios in it are not concrete versions of their original: five are reversed-format scenarios, which ask what happened and do not describe the situation, and one restates its original in abstract terms.

Near-verbatim copyClose to a prompt exampleConcrete version of an abstract originalSame themeHuman raters (no categories)
Emotion management (STEM): flagged new scenarios
ChatGPT-4 (human raters)
5 of 18
original study
ChatGPT-4 (the two judges)
5 of 18
Claude Opus 5.5
1 of 18
Claude Sonnet 5.5
0 of 18
Claude Haiku 5.5
4 of 18
GPT-6 Astra
8 of 18
GPT-6.1 Sol
2 of 18
GPT-6.1 Terra
3 of 18
GPT-6 Luna
5 of 18
Gemini 4 Argon
2 of 18
Gemini 3.8 Flash
2 of 18
DeepSeek V4.1 Flash
7 of 18
Emotion understanding (STEU): flagged new scenarios
ChatGPT-4 (human raters)
2 of 19
original study
ChatGPT-4 (the two judges)
3 of 19
Claude Opus 5.5
14 of 19
Claude Sonnet 5.5
9 of 19
Claude Haiku 5.5
8 of 19
GPT-6 Astra
12 of 19
GPT-6.1 Sol
12 of 19
GPT-6.1 Terra
12 of 19
GPT-6 Luna
16 of 19
Gemini 4 Argon
14 of 19
Gemini 3.8 Flash
11 of 19
DeepSeek V4.1 Flash
11 of 19

Twelve of the 14 abstract originals describe an appraisal pattern, the way of judging a situation that appraisal theory links to an emotion, such as "A pleasant experience ceases unexpectedly and there is not much that can be done about it." The other two only name an emotion, and no flagged scenario was most similar to either. A correct new item for the same emotion has to show the same pattern, so a concrete version of such an item is rated as similar to it. The original study's human raters flagged 2 of ChatGPT-4's STEU scenarios, and each was a concrete version of the abstract original item 24. Each of the five near-verbatim copies reused the scenario of one of the five prompt examples, item 20 four times and item 41 once. Eight more new STEU scenarios reused a prompt example's sentence frame with a different emotion, setting or both. Seven of them were not flagged, and one was flagged in the concrete-version category, because its most similar original was an abstract item.

A judge that favoured its own items would rate them as less similar to the originals than the other judge does. On the items Claude Opus 5.5 wrote, its highest similarity per scenario averaged 0.27 points below GPT-6.1 Sol's on the 1 to 7 scale. On the items of the eight models that were not judges, it averaged 0.18 below. The 0.09-point difference is not reliable (p = 0.927 after correction for the two judges' tests). GPT-6.1 Sol showed the opposite pattern: relative to Claude Opus 5.5, it rated its own items 0.09 points more similar to the originals than it rated the other models' items. The two differences are the same size by coincidence.

Near-verbatim copy of a prompt example (similarity 7.0 of 7)
GPT-6 Luna, new STEU item 16

Matthew has been at his current job for six months. Something happened that caused him to feel regret.

Original STEU item 41

Matthew has been at his current job for six months. Something happened that caused him to feel regret.

Concrete version of an abstract original (similarity 7.0 of 7)
GPT-6 Luna, new STEU item 6

A concert that Lea has been enjoying is abruptly canceled because of a power failure, and there is no chance to resume it.

Original STEU item 1

A pleasant experience ceases unexpectedly and there is not much that can be done about it.

Same theme (similarity 6.5 of 7)
GPT-6 Astra, new STEM item 11

Beth must explain a technical project to clients unfamiliar with her field. She is nervous that they will not understand her presentation, which currently contains many specialist terms.

Original STEM item 10

Darla is nervous about presenting her work to a group of seniors who might not understand it, as they don't know much about her area.

Flag counts by item-writing model and category, both tests
TestWritten byFlaggedNear-verbatim copyClose to a prompt exampleConcrete version of an abstract originalSame theme
STEMChatGPT-4 (original study's items)5 of 180005
STEMClaude Opus 5.51 of 180001
STEMClaude Sonnet 5.50 of 180000
STEMClaude Haiku 5.54 of 180004
STEMGPT-6 Astra8 of 180008
STEMGPT-6.1 Sol2 of 180002
STEMGPT-6.1 Terra3 of 180003
STEMGPT-6 Luna5 of 180005
STEMGemini 4 Argon2 of 180002
STEMGemini 3.8 Flash2 of 180002
STEMDeepSeek V4.1 Flash7 of 180007
STEUChatGPT-4 (original study's items)3 of 190030
STEUClaude Opus 5.514 of 1901121
STEUClaude Sonnet 5.59 of 190063
STEUClaude Haiku 5.58 of 190071
STEUGPT-6 Astra12 of 1910101
STEUGPT-6.1 Sol12 of 1900102
STEUGPT-6.1 Terra12 of 1910101
STEUGPT-6 Luna16 of 1920131
STEUGemini 4 Argon14 of 1900113
STEUGemini 3.8 Flash11 of 1900101
STEUDeepSeek V4.1 Flash11 of 191082

Quality of the model-written items

In the original study, 90 to 94 online participants per test took both the original short form and ChatGPT-4's new version, and after each item rated how realistic and how clear it was on a 0 to 100 slider. No new people were tested. Instead, two model checks were first tested against those participants' data, with pass rules fixed before the first query, and only the checks that passed were applied to the frontier models' new items.

Realism and clarity ratings. Each frontier model, as a rater, rated the realism and clarity of ChatGPT-4's items and the original items 3 times, using the participants' two questions word for word. Each rater was left out in turn, and the average of the other nine raters was correlated with the participants' average rating. A rating passed if every one of these ten correlations was stronger than the correlation of the median participant's ratings with the other participants' average and had a 95% interval above 0. The table gives these correlations: the median participant's, the lowest of the nine-rater averages, and the ceiling, which is the highest correlation a rater can be expected to reach given how much the participants disagree. Seven of the 19 original items in the STEU short form were reworded for it, and the published data do not include their reworded answer options, so they were left out of this check.

TestRatingItemsMedian participantLowest 9-rater average (95% interval)CeilingResult
STEUrealism310.460.59 (0.30 to 0.78)0.98passed
STEUclarity310.46−0.09 (−0.43 to 0.27)0.98failed
STEMrealism360.340.47 (0.17 to 0.69)0.95passed
STEMclarity360.320.72 (0.51 to 0.85)0.94passed

On the STEU, the raters' clarity ratings did not correlate with the participants'. In an analysis that was not planned, they correlated instead with the share of participants who chose the keyed answer (0.57, against −0.06 for the participants' own clarity ratings). On ChatGPT-4's STEU items 7, 8 and 13, whose answer options the original study describes as similar to one another, the raters' mean clarity ratings were 63 to 68 of 100 and the participants' 89 to 91. Over the items used in the check, participants rated ChatGPT-4's STEU items +7.4 points clearer than the originals, and the raters +0.3. The raters may have read clarity as whether one answer is clearly best, and participants as whether the situation is clearly described.

Key check. Each frontier model also took ChatGPT-4's items as a test, 10 times. Together the models form the panel, which disputes an item's key when, counting each model equally, it preferred another option to the key. The original study found 5 of ChatGPT-4's 19 STEU items to perform poorly (items 4, 7, 8, 13 and 18). No participant chose the keyed answer of item 18. The check passed if the panel disputed the keys of at least 3 of these 5, including item 18, and of no more than 3 of the other 14, and if the panel's agreement with the key (the share of its answers that chose the key) correlated with the share of participants who chose it, with a 95% interval above 0. It passed on the STEU. The panel disputed the keys of items 7, 8 and 18 but not of items 4 and 13. On STEU item 18, every panel answer and 90 of the 94 participants chose "compassion" over the keyed "distress". The one other disputed key, on STEU item 10, was not a poorly performing item, but there too the panel chose the participants' most common answer, "shame", over the keyed "distress". The panel's agreement with the key correlated 0.52 (95% interval 0.03 to 0.80) with the share of participants who chose it. ChatGPT-4's STEM items include no poorly performing item, and the panel chose the key in every STEM answer, including on STEM item 18, which 71% of participants answered differently. The STEM key check therefore could not be tested and is not used.

The new items. The nine models that did not write an item set rated it 3 times and answered it as a test 3 times. Three panel runs were used because, in the key check, the first 3 runs disputed the same keys as all 10. In the table, the first two rows give mean ratings on the 0 to 100 scale from all ten models. The other rows give each model's difference from ChatGPT-4's items under the same nine raters, with its 95% interval, and the number of its STEU items whose key the panel disputed. The last row pools all frontier models against ChatGPT-4's items under all ten raters and the full panel, and the p values of its rating differences are corrected for the three comparisons. The STEM realism difference may overstate the gap participants would see: the raters rated ChatGPT-4's STEM items 4.0 points less realistic than the originals, while participants rated them 0.8 points more realistic.

Written bySTEU realismSTEM realismSTEM clarityDisputed STEU keys
ChatGPT-4 (original study's items)91.984.290.74 of 19
Original short-form items86.088.284.0not checked
Claude Opus 5.5−3.0 (−5.8 to −0.3)+8.4 (+5.4 to +11.4)+2.3 (−0.1 to +4.6)2 of 19
Claude Sonnet 5.5−1.2 (−3.8 to +1.3)+8.6 (+5.5 to +11.7)+2.7 (+0.3 to +5.1)0 of 19
Claude Haiku 5.5−0.9 (−2.4 to +0.5)+5.7 (+2.3 to +9.2)+1.9 (−0.8 to +4.6)1 of 19
GPT-6 Astra−2.3 (−4.3 to −0.4)+6.6 (+3.7 to +9.5)+4.0 (+1.8 to +6.2)0 of 19
GPT-6.1 Sol−0.9 (−2.9 to +1.0)+7.2 (+4.3 to +10.1)+3.3 (+1.1 to +5.5)0 of 19
GPT-6.1 Terra−1.1 (−3.0 to +0.8)+5.7 (+2.4 to +8.9)+4.6 (+2.3 to +7.0)2 of 19
GPT-6 Luna−1.6 (−3.2 to +0.1)+7.5 (+4.2 to +10.8)+3.7 (+1.4 to +6.0)2 of 19
Gemini 4 Argon−3.9 (−6.9 to −1.0)+3.9 (+0.8 to +7.0)+2.6 (+0.2 to +4.9)1 of 19
Gemini 3.8 Flash−5.8 (−9.1 to −2.4)+3.1 (−0.4 to +6.6)+3.3 (+0.9 to +5.8)1 of 19
DeepSeek V4.1 Flash−1.5 (−4.4 to +1.3)+6.1 (+3.0 to +9.3)+4.2 (+1.7 to +6.6)1 of 19
All frontier models−2.2 (−3.6 to −0.9), p = 0.005+6.3 (+3.3 to +9.3), p = 0.001+3.3 (+1.0 to +5.5), p = 0.00710 of 190 (p = 0.027)
The 10 new STEU items whose key the panel disputed
Written byItemItem textKeyPanel chosePanel agreement with key
Claude Opus 5.510Ruben's roommate keeps eating Ruben's labeled food from the fridge, even after being asked to stop. The roommate is on the lease, and Ruben cannot move out for another year. Ruben is most likely to feel?dislikefrustrated0.22
Claude Opus 5.511Mei is an experienced baker who has made the same bread recipe hundreds of times. Today she follows it exactly, but the loaf collapses for no apparent reason. Mei is most likely to feel?frustratedconfused0.33
Claude Haiku 5.59Greg learns that the laboratory he trusted has been faking safety test results for years. Greg is most likely to feel?horrorcontempt0.11
GPT-6.1 Terra3Ivara learns that a neighbor deliberately spreads unpleasant rumors about her. The neighbor is moving away, and there is little Ivara can do to repair the damage. Ivara is most likely to feel?dislikedistress0.44
GPT-6.1 Terra6A medical test may reveal that Varo has a serious illness. The results will not arrive for a week, and he cannot influence them. Varo is most likely to feel?fearanxiety0.11
GPT-6 Luna2Someone believes a neighbor deliberately damaged their garden plants. The damage is done, and there is little they can do about it. They are most likely to feel?dislikeanger0.30
GPT-6 Luna3A medical test may reveal a serious illness tomorrow. There is nothing Mateo can do to affect the result. Mateo is most likely to feel?scaredanxiety0.15
Gemini 4 Argon12A landlord arbitrarily decides not to renew the lease for Nadia's apartment because he simply doesn't care for her, forcing her to move out with no legal recourse. Nadia is most likely to feel?dislikeangry0.07
Gemini 3.8 Flash9Claire is trapped in an unpleasant work environment with an unreasonable boss. Despite sending out resumes for a whole year, she has obtained zero interviews, and there seems to be no way out. Claire is most likely to feel?distresseddepressed0.19
DeepSeek V4.1 Flash11Daniel is waiting for the results of a biopsy. The doctor has said the lump could be harmless, but there is a real chance it is serious. Daniel can do nothing until the results arrive. Daniel is most likely to feel?scaredanxiety0.00

Cost and score

The charts plot each score against what it cost at the providers' list prices: for the tests, the mean cost of one run of the whole test, and for the judges, the cost of 1,000 pair ratings. A model at one effort level is on the cost-score frontier when every other model and effort level either costs more or scores lower. The line through those points gives the highest score available at each cost. Hover over a point for its values.

Default effortMaximum effortNot on the cost-score frontierCost-score frontier: the highest score available at each cost
Emotion management (STEM), 44 items: mean proportion correct against API cost per run
$0.001
$0.01
$0.1
$1
0.78
0.82
0.86Claude Haiku 5.5GPT-6.1 TerraGPT-6 LunaGemini 4 ArgonGemini 3.8 FlashGemini 3.8 Flash, max
API cost of one run of the whole test (USD, log scale)
Emotion understanding (STEU), 42 items: mean proportion correct against API cost per run
$0.001
$0.01
$0.1
$1
0.80
0.90
1.00GPT-6.1 SolGPT-6.1 TerraGPT-6 LunaGemini 4 Argon
API cost of one run of the whole test (USD, log scale)
Emotion blends (GEMOK-Blends), 20 items: mean proportion correct against API cost per run
$0.001
$0.01
$0.1
$1
0.90
0.95
1.00GPT-6.1 SolGPT-6 Luna
API cost of one run of the whole test (USD, log scale)
Model judges: correlation with the mean human similarity rating against API cost
$0.01
$0.1
$1
0.60
0.65
0.70
0.75GPT-6 AstraGPT-6.1 SolGPT-6.1 TerraGPT-6 Luna
API cost of 1,000 pair ratings (USD, log scale)

Of the 12 points on the three tests' cost-score frontiers, all but one ran at default effort. The exception is Gemini 3.8 Flash on the STEM, where maximum effort added 0.005 to the mean score for 8 times the cost. One run of a test cost $0.00061 to $1.01. On the STEU, Gemini 4 Argon reached a mean of 1.000 at default effort for $0.0100 per run, and on GEMOK-Blends GPT-6.1 Sol did so for $0.0087 (Gemini 4 Argon did so for 0.2% more). The other models and effort levels with a mean of 1.000 cost up to $0.39 and $1.01 per run. Among the judges, GPT-6.1 Terra and GPT-6.1 Sol had higher correlations with the human ratings (0.73 and 0.74) than Claude Opus 5.5 (0.72), the other judge chosen, at a lower cost: $0.091 and $0.18 per 1,000 pair ratings against $0.63.

Each cost-score frontier is drawn through mean scores. Five of the 9 steps on the three tests' frontiers are smaller than the spread between runs (standard deviation) of one of the two points they join. The price rises announced for Gemini 4 Argon from 1 December 2026 and Gemini 3.8 Flash from 1 January 2027 leave the same points on every cost-score frontier.

Cost and score of every model and effort level
ModelEffortSTEM cost per runSTEM scoreSTEU cost per runSTEU scoreGEMOK-Blends cost per runGEMOK-Blends scoreJudging cost per 1,000 ratingsJudging correlation
Claude Opus 5.5default$0.0400.834$0.0400.993$0.0450.995$0.630.72
Claude Opus 5.5max$0.970.823$0.171.000$0.771.000
Claude Sonnet 5.5default$0.0150.775$0.0240.867$0.0130.900$0.220.67
Claude Sonnet 5.5max$1.010.800$0.391.000$1.011.000
Claude Haiku 5.5default$0.00120.800 frontier$0.00160.786$0.00170.905$0.0260.60
Claude Haiku 5.5max$0.0690.820$0.121.000$0.0590.920
GPT-6 Astradefault$0.0800.825$0.0411.000$0.0421.000$0.890.75 frontier
GPT-6 Astramax$0.550.855$0.0941.000$0.0801.000
GPT-6.1 Soldefault$0.0140.818$0.00910.967 frontier$0.00871.000 frontier$0.180.74 frontier
GPT-6.1 Solmax$0.0980.848$0.0221.000$0.0171.000
GPT-6.1 Terradefault$0.00740.823 frontier$0.00510.912 frontier$0.00530.950$0.0910.73 frontier
GPT-6.1 Terramax$0.150.841$0.0311.000$0.0920.980
GPT-6 Lunadefault$0.00110.775 frontier$0.00120.883 frontier$0.000610.950 frontier$0.0190.72 frontier
GPT-6 Lunamax$0.00950.802$0.00980.869$0.00560.930
Gemini 4 Argondefault$0.0120.859 frontier$0.01001.000 frontier$0.00871.000$0.180.71
Gemini 4 Argonmax$0.0970.848$0.0531.000$0.0721.000
Gemini 3.8 Flashdefault$0.0180.868 frontier$0.00920.938$0.0130.950$0.370.71
Gemini 3.8 Flashmax$0.150.873 frontier$0.0521.000$0.0450.950
DeepSeek V4.1 Flashdefault$0.00660.793$0.00780.912$0.00620.920$0.0800.70
DeepSeek V4.1 Flashmax$0.00780.791$0.0110.952$0.01000.945

Limitations

Method

How the tests were run, scored and compared

Materials. The STEM and STEU were taken from the APA online supplement to MacCann and Roberts (2008). For the STEU, the keyed answers are the options marked correct there. For the STEM, they are the options most experts chose, with three ties broken by the higher mean expert rating. Both keys matched the original study's scoring files for all 37 short-form items. GEMOK-Blends was taken from the supplement to Schlegel and Scherer (2018), where the keyed pair is underlined, and was given in the original study's item order. Its key matched the original study's scoring file for all 20 items, and scoring the original study's human responses with it reproduced their published mean (0.68).

Runs. Each run was one new conversation with one message: the test's own instructions, every item with lettered options, and a request to answer with one line per item. The output limit per response was 128,000 tokens for the Claude models, their maximum, and 393,216 for DeepSeek V4.1 Flash, DeepSeek's documented maximum. The OpenAI models were allowed 128,000 tokens on the STEM and STEU and were sent no limit on GEMOK-Blends. Gemini 4 Argon was allowed 32,768 tokens on the STEM and STEU and 1,000,000 on GEMOK-Blends, and Gemini 3.8 Flash 1,000,000 on all tests. No OpenAI or Gemini 4 Argon response came near its limit. A model that reached its limit before answering was asked to continue in the same conversation, up to five times. One continuation was needed in 13 runs, all from Claude Haiku 5.5 or Sonnet 5.5 at maximum effort. Only connection errors before any output were retried. Except for those of Gemini 3.8 Flash and DeepSeek V4.1 Flash, the STEM and STEU runs were begun under an earlier protocol, which restarted a run whose response hit the output limit. Under that protocol, responses hit the limit in three model, effort and test combinations. These were rerun in full, and the earlier runs are excluded. In the other combinations no response reached a limit: their 287 runs under the earlier protocol were kept, and their remaining 3 runs were made under the later protocol.

Comparisons. The effect of effort averaged over the 10 models was tested with an exact sign-flip test. On each of the STEM and STEU, the frontier models' flag count was compared with ChatGPT-4's under the same judges with Fisher's exact test. The pooled realism and clarity ratings were compared with ChatGPT-4's items with Welch's t-test, items as units, and the disputed STEU keys with Fisher's exact test. All tests were two-sided at 0.05 and Holm-corrected within each family of comparisons.

Item writing. Each model's item-writing conversation continued from its first default-effort run of the test. The model was told the correct answers to the items it had missed and then given the original study's item-writing prompts, word for word except that "revise items 58-61" (ChatGPT-4's own numbering) became "revise the 4 reversed-format items". Claude Sonnet 5.5 extracted the items, and every extracted scenario was checked to appear word for word in the model's response. Item writing, similarity judging, the realism and clarity ratings and the panel all used default effort.

API cost by stage: $119.04 logged

Costs are API spend from the run records and, for item writing, the run logs, at the providers' standard list prices. For DeepSeek V4.1 Flash the price is DeepInfra's on OpenRouter.

StageScopeCost
Test runs used in the analysis600 runs, 10 models$67.24
Test runs under the earlier protocol (excluded)3 model, effort and test combinations, rerun$10.16
Pilot runs to check requests and estimate costfirst 3 items of the STEM and STEU, 8 models$0.23
Checking that DeepSeek V4.1 Flash's host applied the effort setting23 calls$0.09
Testing the judges against human ratings10 judges × 1,590 pairs × 3$12.81
Item writing and extraction10 models × 2 tests$2.91
Judges rating the new items2 judges × 15,900 pairs$15.01
Testing the raters and the panel against participants' data120 rating calls, 200 panel runs$1.99
Rating and answering the new items540 rating calls, 540 panel runs$8.60
Total logged$119.04
Calls interrupted when runs were restartednot loggedup to ~$5.00