Frontier language models outscore human norms on emotional intelligence tests: a replication with newer models
Terms used on this page
- STEM
- Situational Test of Emotion Management: 44 short situations, each with four possible actions. The keyed answer, the one the scoring key marks as correct, is the action most experts chose.
- STEU
- Situational Test of Emotion Understanding: 42 situations. According to the original study, the keyed answer is the emotion or cause that appraisal theory predicts.
- GEMOK-Blends
- Geneva Emotion Knowledge Test, blends subtest: 20 short stories in which a person feels two emotions, described through their thoughts, feelings, urges, expressions and bodily signs. The keyed answer is the pair of emotion words whose signs the test's authors wrote into the story.
- Proportion correct
- Share of a test's items answered with the keyed option.
- Run
- One new conversation in which a model received the whole test and answered every item.
- Effort
- How much a model reasons before answering. At default effort the request sets no effort level, so each model uses its provider's default. Where documented, this is medium for most models and high for Claude Sonnet 5.5 and, on DeepSeek's own service, DeepSeek V4.1 Flash. At maximum effort (max in tables and charts) the request asks for the provider's highest level.
- Original six models
- ChatGPT-4, ChatGPT-o1, Gemini 1.5 flash, Copilot 365, Claude 3.5 Haiku and DeepSeek V3, tested by the original study in December 2024 and January 2025.
- Human norm
- The mean score of the human sample used when each test was first published. For the STEU this was 200 psychology undergraduates in Australia, and for the STEM 112 of them. For GEMOK-Blends it was 180 English-speaking workers on the online work platform Amazon Mechanical Turk.
- Reliable
- Statistically significant at 0.05, two-sided, after correction for the number of comparisons tested together.
- Scenario
- An item's text without its answer options.
- Model judge
- A language model asked to rate how similar two scenarios are, on the 1 to 7 scale the original study's human raters used.
- Flagged
- A new scenario is flagged when its highest rated similarity to any original scenario is above 5 of 7. The original study counted possible paraphrases by this rule.
- Realism and clarity
- The two questions the original study's participants answered after each item, on a 0 to 100 slider: how plausible or realistic, and how clear, the situation and its answer options are.
- Model rater
- A frontier model asked to rate how realistic and how clear an item is, on the original study's 0 to 100 sliders.
- Panel
- The frontier models taking a set of items as a test. The panel disputes an item's key when, counting each model equally, it preferred another option to the key.
Schlegel, Sommer and Mortillaro (2025, Communications Psychology) reported that six language models outscored people on five tests of emotional intelligence, and that ChatGPT-4 could write new test items of similar difficulty. This replication repeats three of those tests with ten models released since, called the frontier models below: Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, GPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna, Gemini 4 Argon, Gemini 3.8 Flash and DeepSeek V4.1 Flash. Each took the full Situational Test of Emotion Management (STEM), the full Situational Test of Emotion Understanding (STEU) and the blends subtest of the Geneva Emotion Knowledge Test (GEMOK-Blends), 10 times at its default reasoning effort (how much the model reasons before it answers) and 10 times at maximum effort. Their scores are compared with the published human norms and with the six models the original study tested. Each frontier model also wrote new STEM and STEU items, as ChatGPT-4 did in the original study, and two of the frontier models, acting as judges, rated how closely those items resemble the originals. The nine models that did not write an item set then rated its realism and clarity and answered it as a test. Each of these model checks was first tested against the original study's human data.
Findings
- Every frontier model scored above the human norm on all three tests at both effort levels. Each scored at least 3.6 human standard deviations above the norm on the STEM, at least 1.4 on the STEU and at least 1.2 on GEMOK-Blends. The lowest score of all, Claude Sonnet 5.5's on GEMOK-Blends at default effort, is at about the 90th percentile of a normal distribution with the human mean and standard deviation. Original vs frontier models
- At maximum effort, 8 of the 10 frontier models answered all 42 STEU items correctly in all 10 runs. GPT-6 Luna and DeepSeek V4.1 Flash were the exceptions. At default effort only GPT-6 Astra and Gemini 4 Argon had all 10 runs perfect. Original vs frontier models
- GEMOK-Blends is close to its ceiling, and most of its remaining errors are on two items. In 8 of the 20 model and effort combinations, every run was perfect. Of the 141 wrong answers in 200 runs, 107 were on items 12 and 5, which the original six models also mostly answered wrongly. On both items, the frontier models' most common wrong answer was also the most common wrong answer of the people in the original study. GEMOK-Blends items
- The frontier models scored reliably higher than the original six models on all three tests at both effort levels. Reliably means statistically significant at 0.05 after correction for the number of comparisons. Counting each model's mean once, the frontier models were ahead by 0.142 proportion correct (p = 0.003) at default and 0.199 (p < 0.001) at maximum effort on the STEU, by 0.037 (p = 0.044) and 0.050 (p = 0.011) on the STEM, and by 0.104 (p < 0.001) and 0.119 (p < 0.001) on GEMOK-Blends. Original vs frontier models
- Maximum effort reliably raised STEU scores for 3 frontier models, and no model's STEM score changed reliably. Claude Sonnet 5.5, Claude Haiku 5.5 and GPT-6.1 Terra gained 0.088 to 0.214 proportion correct on the STEU. Averaged over the ten models, the change from default to maximum effort was +0.013 on the STEM (p = 0.047) and +0.056 on the STEU (p = 0.047). On GEMOK-Blends the average change was +0.016 (p = 0.156), and only Claude Sonnet 5.5's score changed reliably, from 0.900 to 1.000. A STEM run produced 1.2 to 550 times as many output tokens at maximum effort as at default. Effort
- Every frontier model's similarity ratings correlated more strongly with the mean human rating than the median human rater's ratings did with the mean of the other raters. On the scenario pairs that the original study's human raters scored, the ratings of the best model, GPT-6 Astra, correlated 0.75 with the mean human rating, and those of the weakest, Claude Haiku 5.5, correlated 0.60. The median human rater's ratings correlated 0.52 with the mean of the other raters. Given how much the human raters disagree, the highest correlation a rater can be expected to reach is about 0.83. Judges
- The frontier models' new STEU items were flagged as possible paraphrases far more often than ChatGPT-4's, but their new STEM items were not. A new item is flagged when the two model judges' average rating of how similar its scenario (its text without the answer options) is to any original scenario is above 5 of 7. On the STEU, they flagged 119 of 190 new items, against 3 of ChatGPT-4's 19 (p < 0.001). For 97 of the 119, the most similar original was an abstract item (a STEU item that names no person), and all but six of these were concrete versions of it. Five were near-verbatim copies of original items shown as examples in the item-writing prompt. On the STEM, the judges flagged 34 of 180, against 5 of ChatGPT-4's 18 (p = 0.359). Paraphrase check
- A panel of frontier models, first tested against the original study's participants, disputed the keys of a smaller share of the frontier models' new STEU items than of ChatGPT-4's. The panel answers the items as a test and disputes an item's key, the answer marked correct, when it prefers another answer. It disputed 10 of 190 new STEU keys (0 to 2 per model), against 4 of ChatGPT-4's 19 (p = 0.027). The model raters passed their test against participants for realism on both tests and for clarity on the STEM. Their STEU clarity ratings failed, and the STEM key check could not be tested, so neither was used. The raters rated the new STEU items as slightly less realistic than ChatGPT-4's (−2.2 points of 100) and the new STEM items as more realistic (+6.3) and clearer (+3.3). The STEM realism difference may overstate what participants would see, because the raters, unlike participants, rated ChatGPT-4's STEM items less realistic than the originals. Item quality
Original six models against the frontier models
Each bar is one model's mean proportion correct. For the original six models it is the mean of 10 runs, as reported to two decimals in the original study. Each frontier model has one bar per effort level, each the mean of 10 runs. All 600 frontier runs returned a complete set of answers that could be scored.
On the STEU and GEMOK-Blends, every frontier model scored above the best original model from its developer at both effort levels. On the STEM, 13 of the 20 model and effort combinations scored above the best original model from the same developer. At default effort, none of OpenAI's four frontier models scored above ChatGPT-o1 (0.83), and GPT-6 Astra's 0.825 was level with it at two decimals. GPT-6 Luna (0.802) stayed below ChatGPT-o1 at maximum effort. DeepSeek V4.1 Flash (0.793 and 0.791) stayed below DeepSeek V3 (0.80) at both effort levels.
Statistical comparison
The table below compares the frontier models with the original six models, counting each model's mean once (10 frontier against 6 original models). Each difference was tested with an exact permutation test and corrected for the four STEM and STEU comparisons or, separately, the two GEMOK-Blends comparisons. The original study compared its models with the human sample in a t-test that treated each run as a separate person. Repeating that t-test on these data gives very large effects against the human norm. Cohen's d, the difference between the two means in units of their pooled standard deviation, is 5.22 on the STEM at default effort, against 4.08 for the original six models. The t-test overstates certainty, because runs of one model are not independent.
| Test | Effort | Frontier models, mean (range) | Original six models, mean (range) | Difference | 95% interval | p (corrected) | |
|---|---|---|---|---|---|---|---|
| STEM | default | 0.817 (0.775 to 0.868) | 0.780 (0.75 to 0.83) | +0.037 | +0.000 to +0.074 | p = 0.044 | reliable |
| STEM | max | 0.830 (0.791 to 0.873) | 0.780 (0.75 to 0.83) | +0.050 | +0.015 to +0.085 | p = 0.011 | reliable |
| STEU | default | 0.926 (0.786 to 1.000) | 0.783 (0.72 to 0.85) | +0.142 | +0.082 to +0.203 | p = 0.003 | reliable |
| STEU | max | 0.982 (0.869 to 1.000) | 0.783 (0.72 to 0.85) | +0.199 | +0.149 to +0.248 | p < 0.001 | reliable |
| GEMOK-Blends | default | 0.957 (0.900 to 1.000) | 0.853 (0.80 to 0.90) | +0.104 | +0.062 to +0.145 | p < 0.001 | reliable |
| GEMOK-Blends | max | 0.973 (0.920 to 1.000) | 0.853 (0.80 to 0.90) | +0.119 | +0.080 to +0.158 | p < 0.001 | reliable |
No STEM run was perfect, and the best scored 0.909. The STEM key is the option most experts chose, and the experts often disagreed. Averaged over items, the options a model picked had been chosen by 63% to 69% of the experts, depending on the model and effort level.
GEMOK-Blends items with wrong answers
Each row is a GEMOK-Blends item that at least one frontier run answered wrongly. The keyed pair is the pair of emotions whose signs the test's authors wrote into the story. The authors note that it need not be the answer most people choose. The original six models' proportions correct come from the original study's item-level data, and the people are the 91 participants in the original study's GEMOK-Blends study.
| Item | Keyed pair | Most common wrong answer | Frontier runs wrong | Original six models, proportion correct | People, proportion correct |
|---|---|---|---|---|---|
| 12 | happiness and pleasure | joy and happiness | 57 of 200 | 0.13 | 0.58 |
| 5 | surprise and jealousy | jealousy and anger | 50 of 200 | 0.03 | 0.21 |
| 7 | happiness and jealousy | pride and irritation | 16 of 200 | 0.78 | 0.46 |
| 1 | pride and anxiety | pride and fear | 10 of 200 | 0.92 | 0.76 |
| 11 | anger and sadness | contempt and sadness | 5 of 200 | 1.00 | 0.78 |
| 19 | shame and anxiety | guilt and anxiety | 2 of 200 | 0.95 | 0.75 |
| 3 | anxiety and guilt | fear and shame | 1 of 200 | 0.93 | 0.54 |
The original study's supplement (Supplementary Notes 5) says 76% of people answered item 5 correctly. That figure comes from a data file of the study in which the GEMOK-Blends human accuracies are listed against the wrong items (76% is the accuracy on item 10). The study's own item table (its Supplementary Table 11b) and its raw responses give 21%.
All 60 model, effort and test combinations: spread, perfect runs, output length and human percentile
Human percentile is estimated from the published human mean and standard deviation, assuming normally distributed scores. The standard deviations are 0.07 on the STEM, 0.13 on the STEU and 0.18 on GEMOK-Blends.
| Test | Model | Effort | Mean | s.d. | Lowest to highest run | Perfect runs | Output tokens per run | Human percentile |
|---|---|---|---|---|---|---|---|---|
| STEM | Claude Opus 5.5 | default | 0.834 | 0.024 | 0.795 to 0.864 | 0 | 694 | >99.9 |
| STEM | Claude Opus 5.5 | max | 0.823 | 0.021 | 0.795 to 0.841 | 0 | 47,225 | >99.9 |
| STEM | Claude Sonnet 5.5 | default | 0.775 | 0.007 | 0.773 to 0.795 | 0 | 177 | >99.9 |
| STEM | Claude Sonnet 5.5 | max | 0.800 | 0.018 | 0.773 to 0.818 | 0 | 97,356 | >99.9 |
| STEM | Claude Haiku 5.5 | default | 0.800 | 0.035 | 0.750 to 0.841 | 0 | 1,142 | >99.9 |
| STEM | Claude Haiku 5.5 | max | 0.820 | 0.035 | 0.750 to 0.864 | 0 | 94,471 | >99.9 |
| STEM | GPT-6 Astra | default | 0.825 | 0.026 | 0.773 to 0.864 | 0 | 726 | >99.9 |
| STEM | GPT-6 Astra | max | 0.855 | 0.024 | 0.818 to 0.886 | 0 | 10,114 | >99.9 |
| STEM | GPT-6.1 Sol | default | 0.818 | 0.021 | 0.795 to 0.864 | 0 | 486 | >99.9 |
| STEM | GPT-6.1 Sol | max | 0.848 | 0.022 | 0.818 to 0.886 | 0 | 8,908 | >99.9 |
| STEM | GPT-6.1 Terra | default | 0.823 | 0.021 | 0.795 to 0.864 | 0 | 620 | >99.9 |
| STEM | GPT-6.1 Terra | max | 0.841 | 0.028 | 0.795 to 0.886 | 0 | 28,790 | >99.9 |
| STEM | GPT-6 Luna | default | 0.775 | 0.017 | 0.750 to 0.795 | 0 | 1,383 | >99.9 |
| STEM | GPT-6 Luna | max | 0.802 | 0.037 | 0.750 to 0.841 | 0 | 18,067 | >99.9 |
| STEM | Gemini 4 Argon | default | 0.859 | 0.014 | 0.841 to 0.886 | 0 | 214 | >99.9 |
| STEM | Gemini 4 Argon | max | 0.848 | 0.022 | 0.818 to 0.886 | 0 | 8,748 | >99.9 |
| STEM | Gemini 3.8 Flash | default | 0.868 | 0.021 | 0.841 to 0.909 | 0 | 3,900 | >99.9 |
| STEM | Gemini 3.8 Flash | max | 0.873 | 0.012 | 0.864 to 0.886 | 0 | 39,236 | >99.9 |
| STEM | DeepSeek V4.1 Flash | default | 0.793 | 0.023 | 0.750 to 0.818 | 0 | 21,131 | >99.9 |
| STEM | DeepSeek V4.1 Flash | max | 0.791 | 0.051 | 0.682 to 0.841 | 0 | 25,128 | >99.9 |
| STEU | Claude Opus 5.5 | default | 0.993 | 0.012 | 0.976 to 1.000 | 7 | 1,187 | 99.9 |
| STEU | Claude Opus 5.5 | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 7,933 | 99.9 |
| STEU | Claude Sonnet 5.5 | default | 0.867 | 0.017 | 0.833 to 0.881 | 0 | 1,534 | 98.0 |
| STEU | Claude Sonnet 5.5 | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 37,754 | 99.9 |
| STEU | Claude Haiku 5.5 | default | 0.786 | 0.032 | 0.738 to 0.833 | 0 | 2,289 | 92.3 |
| STEU | Claude Haiku 5.5 | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 130,554 | 99.9 |
| STEU | GPT-6 Astra | default | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 278 | 99.9 |
| STEU | GPT-6 Astra | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 1,349 | 99.9 |
| STEU | GPT-6.1 Sol | default | 0.967 | 0.023 | 0.952 to 1.000 | 3 | 367 | 99.8 |
| STEU | GPT-6.1 Sol | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 1,626 | 99.9 |
| STEU | GPT-6.1 Terra | default | 0.912 | 0.025 | 0.881 to 0.952 | 0 | 490 | 99.2 |
| STEU | GPT-6.1 Terra | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 5,627 | 99.9 |
| STEU | GPT-6 Luna | default | 0.883 | 0.033 | 0.833 to 0.929 | 0 | 1,925 | 98.5 |
| STEU | GPT-6 Luna | max | 0.869 | 0.017 | 0.857 to 0.905 | 0 | 19,068 | 98.1 |
| STEU | Gemini 4 Argon | default | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 390 | 99.9 |
| STEU | Gemini 4 Argon | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 4,710 | 99.9 |
| STEU | Gemini 3.8 Flash | default | 0.938 | 0.055 | 0.881 to 1.000 | 4 | 1,859 | 99.5 |
| STEU | Gemini 3.8 Flash | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 13,361 | 99.9 |
| STEU | DeepSeek V4.1 Flash | default | 0.912 | 0.055 | 0.857 to 1.000 | 2 | 25,591 | 99.2 |
| STEU | DeepSeek V4.1 Flash | max | 0.952 | 0.066 | 0.833 to 1.000 | 6 | 35,244 | 99.7 |
| GEMOK-Blends | Claude Opus 5.5 | default | 0.995 | 0.016 | 0.950 to 1.000 | 9 | 1,067 | 96.5 |
| GEMOK-Blends | Claude Opus 5.5 | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 37,568 | 96.7 |
| GEMOK-Blends | Claude Sonnet 5.5 | default | 0.900 | 0.000 | 0.900 to 0.900 | 0 | 81 | 89.9 |
| GEMOK-Blends | Claude Sonnet 5.5 | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 97,430 | 96.7 |
| GEMOK-Blends | Claude Haiku 5.5 | default | 0.905 | 0.028 | 0.850 to 0.950 | 0 | 2,303 | 90.4 |
| GEMOK-Blends | Claude Haiku 5.5 | max | 0.920 | 0.026 | 0.900 to 0.950 | 0 | 103,233 | 91.8 |
| GEMOK-Blends | GPT-6 Astra | default | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 98 | 96.7 |
| GEMOK-Blends | GPT-6 Astra | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 850 | 96.7 |
| GEMOK-Blends | GPT-6.1 Sol | default | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 120 | 96.7 |
| GEMOK-Blends | GPT-6.1 Sol | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 913 | 96.7 |
| GEMOK-Blends | GPT-6.1 Terra | default | 0.950 | 0.033 | 0.900 to 1.000 | 2 | 319 | 94.0 |
| GEMOK-Blends | GPT-6.1 Terra | max | 0.980 | 0.035 | 0.900 to 1.000 | 7 | 17,566 | 95.7 |
| GEMOK-Blends | GPT-6 Luna | default | 0.950 | 0.000 | 0.950 to 0.950 | 0 | 479 | 94.0 |
| GEMOK-Blends | GPT-6 Luna | max | 0.930 | 0.026 | 0.900 to 0.950 | 0 | 10,449 | 92.6 |
| GEMOK-Blends | Gemini 4 Argon | default | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 94 | 96.7 |
| GEMOK-Blends | Gemini 4 Argon | max | 1.000 | 0.000 | 1.000 to 1.000 | 10 | 6,438 | 96.7 |
| GEMOK-Blends | Gemini 3.8 Flash | default | 0.950 | 0.000 | 0.950 to 0.950 | 0 | 2,828 | 94.0 |
| GEMOK-Blends | Gemini 3.8 Flash | max | 0.950 | 0.000 | 0.950 to 0.950 | 0 | 11,189 | 94.0 |
| GEMOK-Blends | DeepSeek V4.1 Flash | default | 0.920 | 0.035 | 0.850 to 0.950 | 0 | 19,808 | 91.8 |
| GEMOK-Blends | DeepSeek V4.1 Flash | max | 0.945 | 0.044 | 0.900 to 1.000 | 3 | 32,583 | 93.7 |
Effect of maximum reasoning effort
The table below gives each frontier model's mean score and output tokens per run at default and at maximum effort. Output tokens include reasoning tokens. Each change in score was tested with an exact permutation test on the 10 runs at each level. The label "reliable" marks a change that survives correction for the 20 STEM and STEU comparisons or, separately, for the 10 GEMOK-Blends comparisons.
| Model | STEM score, default → max | STEM output tokens per run | STEU score, default → max | STEU output tokens per run | GEMOK-Blends score, default → max | GEMOK-Blends output tokens per run | |||
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 0.834 → 0.823 (−0.011) | 694 → 47,225 | 0.993 → 1.000 (+0.007) | 1,187 → 7,933 | 0.995 → 1.000 (+0.005) | 1,067 → 37,568 | |||
| Claude Sonnet 5.5 | 0.775 → 0.800 (+0.025) | 177 → 97,356 | 0.867 → 1.000 (+0.133) | reliable | 1,534 → 37,754 | 0.900 → 1.000 (+0.100) | reliable | 81 → 97,430 | |
| Claude Haiku 5.5 | 0.800 → 0.820 (+0.020) | 1,142 → 94,471 | 0.786 → 1.000 (+0.214) | reliable | 2,289 → 130,554 | 0.905 → 0.920 (+0.015) | 2,303 → 103,233 | ||
| GPT-6 Astra | 0.825 → 0.855 (+0.030) | 726 → 10,114 | 1.000 → 1.000 (+0.000) | 278 → 1,349 | 1.000 → 1.000 (+0.000) | 98 → 850 | |||
| GPT-6.1 Sol | 0.818 → 0.848 (+0.030) | 486 → 8,908 | 0.967 → 1.000 (+0.033) | 367 → 1,626 | 1.000 → 1.000 (+0.000) | 120 → 913 | |||
| GPT-6.1 Terra | 0.823 → 0.841 (+0.018) | 620 → 28,790 | 0.912 → 1.000 (+0.088) | reliable | 490 → 5,627 | 0.950 → 0.980 (+0.030) | 319 → 17,566 | ||
| GPT-6 Luna | 0.775 → 0.802 (+0.027) | 1,383 → 18,067 | 0.883 → 0.869 (−0.014) | 1,925 → 19,068 | 0.950 → 0.930 (−0.020) | 479 → 10,449 | |||
| Gemini 4 Argon | 0.859 → 0.848 (−0.011) | 214 → 8,748 | 1.000 → 1.000 (+0.000) | 390 → 4,710 | 1.000 → 1.000 (+0.000) | 94 → 6,438 | |||
| Gemini 3.8 Flash | 0.868 → 0.873 (+0.005) | 3,900 → 39,236 | 0.938 → 1.000 (+0.062) | 1,859 → 13,361 | 0.950 → 0.950 (+0.000) | 2,828 → 11,189 | |||
| DeepSeek V4.1 Flash | 0.793 → 0.791 (−0.002) | 21,131 → 25,128 | 0.912 → 0.952 (+0.040) | 25,591 → 35,244 | 0.920 → 0.945 (+0.025) | 19,808 → 32,583 |
Item difficulty compared with humans
The original study's item-level data cover short forms of the STEM (18 items, 187 people) and the STEU (19 items, 185 people), each pooled from two of the original study's samples. Across these 37 items, the share of the original six models' answers that were correct correlated 0.71 with the share of people who answered correctly. For the frontier models the correlation was 0.61 at default and 0.51 at maximum effort. At maximum effort, 27 of the 37 items were answered correctly in every frontier run, so the correlation rests on the other 10 items, where it was 0.41. These data do not show whether the frontier models and people find different items hard.
Model judges against human similarity ratings
The paraphrase check below needs a rater of how similar two scenarios are. A scenario is an item's text without its answer options. To test language models as judges, each of the ten frontier models rated the 1,590 scenario pairs that human raters scored in the original study. These pair each of ChatGPT-4's 37 new scenarios with every original scenario of the same test (18 × 44 STEM and 19 × 42 STEU pairs). GPT-6.1 Sol and Claude Opus 5.5 were chosen as the two judges for the new items from these ratings. GPT-6.1 Sol had the second-highest correlation with the mean human rating, at about a fifth of the cost of the highest, GPT-6 Astra. Claude Opus 5.5 was paired with it so that the judges came from different developers. The pair's averaged rating correlated 0.75 with the mean human rating and flagged 6 of the 7 ChatGPT-4 scenarios that the human raters flagged, plus 2 that they did not.
Paraphrase check of the model-written items
Each frontier model wrote 18 new STEM and 19 new STEU items with the original study's item-writing procedure (see Method). The two judges rated every new scenario against every original scenario of the same test, 15,900 pairs in all. A new scenario is flagged when its highest rated similarity to any original is above 5 of 7, the original study's rule for a possible paraphrase.
Each flagged scenario was put in the first of these four categories that fits its most similar original. Near-verbatim copy: the text matches that original almost word for word (a character-level match of at least 75%). Concrete version of an abstract original: the original is one of the 14 STEU items that name no person. Close to a prompt example: the original is one of the five STEU items that the original study's third STEU item-writing prompt shows as examples. Same theme: every other flagged scenario. The concrete-version category is assigned from the original alone, and six of the 97 scenarios in it are not concrete versions of their original: five are reversed-format scenarios, which ask what happened and do not describe the situation, and one restates its original in abstract terms.
Twelve of the 14 abstract originals describe an appraisal pattern, the way of judging a situation that appraisal theory links to an emotion, such as "A pleasant experience ceases unexpectedly and there is not much that can be done about it." The other two only name an emotion, and no flagged scenario was most similar to either. A correct new item for the same emotion has to show the same pattern, so a concrete version of such an item is rated as similar to it. The original study's human raters flagged 2 of ChatGPT-4's STEU scenarios, and each was a concrete version of the abstract original item 24. Each of the five near-verbatim copies reused the scenario of one of the five prompt examples, item 20 four times and item 41 once. Eight more new STEU scenarios reused a prompt example's sentence frame with a different emotion, setting or both. Seven of them were not flagged, and one was flagged in the concrete-version category, because its most similar original was an abstract item.
A judge that favoured its own items would rate them as less similar to the originals than the other judge does. On the items Claude Opus 5.5 wrote, its highest similarity per scenario averaged 0.27 points below GPT-6.1 Sol's on the 1 to 7 scale. On the items of the eight models that were not judges, it averaged 0.18 below. The 0.09-point difference is not reliable (p = 0.927 after correction for the two judges' tests). GPT-6.1 Sol showed the opposite pattern: relative to Claude Opus 5.5, it rated its own items 0.09 points more similar to the originals than it rated the other models' items. The two differences are the same size by coincidence.
Matthew has been at his current job for six months. Something happened that caused him to feel regret.
Matthew has been at his current job for six months. Something happened that caused him to feel regret.
A concert that Lea has been enjoying is abruptly canceled because of a power failure, and there is no chance to resume it.
A pleasant experience ceases unexpectedly and there is not much that can be done about it.
Beth must explain a technical project to clients unfamiliar with her field. She is nervous that they will not understand her presentation, which currently contains many specialist terms.
Darla is nervous about presenting her work to a group of seniors who might not understand it, as they don't know much about her area.
Flag counts by item-writing model and category, both tests
| Test | Written by | Flagged | Near-verbatim copy | Close to a prompt example | Concrete version of an abstract original | Same theme |
|---|---|---|---|---|---|---|
| STEM | ChatGPT-4 (original study's items) | 5 of 18 | 0 | 0 | 0 | 5 |
| STEM | Claude Opus 5.5 | 1 of 18 | 0 | 0 | 0 | 1 |
| STEM | Claude Sonnet 5.5 | 0 of 18 | 0 | 0 | 0 | 0 |
| STEM | Claude Haiku 5.5 | 4 of 18 | 0 | 0 | 0 | 4 |
| STEM | GPT-6 Astra | 8 of 18 | 0 | 0 | 0 | 8 |
| STEM | GPT-6.1 Sol | 2 of 18 | 0 | 0 | 0 | 2 |
| STEM | GPT-6.1 Terra | 3 of 18 | 0 | 0 | 0 | 3 |
| STEM | GPT-6 Luna | 5 of 18 | 0 | 0 | 0 | 5 |
| STEM | Gemini 4 Argon | 2 of 18 | 0 | 0 | 0 | 2 |
| STEM | Gemini 3.8 Flash | 2 of 18 | 0 | 0 | 0 | 2 |
| STEM | DeepSeek V4.1 Flash | 7 of 18 | 0 | 0 | 0 | 7 |
| STEU | ChatGPT-4 (original study's items) | 3 of 19 | 0 | 0 | 3 | 0 |
| STEU | Claude Opus 5.5 | 14 of 19 | 0 | 1 | 12 | 1 |
| STEU | Claude Sonnet 5.5 | 9 of 19 | 0 | 0 | 6 | 3 |
| STEU | Claude Haiku 5.5 | 8 of 19 | 0 | 0 | 7 | 1 |
| STEU | GPT-6 Astra | 12 of 19 | 1 | 0 | 10 | 1 |
| STEU | GPT-6.1 Sol | 12 of 19 | 0 | 0 | 10 | 2 |
| STEU | GPT-6.1 Terra | 12 of 19 | 1 | 0 | 10 | 1 |
| STEU | GPT-6 Luna | 16 of 19 | 2 | 0 | 13 | 1 |
| STEU | Gemini 4 Argon | 14 of 19 | 0 | 0 | 11 | 3 |
| STEU | Gemini 3.8 Flash | 11 of 19 | 0 | 0 | 10 | 1 |
| STEU | DeepSeek V4.1 Flash | 11 of 19 | 1 | 0 | 8 | 2 |
Quality of the model-written items
In the original study, 90 to 94 online participants per test took both the original short form and ChatGPT-4's new version, and after each item rated how realistic and how clear it was on a 0 to 100 slider. No new people were tested. Instead, two model checks were first tested against those participants' data, with pass rules fixed before the first query, and only the checks that passed were applied to the frontier models' new items.
Realism and clarity ratings. Each frontier model, as a rater, rated the realism and clarity of ChatGPT-4's items and the original items 3 times, using the participants' two questions word for word. Each rater was left out in turn, and the average of the other nine raters was correlated with the participants' average rating. A rating passed if every one of these ten correlations was stronger than the correlation of the median participant's ratings with the other participants' average and had a 95% interval above 0. The table gives these correlations: the median participant's, the lowest of the nine-rater averages, and the ceiling, which is the highest correlation a rater can be expected to reach given how much the participants disagree. Seven of the 19 original items in the STEU short form were reworded for it, and the published data do not include their reworded answer options, so they were left out of this check.
| Test | Rating | Items | Median participant | Lowest 9-rater average (95% interval) | Ceiling | Result |
|---|---|---|---|---|---|---|
| STEU | realism | 31 | 0.46 | 0.59 (0.30 to 0.78) | 0.98 | passed |
| STEU | clarity | 31 | 0.46 | −0.09 (−0.43 to 0.27) | 0.98 | failed |
| STEM | realism | 36 | 0.34 | 0.47 (0.17 to 0.69) | 0.95 | passed |
| STEM | clarity | 36 | 0.32 | 0.72 (0.51 to 0.85) | 0.94 | passed |
On the STEU, the raters' clarity ratings did not correlate with the participants'. In an analysis that was not planned, they correlated instead with the share of participants who chose the keyed answer (0.57, against −0.06 for the participants' own clarity ratings). On ChatGPT-4's STEU items 7, 8 and 13, whose answer options the original study describes as similar to one another, the raters' mean clarity ratings were 63 to 68 of 100 and the participants' 89 to 91. Over the items used in the check, participants rated ChatGPT-4's STEU items +7.4 points clearer than the originals, and the raters +0.3. The raters may have read clarity as whether one answer is clearly best, and participants as whether the situation is clearly described.
Key check. Each frontier model also took ChatGPT-4's items as a test, 10 times. Together the models form the panel, which disputes an item's key when, counting each model equally, it preferred another option to the key. The original study found 5 of ChatGPT-4's 19 STEU items to perform poorly (items 4, 7, 8, 13 and 18). No participant chose the keyed answer of item 18. The check passed if the panel disputed the keys of at least 3 of these 5, including item 18, and of no more than 3 of the other 14, and if the panel's agreement with the key (the share of its answers that chose the key) correlated with the share of participants who chose it, with a 95% interval above 0. It passed on the STEU. The panel disputed the keys of items 7, 8 and 18 but not of items 4 and 13. On STEU item 18, every panel answer and 90 of the 94 participants chose "compassion" over the keyed "distress". The one other disputed key, on STEU item 10, was not a poorly performing item, but there too the panel chose the participants' most common answer, "shame", over the keyed "distress". The panel's agreement with the key correlated 0.52 (95% interval 0.03 to 0.80) with the share of participants who chose it. ChatGPT-4's STEM items include no poorly performing item, and the panel chose the key in every STEM answer, including on STEM item 18, which 71% of participants answered differently. The STEM key check therefore could not be tested and is not used.
The new items. The nine models that did not write an item set rated it 3 times and answered it as a test 3 times. Three panel runs were used because, in the key check, the first 3 runs disputed the same keys as all 10. In the table, the first two rows give mean ratings on the 0 to 100 scale from all ten models. The other rows give each model's difference from ChatGPT-4's items under the same nine raters, with its 95% interval, and the number of its STEU items whose key the panel disputed. The last row pools all frontier models against ChatGPT-4's items under all ten raters and the full panel, and the p values of its rating differences are corrected for the three comparisons. The STEM realism difference may overstate the gap participants would see: the raters rated ChatGPT-4's STEM items 4.0 points less realistic than the originals, while participants rated them 0.8 points more realistic.
| Written by | STEU realism | STEM realism | STEM clarity | Disputed STEU keys |
|---|---|---|---|---|
| ChatGPT-4 (original study's items) | 91.9 | 84.2 | 90.7 | 4 of 19 |
| Original short-form items | 86.0 | 88.2 | 84.0 | not checked |
| Claude Opus 5.5 | −3.0 (−5.8 to −0.3) | +8.4 (+5.4 to +11.4) | +2.3 (−0.1 to +4.6) | 2 of 19 |
| Claude Sonnet 5.5 | −1.2 (−3.8 to +1.3) | +8.6 (+5.5 to +11.7) | +2.7 (+0.3 to +5.1) | 0 of 19 |
| Claude Haiku 5.5 | −0.9 (−2.4 to +0.5) | +5.7 (+2.3 to +9.2) | +1.9 (−0.8 to +4.6) | 1 of 19 |
| GPT-6 Astra | −2.3 (−4.3 to −0.4) | +6.6 (+3.7 to +9.5) | +4.0 (+1.8 to +6.2) | 0 of 19 |
| GPT-6.1 Sol | −0.9 (−2.9 to +1.0) | +7.2 (+4.3 to +10.1) | +3.3 (+1.1 to +5.5) | 0 of 19 |
| GPT-6.1 Terra | −1.1 (−3.0 to +0.8) | +5.7 (+2.4 to +8.9) | +4.6 (+2.3 to +7.0) | 2 of 19 |
| GPT-6 Luna | −1.6 (−3.2 to +0.1) | +7.5 (+4.2 to +10.8) | +3.7 (+1.4 to +6.0) | 2 of 19 |
| Gemini 4 Argon | −3.9 (−6.9 to −1.0) | +3.9 (+0.8 to +7.0) | +2.6 (+0.2 to +4.9) | 1 of 19 |
| Gemini 3.8 Flash | −5.8 (−9.1 to −2.4) | +3.1 (−0.4 to +6.6) | +3.3 (+0.9 to +5.8) | 1 of 19 |
| DeepSeek V4.1 Flash | −1.5 (−4.4 to +1.3) | +6.1 (+3.0 to +9.3) | +4.2 (+1.7 to +6.6) | 1 of 19 |
| All frontier models | −2.2 (−3.6 to −0.9), p = 0.005 | +6.3 (+3.3 to +9.3), p = 0.001 | +3.3 (+1.0 to +5.5), p = 0.007 | 10 of 190 (p = 0.027) |
The 10 new STEU items whose key the panel disputed
| Written by | Item | Item text | Key | Panel chose | Panel agreement with key |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 10 | Ruben's roommate keeps eating Ruben's labeled food from the fridge, even after being asked to stop. The roommate is on the lease, and Ruben cannot move out for another year. Ruben is most likely to feel? | dislike | frustrated | 0.22 |
| Claude Opus 5.5 | 11 | Mei is an experienced baker who has made the same bread recipe hundreds of times. Today she follows it exactly, but the loaf collapses for no apparent reason. Mei is most likely to feel? | frustrated | confused | 0.33 |
| Claude Haiku 5.5 | 9 | Greg learns that the laboratory he trusted has been faking safety test results for years. Greg is most likely to feel? | horror | contempt | 0.11 |
| GPT-6.1 Terra | 3 | Ivara learns that a neighbor deliberately spreads unpleasant rumors about her. The neighbor is moving away, and there is little Ivara can do to repair the damage. Ivara is most likely to feel? | dislike | distress | 0.44 |
| GPT-6.1 Terra | 6 | A medical test may reveal that Varo has a serious illness. The results will not arrive for a week, and he cannot influence them. Varo is most likely to feel? | fear | anxiety | 0.11 |
| GPT-6 Luna | 2 | Someone believes a neighbor deliberately damaged their garden plants. The damage is done, and there is little they can do about it. They are most likely to feel? | dislike | anger | 0.30 |
| GPT-6 Luna | 3 | A medical test may reveal a serious illness tomorrow. There is nothing Mateo can do to affect the result. Mateo is most likely to feel? | scared | anxiety | 0.15 |
| Gemini 4 Argon | 12 | A landlord arbitrarily decides not to renew the lease for Nadia's apartment because he simply doesn't care for her, forcing her to move out with no legal recourse. Nadia is most likely to feel? | dislike | angry | 0.07 |
| Gemini 3.8 Flash | 9 | Claire is trapped in an unpleasant work environment with an unreasonable boss. Despite sending out resumes for a whole year, she has obtained zero interviews, and there seems to be no way out. Claire is most likely to feel? | distressed | depressed | 0.19 |
| DeepSeek V4.1 Flash | 11 | Daniel is waiting for the results of a biopsy. The doctor has said the lump could be harmless, but there is a real chance it is serious. Daniel can do nothing until the results arrive. Daniel is most likely to feel? | scared | anxiety | 0.00 |
Cost and score
The charts plot each score against what it cost at the providers' list prices: for the tests, the mean cost of one run of the whole test, and for the judges, the cost of 1,000 pair ratings. A model at one effort level is on the cost-score frontier when every other model and effort level either costs more or scores lower. The line through those points gives the highest score available at each cost. Hover over a point for its values.
Of the 12 points on the three tests' cost-score frontiers, all but one ran at default effort. The exception is Gemini 3.8 Flash on the STEM, where maximum effort added 0.005 to the mean score for 8 times the cost. One run of a test cost $0.00061 to $1.01. On the STEU, Gemini 4 Argon reached a mean of 1.000 at default effort for $0.0100 per run, and on GEMOK-Blends GPT-6.1 Sol did so for $0.0087 (Gemini 4 Argon did so for 0.2% more). The other models and effort levels with a mean of 1.000 cost up to $0.39 and $1.01 per run. Among the judges, GPT-6.1 Terra and GPT-6.1 Sol had higher correlations with the human ratings (0.73 and 0.74) than Claude Opus 5.5 (0.72), the other judge chosen, at a lower cost: $0.091 and $0.18 per 1,000 pair ratings against $0.63.
Each cost-score frontier is drawn through mean scores. Five of the 9 steps on the three tests' frontiers are smaller than the spread between runs (standard deviation) of one of the two points they join. The price rises announced for Gemini 4 Argon from 1 December 2026 and Gemini 3.8 Flash from 1 January 2027 leave the same points on every cost-score frontier.
Cost and score of every model and effort level
| Model | Effort | STEM cost per run | STEM score | STEU cost per run | STEU score | GEMOK-Blends cost per run | GEMOK-Blends score | Judging cost per 1,000 ratings | Judging correlation |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | default | $0.040 | 0.834 | $0.040 | 0.993 | $0.045 | 0.995 | $0.63 | 0.72 |
| Claude Opus 5.5 | max | $0.97 | 0.823 | $0.17 | 1.000 | $0.77 | 1.000 | ||
| Claude Sonnet 5.5 | default | $0.015 | 0.775 | $0.024 | 0.867 | $0.013 | 0.900 | $0.22 | 0.67 |
| Claude Sonnet 5.5 | max | $1.01 | 0.800 | $0.39 | 1.000 | $1.01 | 1.000 | ||
| Claude Haiku 5.5 | default | $0.0012 | 0.800 frontier | $0.0016 | 0.786 | $0.0017 | 0.905 | $0.026 | 0.60 |
| Claude Haiku 5.5 | max | $0.069 | 0.820 | $0.12 | 1.000 | $0.059 | 0.920 | ||
| GPT-6 Astra | default | $0.080 | 0.825 | $0.041 | 1.000 | $0.042 | 1.000 | $0.89 | 0.75 frontier |
| GPT-6 Astra | max | $0.55 | 0.855 | $0.094 | 1.000 | $0.080 | 1.000 | ||
| GPT-6.1 Sol | default | $0.014 | 0.818 | $0.0091 | 0.967 frontier | $0.0087 | 1.000 frontier | $0.18 | 0.74 frontier |
| GPT-6.1 Sol | max | $0.098 | 0.848 | $0.022 | 1.000 | $0.017 | 1.000 | ||
| GPT-6.1 Terra | default | $0.0074 | 0.823 frontier | $0.0051 | 0.912 frontier | $0.0053 | 0.950 | $0.091 | 0.73 frontier |
| GPT-6.1 Terra | max | $0.15 | 0.841 | $0.031 | 1.000 | $0.092 | 0.980 | ||
| GPT-6 Luna | default | $0.0011 | 0.775 frontier | $0.0012 | 0.883 frontier | $0.00061 | 0.950 frontier | $0.019 | 0.72 frontier |
| GPT-6 Luna | max | $0.0095 | 0.802 | $0.0098 | 0.869 | $0.0056 | 0.930 | ||
| Gemini 4 Argon | default | $0.012 | 0.859 frontier | $0.0100 | 1.000 frontier | $0.0087 | 1.000 | $0.18 | 0.71 |
| Gemini 4 Argon | max | $0.097 | 0.848 | $0.053 | 1.000 | $0.072 | 1.000 | ||
| Gemini 3.8 Flash | default | $0.018 | 0.868 frontier | $0.0092 | 0.938 | $0.013 | 0.950 | $0.37 | 0.71 |
| Gemini 3.8 Flash | max | $0.15 | 0.873 frontier | $0.052 | 1.000 | $0.045 | 0.950 | ||
| DeepSeek V4.1 Flash | default | $0.0066 | 0.793 | $0.0078 | 0.912 | $0.0062 | 0.920 | $0.080 | 0.70 |
| DeepSeek V4.1 Flash | max | $0.0078 | 0.791 | $0.011 | 0.952 | $0.0100 | 0.945 |
Limitations
- Both the original six models and the frontier models could have seen the tests in training. The full STEU and STEM items and answer keys have been public since 2008, and the GEMOK-Blends items and key since 2017. The comparison between the two groups of models is therefore like for like with respect to the test items, but neither set of scores separates understanding from memorisation. The frontier models could also have seen material published after the original six models were tested: the original study, its supplementary material on specific items and its item-level data. Perfect STEU scores fit memorisation but do not prove it, because, as the original study describes it, the STEU key follows from appraisal theory, which a model could apply to unseen items. No memorisation test was run.
- The human norm samples are small and old. The STEM and STEU norms come from psychology undergraduates in Australia tested for the 2008 publication, and the GEMOK-Blends norm from 180 Mechanical Turk workers. The item-level human data come from the original study's online samples, who took the short forms, with different wording for seven of the 19 STEU short-form items.
- Procedures differ from the original study. The frontier models received each whole test in one message through the providers' APIs. The original six models received a test split across several messages when it exceeded a character limit. DeepSeek V4.1 Flash, whose weights are public, was served by a third-party host (DeepInfra, through the OpenRouter service) at the precision of the released weights. Its scores may differ from those of DeepSeek's own service.
- Three of the original study's five tests were run. The emotion regulation and emotion management subtests of the Geneva Emotional Competence Test (GECo) were not run, because their items are available only under a signed user agreement with the University of Geneva. GEMOK-Blends was used only for scoring, not for item writing or the paraphrase check.
- Some choices were made after seeing data. The two judges were chosen after the models had been tested as judges against the human similarity ratings, and the four categories of flagged scenarios were defined after reading the flagged scenarios.
- No new human data were collected. The model checks of the new items cover realism on both tests, clarity on the STEM and keys on the STEU, but not whether the new tests give consistent scores or measure the same ability in people as the originals. A panel of current models may also share the frontier models' view of a contested option more than ChatGPT-4's, which would lower the dispute rate for their STEU keys without the keys being better.
Method
How the tests were run, scored and compared
Materials. The STEM and STEU were taken from the APA online supplement to MacCann and Roberts (2008). For the STEU, the keyed answers are the options marked correct there. For the STEM, they are the options most experts chose, with three ties broken by the higher mean expert rating. Both keys matched the original study's scoring files for all 37 short-form items. GEMOK-Blends was taken from the supplement to Schlegel and Scherer (2018), where the keyed pair is underlined, and was given in the original study's item order. Its key matched the original study's scoring file for all 20 items, and scoring the original study's human responses with it reproduced their published mean (0.68).
Runs. Each run was one new conversation with one message: the test's own instructions, every item with lettered options, and a request to answer with one line per item. The output limit per response was 128,000 tokens for the Claude models, their maximum, and 393,216 for DeepSeek V4.1 Flash, DeepSeek's documented maximum. The OpenAI models were allowed 128,000 tokens on the STEM and STEU and were sent no limit on GEMOK-Blends. Gemini 4 Argon was allowed 32,768 tokens on the STEM and STEU and 1,000,000 on GEMOK-Blends, and Gemini 3.8 Flash 1,000,000 on all tests. No OpenAI or Gemini 4 Argon response came near its limit. A model that reached its limit before answering was asked to continue in the same conversation, up to five times. One continuation was needed in 13 runs, all from Claude Haiku 5.5 or Sonnet 5.5 at maximum effort. Only connection errors before any output were retried. Except for those of Gemini 3.8 Flash and DeepSeek V4.1 Flash, the STEM and STEU runs were begun under an earlier protocol, which restarted a run whose response hit the output limit. Under that protocol, responses hit the limit in three model, effort and test combinations. These were rerun in full, and the earlier runs are excluded. In the other combinations no response reached a limit: their 287 runs under the earlier protocol were kept, and their remaining 3 runs were made under the later protocol.
Comparisons. The effect of effort averaged over the 10 models was tested with an exact sign-flip test. On each of the STEM and STEU, the frontier models' flag count was compared with ChatGPT-4's under the same judges with Fisher's exact test. The pooled realism and clarity ratings were compared with ChatGPT-4's items with Welch's t-test, items as units, and the disputed STEU keys with Fisher's exact test. All tests were two-sided at 0.05 and Holm-corrected within each family of comparisons.
Item writing. Each model's item-writing conversation continued from its first default-effort run of the test. The model was told the correct answers to the items it had missed and then given the original study's item-writing prompts, word for word except that "revise items 58-61" (ChatGPT-4's own numbering) became "revise the 4 reversed-format items". Claude Sonnet 5.5 extracted the items, and every extracted scenario was checked to appear word for word in the model's response. Item writing, similarity judging, the realism and clarity ratings and the panel all used default effort.
API cost by stage: $119.04 logged
Costs are API spend from the run records and, for item writing, the run logs, at the providers' standard list prices. For DeepSeek V4.1 Flash the price is DeepInfra's on OpenRouter.
| Stage | Scope | Cost |
|---|---|---|
| Test runs used in the analysis | 600 runs, 10 models | $67.24 |
| Test runs under the earlier protocol (excluded) | 3 model, effort and test combinations, rerun | $10.16 |
| Pilot runs to check requests and estimate cost | first 3 items of the STEM and STEU, 8 models | $0.23 |
| Checking that DeepSeek V4.1 Flash's host applied the effort setting | 23 calls | $0.09 |
| Testing the judges against human ratings | 10 judges × 1,590 pairs × 3 | $12.81 |
| Item writing and extraction | 10 models × 2 tests | $2.91 |
| Judges rating the new items | 2 judges × 15,900 pairs | $15.01 |
| Testing the raters and the panel against participants' data | 120 rating calls, 200 panel runs | $1.99 |
| Rating and answering the new items | 540 rating calls, 540 panel runs | $8.60 |
| Total logged | $119.04 | |
| Calls interrupted when runs were restarted | not logged | up to ~$5.00 |