GPT-6 Sol vs Claude Fable 5.1

Provider model evaluated with adaptive thinking at max effort; average over five trials. This is not the catalog default medium-effort configuration.

Provider model evaluated with adaptive thinking at max effort; average over five trials. This is not the catalog default medium-effort configuration.

Provider model evaluated with adaptive thinking at max effort; average over five trials. This is not the catalog default medium-effort configuration.

Provider model evaluated with adaptive thinking at max effort; average over five trials. This is not the catalog default medium-effort configuration.

System card section 8.8 reports Cursor’s production agent harness at max effort; historical 3.2.0 test, not CursorBench 4.0.

Section 8.11.1: 166 filtered tasks; hidden test pass rate, mini-swe-agent without the six-hour time limit; max effort. Not interchangeable with the unfiltered public evaluation.

Published current mean reward 0.5653 over 170 runs, displayed as a percentage; not the older supplied 56.3 result. Effort is not established by this leaderboard.

Vals evaluation: max compute effort, temperature 1, maximum output 128,000; reported uncertainty ±0.86 percentage points. Not the default medium version.

Cursor’s live table explicitly lists Fable 5.1 Max at 51.8; Extra High is a separate 51.6 result. Max must not be equated with the catalog xhigh variant.

Row explicitly labeled 'Fable 5.1 Extra High' (xhigh) at 51.6%, $13.01/task, confirmed directly on Cursor's live table. Distinct from the already-recorded 51.8% 'Fable 5.1 Max' row and the 49.2% 'Fable 5.1 High' row on the same leaderboard — effort levels are not interchangeable.

Section 8.16: ARC Prize verified semi-private dataset, max effort. Not the medium-effort default catalog version.

Section 8.16: ARC Prize verified semi-private dataset, max effort. Not the medium-effort default catalog version.

Section 8.18.1: 42 languages, adaptive thinking max effort, single trial, no tools or custom system prompt. Not the medium-effort default catalog version.

Section 8.18.2: 11 languages, adaptive thinking max effort, single trial. Not the medium-effort default catalog version.

Section 8.17: max effort, five trials, Opus 4.8 grader; safety classifiers and Opus 5 refusal fallback. Raw score, not length adjusted. Not the medium-effort default catalog version.

Section 8.17: same max-effort evaluation, length-adjusted score. Not the medium-effort default catalog version.

Section 8.17: five trials, max effort, Opus 4.8 grader, safety classifiers and Opus 5 fallback. Raw score. Not the medium-effort default catalog version.

Section 8.17: same max-effort evaluation, length-adjusted score. Not the medium-effort default catalog version.

Section 8.14.2: random 1,000-file subset, average five runs, adaptive thinking max effort, no tools. Not the medium-effort default catalog version.

Section 8.14.2: random 1,000-file subset, average five runs, adaptive thinking max effort, container and image cropping tool. Not the medium-effort default catalog version.

Section 8.15.3: AA independent agentic professional work evaluation, max effort. Historical v2, not v2.1. Not the medium-effort default catalog version.

Section 8.15.4: independent AA long-horizon work evaluation, max effort; historical system-card snapshot. Not the medium-effort default catalog version.

Cognition FrontierCode 1.1; Claude Fable 5.1; medium effort; claude-code harness; Extended subset (150 tasks). Published weighted rubric score (new_score × 100), not the all-or-nothing pass rate; solution-source violations score zero. Source field new_score=0.636. Publisher evaluation, not a TokenRex measurement; provider effort configuration is not a guarantee of the API's fixed thinking budget.

Cognition FrontierCode 1.1; Claude Fable 5.1; medium effort; claude-code harness; Main subset (100 tasks). Published weighted rubric score (new_score × 100), not the all-or-nothing pass rate; solution-source violations score zero. Source field new_score=0.5091. Publisher evaluation, not a TokenRex measurement; provider effort configuration is not a guarantee of the API's fixed thinking budget.

Section 8.6: Claude Code --bare, maximum thinking effort; 15 trials/task (990 trials), SE ±1.6–2 points. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Section 8.7: Claude Code --bare, maximum thinking effort; 10 trials/task (700 trials), SE ±3.5–4.5 points. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Table 8.1.A and section 8.12.1: web search, web fetch and code execution; thinking auto, 1M token cap, Opus 4.6 grader and source blocklist. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Table 8.1.A: no tools; standard adaptive thinking at max effort, default sampling, five trials. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Table 8.1.A: adaptive thinking at max effort, default sampling, five trials; production safeguards enabled. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Section 8.14.1: adaptive thinking at max effort; container and image-cropping tool, five runs. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Section 8.14.1: adaptive thinking at max effort, no tools, five runs. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Section 8.14.3: August 2026 tasks with subsequent fixes; 1080p, maximum 500 action steps, maximum reasoning effort, five runs, Opus 4.8 grader. Not comparable to OSWorld 2.1 or earlier task releases. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.

Section 8.14.3: August 2026 tasks with subsequent fixes; 1080p, maximum 500 action steps, maximum reasoning effort, five runs, Opus 4.8 grader. Not comparable to OSWorld 2.1 or earlier task releases. Published Fable 5.1 evaluation, not the default medium catalog configuration or a TokenRex run; benchmark tools are not hosted API capabilities.