The AI Productivity Index for SWEs
The AI Productivity Index for Software Engineers (APEX-SWE) measures whether frontier AI systems can execute economically valuable software engineering work. It covers Integration and Observability tasks.
The APEX-SWE leaderboard
We created APEX-SWE to evaluate the real day-to-day work of software engineers, unlike unit-level and single-repository bug-fix benchmarks. It comprises n=200 cases and spans two complementary settings that mirror professional SWE work: (1) Integration tasks that require end-to-end system construction and deployment across heterogeneous services and (2) Observability tasks that require debugging with production-style telemetry.
Each task includes a human-authored rubric that grades agent outputs for functional requirements, robustness, and code style, alongside unit tests.
To support open research, we have open-sourced n=50 cases that are in-distribution of APEX-SWE on Hugging Face with all metadata labels. We have also shared our eval harness for reproducibility.
Evaluates a model's ability to orchestrate end-to-end workflows and synchronize data across heterogeneous services.
View more
Grok 4.5Grok 4.5
65.0%
65.0%
GLM-5.2 (Thinking)GLM-5.2 (Thinking)
55.3%
55.3%
Sonnet 5 (High)Sonnet 5 (High)
54.3%
54.3%
Evaluates a model's ability to diagnose and remediate real-world software engineering production failures.
View more
Opus 4.8 (High)Opus 4.8 (High)
43.3%
43.3%
Grok 4.5Grok 4.5
37.3%
37.3%
GPT 5.3 Codex (High)GPT 5.3 Codex (High)
33.3%
33.3%