The AI Consumer Index (ACE) assesses whether frontier AI models can perform everyday consumer tasks in shopping, food, gaming, and DIY.

OpenAI

Google

Anthropic

We created ACE because consumer applications are among the most widespread and fastest-growing uses of AI—yet existing benchmarks have paid too little attention. ACE evaluates whether AI models can help people with everyday activities such as finding products, enjoying their hobbies, fixing items at home, and cooking. It comprises a hidden held-out set of 400 prompts.

ACE tests frontier models with web search enabled, mimicking real-world consumer behavior. It uses a novel evaluation methodology that combines expert-crafted rubrics with hurdle criteria and grounding checks. Read the paper to learn more about our approach.

To support open research, we have open-sourced 80 in-distribution cases on Hugging Face, including all metadata labels. We have also released our evaluation harness for full reproducibility.

The AI Consumer Index

ACE ranks AI models on everyday consumer tasks — shopping, food, gaming, and DIY. See which AI models perform best on tasks real people use daily.

AI Consumer Index: Best AI Models for Everyday Tasks | Mercor

Measuring AI agents ability to complete professional accounting tasks across tools like accounting software, spreadsheets, and PDFs.

Zhipu

Kimi

Alibaba

We created APEX-Accounting to evaluate agents on the real day-to-day work of accountants closing the books each month. The tasks require agents to reason over incomplete records, apply company-specific context, use multiple applications, and carry figures accurately through a close.

APEX-Accounting was built in three steps. First, accountants created 10 data-rich worlds, each a business frozen at month-end close. Second, they created challenging period-close tasks using the files from within each world, then solved them to produce the rubrics. Third, we gave agents access to execute the tasks with the software a human accountant would use.

There are 10 worlds in APEX-Accounting, comprising 160 tasks and grading rubrics. An eleventh world is open-source, along with Archipelago, our infra service for executing and evaluating agent trajectories.


The AI Productivity Index for Accounting (APEX-Accounting) measures whether frontier AI agents can execute long-horizon, cross-application tasks in professional accounting.

The AI Productivity Index for Accounting

Tests whether AI agents can complete multi-hour professional tasks across investment banking, corporate law, and management consulting, using real tools in Google Suite.

Applied Compute

MiniMax

DeepSeek

NVIDIA

We created APEX-Agents to evaluate agents on the real day-to-day work of professionals: investment banking analysts, management consultants, and corporate lawyers. The tasks require agents to reason, demonstrate advanced knowledge, use multiple applications, and plan over long horizons.

APEX-Agents was built in three steps. First, industry professionals created a data-rich world over 5-10 days, based on a unique project scenario. Second, they created realistic, challenging tasks using the files from within the world. Third, we gave agents access so they could execute the tasks (with all of the software that a human would use).

There are 33 worlds in APEX-agents, comprising 480 tasks and grading rubrics. The entire APEX-agents dataset is available open-source, along with Archipelago, our infra service for executing and evaluating agent trajectories.

The AI Productivity Index for Agents (APEX-Agents) measures whether frontier AI agents can execute long-horizon, cross-application tasks across three jobs in professional services.

The AI Productivity Index for Agents

Explore APEX-Agents benchmarks and leaderboards for AI productivity. See rankings for agentic performance on long-horizon, multi-tool professional services tasks.

AI Agent Benchmarks & Leaderboards: APEX-Agents | Mercor

Measures AI performance on real-world software engineering tasks, from bug fixes to feature builds. Built in collaboration with Cognition.

Cognition

The AI Productivity Index for Software Engineers (APEX-SWE) measures whether frontier AI agents can execute high-value engineering work, split across Observability and Integration tasks. Created by Mercor in collaboration with Cognition.

APEX–SWE Integration (n = 100) evaluates a model’s ability to orchestrate end-to-end workflows and synchronize data across heterogeneous services. Models are required to write application code, configure infrastructure, and deploy functioning services. The test stack includes cloud primitives (AWS LocalStack: S3, Lambda, DynamoDB, Kinesis) and production-grade business applications (EspoCRM, Medusa, Zammad, Plane).

APEX–SWE Observability (n = 100) evaluates a model’s ability to diagnose and remediate real-world production failures. Unlike conventional bug-fix benchmarks, models are not provided with failing unit tests. Instead, they must interrogate production logs (via Grafana/Loki), correlate evidence from developer chat discussions, and trace root causes through the codebase.

The AI Productivity Index for Software Engineers (APEX-SWE) measures whether frontier AI agents can execute high-value engineering work, split across Observability and Integration tasks. Created by Mercor in collaboration with Cognition.

The AI Productivity Index for SWEs

Benchmark LLMs & AI coding agents performance on software engineering work with Mercor's SWE leaderboard. Evaluate coding, debugging, system integration, etc.

LLM Coding Benchmarks & SWE Leaderboards: APEX-SWE | Mercor

Tests whether frontier models can perform economically valuable tasks across professional domains including investment banking, corporate law, management consulting, and medicine.

We created APEX to bridge the gap between what professionals want from AI systems and what benchmarks test for. The prompts are realistic, challenging and diverse, and are provided with source documents. Each case was created by a veteran industry expert to capture their day-to-day work.

The leaderboard is based on a hidden heldout set of 400 tasks (n=100 per job). For each task we collect responses from each model 8 times. We grade them using a Judge LM and report the mean value.

To support open research, we have open-sourced n=100 cases that are in-distribution of APEX on Hugging Face, and our eval harness.

The AI Productivity Index (APEX) assesses whether frontier models are capable of performing economically valuable tasks across four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD).

The AI Productivity Index

Evaluate frontier LLMs on our APEX AI model benchmarks & leaderboards. Explore single-turn productivity & rankings across real-world professional tasks.

LLM Leaderboard & Frontier AI Model Benchmarks: APEX | Mercor

Evaluate frontier AI models with Mercor's APEX AI Benchmarks & Leaderboards. Measure AI productivity across software engineering, law, consulting, medicine, etc.

APEX Benchmarks: AI Productivity Index Leaderboards | Mercor

Introducing the AI Productivity Index for Accounting

AI can't read an investor deck

Generalization results from training on the APEX-Agents dev set

https://huggingface.co/datasets/mercor/apex-agents

Hugging Face

https://github.com/Mercor-Intelligence/archipelago

Archipelago (GitHub)

APEX-Agents

https://huggingface.co/datasets/mercor/APEX-SWE

https://github.com/Mercor-Intelligence/apex-swe

GitHub

APEX-SWE

https://huggingface.co/datasets/Mercor/APEX-v1

https://github.com/Mercor-Intelligence/apex-evals/tree/main/apex-evals-v1-extended

APEX

The APEX family of benchmarks assesses whether frontier AI models can perform economically valuable tasks across professional services, medicine, software engineering, and consumer activities.

APEX Benchmarks

Our benchmarks

APEX-Agents

APEX-Accounting

APEX-SWE

APEX

Read the blog

Introducing the AI Productivity Index for Accounting

AI can't read an investor deck

Generalization results from training on the APEX-Agents dev set

APEX Benchmarks

Our benchmarks

APEX-Agents

APEX-Accounting

APEX-SWE

APEX

Read the blog

Introducing the AI Productivity Index for Accounting

AI can't read an investor deck

Generalization results from training on the APEX-Agents dev set

The latest on frontier AI performance, straight to your inbox.