The AI Consumer Index (ACE) assesses whether frontier AI models can perform everyday consumer tasks in shopping, food, gaming, and DIY.

We created ACE because consumer applications are among the most widespread and fastest-growing uses of AI—yet existing benchmarks have paid too little attention. ACE evaluates whether AI models can help people with everyday activities such as finding products, enjoying their hobbies, fixing items at home, and cooking. It comprises a hidden held-out set of 400 prompts.

ACE tests frontier models with web search enabled, mimicking real-world consumer behavior. It uses a novel evaluation methodology that combines expert-crafted rubrics with hurdle criteria and grounding checks. Read the paper to learn more about our approach.

To support open research, we have open-sourced 80 in-distribution cases on Hugging Face, including all metadata labels. We have also released our evaluation harness for full reproducibility.

The AI Consumer Index

ACE leaderboard presents the global ranking of AI models based on the consumer index and their averaged results across all task domains.

ACE: AI Consumer Index

APEX-Agents

The AI Productivity Index for Agents (APEX-Agents) measures whether frontier AI agents can execute long-horizon, cross-application tasks across three jobs in professional services.

We created APEX-Agents to evaluate agents at executing the real day-to-day work of professionals: investment banking analysts, management consultants, and corporate lawyers. The tasks require agents to reason, demonstrate advanced knowledge, use multiple applications, and plan over long horizons.

APEX-Agents was built in three steps. First, industry professionals created a data-rich world over 5-10 days, based on a unique project scenario. Second, they created realistic, challenging tasks using the files from within the world. Third, we gave agents access so they could execute the tasks (with all of the data and software that a human would use).

There are 33 worlds in APEX-agents, comprising 480 tasks and grading rubrics. The entire APEX-agents dataset is available open-source, along with Archipelago, our infra service for executing and evaluating agent trajectories.

The AI Productivity Index for Agents

APEX-Agents leaderboard presents the global ranking of AI models based on their performance at agent tasks.

APEX-Agents: AI Productivity Index for Agents

APEX

The AI Productivity Index (APEX) assesses whether frontier models are capable of performing economically valuable tasks across four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD).

We created APEX to bridge the gap between what professionals want from AI systems and what benchmarks test for. The prompts are realistic, challenging and diverse, and are provided with source documents. Each case was created by a veteran industry expert to capture their day-to-day work.

The leaderboard is based on a hidden heldout set of 400 tasks (n=100 per job). For each task we collect responses from each model 8 times. We grade them using a Judge LM and report the mean value.

To support open research, we have open-sourced n=100 cases that are in-distribution of APEX on Hugging Face, and our eval harness.

APEX-v1

APEX leaderboard presents the global ranking of AI models based on their averaged results across all task domains, showing which models are best at performing economically valuable professional work.

APEX Benchmarks

APEX-Agents

APEX

ACE