The ACE leaderboard shows the ranking of frontier models measured against the hidden held out eval set for ACE. Scores are models' average across the four activities. 

We created ACE because consumer applications are among the most widespread and fastest-growing uses of AI—yet existing benchmarks have paid too little attention. ACE evaluates whether AI models can help people with everyday activities such as finding products, enjoying their hobbies, fixing items at home, and cooking. It comprises a hidden held-out set of 400 prompts.

ACE tests frontier models with web search enabled, mimicking real-world consumer behavior. It uses a novel evaluation methodology that combines expert-crafted rubrics with hurdle criteria and grounding checks. Read the paper to learn more about our approach.

To support open research, we have open-sourced 80 in-distribution cases on Hugging Face, including all metadata labels. We have also released our evaluation harness for full reproducibility.

The AI Consumer Index (ACE) assesses whether frontier AI models can perform everyday consumer tasks in shopping, food, gaming, and DIY.

The AI Consumer Index

ACE leaderboard presents the global ranking of AI models based on the consumer index and their averaged results across all task domains.

ACE: AI Consumer Index

APEX

The APEX leaderboard shows the ranking of frontier models measured against the hidden heldout eval set for APEX. Scores are models' average across the four jobs.

We created APEX to bridge the gap between what professionals want from AI systems and what benchmarks test for. The prompts are realistic, challenging and diverse, and are provided with source documents. Each case was created by a veteran industry expert to capture their day-to-day work.

The leaderboard is based on a hidden heldout set of 400 tasks (n=100 per job). For each task we collect responses from each model 8 times. We grade them using a Judge LM and report the mean value.

To support open research, we have open-sourced n=100 cases that are in-distribution of APEX on Hugging Face. We have also shared our eval harness for reproducibility

APEX-v1

The AI Productivity Index (APEX) assesses whether frontier models are capable of performing economically valuable tasks across four jobs: investment banking associate, management consultant, big law associate, and primary care physician (MD).

The AI Productivity Index

APEX leaderboard presents the global ranking of AI models based on their averaged results across all task domains, showing which models are best at performing economically valuable professional work.