Today, we’re introducing <a href="https://www.mercor.com/apex/apex-agents-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX-Agents</a>, our new benchmark designed to test how well AI agents complete real, long-horizon tasks in investment banking, consulting, and corporate law.
It’s the latest addition to the AI Productivity Index (APEX), Mercor’s family of benchmarks that measure economically valuable capabilities, joining <a href="https://www.mercor.com/apex/apex-v1-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">APEX</a> for tool-free evals and <a href="https://www.mercor.com/apex/ace-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">ACE</a> for consumer applications.
With APEX-Agents, we set out to find if today’s AI agents do economically valuable tasks. Are they ready to work with teams, in real software tools, and deliver client-ready work?
Workplace context is messy, incomplete, and spread across documents and chat threads. Tasks take hours, not seconds.
Most existing benchmarks don’t reflect that. They evaluate models on isolated prompts or narrow skills. They don’t measure whether an agent can navigate multiple workflows and produce something a manager or client would accept.
That is why we approached APEX-Agents differently. Every task is designed to mirror the complex work professionals do, the tasks they wish an AI agent could help with, and identify where current models struggle to perform reliably.
<iframe src="https://player.vimeo.com/video/1156906982" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="Mercor APEX-Agents" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
<h2>Our approach</h2>
APEX-Agents simulates the demands of the profession to challenge agents to navigate instructions, manage complicated file systems, and produce outputs that justify a professional fee.
Our four-step approach:
Surveys: We started out by surveying hundreds of experts from professional services including Goldman Sachs, McKinsey, and Cravath to understand how they spend their time.
Scenarios: A team of Mercor experts—Vice Presidents, Managing Directors, and Managers with five-to-ten years’ experience at top-tier firms—worked in Google Workspace, simulating how coworkers would collaborate on a project.
This might look like a week-long consulting project for a fictitious European oil &amp; gas company focused on cost-cutting measures.
We worked with <a href="https://www.box.com/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Box</a> to define what a real-world file system looks like. We mapped rigorous, domain-specific challenges to the complex file structures found in a mix of industries, resulting in datarooms that truly simulate the daily workflow of a professional.
The result is a high-context workspace that mimics what professionals navigate every day.
Task creation: Using these custom scenarios, the experts defined specific tasks and the exact grading criteria that define what “client-ready” means. Each task includes 1–10 pass/fail criteria.
<a href="https://www.harvey.ai/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Harvey AI</a> provided early feedback on the design, scope, and realism of the corporate law worlds. They confirmed the complexity and value of the tasks in APEX-Agents, validating that they reflect the work of top lawyers at Fortune 500 enterprises and law firms.
Evals: We deployed AI agents inside these worlds using our open source evaluation infrastructure, Archipelago, to measure whether they finish the work correctly.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/7bc182cda45d843f72fa787ae2fc5e49c2bcbd63-2756x1230.png" alt="APEX-Agents" style="width: 100%; border-radius: 10px;" />
<h2>The findings</h2>
Frontier models successfully complete less than 25% of tasks that would typically take professionals hours. With multiple attempts, performance improves but the gap remains large. Even with 8 tries, the best agents can only complete 40% of the tasks.
Many agents fail due to lack of capability, but because they can’t manage ambiguity, find the right file, or hold context across the entire workflow.
No model is ready to replace a professional end-to-end.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/f0ac6afea54245c059dc1cacb8fbe316c936e850-3840x2142.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Open source</h2>
We have released the entire benchmark open source via <a href="http://huggingface.co/datasets/mercor/apex-agents" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> with a CC-BY license and Archipelago, our infra, eval service, and set of apps, is available as an open source repo on <a href="https://github.com/Mercor-Intelligence/archipelago" target="_blank" rel="noopener noreferrer" class="text-indigo-600">GitHub</a>.
Read the full research paper on <a href="https://arxiv.org/pdf/2601.14242" target="_blank" rel="noopener noreferrer" class="text-indigo-600">arXiv</a>.

Bertie Vidgen

Researcher

Osvald Nitski

Product Manager

Mercor APEX-Agents is a benchmark designed to test how well AI agents complete real, long-horizon tasks in investment banking, consulting, and corporate law.

Introducing APEX-Agents

We created the first version of the AI Productivity Index (APEX) to assess whether frontier models can perform high-value work in Investment banking, Management consulting, Law, and Medicine. We found that even the very best models struggle on complex real-world tasks, failing to meet the production bar.
Mercor has now doubled the size of the heldout evaluation set in APEX from n=200 to n=400. This larger eval set allows us to more consistently evaluate frontier models’ ability to perform tasks that create economic value. The design of the cases remains the same (comprising prompts and source documents, and a grading rubric) but we have increased their complexity and variety. On average, tasks in APEX take over two-and-half hours for seasoned professionals to complete in the real-world. The contributors to APEX typically had over 7 years of experience.
We have also refined the evaluation methodology in APEX, increasing the number of runs we execute to 8, simplifying the grading process, and adding confidence intervals to our results. Read the <a href="https://arxiv.org/abs/2509.25721" target="_blank" rel="noopener noreferrer" class="text-indigo-600">technical report</a> to find out more about our approach to evaluation and how the dataset was created.
<h2>Results</h2>
The best performing model (GPT 5, Thinking = High) has the highest mean score at 67%, followed by Gemini 3 Pro (Thinking = High) at 64.3% and Grok 4 at 63.5%. Models&#39; scores are lowest on Investment banking, with the highest scoring model achieving 63.0%, followed by Management consulting (top score = 64%), Medicine (top score = 65.5%), and, with substantially higher scores, Law (top score = 77.9%). APEX shows that models struggle with real-world tasks that professionals undertake everyday. The new leaderboard is available <a href="https://www.mercor.com/apex/apex-v1-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">here</a>.
Gemini 3 Pro (Thinking = High), released in November 2025, is a substantial improvement on 2.5, beating it out by over 5 percentage points overall, and on Investment banking by 9 percentage points. Similarly, we see Opus 4.5 (Thinking = On), also released in November, beating Sonnet 4.5 and Opus 4.1 by 6 percentage points and 12 percentage points respectively. These substantial improvements speak to a meaningful step forward in models’ ability to perform high-value tasks. GPT 5.1 is an exception and does not improve on GPT 5 (which remains the leaderboard champion), but this is perhaps not surprising given that 5.1 is primarily meant to be more conversational and explanative, rather than better at complex reasoning. Finally, given it is the only model without an explicit reasoning setting, Grok 4, is remarkably strong and comes 4th overall.
If you want to add your model to the leaderboard or run a loss analysis, contact the Mercor Applied AI Research team at apex@mercor.com.
<h2>Open source eval set</h2>
We are releasing n=100 open-source cases (APEX-v1-devset) on <a href="https://huggingface.co/datasets/mercor/APEX-v1-extended" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a> with a CC-BY licence for anyone to train on, evaluate against, and research. The open-source cases were created by the same annotators through the same production pipeline. We are also open-sourcing our <a href="https://github.com/Mercor-Intelligence/apex-evals/tree/main/apex-oss-eval" target="_blank" rel="noopener noreferrer" class="text-indigo-600">evaluation harness</a> so you can exactly match our grading approach. We are looking forward to seeing what the community builds and would love to hear from any researchers using our data.

Doubling the size of the AI Productivity Index to better measure AI models’ economic value.


Expanding the Mercor AI Productivity Index

Expanding the AI Productivity Index (APEX)

Today, we&#39;re releasing our first version of the AI Consumer Index (ACE).
ACE tests what people actually want AI to do—from finding a gift for a friend to getting a custom recipe recommendation or fixing a hole in their drywall.
ACE contains realistic and challenging evals, split across shopping, food, gaming and DIY, created by experts from the Mercor platform. We are excited to share our new <a href="https://www.mercor.com/apex/ace-leaderboard/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">leaderboard</a>, <a href="https://arxiv.org/abs/2512.04921" target="_blank" rel="noopener noreferrer" class="text-indigo-600">technical report</a>, <a href="https://huggingface.co/datasets/mercor/ACE" target="_blank" rel="noopener noreferrer" class="text-indigo-600">open source dataset</a>, and <a href="https://github.com/Mercor-Intelligence/apex-evals" target="_blank" rel="noopener noreferrer" class="text-indigo-600">eval harness</a>.
<h2>Results</h2>
We show that models routinely fail on consumer tasks—GPT 5.1 (Thinking = High) is the top model but scores only 56.1%. The next best models are GPT 5 (Thinking = High) and o3 Pro (Thinking = On). The best performing model from Google is Gemini 3 Pro (Thinking = High) and from Anthropic is Opus 4.5 (Thinking = On), showing how the latest model releases are steadily improving at consumer tasks.
We see substantial differences in model performance across the four domains. No models score over 50% on Shopping tasks, an opportunity worth $5+ trillion globally – but models perform much better in Food. GPT 5 (Thinking = High) hits an impressive 70%, beating out the next best model by 10 percentage points. In DIY and Gaming we see the biggest range, with model scores ranging from 28% to 61%. Beyond these high-level stats we see a lot of tasks where models fail, often scoring under 25%.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/e93be49858bb82d553eac79045d8872cd61082ca-1974x2074.png" alt="Mercor AI Consumer Index (ACE)" style="width: 100%; border-radius: 10px;" />
<h2>Grading approach</h2>
We introduce a novel rubric-based evaluation methodology with ACE.
Each task has a rubric of prompt-specific criteria to scalably grade model responses. Every rubric contains at least one hurdle criterion that must be passed before further rewards can be unlocked. The hurdles require the model to meet the user’s core objective -- such as, in Shopping, returning the requested type of product or, in DIY, providing a solution to the user&#39;s problem. The hurdles are important for minimizing the risk of reward hacking. We do not want to reward responses that are mostly irrelevant but still meet a specific requirement (e.g., the response returns any item under $50).
We also add grounding criteria that penalize models for making claims that are not supported by the retrieved web sources (i.e., hallucinating) or providing non-working links. These account for 42% of Gaming criteria and 74% of Shopping criteria, and do not appear in DIY or Food tasks. GPT 5.1 (Thinking = High) is the most grounded model, failing just 29% of the grounding criteria, whereas Gemini 3 Pro (Thinking = High) is the least grounded, failing on 62%.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/080cfaea1586645252936f4e259b6437e81e6098-1744x1108.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Loss analysis</h2>
We built ACE to provide fine-grained insight into the performance of models – and every item in the rubrics are tagged with the criteria type, which can be used for loss analysis of model performance.
For instance, ACE shows that models perform well at meeting quantity requirements (nearly all models score 80% on this criteria type in Food) but they are much worse at meeting nuanced requests, like compatibility requirements in Gaming (most models score under 40%) or providing suitable safety warnings in DIY (most models score under 50%). For some criteria types, models find them so difficult that their mean score is actually negative – like providing working links or giving the price.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/e230c5984a08bbb2096d1df3930ca9274a9e61ce-1992x1500.png" alt="" style="width: 100%; border-radius: 10px;" />
<h2>Open sourcing</h2>
We are open sourcing 20 cases in each of the four domains (n = 80 total) on <a href="https://huggingface.co/datasets/mercor/ACE" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Hugging Face</a>. The data has a similar composition and is similarly difficult to the hidden heldout set. You can also use our evaluation harness (now on <a href="https://github.com/Mercor-Intelligence/apex-evals" target="_blank" rel="noopener noreferrer" class="text-indigo-600">Github</a>) to reproduce our grading approach. If you build with ACE, please let us know. To submit your model for testing, email apex@mercor.com
ACE is only possible because of the incredible work of our experts. Thank you to everyone who took part in the project!
<img src="https://cdn.sanity.io/images/h6s14f4z/production/6745667436ccc0ed66fe3708aaa5dd6c0065bd26-2240x1184.png" alt="" style="width: 100%; border-radius: 10px;" />

AI Consumer Index tests what people actually want AI to do across shopping, food, gaming and DIY, created by experts from the Mercor platform.

Introducing the AI Consumer Index | Mercor

Introducing the AI Consumer Index

Knowledge is passed from one person to another. Students learn from teachers. Employees learn from managers. Apprentices learn from masters. Progress is made through people.
Since we founded Mercor almost three years ago, AI has advanced at an astonishing pace. But it still struggles with the subtleties that drive economically valuable work—balancing trade-offs, understanding intent, developing taste, and deciding what should be done, not just what can be done.
That&#39;s where our work begins.
<h2>Unlocking Human Potential</h2>
Mercor sits at the intersection of labor markets and AI research. We connect human expertise with leading AI labs and enterprises that drive the AI economy.
At Mercor, our vast talent network trains frontier AI models in the same way teachers train students: by sharing knowledge, experience, and context that can&#39;t be captured in code alone. Each project expands what models understand.
Each advance in AI, in turn, unlocks human potential.
We see this through the progress our experts make every day:
<ul><li>A doctor shaping how AI recognizes early warning signs that could be missed by the human eye.</li><li>A banker training an agent to do financial analysis far more efficiently than redundantly analyzing data rooms.</li><li>A lawyer refining how models reason through precedent to improve legal judgment.</li></ul>
<h2>Defining a New Category of Work</h2>
AI is creating opportunity and accelerating human capability.
Millions of people will spend the next decade teaching machines the judgment, nuance, and taste that only humans possess. Instead of doing predictable work repeatedly, they&#39;ll teach agents how to do it once, so the agent can do it a million times.
Enterprise value chains are already shifting this way. Professionals can now codify workflows as evaluations that models can learn and improve from.
In turn, our experts are moving up the value chain. They&#39;re unlocking their potential by offloading rote tasks to agents and focusing on more economically valuable work that AI can&#39;t reliably do.
<img src="https://cdn.sanity.io/images/h6s14f4z/production/fa6d7579746d6250ae023920130bc6c75266d50a-3386x924.png" alt="series-c-metrics" style="width: 100%; border-radius: 10px;" />
<h2>Investing In Our Future</h2>
Today, we&#39;re announcing our $350 million Series C funding, led by Felicis with participation from Benchmark, General Catalyst, and Robinhood Ventures. This round values Mercor at $10 billion, 5x our Series B valuation.
The investment accelerates our mission across <a href="https://mercor.com/blog/big-things/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">three focus areas</a>:
<ul><li>Vast talent network</li><li>Better matching</li><li>Faster delivery</li></ul>
We are uniquely positioned to create this new category of work and shape how human and artificial intelligence create more economic value.
We&#39;re hiring. Join us: <a href="https://mercor.com/careers/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">https://mercor.com/careers/</a>

Announcing Mercor's $350 million Series C at a $10 billion valuation.

Unlocking Human Potential in the AI Economy

Many believe gains made in AI must be zero-sum that as AI improves, human opportunity will decline. But that&#39;s not what we&#39;re seeing.
Our experts collectively earn more than $1M a day. Our customers are today&#39;s most consequential companies. We are the fastest-growing company in history.
We are delivering on a mission to unlock human potential. We are uniquely positioned to create a new category of work that values and progresses human expertise in the AI economy.
We will focus on the following three Big Things:
<h2>Three Big Things</h2>
Vast talent network: Our competitive moat is the network effects from being obsessed with our talent experience. We are proud that more than half of our new experts come through referrals. We won&#39;t lose sight of that. That’s why we will continue to make long-term investments in sourcing, identifying, and retaining top performers who drive disproportionate value for our customers.
Better matching: The right person matched to the right opportunity can create 10x more value. We&#39;ll keep advancing our matching systems to understand each expert&#39;s capabilities and pair them with the opportunities where they&#39;ll have the greatest impact.
Faster delivery: Speed extends across the entire life cycle of data production. We&#39;re building new products and automating everything from fraud detection and onboarding to analytics, review, and testing.
<h2>The Future</h2>
Human data is the foundation of the new economy. Human insight will guide AI, not compete with it.
The total market for human data is defined by the things humans can do that models can&#39;t. Models might be superhuman at Olympiad math, but they still can&#39;t draft an email, negotiate a deal, or understand the tone of a legal argument. Humans give models that missing context. As AI continues to expand what is possible, new categories of work will be created.
This is why the <a href="https://mercor.com/blog/the-economy-will-become-an-rl-environment-machine/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">economy will become an RL environment machine</a>, and every enterprise value chain will converge on evals.
Our job is to unlock the human potential that will drive the AI economy forward.
These are our &quot;Big Things&quot; and they will be for decades to come.

Our focus to deliver on a mission to unlock human potential.

Big things

The biggest obstacle to AI delivering on its economic potential is the gap between existing AI evaluations and what professionals do in the real world. We&#39;re launching the AI Productivity Index (APEX) to start bridging this gap.
APEX is a first-of-its-kind benchmark that evaluates AI models based on their ability to perform economically valuable knowledge work. With future releases, we will expand coverage to other industries, roles, and countries.
APEX is gauging how AI systems are increasing productivity, reshaping the workforce and creating economic value. Alongside other initiatives in the research community, such as OpenAI’s GDPval, we are proud to guide the development of the next generation of AI models to make them more useful. AI is already superhuman at Olympiad math, but these capabilities can be disconnected from what drives the economy. It&#39;s great to have 10,000 PhDs in your pocket—it&#39;s even better to have a model that can reliably do your taxes.
<h2>Constructing APEX</h2>
APEX v1.0 consists of 200 cases split evenly across investment banking, law, consulting, and medical practice. Each case consists of a prompt (task description), sources (information needed to complete the task), and a rubric (criteria for grading model responses). We constructed APEX v1.0 in five steps:
<ol><li>Sourcing experts: we assembled a team of ~100 experts with top-tier experience (e.g., investment bankers from Goldman Sachs) across four professions.</li><li>Prompt generation: experts generate task descriptions, or prompts, describing common workflows in each domain. These workflows are aligned with economic value creation in two ways. Tasks are based on deliverables: prompts ask models to generate deliverables, e.g., a patient diagnosis by a doctor or a competitive research memo by a consultant. Thus, each task in APEX, if completed, would generate genuine economic value. Task distributions: the distribution of prompts exactly matches the experts&#39; estimate of the share of time spent on each workflow. For example, in investment banking, financial modeling and valuation (FMV) tasks were estimated to take up 30% of analysts&#39; time. Thus, FMV tasks comprise 30% of the APEX investment banking benchmark.</li><li>Source generation: experts produce source documents that contain relevant evidence needed to respond to the prompts. For example, in medical practice, a source document might be a real or synthetic set of CDC recommendations. On average, each prompt is accompanied by 5.83 source documents, comprising (on average) 26,000 tokens.</li><li>Rubric generation: experts produce a rubric of prompt-specific criteria. Each criterion is an objective and self-contained statement about the response. For example, if the prompt asks the model to analyze growth opportunities for Delta airlines in 2025, one criterion could stipulate &quot;the response mentions Delta airline&#39;s 2025 revenue.&quot; Rubrics have an average of 29.09 separate criteria.</li><li>Quality control: After prompts, sources, and rubrics have been generated by an expert, a separate expert reviews them to ensure quality control. In total, 300 prompts were started by contributors, of which 200 were approved by reviewers and added to APEX v1.0.</li></ol>
Experts estimated that tasks in APEX would take a professional between 1 and 8 hours (3.5 on average).
<img src="https://cdn.sanity.io/images/h6s14f4z/production/1460f4cf0e83f35cd47cb8436633ff0922afb3c8-1296x346.png" alt="table-1-apex" style="width: 100%; border-radius: 10px;" />
<h2>Evaluation &amp; Results</h2>
<img src="https://cdn.sanity.io/images/h6s14f4z/production/623da020d622db1aa3f9a1766fb3c131e1e55090-2040x1200.png" alt="top-3-models" style="width: 100%; border-radius: 10px;" />
We evaluate 21 state-of-the-art models using APEX v1.0, including top-performing closed and open source models. Each model is given the prompt and sources as an input, and then outputs its response as long-form text. The text responses are graded by a panel of LLM judges (majority vote) according to the expert-generated rubric. The overall score for each response is defined as the average number of criteria satisfied.
This rubric-based evaluation system–also used in OpenAI’s excellent <a href="https://openai.com/index/healthbench/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">HealthBench</a> work–allows us to automatically grade new models as they are released while maintaining consistency and objectivity. We report our findings on auto-grading consistency in our white paper.
GPT-5 achieved the top score of 64.2% and the best performing open source model, Qwen3, was ranked 7th overall with 59.8%. The top scores were highest in Law (70.5%) and worst for Investment Banking (59.7% ). None of the models meet the production bar for automating real-world tasks in the four professions we looked at. In a live setting, all would require substantial human oversight.
That said, the initial results provide a basis for optimism. APEX-v1.0 is highly complex, and its tasks require advanced reasoning, synthesis, multi-hop knowledge handling, and expert-level critical-thinking. This is reflected in the time that annotators estimated it would take them to complete the tasks (3.5 hour average). Models that could autonomously complete these tasks have the potential to unlock hundreds of billions of dollars of value across the U.S. economy.
<h2>Future work</h2>
APEX-v1.0 represents a step forward towards economically meaningful evaluations of AI models. However, it is an imperfect benchmark, and we plan to improve APEX over time:
APEX world: We are working on introducing simulated environments to allow AI models to interact with clones of common applications like SharePoint, Google Workspace, and other external tools via MCP, API, and GUI. We are hiring experts to populate these worlds with data as if they are working in mock companies. Future iterations of APEX will then evaluate models&#39; ability to interact in these worlds, with many more tools and files available to the models. We aim to bridge the sim2real gap in world fidelity, data distribution, and task realism.
Expanding the benchmark: We will broaden the range of professions and task types included, particularly in creative and technical fields. In partnership with academic and industry leaders, we will increase the granularity of the first four domains to show results in specific groups of workflows and areas of practice.
What does 60% mean? No models are close to achieving a 100% score on APEX. That said, in some cases, a performance of 60% may already add substantial economic value–for example, perhaps a consultant can more easily complete a competitor analysis if given an initial draft from an AI. On the other hand, in some domains, inaccuracies may be actively harmful–an AI-generated diagnostic report might ultimately waste a doctor&#39;s time if it needs to be carefully fact-checked for inaccuracies before it can be used. More research is needed to understand the economic impact of imperfect models.
For questions, feedback, or to get involved, reach out to us at apex@mercor.com.

A New Benchmark Measuring the Economic Impact of AI

Introducing APEX: The AI Productivity Index

Every technological revolution has sparked fears of job loss. The industrial revolution displaced domestic producers with machines. The computer revolution displaced manual clerical work with spreadsheets and databases.
And yet, unemployment rates are lower today than both before and during these events, which each yielded entirely new categories of work. The vast majority of job categories recognized by the Bureau of Labor Statistics didn’t exist before the industrial revolution.
<h2>Humans Will Do Things Once</h2>
The history of technology is a story of democratizing access: the printing press spread ideas, industry scaled labor, and computers digitized knowledge. Each revolution forged entire industries around it. Today, AI makes human capability itself sharable.
<blockquote>“If you wish to achieve some kind of intellectual immortality, writing for the AIs is probably your best chance.” - Tyler Cowen</blockquote>
The value of human work will shift. Think about the difference between filing taxes once and teaching an AI model how to file taxes for you forever. The first is a variable cost, paid millions of times over by individuals and businesses. The second is a fixed cost; once we encode that knowledge, it can be applied an unlimited number of times.
<h2>Real-World Environments</h2>
Reinforcement learning (RL) is becoming so effective that it can saturate any eval, but academic benchmarks aren&#39;t reflective of the outcomes that consumers and enterprises care about. There is a sim-to-real gap in our benchmarks. Did the tax filing minimize liability? Did the medical advice improve patient outcomes? Did the lesson plan help students actually learn?
The real world has richer data rooms, more complex environments of applications and tools, and requests from both programmers and accountants. The frontier of model evaluation now lies in building richer environments: data rooms that mirror your Google Drive workspace, scaffolding that mimics the many applications you have on your laptop or phone, and reward functions that can assess the near-infinite number of actions you can take in the real world.
Models also need to be evaluated on longer-horizon tasks and collaborative environments: longitudinal patient cases assessed by boards of physicians, multi-party negotiations in M&amp;A deals, and risk-hedging as markets move through cycles.
<h2>An Expanding Frontier</h2>
The market for humans teaching models is based on the amount of tasks humans can do which agents can’t do. Many researchers who believe in the inevitability of ASI downplay the role of human data. Once AI exceeds humans in every task, they ask, why would human data matter? Will the pool of people able to contribute to model improvement shrink substantially?
We worked on a project where a team of 100 people worked to find mistakes made by a frontier agent while using a tool. They created rubrics to evaluate the model’s mistakes. At first, everyone easily stumped the model because it failed frequently. Six months later, only 20 people could still stump the model, reinforcing the case made by skeptics of human data.
We then added more tools that the agent could access and started pushing for longer-horizon tasks that would take humans over ten hours to complete. Suddenly, the model began failing across these challenges, and all 100 participants were able to once again contribute meaningfully to the project. As long as there are tasks in the economy that humans can perform but agents cannot, we will continue to need humans to create evaluations and train agents.
<h2>The Long-Term Outlook</h2>
Everyone is focused on the jobs AI might eliminate, such as copywriting, paralegal work, and medical billing. But not nearly enough attention is dedicated to the industry it will create, driven by people who will shape AI’s judgment, design its training environments, and ensure its outputs meet human standards.
We are entering the era of experience, with models learning to optimize for rewards in the real-world. Just as humans learn through the guidance of others, AI will require robust feedback. Professors create tests and rubrics to help us improve, while managers provide us with performance reviews to track how we’re doing in our jobs. The same type of scaffolding will be needed by the next generation of AI models.
The industrial revolution created a new class of workers who designed machines and kept them running. Similarly, the AI revolution will create a new class of workers tasked with guiding machines and democratizing access to their abilities. This is the great paradox: the future of AI is human.

While everyone fears job loss, we’re creating a new category of knowledge work faster than any other time in history. The future of work will converge on training agents.

The Economy will Become an RL Environment Machine

People often ask why I left a burgeoning career in law to work for Mercor. After all, I liked practice (which is, unfortunately, a rather rare sentiment). So why leave a stable career path that I’d spent half a decade building and move myself and my husband across the country to become a Strategic Projects Lead? The truth is that I became impatient: impatient for impact, impatient for growth, and impatient to be at the frontlines of a technology that was already changing the world, with far more to come.
Over the last few years, I saw intermittent glimpses of what AI could mean for humanity. It is a fundamental fact that most professions critical to the public good—healthcare, legal defense, teaching—are woefully understaffed. And, for the first time, it seemed like we were on the verge of a technological wave that could truly (and quickly) help fill those gaps. As a Second Circuit law clerk, I would sometimes compare pro se legal briefs (those written by individuals without an attorney, almost always because they cannot afford one) to the arguments AI models made if I fed them the (PII-scrubbed) relevant legal questions. While the models would occasionally hallucinate, their legal arguments were almost always better than those in the pro se briefs. They were, however, still far below the quality of legal analysis that the $1k+-an-hour lawyers at white shoe firms could produce. I got into the human data space—where people create the data that trains AI—because I wanted to help models deliver top-notch legal analysis, finally giving the Davids of the world the sling they need to take on the Goliaths.
And I joined Mercor specifically because of its incredible position leading a key transformation within that space: a shift from simple data labeling (e.g., deciding whether an image is of a stop sign or a child) to semantic intelligence work (where experts generate complex tasks, which can take 20+ hours to complete, based on their domain-specific knowledge). The company’s core thesis was that, to create next level AI models, you need great data and, to create great data, you need top human talent. Thus, Mercor’s founders focused on (1) building products that could effectively and efficiently screen for true domain experts (products which have use cases that <a href="https://mercor.com/blog/secret-master-plan/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">go far beyond the human data space</a>) and (2) creating a deep bench of those experts across a broad range of domains.
The company’s success in this new era of semantic intelligence work has made my job as a Strategic Projects Lead incredibly exciting for three reasons: (1) we work for the best clients; (2) we engage the best experts; and (3) we become the best operators. Mercor now works with the <a href="https://x.com/BrendanFoody/status/1939783540394402083" target="_blank" rel="noopener noreferrer" class="text-indigo-600">top five AI labs and six of the Magnificent Seven</a>; many of the projects we run for these companies involve the world’s best and brightest, including Nobel Prize recipients, Emmy winners, Marshall and Rhodes Scholars, FAANG software engineers, and IMO medalists. To make these projects successful, Mercor also invests in world-class internal talent. My colleagues are among the sharpest and most committed people I’ve had the privilege of working with. And we’ve all grown into even stronger operators (and <a href="https://mercor.com/blog/why-i-joined-mercor-srini-rajagopal/" target="_blank" rel="noopener noreferrer" class="text-indigo-600">engineers</a>) by taking on far more challenging and meaningful problems at Mercor than we could find elsewhere—backed by exceptional mentorship from leaders like the Mercor founders, who built a company that has engaged 10,000+ experts from 45+ countries in under two years, and Sundeep Jain, Mercor’s President and former Chief Product Officer at Uber.
To put things in perspective: in my second week at Mercor, I was tasked with staffing and running a project of 200+ software developers for one of the world’s top AI companies. This project not only created thousands of high quality data samples that will move the needle on AI progress—it also gave hundreds of people flexible, high-paid work for nearly five months, with a total of over $4,000,000 DPT (Dollars Paid to Talent). In my second quarter at Mercor, I interfaced almost daily with one of the top researchers at the same frontier lab to manage what the lab has referred to as one of the most ambitious human data projects it’s ever undertaken (notably led by an all female team, on both the lab’s side and Mercor’s). In sum, I can’t think of a better place to both learn about the human data space, with a bird’s eye view of how the best AI companies utilize human data to improve their models, and to challenge myself as an operator, responsible for supporting hundreds of experts and executing projects worth millions of dollars.
Looking back, the decision to leave law wasn’t about walking away from anything: it was about running toward something urgent and somewhere that I could flourish. I wanted to be in a place where innovation was the norm, where I could take full ownership from Day One, and where the work could scale beyond a single courtroom or client to help shape systems at the global level. Mercor has been that place.
For anyone standing at a similar crossroads who is curious about how they can contribute to something bigger, faster, and more impactful, I hope my story offers a glimpse of what’s possible. If you’re drawn to the same things that I am—a desire to have global impact; to work on the cutting edge of developments in AI; and to push the boundaries of your capabilities—I’d encourage you to check out <a href="https://mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">our open internal roles</a>.

People often ask why I left a burgeoning career in law to work for Mercor...

Why I Joined Mercor

At Mercor, we connect graduate-level experts with meaningful, well-paid work that leverages their domain knowledge, while providing AI labs with access to our highly specialized and trusted network of talent. The experts on our platform train frontier AI models using their domain expertise by developing discipline-specific prompts, evaluating responses, and delivering high-quality human feedback across fields ranging from law, linguistics, engineering, and medicine.
For many graduate students and alumni, it’s both a valuable opportunity to apply their skills at the cutting edge of technology and a way to monetize their expertise without leaving academia behind. To invest in the future of academic research and human insight, we launched the Mercor Graduate Fellowship to support outstanding graduate students who are on-track to do exceptional work in their field.
This year, we received over 2,000 applications from across the country and are proud to announce our 50 finalists and two fellowship winners:
<ul><li>Neil Band, a Rhodes Scholar and third-year Computer Science PhD student at Stanford</li><li>Mireya Gonzales-Rivera, an incoming Physics PhD student at UC Berkeley, graduated from CSU San Marcos</li></ul>
The fellowship comes with a $50,000 grant and public recognition as part of our commitment to supporting graduate researchers and unlocking access to opportunity.
<h2>Meet the 2025 Winners</h2>
Neil Band
Neil is a third-year Computer Science PhD student at Stanford and a former Rhodes Scholar. His research focuses on making large language models more trustworthy and reliable, tackling problems such as hallucination, uncertainty calibration, and reasoning in LLMs. He currently works in one of Stanford’s top AI research labs, exploring how statistical thinking can be applied to modern generative systems.
With the support of this fellowship, Neil plans to take on more ambitious research, further contributing to the future of safe and reliable AI.
<iframe src="https://www.youtube.com/embed/Bl7ZWadMHT4?si=UnUogPXhCyEF9W49" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="YouTube video player" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
Mireya Gonzales-Rivera
Mireya earned her bachelor’s in applied physics from California State University, San Marcos, where she conducted undergraduate research in quantum nanoelectronics and silicon-based qubit architectures. This fall, she’s beginning her Physics PhD at UC Berkeley, with a focus on quantum information science.
Mireya aims to build a lab that expands access to research and mentorship for underrepresented students in STEM. Her passion for equity, mentorship, and scientific discovery made her stand out in this year’s applicant pool as a clear example of undiscovered talent.
<iframe src="https://www.youtube.com/embed/xFyESncCyls?si=ZC-ycwAVY3aQ7pgH" frameborder="0" allow="autoplay; fullscreen; picture-in-picture; clipboard-write; encrypted-media; web-share" referrerpolicy="strict-origin-when-cross-origin" title="YouTube video player" class="mb-8 mt-4 aspect-video w-full max-w-[1080px] rounded-lg bg-gray-100 object-cover sm:mt-6"></iframe>
<h2>A Platform Built for Academic Talent</h2>
Through our platform, thousands of PhDs and postdocs are already contributing to advancements in AI and getting paid well for it. For some, it’s a full-time career path. For others, it’s a part-time opportunity to apply their expertise beyond the lab.
Whether through work or funding, our mission remains the same: to support graduate researchers, amplify their impact, and help them shape the future.

At Mercor, we connect graduate-level experts with meaningful, well-paid work that leverages their domain knowledge, while providing AI labs with access to our highly specialized and trusted network of talent.

Mercor Graduate Fellowship Winners

Reinforcement Learning (RL) is driving the most exciting advancements in AI. RL is becoming so effective that models will be able to saturate any evaluation. This means that the primary barrier to applying agents to the entire economy is building evals for everything. However, AI labs are facing a dire shortage of relevant evaluations. Academic evaluations that labs goal on don’t reflect what consumers and enterprises demand in the economy.
Evals are the new PRD. Progress in accelerating knowledge work will converge on building environments and evaluations that map real workspaces and deliverables. This new RL-centric paradigm of human data is vastly more data efficient than pretraining, SFT, or RLHF. Most knowledge work includes recurring workflows as variable costs, but creating an environment or evaluation can transform that into a one-time fixed cost.
<h2>Training on Verifiable Rewards</h2>
RL environments allow for rewarding outcomes and intermediate steps in an evaluation. Models take many attempts at a problem, using test-time compute to &quot;think&quot; before it answers. Human created autograders reward the attempts which were &quot;good&quot;. Reinforcing on those &quot;good&quot; trajectories upweights the chains of thought that were used to get to the answer. This teaches models to think correctly about different types of problems as researchers iteratively hill climb evals.
These environments can be thought of as existing on a spectrum of rigidity between two categories:
<ul><li>Objective domains: Games, like pac-man, chess, and Go, have clear states spaces, action spaces, and desired outcomes. Math, code, and even some tasks in biology, can often be formulated with near game-like verifiability. This is where RL has achieved early massive success already, notably, AlphaProof, AlphaFold, and DeepSeek R1 and the many code generation models on the market today.</li><li>Subjective domains: It’s more difficult to measure accuracy in many real world tasks such as generating investment memos, making legal briefs, providing therapy. This makes it difficult to verify that a model achieved desired outcomes. Additionally, experts often support multiple valid opinions about desired processes and outcomes. Rubric-based rewards serve as a way to learn from the messiness of expert human opinions. How to evaluate and train with rubrics as environments is an exciting area of research with roots laid as early as constitutional AI and RLAIF work from Anthropic.</li></ul>
Computer-use agents sit somewhere in the middle. For most of the tasks humans do on computers, goals start to become ambiguous and multi-faceted. Once defined, the actions and outcomes are programmatic and verifiable. These could include planning trips, responding to emails, shopping, or posting on social media. In all of these cases, containerized environments allow for horizontal scaling to learn online from thousands of interactions in parallel.
<h2>Environments Create Experience</h2>
Eventually, our AI systems will learn automatically from signals in the real world like pupils’ test scores increasing, sales closing, maybe even bridges being built. However, intermediate rewards will always remain critical. Similar to how humans learn from other people, models will need guidance on which styles of teaching and sales techniques are most effective. Humans will remain an integral part of the environments models learn from.
We will never escape the era of data; it must follow us to the frontier. That frontier is human created environments that provide durable sources of experiential data. These environments can serve to train and evaluate models.
<h2>The Path Forward</h2>
Meeting today’s data demand requires rethinking the way we generate signal from human efforts. Creating evals and RL environments is the highest leverage and most durable use of people’s time. Mercor has helped pioneer environment generation using autograders and continues to push the boundaries of RL data with simulated workspaces, multi-turn support, and multi-modality.
Knowledge work will quickly converge on building RL environments and evaluations for agents to learn from. As AI enters the workforce and operates over proprietary information and under unique professional contexts, these environments codify knowledge and goals for agents. Once individual steps of agentic workflows reach sufficient reliability, all that will be left will be RL training on the goals laid out by humankind.

Reinforcement Learning (RL) is driving the most exciting advancements in AI. RL is becoming so effective that models will be able to saturate any evaluation. This means that the primary barrier to applying agents to the entire economy is building evals for everything. However, AI labs are facing a dire shortage of relevant evaluations.

Welcome to The Era of Evals

Throughout my career, I’ve been drawn to foundational challenges. The kind that, if solved right, can transform the biggest systems in the world. With each technological shift–web, cloud, mobile, and now AI–we’ve seen a new class of category-defining companies emerge. The web gave us Google. Cloud established AWS. Mobile unlocked Uber. Now, AI is reshaping the way we work, hire, and build. The next breakout companies will be born from this inflection point, and I believe Mercor is one of them.
When I was studying computer science at IIT Delhi, the internet was just beginning to take shape. I was captivated by how data moved through networks and what became possible when digital infrastructure connected people at global scale. That fascination led me to ONI Systems, where I worked on optical networks. The internet was growing faster than infrastructure, choked by bandwidth and throughput constraints, could keep up.
Two decades later, I see history repeating itself. The bottlenecks are no longer physical routers and fiber optics, but they’re just as real. Today’s AI systems are limited by three key factors: compute, algorithmic breakthroughs, and access to new, high-quality data. Foundational models have made astonishing progress, but unlocking their next phase of capability depends on how well we can guide and refine them with structured, domain-specific feedback from human experts.
That’s what drew me to Mercor.
Mercor is tackling one of the most important and overlooked infrastructure challenges of our time: how we evaluate and allocate human talent. We’re starting with AI labs because training frontier models demands the expertise of humans, like lawyers who probe reasoning, engineers who review code, and physicians who validate clinical outputs. Mercor matches these experts to projects and uses feedback to continually refine the matching itself. Place talent, observe outcomes, refine predictions—repeat. The result is a data-rich feedback loop that learns more about human performance in weeks than traditional hiring systems learn in years.
Observing Mercor’s early days, I feel a strong sense of déjà vu. I’ve witnessed rapid growth firsthand at Amazon, Google, Uber, Instacart, and Stripe. Each tackled foundational infrastructure challenges in their own domain. At Amazon, I had a front-row seat as AWS evolved from an internal platform to the backbone of the internet. At Uber, we started with a specific use case–ridehailing–and scaled it into a global transportation network. I still remember the mantra: make transportation as reliable as running water. At Google, the mission to organize the world’s information drove us to build systems that could scale with the internet itself. Similarly, at Stripe, we built programmable financial infrastructure designed to be global from day one.
At the heart of each of these companies was the same thesis. If you solve a foundational problem deeply, you don’t just build a product–you build a platform that reshapes entire industries. Mercor now fits that legendary lineage. But in an era when AI is reshaping the nature of work, I believe Mercor is seizing an opportunity far more consequential than any other: the market for human potential.
Currently, the broader labor market is fractured and inefficient. Candidates see only a fraction of opportunities. Companies rely on weak proxies like résumés. Mercor is building the most structured and comprehensive performance dataset ever assembled, allowing us to match any person to any opportunity with incredible precision. The engine that helps an AI lab hire 300 PhDs in a weekend will eventually power a global talent marketplace where every company can hire confidently and every candidate gets a fair shot. In the long run, I believe Mercor will redefine the endgame for human work.
Since joining Mercor, I’ve helped scale our systems from 1 to 100 million in payments volume in under a year. I’ve worked on the infrastructure that underpins large-scale evaluations and makes talent assessment fair and effective. It’s among the most complex and meaningful challenges I’ve taken on.
Mercor is already one of the fastest-growing startups in Silicon Valley history—and we’re just getting started. We’re hiring across engineering and operations. If you’re excited about shaping the future of work and building systems that unlock human potential, we’d love for you to apply at <a href="https://mercor.com/careers" target="_blank" rel="noopener noreferrer" class="text-indigo-600">mercor.com/careers</a>.

Throughout my career, I’ve been drawn to foundational challenges. The kind that, if solved right, can transform the biggest systems in the world. With each technological shift–web, cloud, mobile, and now AI–we’ve seen a new class of category-defining companies emerge.

Imagine a world where Jeff Bezos is a hedge fund investor, Howard Shultz is a salesman, and Reed Hastings is a teacher. That was the world we lived in, not so long ago. These are the jobs they were doing before they found the best use for their talents.
We founded Mercor because the labor market is the largest, most inefficient market in the world. Better matching people with the work they do everyday is the largest lever on maximizing global utility. While we gained incredible traction with our initial focus on contracting experts to train AI models, this is only the first step in our plan to solve global labor allocation.
The Wedge
Marketplaces are hard to get off the ground, but if they do take off they become huge. The successful ones have a wedge into a large and pressing unmet need. For Uber, the wedge was black cars. For Airbnb, it was conferences.
We started 2024 in our apartment with no US employees, under $1M in annual revenue, and only seed companies as customers. Last year we grew 6400% and we now work with the most sophisticated technology companies in the world, making us one of the fastest growing companies in Silicon Valley history.
We believe we can create hundreds of thousands of opportunities with AI labs alone, but that pales in comparison to the billions of knowledge work opportunities in the world. The technology that we’ve been building is generally applicable.
Structural Inefficiency
Labor inefficiency stems from two structural challenges in the market:
<ol><li>Fragmentation–Candidates apply to a handful of opportunities and companies consider a fraction of a percent of candidates in the market. This is because matching supply and demand needs to be solved manually (and previously in person). Companies manually review resumes, conduct interviews, and predict who they believe will perform well. Human time is the limiting factor. However, if you can solve this matching problem at the cost of software it allows you to interview everyone, making way for a global, unified labor market that every candidate applies to and every company finds talent from.</li><li>Imperfect Information–When you order a ride on Uber, you know what you’re getting. When you book an Airbnb, the pictures usually do a pretty good job. When you’re hiring someone, it’s extremely difficult to accurately predict how well they will perform. Imperfect human judgement is embedded within every transaction. While LLMs are not perfect at talent assessment, models are quickly surpassing human capabilities. This trend will continue to make transactions more efficient.</li></ol>
Correspondingly, our main objectives are to attract high caliber applicants to come to Mercor and accurately predict candidate’s job performance. Achieving these objectives will solve global labor efficiency more broadly.
Contracting experts to train AI models is the perfect forcing function on these objectives. First, we collect performance data from AI labs within days, compared to the 3 month lag from a traditional enterprise. This allows us to immediately calibrate on the effectiveness of our models and continuously experiment to find the features predictive of success. Second, we need to engage with a broad pool of candidates across all knowledge work roles (law, consulting, medicine, engineering, etc.). This builds the strength of our talent pool across every professional and academic domain. Third, we will service “unreasonable asks” from AI labs like needing to find 300 people in two days. These high volume requests for quality people on extremely short timelines can’t be fulfilled with a services operation. They force us to build the automations at each layer of the engagement process to deliver.
Hiring for All Work
We have the largest comparative advantage from automating talent assessment when the ratio of time spent assessing someone relative to the time spent working with them is the highest. When hiring someone for 5 years, it’s easier to interview them manually. When hiring someone for 5 weeks, efficient matching automation creates a huge comparative advantage. Because of this, we’ve started with shorter duration contract work, but will expand progressively towards longer duration, full-time jobs as our technology matures.
So, in short, the master plan is:
<ol><li>Hire people to train AI models</li><li>Use those contracts to learn how to predict job performance</li><li>Expand to short-duration contract roles</li><li>Hire people for all jobs</li></ol>
Don’t tell anyone.

Imagine a world where Jeff Bezos is a hedge fund investor, Howard Shultz is a salesman, and Reed Hastings is a teacher. That was the world we lived in, not so long ago. These are the jobs they were doing before they found the best use for their talents.

The Secret Mercor Master Plan

Throughout history, many of humanity’s great minds took a while to find their calling. Jeff Bezos started as a hedge fund investor. Walt Disney was fired as a newspaper editor for “lacking creativity.” Before becoming a priest, Pope Francis worked as a nightclub bouncer. (Yes, really.) In each case, greatness was unlocked by the right person ending up in the right place at the right time. But, tragically, many never do.
The labor market has consistently been the largest, most inefficient market in the world. That unsolved problem becomes immediate and urgent now as we face the advent of AGI and prepare for the most transformative and challenging period in how human talent is allocated.
AI will change and even displace many traditional careers. At the same time, in other professions, AI will actually create much more need and opportunity for human ability.
We’re not the first to recognize these changes. But we’re the first to build a scalable solution. Bringing the right talent to these roles — defining “skilled labor” in the AI era — is where we focus our efforts.
Our current focus is on contracting experts to help advance the frontier of AI, but our sights are on every sector.
Mercor is two years old and growing more than 51% month over month. Our team includes the former Head of Human Data Operations at OpenAI and the previous Head of Growth at Scale. Over half of us are former founders, and our median age is 22.
Our latest round was led by Felicis, with participation from General Catalyst, DST Global Partners, Benchmark, and Menlo Ventures — and it will accelerate our ability to match billions of people with their calling, applying human talent to its highest potential.
If this mission resonates with you, we are looking for exceptional individuals — elite engineers, seasoned operators, and former founders — to join us. Because it takes talent to recognize talent. Reach out: <a href="mailto:careers@mercor.com" target="_blank" rel="noopener noreferrer" class="text-indigo-600">careers@mercor.com</a>.

Mercor is the global allocator for extraordinary human talent in the AI economy, and one of the fastest growing companies in Silicon Valley history. We’ve just raised a $100M Series B to solve the hardest problem in capitalism: matching human ability to its greatest use.

Announcing Mercor's Series B

Today, we're excited to announce $3.6M in funding and the launch of our fully-automated platform, which uses AI to assess and match talent with companies. Our round is led by General Catalyst and includes participation from Scott Sandell (Chairman, CEO and CIO of NEA) and others.

Introducing APEX-Agents

Expanding the AI Productivity Index (APEX)

Introducing the AI Consumer Index

Unlocking Human Potential in the AI Economy

Big things

Introducing APEX: The AI Productivity Index

The Economy will Become an RL Environment Machine

Why I Joined Mercor

Mercor Graduate Fellowship Winners

Welcome to The Era of Evals

Why I Joined Mercor

The Secret Mercor Master Plan

Announcing Mercor's Series B

Introducing Mercor: Redefining Hiring With AI