Aug 2025Healthcare
General Practitioner (MD) Benchmark
AI Benchmark created by experts from University of Pennsylvania, Northwestern, Cornell, Brigham & Women’s, Mount Sinai and advised by Eric Topol—Cardiologist, geneticist, and founder of the Scripps Research Translational Institute, leading voice in digital and precision medicine.
Model
Score
gpt-oss
65.7%
65.7%
gpt-5
62.9%
62.9%
grok-3
62.9%
62.9%
grok-4
62.9%
62.9%
gemini-2-5-pro
60.0%
60.0%
o4-mini
57.1%
57.1%
gemini-2-5-flash
54.3%
54.3%
o3-pro
54.3%
54.3%
claude-opus-4
51.4%
51.4%
claude-opus-4-1
48.6%
48.6%
claude-sonnet-4
48.6%
48.6%
deepseek-r1
48.6%
48.6%
kimi-k2
48.6%
48.6%
o3
48.6%
48.6%
qwen-3
48.6%
48.6%
gemma-3-27B
34.3%
34.3%
mistral-medium-3
34.3%
34.3%
llama-4-maverick
31.4%
31.4%
nova-pro
28.6%
28.6%
gpt-4o
25.7%
25.7%
llama-nemotron
20.0%
20.0%
glm-4-5
5.7%
5.7%
phi-4
0.0%
0.0%
Evaluation process
Model performance was measured using a structured evaluation process grounded in real workflows. Each task included an example problem, source materials, and a detailed rubric of criteria for grading. Model responses were then scored against this rubric to capture accuracy and completeness, with representative sample responses included below to illustrate both strong and weak performance. This methodology ensures consistency while reflecting the complexity and economic value of real-world consulting work.
Eval example
Prompt input A 6-year-old boy (new patient) presents to your clinic with wheezing. Per parents, he is on Fluticasone Propionate/Salmeterol (Advair® HFA), 2 puffs twice daily (total 460 mcg fluticasone/day) and montelukast. He also uses albuterol as needed. In the past 12 months, he has been to the ER 10 times for wheezing and given inhalers and oral medications. Three of the ER visits resulted in prolonged hospital stays for a “lung infection,” and he is “always sick”. Other than asthma, he has no medical problems that they are aware of. Parents also state that he is UTD on all his vaccines except for his 4-year vaccines. Parents have been hearing more about Tetanus and want to ensure that he is immune.
On physical examination, he exhibits diffuse wheezing, which improves with albuterol and oral steroids in the office. He is sent home with an oral steroid burst, albuterol, and continuation of Advair. The rest of the ROS and physical exam are negative, and he has a normal height and weight.
Initial labs and imaging are completed CBC with Diff: elevated eosinophils at 300 cells/μL Reassuring CMP Immunoglobulins: normal IgE and IgM, lower IgG (<2 standard deviations below the norm), low IgA Tetanus non-immune Chest x ray: normal
Two months later, the family returns and wants to discuss labs and additional medications for his symptoms. What labs should be drawn at this visit? Please include both visit specific labs and any follow-up labs needed from the last appointment. Based on the patient’s history, presenting symptoms, and available lab work, what is the most likely diagnosis? What is the most appropriate medication to start immediately, and what other medications and treatments should be considered? Information should be based on the latest peer-reviewed medical literature as of May 2025.
Model response:
gpt-oss
1. What laboratory studies should be ordered at this 2‑month follow‑up visit?
(The work‑up should address two separate but overlapping problems – the possible primary antibody deficiency and the refractory wheezing/asthma.)
| Category | Test | Why it is needed | When to repeat |
|---|---|---|---|
| Quantitative immunoglobulins | IgG, IgA, IgM (repeat the previous draw) | Confirms persistent hypogammaglobulinaemia and rules out a transient fall (e.g., due to acute infection, steroids). | 4–6 weeks after any change in therapy (e.g., after the first IVIG infusion) and then every 3–6 months while on replacement. |
| Specific antibody titres (vaccine‑response) | • Anti‑tetanus toxoid IgG <br>• Anti‑diptheria toxoid IgG <br>• Anti‑pneumococcal polysaccharide IgG (23‑valent) <br>• (Optional) anti‑Haemophilus influenzae type b IgG | A core ESID/ICON criterion for CVID is poor or absent protective antibody titres despite prior vaccination. These titres also tell us whether the child can be protected by routine boosters before starting IVIG. | Baseline now; repeat 4–6 weeks after a repeat immunisation if the first titres are non‑protective. |
| Isohemagglutinin (IgM anti‑A/B) titres | IgM anti‑A and anti‑B | Helps to differentiate primary hypogammaglobulinaemia from transient secondary loss; a low/absent isohemagglutinin is typical in CVID. | Once, at diagnosis. |
| Lymphocyte subset flow cytometry | CD19⁺ B‑cells (total and switched memory CD27⁺IgD⁻), CD3⁺CD4⁺ and CD3⁺CD8⁺ T‑cells, NK cells, CD21ⁿ⁽ⁿᵉᵍ⁾ B‑cells, CD45RA/RO | CVID patients often have normal total B‑cell numbers but reduced switched memory B‑cells; T‑cell abnormalities (low CD4, ↑ activation markers) can influence management and signal a combined immunodeficiency. | At baseline; repeat every 12 months or if clinical status changes. |
| Complete blood count with differential | Repeat CBC + diff (focus on eosinophil count) | Confirms current eosinophil level (≥300 cells/µL supports an eosinophilic asthma phenotype) and screens for cytopenias that can accompany CVID. | Every visit (every 2–3 months while on steroids/IVIG). |
| Serum eosinophils & FeNO | Absolute eosinophil count + fractional exhaled NO (≥35 ppb in children 5–12 y) | Biomarker panel for type‑2 asthma to decide on anti‑IL‑5/IL‑4R therapy. | At each asthma review (every 3 months). |
| Total serum IgE (already normal, repeat for completeness) | Confirms non‑allergic (IgE‑mediated) asthma phenotype. | Helps choose between anti‑IL‑5 (eosinophilic) vs anti‑IgE (omalizumab) therapy. | Every 6 months if considering change in biologic. |
| Pulmonary function testing | Spirometry with bronchodilator response, FEV₁/FVC, and if possible lung volume & diffusion (DLCO) | Baseline severity of airflow limitation and to monitor response to any new asthma biologic. | Every 6 months (or sooner if clinically indicated). |
| High‑resolution chest CT (if not done recently) | HRCT of the thorax | Detects bronchiectasis, interstitial lung disease, or other chronic lung sequelae that are common in CVID and may be contributing to wheeze. | Once now; repeat only if clinically indicated. |
| Stool ova/parasite panel and Giardia antigen | Stool microscopy/ELISA | CVID patients are prone to chronic Giardia and other GI parasites that can worsen wheeze via reflux or immune activation. | If GI symptoms develop (diarrhoea, abdominal pain). |
| Hepatitis B surface antibody (anti‑HBs) | To verify protective immunity (important before any future biologic or immunosuppressive therapy). | Standard of care for immunodeficient children. | At baseline; repeat if no protective titre. |
| Baseline liver function tests & viral serologies (CMV, EBV) | Only if clinical suspicion (e.g., unexplained transaminitis). | CMV re‑activation can drive immune activation in CVID. | As clinically indicated. |
Key point: All antibody‑response tests (tetanus, diphtheria, pneumococcus) must be drawn before the first IVIG infusion because passive IgG will mask the child’s own response.
2. What is the most likely diagnosis?
| Finding | Interpretation |
|---|---|
| Recurrent wheezing, ≥10 ER visits/yr with ≥3 “lung infections” | Severe, poorly controlled asthma + frequent lower‑respiratory infections. |
| Low IgG (<2 SD) and low IgA (both persist on two separate draws) | Meets ESID/ICON criteria for Common Variable Immunodeficiency (CVID) (age > 4 y, ≥2 SD low IgG, low IgA, and poor vaccine response to be proven). |
| Normal IgE, mild eosinophilia (300 cells/µL) | Consistent with an eosinophilic asthma phenotype but does not explain the immunoglobulin deficiency. |
| Normal chest X‑ray, no structural lung disease yet | In early CVID pulmonary disease may be silent. |
| Montelukast use & high‑dose ICS/LABA without lasting control | Suggests an intrinsic disease (immune deficiency) driving airway inflammation. |
Conclusion: The child most likely has CVID with associated severe eosinophilic asthma. The combination of hypogammaglobulinaemia, recurrent respiratory infections, and poor response to standard asthma therapy points to CVID as the underlying driver of his wheezing burden.
3. Therapeutic priorities – what should be started immediately and what else should be considered?
3.1 Immediate, essential therapy
| Intervention | Rationale (peer‑reviewed evidence up to May 2025) |
|---|---|
| Intravenous immunoglobulin (IVIG) replacement – 400–600 mg kg⁻¹ every 3–4 weeks (target trough IgG ≈ 8–10 g/L). | • The cornerstone of CVID management (ICON 2016, ESID 2020). <br>• Prospective data (Gill et al., Allergy Asthma Clin Immunol 2018) show a ~2.6 infections/pt‑yr rate with IVIG, and a “wear‑off” effect is mitigated by maintaining trough >8 g/L. <br>• Reduces bacterial sinus/lung infections, improves quality of life and may blunt the immune‑activation cascade described by Paquin‑Proulx & Sandberg (Front Immunol 2014). |
| Accelerated tetanus‑diphtheria‑pertussis (Tdap) catch‑up series (first dose now, booster at 4 weeks, then routine Td/Tdap at age 11 y). | CDC 2025 schedule: any child <7 y who is non‑immune receives a 3‑dose primary series of DTaP; an 11‑y booster with Tdap is recommended. Because the child is receiving IVIG, give Tdap before the next IVIG infusion (≥2 weeks) to allow an adequate antibody response. |
| Pneumococcal conjugate vaccine (PCV15/20) – single dose now and PPSV23 ≥8 weeks later (once IVIG steady). | Children with CVID respond poorly to polysaccharide antigens, but vaccinating before or mid‑therapy (with ≥2 weeks before next IVIG) is recommended (ACIP 2025). This reduces risk of invasive pneumococcal disease – a leading pathogen in CVID. |
| High‑dose inhaled corticosteroid/LABA regimen (continue current Advair) + short‑course oral corticosteroid burst (as already prescribed) for acute exacerbations. | Standard of care for severe asthma while evaluating biologics. |
| Start a type‑2 (eosinophilic) asthma biologic – Mepolizumab 40 mg SC q4 weeks (weight < 40 kg) or 100 mg q4 weeks (≥40 kg). | • FDA/EMA approved for children ≥ 6 y with severe eosinophilic asthma (≥150 cells/µL, or ≥300 cells/µL in past 12 mo) (Mepolizumab label 2019/2024). <br>• MUPPITS‑2 (Jackson et al., NEJM 2022) showed a 27 % reduction in exacerbation rate in 6‑17‑y olds (RR 0.73). <br>• Safety profile favourable in pediatric studies (no increase in infection risk when IVIG is already covering humoral immunity). <br>• Works upstream of IL‑5, directly addressing the eosinophilia that is contributing to airway inflammation. |
| Education & adherence support (inhaler technique, symptom diary, action plan). | Poor technique is a common cause of “steroid‑resistant” wheeze; systematic review (Liu et al., Pediatr Pulmonol 2023) showed a 30 % reduction in ER visits after a structured education program. |
3.2 Additional/adjunctive options (to be considered after the above are in place)
| Option | Indication & Evidence | Practical notes |
|---|---|---|
| **Mouth‑to‑mouth or subcutaneous **IgG‑enriched IVIG (IgA‑deficient preparations) if the child has anti‑IgA antibodies. | In CVID patients with anti‑IgA, standard IVIG can cause anaphylaxis; low‑IgA preparations are recommended (see Paquin‑Proulx & Sandberg, 2014). | Screen for anti‑IgA antibodies before the first infusion. |
| Prophylactic antibiotics (e.g., azithromycin 3× weekly) if infections persist despite adequate IgG troughs. | Systematic review (Cunningham‑Rundles et al., Clin Immunol 2022) supports prophylaxis in CVID with ≥2 severe infections/year despite IVIG. Use only after confirming IVIG adequacy. | |
| Bronchiectasis surveillance – repeat high‑resolution CT at 12 months, consider airway clearance techniques (physiotherapy, hypertonic saline) if bronchiectasis identified. | Up to 30 % of pediatric CVID develop bronchiectasis (Lee et al., Ann Allergy Asthma Immunol 2022). Early physiotherapy improves lung function. | |
| Alternative asthma biologics (Dupilumab, Omalizumab, Benralizumab) if mepolizumab is ineffective. | • Dupilumab (IL‑4Rα blocker) approved for children ≥ 6 y with T2‑high asthma (VOYAGE trial, 2021). <br>• Omalizumab (anti‑IgE) useful when IgE > 30 IU/mL and clear atopic sensitisation (but IgE is normal here). <br>• Benralizumab (IL‑5Rα) shows profound eosinophil depletion; data in children ≥ 12 y (Phase‑3), but limited experience <12 y. (See Pediatric Allergy Immunol 2024). | |
| Consider ** abatacept or CTLA‑4‑Ig if genetic testing later reveals LRBA or CTLA‑4 deficiency (often overlap with CVID).** | Targeted therapy for CVID with immune‑regulatory gene defects (Lee et al., J Allergy Clin Immunol 2023). | Requires genetic confirmation before use. |
| Vaccination of household contacts (influenza, COVID‑19, RSV monoclonal antibody) to reduce exposure. | Standard infection‑prevention in immunodeficient households (CDC 2025). | |
| Seasonal RSV prophylaxis (nirsevimab – Beyfortus) if the child has recurrent RSV‑related wheeze (e.g., winter spikes). | ACIP 2025 recommendation for all infants; off‑label extension for high‑risk children 6 mo–2 y (see Nirsevimab label). Not mandatory now but can be considered if RSV is documented. | |
| Referral to immunology & pediatric pulmonology for coordinated care, including longitudinal monitoring for non‑infectious CVID complications (autoimmune cytopenias, liver disease, GLILD). | CVID has high morbidity from autoimmune/lymphoproliferative disease (CVID‑c vs CVID‑u phenotype). Early specialist involvement improves outcomes (Lee et al., Ann Allergy 2022). |
4. Putting it all together – a practical action plan for the next visit
- Laboratory draw – order the full panel listed in section 1 (immunoglobulins, vaccine titres, isohemagglutinins, lymphocyte subsets, CBC + diff, eosinophils, FeNO, total IgE, pulmonary function testing).
- Vaccination – give the first dose of Tdap today (preferably at least 2 weeks before the first IVIG infusion). Schedule PCV15/20 today and plan PPSV23 in 8 weeks.
- IVIG initiation – arrange the first IVIG infusion (400 mg/kg) within the next 1–2 weeks (after the Tdap titre is drawn). Check anti‑IgA status first.
- Mepolizumab – start the appropriate dose today (40 mg if < 40 kg; otherwise 100 mg) and schedule the first sub‑cutaneous injection for the next week (preferably on a non‑IVIG day).
- Asthma review – obtain FeNO, repeat eosinophil count, perform spirometry, reinforce inhaler technique, and give written asthma action plan.
- Imaging – schedule a low‑dose HRCT of the chest (if not done within the past 12 months) to look for early bronchiectasis or interstitial changes.
- Counselling – discuss the need for regular IVIG, the expected reduction in infection frequency, and the add‑on benefit of mepolizumab for asthma control; review potential side‑effects (headache, injection‑site reactions, rare anaphylaxis).
- Follow‑up schedule –
- 2 weeks: review tolerance to first IVIG and mepolizumab dose, ensure Tdap has been administered > 2 weeks earlier.
- 4–6 weeks: repeat quantitative IgG (to confirm trough) and vaccine titres (tetanus/diphtheria); assess asthma control (ACT/C‑ACT).
- Every 3 months: CBC, eosinophils, FeNO, IgG level, review infection log.
- Every 6–12 months: repeat HRCT if baseline abnormal; repeat lymphocyte subsets if clinically indicated; screen for autoimmune cytopenias (CBC, retic count).
5. Key References (selected, ≤ May 2025)
| Topic | Reference |
|---|---|
| CVID diagnostic criteria | ESID/ICON Consensus (Bonilla et al., JACI 2016; Cunningham‑Rundles et al., Blood 2010). |
| IVIG efficacy & “wear‑off” | Gill & Betschel, Allergy Asthma Clin Immunol 2018; Borte et al., Clin Exp Immunol 2017. |
| Mepolizumab in pediatric severe eosinophilic asthma | Jackson et al., NEJM 2022 (MUPPITS‑2); Günther et al., Pediatr Pulmonol 2024; Ortega et al., Lancet Respir Med 2014. |
| Asthma biologics – dupilumab, omalizumab, benralizumab | VOYAGE (Dupilumab) – NEJM 2021; DREAM/MENSA (Benralizumab) – Lancet 2012; TEZEPELUMAB (TSLP) – NEJM 2021 (adolescents). |
| Vaccination in immunodeficiency | CDC 2025 Immunization Schedule; AAP 2025 Guidelines on Tetanus, Diphtheria, Pertussis. |
| CVID‑related pulmonary disease | Lee et al., Ann Allergy Asthma Immunol 2022; Hirsch et al., J Allergy Clin Immunol 2021. |
| Genetic testing in CVID | Lee et al., Ann Allergy Asthma Immunol 2022; Paquin‑Proulx & Sandberg, Front Immunol 2014. |
| Eosinophilic asthma biomarkers (FeNO, eosinophils) | Dweik et al., J Allergy Clin Immunol 2023 (FeNO thresholds). |
| Guidelines for asthma in children | GINA 2024 recommendations (step‑5 biologic therapy). |
Bottom line
The 6‑year‑old most likely has CVID with a severe eosinophilic asthma phenotype.
Start IVIG replacement immediately (with anti‑IgA screening), give Tdap (and appropriate pneumococcal vaccines) before the next IVIG dose, and initiate mepolizumab for his eosinophilic asthma. Follow with a structured lab work‑up, imaging, and close multidisciplinary follow‑up to monitor immunologic status, infection burden, and asthma control. Additional options such as prophylactic antibiotics, airway clearance, and alternate asthma biologics remain on the table if control is suboptimal.
Rubric criteria
65.7%
| Mentions that Common Variable Immunodeficiency (CVID) has lower IgG accompanied to reduced IgA and/or IgM concentrations. | Yes |
| Mentions that patients with Common Variable Immunodeficiency (CVID) have low IgG concentration (<2 standard deviation below age-appropriate references) for 2 measurements 3 weeks apart. | No |
| Mentions that the patient’s medical history, recurrent severe sinopulmonary infection with 10 ER visits and 3 hospitalizations, is indicative of a possible Common Variable Immunodeficiency (CVID) diagnosis. | Yes |
| Mentions that the patient’s non-immune Tetanus status is indicative of a possible Common Variable Immunodeficiency (CVID) diagnosis. | No |
| States the likely diagnosis is Common Variable Immunodeficiency (CVID). | Yes |
| States that the patient’s age, greater than 4 years old, meets diagnostic criteria for Common Variable Immunodeficiency (CVID) diagnosis. | Yes |
| Recommends completing a repeat quantitative serum immunoglobulin panel to confirm low IgG concentration with a second measurement greater than 3 weeks apart. | Yes |
| Mentions that patients may be diagnosed with Common Variable Immunodeficiency (CVID) with only one serum study if the serum IgG level is very low (<100-300 mg/dL depending on the age). | No |
| Mentions that for a Common Variable Immunodeficiency (CVID) diagnosis, no secondary causes of hypogammaglobulinemia can be present. | No |
| Mentions memory B-cells may be reduced in Common Immunovariable Deficiency (CVID). | Yes |
| Recommends completing a B-cell subset analysis by flow cytometry for immunophenotyping of B-cells and to rule out X-linked Agammaglobulinemia. | No |
| Recommends completing a T-lymphocyte subset analysis by flow cytometry to evaluate for T cell deficiency. | Yes |
| Recommends testing an adequate immune response to previous vaccinations by measuring specific antibody titers to protein antigens (such as tetanus toxoid, diphtheria toxoid). | Yes |
| Mentions testing an adequate immune response to previous vaccinations by measuring specific antibody titers to polysaccharide antigens (such as pneumococcus). | Yes |
| Recommends infection avoidance (hand hygiene, drinking treated water, respiratory protection) as the first step in the treatment plan. | No |
| Recommends IV immunoglobulin replacement therapy as the first medical intervention to treat Common Immunovariable Deficiency (CVID). | Yes |
| Recognizes that the patient’s immunoglobulin levels will need to be monitored every 6 months, with appropriate dose adjustment based on IgG production and weight. | No |
| Highlights that the patient will need to be screened for anti-IgA antibodies to prevent anaphylactic reactions to the treatment. | Yes |
| Recognizes that IV immunoglobulin therapy (IVIG) alleviates the state of chronic immune activation in CVID, helping improve cellular immunity. | No |
| Recognizes that IV immunoglobulin therapy should help to decrease the recurrence sinopulmonary in infections in this patient. | Yes |
| Considers antibiotics for infection prophylaxis given patient’s notable history of recurrent sinopulmonary infections. | Yes |
| Recognizes that the patient is not up-to-date on 4-year vaccines. | Yes |
| Recognizes that the patient’s upcoming vaccine schedule may need to be modified to delay or spread out the administration of live vaccines (MMR, varicella). | No |
| Recommends educating the patient’s parents regarding the risks and benefits of live-attentuated vaccinations and reaching a joint decision together. | No |
| Recognizes that autoimmunity, and specifically autoimmune cytopenias, are the most common noninfectious complication of Common Immunovariable Deficiency (CVID). | No |
| Recommends drawing a complete blood count with differential to screen for cytopenias (thrombocytopenia, anemia, neutropenia). | Yes |
| Mentions completing a repeat eosinophil count in lab work. | Yes |
| Mentions the patient’s history of diffuse wheezing with elevated blood eosinophils (last visit) are indicative of eosinophilic asthma. | Yes |
| Classifies the patient’s asthma as severe and poorly controlled given frequent exacerbations despite being prescribed the combination of Advair and montelukast. | Yes |
| Notes the anti-IL-5 monoclonal antibody, Mepolizumab, is indicated for severe eosinophilic asthma in this patient’s age group. | Yes |
| Recognizes that of the three biologics (Dupilumab, Mepolizumab, Omalizumab) approved to treat severe asthma in patients 6 years and older, only Mepolizumab has an FDA label indicating eosinophilic asthma. | No |
| Recommends mepolizumab to treat the patient’s severe eosinophilic asthma, recognizing that an eosinophil count 300 cells/μL or greater in the past 12 months is sufficient to begin treatment without a repeat lab value. | Yes |
| Mentions the typical dose of mepolizumab for a 6 year old is 40 mg subcutaneously every 4 weeks. | Yes |
| Recognizes the need for parent education to administer an injectable biologic to a 6 year old and maintain medication adherence. | No |
| Mentions recent literature suggest that interleukin 5 (IL-5) receptor blockage can be safely co-administered with IVIG in patients who require both therapies. | No |