Benchmarking Autonomous AI Doctors: Real-World Validation Against Board-Certified Clinicians in Virtual Acute Care

Highlights

• Autonomous, multi-agent LLM-driven AI system demonstrated diagnostic and therapeutic performance comparable to board-certified clinicians in 500 real-world virtual acute care cases.
• The AI system achieved 99.2% guideline-concordant treatment compatibility and zero clinically unsupported (hallucinatory) recommendations.
• Expert review found the AI outperformed human clinicians in following up-to-date guidelines and managing complex, atypical cases in over one-third of discordant cases.
• AI-generated clinical documentation exhibited high semantic alignment with human notes, despite differences in language and structure.

Study Background and Clinical Challenge