klemenvod

*A lot of our effort goes into creating benchmarks to objectively evaluate our model’s performance, and we’ve open-sourced them here: https://github.com/medaks/medask-benchmarks

We’ve developed both an OSCE-style diagnostic benchmark and a medical triage classification task. The 12% improvement over o3 comes from our triage benchmark, which evaluates models on clinical vignettes across emergency, non-emergency, and self-care classifications.

Medical triage in our context means whether symptoms require emergency care, urgent care, or can be managed with self-care. This matters because LLMs are increasingly becoming the “digital front door” for health concerns—replacing the instinct to just Google it.

Getting triage wrong can be dangerous (missed emergencies) or costly (unnecessary ER visits).

We’ve open-sourced TriageBench, a reproducible framework for evaluating LLM triage accuracy. It includes:

- A standard clinical dataset (Semigran vignettes)

- Paired McNemar’s test to detect model performance differences on small datasets

- Full methodology and evaluation code

GitHub: https://github.com/medaks/medask-benchmark

As a demonstration, we benchmarked our own model (MedAsk) against several OpenAI models:

- MedAsk: 87.6% accuracy

- o3: 75.6%

- GPT‑4.5: 68.9%

The main limitation is dataset size (45 vignettes). We're looking for collaborators to help expand this - the field needs larger, more diverse clinical datasets.

Blog post with full results: https://medask.tech/blogs/medical-ai-triage-accuracy-2025-me...