SOTAVerified the open verification layer for ML research

sotaverified.org

2 points·uberdavid··4 comments

Cool stuff, the next will be a place for AI agents to publish research.

Thank you! The progress on research agents is exciting, but understanding what papers are reproducible on different datasets and architectures is often the bottleneck.

I think we are leaning towards autonomous research, and if the concept of agents sharing their research work or like any sort of information on a public domain where other agents or researchers are doing a similar kind of work, will help setting up a research community of agents.

And about the reproducible part, we can opt out in agents sharing repos and sandboxes.

Hi HN, I'm David, an ML researcher at Meta. I built SOTAVerified as an independent project after Papers with Code shut down last year and took 575k papers worth of benchmark data with it.

SOTAVerified inherits that dataset (658k papers, 257k code links, 59k benchmark results) and adds what PWC never had: a verification layer. Anyone can submit reproductions with hardware specs and run logs, and the verification score updates immediately.

I've been doing reproductions myself on my RTX 3090: Fort et al. 2019 deep ensembles and Havasi et al. 2021 MIMO so far, with wandb logs linked. The goal is making this the ground-truth registry that both researchers and autonomous research agents can query.

Stack: Next.js, PostgreSQL, Vercel, Railway. Open source: https://github.com/sotarepro/sotaverified

Built for: - Authors who want to claim their papers and submit official metrics - Researchers who want to understand the SOTA techniques for a task - Autonomous research agents to check if a result reproduces before investing GPU hours

Would love feedback from the HN community. What features would make this useful for your workflow?

Reply on news.ycombinator.com