HNHacker News
TopNewBestAskShowJobs

florianstandhar

50 karma · joined September 22, 2026

https://x.com/airesearch12
submissionscomments
florianstandhar··on Show HN: JevBench, a reproducible benchmark for typed decision models
Happy to — please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set. I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination. We can call API hosted models though, we'll just flag the leaderboard entries appropriately.
florianstandhar··on Show HN: JevBench, a reproducible benchmark for typed decision models
Thanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree
florianstandhar··on Show HN: JevBench, a reproducible benchmark for typed decision models
Agree the coverage question is real. Today it's text decisions (intent, routing, extraction, scoring, harder multi-constraint cases). We're working on screen/computer-use and image decisions (a noindex preview exists) because that's closer to where Jev-class models will actually be used. Suggestions for task families are very welcome as GitHub issues.
florianstandhar··on Show HN: JevBench, a reproducible benchmark for typed decision models
Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.
florianstandhar··on OpenAI is about to eat Jev's lunch – Arcturus Labs
maybe open source even eats Jevs lunch first

see here: https://news.ycombinator.com/item?id=49800574