Hacker News Reader

Comment by danielhanchen | Hacker News Reader

Hey there! Yes that's exactly what I thought! I was writing up a blog post at https://colab.research.google.com/drive/1AOuhMVILE06mD-Go7-R... which step by step shows every change I did plus gave timings / mem savings.

I'll post it once its completed if you're interested!

View on HN

Thanks. There's been a few substantial and laudable efforts which are much appreciated but what I'm suggesting is an actual continuous infrastructure, like how those benchmarking sites have software for people to run on their machines that phone home so that people who make new benchmarks or new variations can submit them and refine the results.

For instance, are any of your prompting tests in say, Korean? What about winograd schema challenges in languages other than English? Japanese for instance, comes with its own unique set of context ambiguities that do not appear in English. I'm sure dozens of languages are similar. It'd be nice to have user contributable tests to cover the breadth of use cases here.

A great optimization that moves a score let's say from 95% -> 5% on "winograd-persian" may be fine or may be a show stopper, depends on what you care about.

That's why it's gotta be normalized, future-proof, and crowdsourced.

Ohhh interesting! So like ML Perf or Faster CPython https://github.com/faster-cpython/benchmarking-public?

Sure. This thing I propose might already exist in a nascent form.

I haven't checked in a few months and the way things are moving now...

Interesting!! I'll talk with my brother about this - sounds very cool!

Hey Daniel, I would love to help out on this. I'm learning about LLMs and this benchmarking project sounds like a fun way to further my knowledge and skills. I sent you a message on LinkedIn.

Cheers!

Cool lets chat on Linkedin! :)

Reply on news.ycombinator.com