It’s nice to see this posted and interesting that it’s in tensorflow. I wonder for how many years the capture was already broken but not just posted about publicly.
They mentioned that in the last investment too. That seems like marketing to me as no one is doing bleeding edge research outside of the NVIDIA CUDA run ecosystem.
Good post. I’ve been thinking about doing offline testing of LLM tasks a bit these days and have come to the conclusion that old school testing is the best until more mature features can be developed. Specifically, I mean running a power analysis to determine sample size, random sampling based on that and then running tests like a z test to see if there is a difference and between what bounds. Tests are expensive and I wish there was a better way for realizable offline evals.
I am actually not surprised by JAX converting better to XLA. Also deep respect for anybody in this space as their is lot of complexity (?) to deal with at the framework and compiler level.
I’ve use it since last year as a paid subscriber. I like it because of the technical nature as it will help you know the exact steps on how to get something done. I also use it for random things like bouncing ideas off or to enhance my knowledge retention of a subject.
Interesting. I’d be curious if someone solved this at scale for good cost. Double call seems expensive to me when alternatives can do it in one but are still quite costly.
I remember a friend in college telling me a story like this once: some walled off room with wires running in it was opened up and they found an old sever running behind it. No idea what it was doing. This was in 2001.