Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.
Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.
Good luck and godspeed!