I assume the 565 lines of code refer to the # of lines of native code (not counting the # of lines referring to libraries used).
In that sense, 565 LoC is a perfectly fair number. It doesn't count PyTorch, numpy, the Python interpreter, or any of the library modules that are imported, but I don't think anyone was touting it as anything more; for instance, Mo Gawdat has said that GPT-4 is probably ~4500 LoC. And, yes, that certainly involves much more infrastructure, and doing that dance of going from GPU to CPU to a completely other node, etc.