Based on that discussion, it definitely sounds like some sort of bug is hiding. Perhaps run some evaluations to compare perplexity to the standard implementation?
Edit: looks like there's now confirmation that running it on a 10GB VM slows inference down massively, so looks like the only thing strange is the memory usage reading on some systems.