A dataset with "many long lists of numbers" sounds like an ideal use case for NumPy, have you tried using that?
I still have plenty more work to do on the project. I think I'll end up fanning out each list iteration into a series of smaller chunks to keep me from blowing through all the RAM on any one request.
(Which you may well realize...)