You're not going to do better with pandas or similar tools. If it can't fit in memory, it's going to be painful. SQLite is the least painful in my experience, and it sets you up for working with the data in a proper DB like postgres or similar for when you get fed up with the memory constraints.
I wouldn’t use pandas in that situation either.
In the original article, they were trying to read a 10GB file into 30GB RAM.