How did I convert the 33 GB Dataset into a 3 GB file Using Pandas?

This is err.. Interesting. There are a couple of issues and I think it would be good if the author interrogated some of the tradeoffs made here.

I'm not sure the GC thing is correct, and certainly won't work in an interactive shell, you're collecting the garbage when there's still a reference to chunk so it won't free that memory, but (AFAIK) since you're in a loop, the next assignment to chunk will remove the previous assignment freeing the memory automatically anyway. So (I could be wrong about this) it works, but not for the reason you think.

Also,

> So we have to convert it to ‘float16’ or ‘float32’ to minimise memory usage.

Well, I mean, technically float16 does use less space, yes. But it also does a few other things you might want to mention - like reducing the maximum range of your values to +/-65k.