Hacker News Reader

Comment by vlovich123 | Hacker News Reader

Isn’t that almost always true unless the format is specifically designed to take the same amount on disk and in RAM? If that’s the case I think typically it means the file on disk is larger than it needs to be. That’s because you have to convert the file to some kind of in-memory representation that’s easier to work with to manipulate it and what not which requires structure that isn’t necessary at rest.

Can you give an example where the file on disk is going to be larger than what it is in memory? Provided you’re not just reading it and working with it as an opaque binary blob.

View on HN

Depends on the data, obviously; but record format data should be compact in memory in ways that text isn't. My favorite example is the meg's of commas and quotes that just aren't needed in memory.

So, my examples are largely csv files. JSON and xml are the same, in many ways.

The big curve ball will be text heavy data. But all too often textual data is categorical, such that even that should be smaller in memory.

In large, this is why parquet files are a ridiculous win in space.

My favorite example is the meg's of commas and quotes that just aren't needed in memory.

Depending on the language and internal represenation, for any comma and quote not needed (which for a JSON document with, say, an object without nesting, is just 5 bytes per entry), you might get your strings doubled in size (because e.g. the language converted your fitting into 8bit utf-8 strings to its 16bit native string format), or have some huge constant boxing overhead (in say, Python), and several other fun things besides...

Right, is why I said strings can be a curve ball. But most of the data I see can avoid that fairly easily.

Coming back to this, I was curious and decided to challenge my assertion some. https://taeric.github.io/challenging-my-filesize-intuition.h... is a few examples where I was fishing around. I didn't find any where my assertion is not true.

Curious if you have any easy examples to look at that do show the opposite.

XML would probably fall into that category.

But I do agree with your point more generally speaking.

I think it comes more from the thought that human text readable format is usually more verbose than binary representations.

Reply on news.ycombinator.com