Hacker News Reader

Part of serialization is getting to pick how you encode numbers. And, one should do so with a good match on the range or required precision. I'm not sure double values are required here or necessary.

> Text compression of number data is extremely efficient

Of course it isn't. It is arguably one of the worst there is. It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor. But those constraint would make the whole discussion kinda silly.

Serializing text, is after all binary encoding data. If you follow this with compression algorithms, you'll likely not end up too far away in either case.

Just take the example you gave. "A 5 digit integer is 3 bytes less than the double to store it". Yes, true. However, a 5 digit integer can also be represented by 12 bits.

And you can even pack them if this is all you really care about (there is a good reason why you wouldn't want to do this, as word alignment is much more important, as you would be able to directly map data to memory allocations, but let's for argument sake say that storage was king). For each 5 digit numbers as a text file, you'll need another byte to encode the line break or delimiter. I.e. 6 bytes instead of 12 bits, or 4 times wasted space. For a million values, 5.7 MiB instead of 1.4 MiB.

> It can only ever by more efficient when representing a very low subset of values and the alternative binary representation is exceedingly poor.

In almost all cases in practice one is only serializing a very low subset of values. A 1 TB file, no matter how efficient the encoding, could only contain serialization of less than 0.00005% of the possible IEEE 754 double values.

That is of course true. My point was to highlight the kind of situations where straight serialization of text would be somewhat closer in required storage, when compared to similar serialization of number values. That serializations of such numbers would only encode a subset of the values it can represent, I would say, is besides the point. It is after all how number representations work.

To reiterate my original point: It is never the case that serializing text that represents numbers is the most efficient representation. The posts here seem to either suggest or think this is the case. And this was only the size consideration. It is even worse in terms of processing. You need to do O(n) operations on the length of text data just in order to decode the values. A step that is entirely avoided with a correct binary format.

Just a anecdotal example here on the computational aspect. Maybe 10 years ago or so, I used an OBJ loading library that would straight up decode 3d models represented as ASCII text. I wrote a simple de/serialization program, just 70 lines of code, to both dump and read dumped data. The load speed for a complicated scene went from 38 to 0.7 seconds.

This isn't to say there aren't good use cases for text formats. However, it is always a tradeoff for human readable data, at the expense of efficiency (storage and computational). The former is much less so when using a simple compression step. Very often this tradeoff is acceptable.

> It is never the case that serializing text that represents numbers is the most efficient representation.

This is false.

Can you elaborate? GP's argument is verbose with a lot of explanations and examples that I agree with. What text encoding and how would it be more efficient?

When the text file is a source of data rather than a reflection of the data that you want to operate on, and particularly, when a lot of that data has been introduced by humans (or based on human input) in some suitable units, it very often happens that many numbers that are generate are short, shorter than a regular double.

I know it is difficult to believe, so, don't; I was not trying in my comment to say that text is better everywhere always; of course not; I myself let users have text as a reference while creating a one-to-one representation in a well known binary format for speed of access. When you really have huge amounts of data, each use of the data usually requires filtering/selection/manipulation/new memory layout/etc. that makes the original optimizations in the binary format useless and complicated to maintain, and more difficult to allow for low friction use by anyone even 10 years in the future. I consider the textual representation as the source of truth; but the binary allows me to do the filtering faster, although in many occasions it takes more space. Yes, one could choose better encodings for the binary data (the 12-bit referenced above, etc.) but the complexity of the code increases and it becomes even more obscure to the humans that have to use (sometime debug) on that data.

I understand is difficult to believe. Well, do not believe. Probably my experience is very niche.

Seems we then spoke past each other. I did not focus on practical aspects or other such tradeoffs. Just, what is the most efficient way to represent numbers. If your argument is that for many practical applications, text is good enough. Then, I don't disagree.

But, if the argument was that text is a more efficient way to store numerical data. Then, there is no need to bring beliefs into this discussion. It is just not the case. Serialisation of numbers have been solved by engineers many decades ago, and there are many to chose from based on extra information on what the numbers should represent. One such representation is actually useful as a lookup table for text. But, to suggest that breaking up a single numerical value into individual lookup table values, is somehow more efficient than storing the original value with the right choice of encoding it... I don't know. I don't care all that much. But, surely you can see my argument?

The whole thing is a tautology. For the sake of silliness, if ascii encoding numbers somehow was the most efficient way of encoding numbers (it isn't, but, let's say it was) then it would only then be exactly as efficient as "choosing the best way to serialize a number". But, surely, whatevever that data was, would then be most efficiently represented by the ascii representation. So, we should store the ascii representation of the bytes that store the lookup table values, as encoded ascii values. Doesn't take more than a handful iterations before you run out of storage space available on the planet.

I agree with everything; for users that are experts in their domain (not computer science), if you don't use text in the frontend that they can parse with little (but very inefficient) code in their scripting language of choice, then, they will be reinventing some even more difficult/complex/yet-another way to handle text that you will have to support in the backend for performance - and in the backend I choose binary formats (although my experience, without focusing too much in any other aspects of performance than retrieval, because most uses with large amounts of data require specific filtering/reordering/new memory layout for efficient algorithms, etc.). And, counterintuitively, when you do that, and you have hundreds of files which are hundreds of megabytes where some columns are integers, and you see the size of your binary data you wonder... why is my binary data few times larger?...

That is a good point. It is a good observation that just binary encoding numbers does not mean you have an efficient serialized representation. And, that sometimes that can be so bad that even encoding numbers as serialized text beats it out.

I think what threw me off was the context of this discussion, which was 3MF file format. For such cases, the usefulness of humans opening up the data in text editors is somewhat contrived. I believe that tradeoff for the 3MF format was a mistake, and will likely hinder broad adoption. I'm however mostly concerned with the extra processing required in order to decode the human readable files. We're talking a likely speedup of a factor of 50-100. For models that push this limit, imagine a file that takes 60 seconds to process. This isn't entirely unreasonable for a model with insane level of details. If they instead had gone for a binary file representation, the load time would likely be less than a second. Failing to recognize this tradeoff for human readable text... is unfortunate.

Hmm, I think I see your point. So, for example, if most numbers were something like 0.1, 0.25, 3.25 and the like, they would take 4 bytes in most cases, whereas storing doubles would always take 8 bytes.

We don't want to take float32 because of the inherently bad precision of working with them, so storing as text is better.

Thanks for the explanation.

If you’re reading or writing a file format, whether it’s text, floating point or some other encoding, you don’t have to store the data in memory the same way when you work with it.

…which frees you to write your data to disk however you want.

And given that freedom to choose any serialization format, you definitely should not manually be figuring out every byte you write — you probably write an abstraction, so wherever complexity there is should be completely transparent because you unit tested the hell out of your sterilization routines.

…which to me makes the whole complexity point moot, and which also makes the whole “text is better for memory minimization” argument also moot.

I do think text formats have a place — primarily readability and interoperability — but to choose it for purely reducing your file sizes sounds insane to me.

of course... I never thought that text files to reduce file sizes is sane... Maybe because I don't want to write in a wishy-washy way as if I was stepping into a mine field I came across like that [look at my first comment... I think I started as "I know it is counterintuitive" - I wanted to point point out the binary (as naturally interpreted: storing floats/doubles) does not necessarily mean smaller file sizes for files generated by humans.

There is a good reason for why one would want to store data the same way as when you work on it. It removes the need to process the data itself. A common approach is to follow it up with a fast compression algorithm when storage space is important.

The difference in post-processing when having to decode text is around two orders of magnitude, which is very significant. In the case of the 3MF format in question in this article, I find the tradeoff for human readable files to be a very strange one. The benefit of that at the cost of say waiting a minute to load a file that would otherwise take a second, is a puzzling one. But, it's all a tradeoff after all. If humans work with the files, it is a strong argument work with text. When software stores and loads files, it will likely be a poor choice to store data as text.

But, you are right. The file size consideration is not all that relevant. However, what was mostly discussed was the argument that serializing text that represents numbers is more space efficient. Which as I mentioned a few times, can only be the case when the straight serialization of the numbers is done poorly.

I too am curious. Not that it really matters. The statement itself is a tautology. There is no "opinion" here. Just straight logic and math. Text serialization is a way to serialize numbers. It's just a very poor and roundabout way made for human readable text. The idea that such a roundabout way of storing numbers as other numbers should somehow make things more efficient than storing the actual numbers... Is kinda silly.

In my original comment to which you replied I wrote "The best of both worlds is to have a text representation coupled with one-to-one binary encoding;" for the reasons I wrote in that comment. I am aware of all your points on ser/des of text and I fully agree with them.

Also, I was constraining the use case for text to data that is either the input of humans, or generated from those inputs.

For data that is gathered from devices, etc. I don't think a text representation would be of much use.

Also, between a set of parameters entered by a human, and an audio or a video, there are a lot of use cases in between (all of the points in space of N-sided polygon after a human entered 4 or 5 parameters, ...., generation of statistical data from parameters or other inputs, ...). I think, as a rule of thumb, that the less close the data is to the inputs of humans, the less sense it makes to store the textual representation.

The computation burden on these isn't so stiff as to force optimization, just FYI. Even high count models load and process on a reasonable machine in a reasonable amount of time.

> much more space than storing the text

In other cases it’s the opposite. Convert a model from millimeters to inches, and that 1.0/25.4 multiplier gonna cause decimal representation to be much longer than 8 digits. Mechanical engineers are doing things like that pretty often.

> when humans enter data in some specific units

I don’t think it’s a reasonable expectation for a human to type 100 MB, or 1GB of numbers. People do work with triangle meshes that large. These models were made by specialized hardware or software tools, not by entering numbers. It’s the same story with images: they have too many megapixels now, and people aren’t entering millions of RGB numbers. Instead, people using scanners, digital cameras, and Photoshop to deal with these volumes of data.

> Text compression of number data is extremely efficient

The most expensive part is not compression, it’s printing and parsing. That code is very hard to vectorize or parallelize.

When software reads or writes large continuous buffer of binary data like std::vector in C++, the throughput is often limited by physical disk. On many modern computers that’s at least 100 MBytes/sec, often much faster.

I think it’s borderline impossible to print or parse float numbers that fast. Especially given that in 3MF, there’s some XML producer/parser down the stack who needs to escape/unescape these strings.

This is an issue that shows up often. ASCII is used because the performance cost is less than the benefit of guaranteeing that the exact value retrieved is the exact value stored.

I often wonder why BCD file formats are not used.

The problem there is using inches.

While many agree, and I for sure do, fact is manufacturing in a lot of the US is a mixed unit scenario at best.

Much of Additive uses SI units, fortunately. One of the bigger players remains with Imperial units, unfortunately

I do sales, service, support in this industry. One of the more hilarious queries I got recently was, "Does your machine support freedom units?"

We all had a nice chuckle, and yes that shop works mixed and it's a pain point for them being a mix of older school and new school tech and people all over the map. They don't see a transition in the near future either. Too much of the work remains Imperial as does equipment that has a long service life ahead.

The real problem lies in formats and software that makes assumptions about units, and or makes it all difficult to manage, or input / extract mixed unit data.

I see meters used fairly often too. Not the same non integer conversion, but the other problems do remain.

I understand all your points (I have implemented all sort of parsers, even in mmaped memory regions, jumped around text parsers to not look for '\0' or do in-place changes, etc...) and I agree with them.

I think text is a good format for the "truth"; a binary one-to-one representation of that text to avoid floats parsing is the best alternative. Counterintuitive as it is, many times, the text representation takes less space (sometimes there are huge lot of little round values, for examples 0s or 1s or 2s, in some columns).

in the inches/mm case... my choice would be, if we can support units, then write the file data as given by the user - use that data as the ultimate source of truth; use a more efficient/standard representation in the backend and have processes that test that you can re-generate the file from the user as accurately as possible with the backend data. But of course, it depends on the exact case and how much users expect to edit by hand their files.

This is best practice, IMHO. Consider the input data immutable and work from there.

Maybe creating a standard textual hexadecimal-double format could be a good middle ground. "a.2aE3b" easy to include in text and to parse and maybe easy to read once used to it.

   hex   => dec
   a.8   => 10.5
   f.88  => 15.55
   9.e   => 9.875
   11.8  => 17.5
   a.8E1 => 168  (15.5*16)
   a.8E2 => 2688 (15.5*16^2)

Rather than constraining to floating point, why not just use a 4-bit BCD format with a zeros place symbol and terminator symbol? That way you can express exact values efficiently. Of course it becomes unwieldy when specifying extreme values (such as 10^100), but for most reasonable values (10^20 to 10^-20) it beats both floating point and ASCII for size and speed.

I acknowledge that in a correctly designed binary format given data may in some cases be more compact than when represented as compressed XML.

I ask myself though will it practically matter in the 3MF vs STL case, given 3MF uses a different approach to represent geometries that produces both more accurate and far more compact results than STL for a similarly high precision[0]?

[0] https://hawkridgesys.com/blog/3mf-the-file-format-for-the-fu...

You could store decimals (either power 10 or power 2) with a given precision. Essentially, when storing floats as text, you are already doing that.

The binary stl format uses float precision. But the point is much more that parsing takes CPU cycles.