Hacker News Reader

Comment by tgsovlerkhgsel | Hacker News Reader

A better fix could have been switching to some other (binary) serialization protocol (BSON or similar) - the problem here doesn't seem to have been the size of the history itself, but the exponential growth of the escaping.

View on HN

Or, y'know, just

    jo.put("previous", previous);

Rather than

    jo.put("previous", previous.toJson());

Presumably the reason they didn't do this is because they couldn't handle a recursive schema?

This is reminiscent of some APIs I've had the great pleasure of needing to integrate with, where a SOAP API would take XML containing Username, Password, Method, Data... where Data was a string, itself containing barely-documented XML, but serialized as a string before being placed in the XML. You'd think that if they knew how to stand up a SOAP API, they'd know that XML is literally designed to be nested. Alas, such was not the case. I shudder to think about their security under the hood...

I remember dealing with a small ISV that was told to make an RPC API, so they used SOAP (via WCF). That would have been fine, except that every function took a single "string" parameter that was invalid XML that a normal parser couldn't read. You had to use their hand-rolled parser.

And then the whole thing was "encrypted"... with a static RSA key. I don't mean they used a stream cipher based on a hard-coded key, which would bad enough. No, they used the RSA keys directly for bulk data encryption.

I flat rejected their solution, then was overruled as being "too picky", and this monstrosity went into production.

In the post they say they didn’t do this bc they’d just be trading long strings for deeply nested objects.

Deeply nested objects grow linearly with the size of history, not exponentially.

Agreed, I'd take deeply nested objects, over deeply nested strings (of strings...) of objects.

For the record I do agree that the schema

    data Screen = Screen {
        previous :: Maybe Screen,
        ...
    }

is inferior to what they eventually had,

    data ScreenHistory = ScreenHistory {
        previous10 :: [Screen],
        current :: Screen
    }
    -- Screen itself is nonrecursive

But I do think that storing JSON as a string was a weird antipattern in the first case, smacking of a premature optimization.

That would have also fixed it, though IDK what is necessarily "better."

Recursively serialized JSON, XML, whatever is a code smell in general.

(Note that outside certain examples like this, in general BSON is not smaller than JSON. You'd need something like protobufs. )

Recursively serialized data isn't a smell. How would you even serialize a tree structure without recursion?

A recursive data structure is fine, if you implement the serialization in a sane way. The escaping of your escaped strings containing other escaped strings seems like a code smell.

like this basically https://simdjson.org/api/0.4.0/md_doc_tape.html

You’d still be doing quadratic work over a user’s session but perhaps navigation is infrequent enough for it to not matter.

sighs in bencode

Reply on news.ycombinator.com