See the msgpack-cli example at the bottom. Say you have a function that returns a generator for tokens in Python. You would need another function that builds objects out of them. How do you customize how objects are being built? A class makes that simpler because each of the methods are extension points you can override.
But yeah, a token stream would be much appreciated.
> - Why would I ever design the JSON library to be extendable at the tokenizer level?
For instance if you want to skip past objects. That's what we're doing for instance for unwanted input data. That implicitly also prevents hash collision DOS attacks because we never build hash tables for things that we don't want. It also gets rid of the suspension of execution when a garbage collector runs and cleans up a nested structure. I can make you tons of examples where the "build a tree, reject a tree" approach destroys an evented Python server.
> - The 2GB JSON example is just weird. If you care about such use cases, you a) most likely have a limit on data size at webserver level, b) use proper formats for handling that size of data (I really doubt there's no better data representation once you get to GB sizes).
Try using a streaming JSON API like Twitter's firehose. Most people just give up and use XML because there is SAX or if they go with JSON they newline delimit it because most libraries are horrible or useless for parsing streamed JSON.