The problem start with knowing the encoding (base64, for small sizes sometimes hex, for one or two bytes sometimes arrays with all there ambiguities).
Then that you can't easily differentiate between a string a bytearray anymore (through you often shouldn't need to).
Then that it becomes noticable bigger, which especially for large blobs is a problem
Then that you always have to scan over this now larger large chunk to know it's end. (Inserted of just skipping ahead).
Then that you have to encoded/decide it which might require some additional annotations depending on serialization library and can imply other annoyances.
The later point can also in some languages lead to you accidentally using a base64 string as bytes or the other way around.
Well I guess that was enough ranting ;=)
JSON works where you need simplicity on the verge of being dumb, and human-readability.
When I was developing a three component system, protos were not working for me. Too many schema changes and implementation differences to make the reduced binary size worth it. This was partly because the best protobuffer implementation in C (nanoPB) is just the hobby project of some dude, but mostly because coordinating schemas was annoying.
Have you tried BSON or Protobufs to solve your annoyances? How did it go?
Outside of Mongo, I haven’t seen BSON used anywhere.
CBOR however, is up and coming. Starting to look like it may be a first class citizen in AWS someday. On the IoT side they are already preferring it over JSON for Device Defender.
Practically, base64 encode a video file and tell me how exactly you're going to allow the user to seek to any place they like within that video using only common libraries on common platforms... Theoretically easy, practically hard enough nobody does it.
2. Do whatever you said.
chunkOffset = byteIndex/3*4
base64decode(base64data[chunkOffset:chunkOffset+4])[byteIndex%3]
This gets you the byte located at an arbitrary index of the original buffer while operating on the base64 data. 2537012 Nov 6 10:10 test.tar.hex.gz
2954608 Nov 6 10:03 test.tar.b64.gz
All the caveats, test with your data, etc.What would be nice is an email/http resilient compressed format that could be stuck inside of json strings that can be easily recovered.
wget http://mattmahoney.net/dc/enwik8.zip -O - | gunzip | head -c 1000000 | gzip -9 | wc -c
355350
wget http://mattmahoney.net/dc/enwik8.zip -O - | gunzip | head -c 1000000 | base64 | gzip -9 | wc -c
528781
base64 is 48.8% larger after compression on english text, whereas it would only be 33.3% larger without compression. The reason is because compression finds and eliminates repeating patterns, but base64 can make the same input data look totally different depending on which of the 4 input alignments it has.A general-purpose compression scheme can easily detect when its input is Base64, so I don't see why Base64 should be particularly hard to compress, in principle at least.