You are almost certainly passing these through tools (even built into the browser) that are doing extra processing to make it more readable.
Let's assume, for example, CBOR ends up taking over JSON. Do you not think browsers wouldn't have a CBOR parser to make it more readable?
This isn't bolt vs welding, this is bolt vs bolt with a washer. Yes there's a small extra step, but not some sort of insurmountable hurdle.
There's no reason a binary format couldn't be as easy to read as a JSON format. All you'd really need is a "binary->json" gui and you're off.
It would take a little effort to make such a tool and you could have it integrated into every browser.
It's not like something like that is unprecedented even, After all, there's no browser tool out there that's showing you the gzipped resource on a compressed endpoint. It's always already doing the step of gunzipping. binary->Json would similarly be just an additional step the browser could do in the tools.