Why not build a tool for every file format that simply dumps the contents of the file in a human-readable way. Then feed the output through "diff". That approach would sound more "unix-ey" to me.
Great idea! That would work well for a lot of stuff, but I don't think it would scale to diffing, say, two 4Gb iso files. Also, some structured content has user-hidden IDs which can help a differ, but they probably wouldn't show up in a "human-readable" representation.
The human-readable representation can be as long as you want. Also, one could embed checksums in the output, so that if non-displayed contents differs that would trigger an output by the diff command (granted, this way it is not so human-readable anymore because of the checksum, but it would still be a more modular approach).
I don't think diff will perform well on two 4gb inputs. I agree there should be a modular approach, but I don't think flat text is the best lowest-common-denominator. Your proposal would certainly work for a lot of cases though, and it is very unix-y!