People are working on smaller unixy tools to do fewer things well, but at least for the main codebase we are limited by go itself. Making binaries smaller is not simple at all...
You might be interested in checking out the rust implementation: https://github.com/rs-ipfs/rust-ipfs
Im not actually sure how well it would work for what youre after, but it is an option!