Swift is a big part of the app binary size bloat. It creates about 3-4x the amount of binary for about the same lines of code count as objective-c. Generic specialization happens everywhere in swift and using optionals takes a lot more ASM instructions than you would think it does. Swift also adds about 5-20MB of standard libraries for each app that uses it. Extensions like maps or watch apps also take a few MB each.
Also apple makes decisions like encrypting binaries and then compressing, removing a lot of over the air space savings. They don't give free apps the option to just sign their apps and not encrypt them for smaller IPA sizes.
I wouldn't be surprised if a similar decision was made in the xcode toolchain that caused a similar issue.