But what you described is literally "the worst case scenario is as bad as before (building my own images), but it would commonly be better". I'll take that any day.
Also, a lot of apps don't need a full linux distro shipped with them. If your binary is statically linked (like, for example, anything written in Go), the docker image can start from scratch, resulting in an image only a few megabytes large and with no issues like you outline.