sorry, missed your reply (so perhaps you won't see this for the same reasons) but anyway:
Mainly the feeling of giving up simplicity. Adding things (such as sourcemaps and any relevant configuration changes) usually detracts from the sense of "no middleman", although I know "no middleman" is always an illusion, it's less obvious when you don't have to do anything to create it.