You also never know when you need a really good admin panel.
Any sufficiently complicated flask application contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of django.
For sites that are just a few endpoints, it's great. I use it on an application that uses redis+gearman to spawn docker containers on demand. django would have worked too, but I'm not using anything other than routing and templating.
On the other hand, I have an application that I built on flask that I needed to shift the timezone of the dates.
in django you change TIME_ZONE and you're done. My flask app I had to fix it all myself, it was a nightmare.
For example, I develop single-page apps with Django+DRF and Ember, so I never use Django's template engine. If you really want to minimize your setup, you can even go into your settings.py file and turn off the middleware you don't need.
Put another way, it's not like client-side JavaScript where the entire framework is sent to the user's browser.