It can be used as the main backend for an API just fine. It's also very easy to override just about every piece of what DRF offers. This is pure, unsubstantiated, subjective FUD.
Use the tool that you are most comfortable with, change it up if/when you get big or busy enough to have to. There are some huge sites running Django/DRF successfully.