As a front end developer, I would say if you need an MVP to get out the door and you've got a bunch of plates in the air, then yes, Bootstrap (and also Font Awesome!) works just fine for all the things that you don't really care to worry about at the present moment. Its also easy enough to remove if you do eventually bring in a UX person or a designer to refine things.
That being said, the little things that you can do make a huge difference between a site that looks like its made with bootstrap vs a site that has a bit more going on under the design hood. Pay a bit of attention to your fonts and line heights, add in -webkit-font-smoothing, and lock down your colors before you start (e.g. black, white, brand colors, grays, etc.) Consistency will make your life so much easier to get that UI looking amazing, and it takes about 15 minutes or less to write down some variables (assuming you're using a CSS preprocessor). Also, optimize your assets (images, etc) and make sure you have fallbacks if something doesn't load fast.
Lastly, make sure you know what you (roughly) want your app to look like on mobile, tablet and desktop before just diving in to Bootstrap- even just roughly sketching out boxes on paper helps me sometimes with that. I like to use the Chrome emulator for a rough idea of any device as I work on phone, tablet and desktop in either orientation as well.
All in all, what I've described you can easily implement in a a few hours to a few days (depending on the complexity of your site) and makes a huge difference. Once you're making millions, you can hire a great designer to push your site to the next level!