The thing about writing the CSS to make a button pretty is that you only have to write it once (it's also usually no more than 10 lines), and then you can reuse it as much as you want. So forgoing a framework has some up-front cost, but in the long run it amortizes out and makes things easier to maintain imo.

Now, if I were churning out a new website every week or every month I would not be starting the CSS from scratch each time. I would probably carry around some starter CSS from project to project myself, though this would also be a good case for a framework.