I had a lot of smart people tell me early on that my last company was a waste of my time because the opportunity wasn't large enough. Large enough for whom though? Hearing all that feedback was incredibly discouraging when I was starting, but I passionately believed that I was right about the opportunity and the value we could create, even if it may never be a billion dollar business. So, I hired a co-founder, recruited a very small core team, worked 24x7 for 18 months, and we were acquired. At the time we sold, we actually had pretty reasonable Series A terms sheets on the table, but we decided selling was the right path, for a number of reasons.
Many (definitely not all) silicon valley investment types would probably look at that and the amount we sold for and assume the company was a failure. I had a lot of people question not going the VC route at the end and trying to blow it out (many of these doubters were the ones who doubted going after it at all initially). I would have done it if I thought it was right for the business, but it was not. And I now own a home in Marin (and one in Sonoma), my kids are in private school, and I just don't worry about the shit I worried about 10 years ago. My life is radically different and better. AND I had an incredible experience creating a real business, learned a ton, and am a far better entrepreneur now than I was when I started.
Now I'm back at it, and swinging much bigger this time, but with that experience (and added confidence) under my belt. We actually just did YC (it was awesome), raised money, and are going after a massive opportunity. But I think it's right this time, for this founding team, for this opportunity.
The best professional advice I ever received was from Chris Moore at Redpoint Ventures. He told me a few years before my first startup to think about my career as a process, and not to be so focused on just jumping to the end point. For me, this was the right advice at the right time. I wouldn't be where I am today had I not accumulated a number of experiences along the way which helped me to succeed later. Everyone's different. Explore your passions, but also be true to yourself. Know who you are and what you want. And surround yourself with smart people to bounce things off - but just be sure to treat it only as input. No one can tell you what you should do.
Also, there's just no shame in creating something that's 'smaller'. You are right to not give a shit what other people think (and it probably helps that you're in Philly in that regard). There are a ton more buyers of companies at $10MM or $20MM than there are at $250MM or $500M. Hell, even an exit for a couple million can really change your life (with the right cap table). If you raise a shit ton of money because "that's how things are done", you are most likely shutting off any early exit (at least, in a way that will be meaningful for you). So, just be sure it's the right thing for you and your business, and don't focus too much on what other people think is the right path.