Say, for example, you go into a carpenter's shop, and you see that he writes down what you want built on a scrap of paper. That's your "in". Ask if all orders are stored on paper. "Oh? I'm nikentic, and I build software for businesses. Where do you store these papers? How are you tracking receivables? How are you tracking the progress of each piece? Do you store your customer's email?"
Ask them these questions, in as much detail as you need, using your knowledge of what's possible with technology to make suggestions. You will be pleasantly surprised how much details they are willing to share about their businesses. Ask them if they would pay XXX/month for a system that covers all the major pain points they mentioned. If they would, find other carpenters and ask them if they would be interested. If you get enough interest you might be on to something. Also, how many devs are out actively looking to create software for carpenters?*
*Where carpenters = small - medium companies that are not in the technology sector.
It took a bit of market research to align the two, but once I did it was great. With the idea itself, I just had a few simple metrics:
1. B2B (I have a raw hate for B2C just due to how much harder they are and it's harder to get a sustainable business off the ground vs businesses that just pay money)
2. Repeatable revenue source: subscription based services are easy to forecast, and you typically need 3 tiers for a pricing scheme to segment customers.
3. Ease of validation: Business problems are easy to predict. You have to ask the right questions, but your only real criteria has to be: Does this save the customer money or time? If it does, you only have to survey some different businesses to ensure the idea is wanted by enough of them. Congrats, you have something.
For customer development, just do some google searches in your target niche and start firing off some emails. Better yet: try to find where these customers gather. Whether it be on support forums, or even freelancing market places.
That's how zapier took off if I remember right. In my case, I had a lot of luck with freelancing forums since I'm looking to automate certain tasks that virtual assistants usually do.
A bit of a simplistic explanation, but easy enough to get started. What might work for me might not work for others, but there it is.
The more you try this the easier it will become, and after a few failed attempts you'll know how to get from the "new idea" to a finished product. Yes failed attemps, it'll probably take a while until you learn which things are worth building and which aren't.
Don't be afraid to be very specific. Pick a niche problem, don't try to build a solution that fits everyone. Just go after one specific use case and do it right.
You can try solving someone else's problem, or a problem you see in the company you work for, or a problem your customers are having. But be sure to understand the pain you're trying to solve, feel the pain, you need to know why are you building this thing. Don't just do it because it is cool, do it for a reason.
In addition to the pluses mentioned above, solving a pain you experience means you will be in a position to "dog-food" your own solution. This is obvious but worth reiterating. If, for example, you solve a problem that people other than you have, then you will not be using your own solution, and this is likely to lead you astray.
Having said that, be careful about being too "niche" or too personal with your solution. After you have come up with a problem you would like to solve for yourself, and you have ideas on how to solve it, go out and ask others: - do you have this pain? - how bad is it? - if someone offered solution X for Y dollars, would you consider it? - what would be some of the requirements and hangups that will keep you from adopting solution X?
In other words you don't want to solve your pain and then find out nobody else perceived the same pain in the same way. That's a one-way ticket to the deadpool.
> bring anything new to the table".
Well if you're going to compete with established players (which is not a bad idea), you have to answer this question.
The good news is that a lot of software is painful to use. When you figure out what you want to make, look at your competitors. Search for "competitor X sucks" and things like that; figure out what people really dislike about using their software. Then make sure your own software doesn't fall into the same traps.
That said... I feel like you just have to start with something that interests you. In my case, I was just plain fascinated by the idea of "social networks in the enterprise"... I was also really interested in machine learning / artificial intelligence, knowledge management and the idea of "collective intelligence". And over time, different events just led to all the dots starting to connect, and I started to see the kinds of products I wanted to build. Of course, the vision was pretty fuzzy at first, and we've iterated on it, and de-emphasized some aspects, added new aspects, promoted aspects that were originally less emphasized, etc.
The thing is, there are absolutely people already doing some - if not exactly all - of what we're doing. There are "enterprise social network" products out there... hell, there's actually a laundry list of them: Jive Software, Yammer, Hall.com, Tibco Tibbr, IBM Connect, Salesforce Chatter, etc., etc., etc. But, to that point, two thoughts:
1. Despite how many ESNs there are, we still have some aspects to our approach that are actually different from the other ESN products out there. Our vision also goes bigger than just the ESN... that'll be our first product, but we see it as just part of a much larger whole. And it's that "whole" in sort of the "whole product"[1] sense that we think will really set us apart when it's in place.
2. Competition doesn't bother us. I remember reading a blog post from Bob Parsons (of GoDaddy fame) a few years ago, that said something like (paraphrased):
"Don't be scared of crowded markets, just be better than everybody else".
That resonated with me. Call me arrogant, but yeah, we think we can build a better product than Jive, Yammer, Hall, Tibco, IBM, Salesforce or whoever. And we think we're operating with a superior big-picture vision. So basically, we don't care that there's "somebody else doing that", we just intend to beat them.
So I guess if I had any advice for you, I'd say:
Start with something you're already interested in. Look for places where two or more of your interests intersect, or can be combined in such a fashion as to create something novel. Once you have the basic idea in place, do a deep dive into that world... read the literature (if there is any) on the domain in question, talk to people who might be potential users of your product (or, at the earliest phases, anybody who'll talk to you) and solicit their ideas. Look for the "X sucks" forum posts for your competition and see what people are complaining about. There's your opportunity to do better.
Another idea I'm big on could be paraphrased by saying "everything old is new again" or "the only way forwards is backwards". I know I've ranted about this before, but I think there's a ton of value in going back and reading the works of early computer industry pioneers - people like Vannevar Bush, Douglas Engelbart, J.C.R. Licklider, Norbert Wiener, Herbert Simon, Alan Newell, Ted Nelson, Marvin Minsky, etc. and seek inspiration in their writings. Surprisingly, there are ideas that were being batted around in the 1960's (or earlier) that haven't been fully realized yet. Take that inspiration, merge it with your interest area, and see if you can't come up with some ideas for ways to differentiate your product.
Also, watch that Brett Victor video[2] that is posted here on HN quite frequently. It's chock full of amazing insights.
Edit: to add one more thing: I don't know that being passionate about the thing you're building is sufficient (it almost certainly isn't) but I'd almost be willing to be that it is necessary. At the very least, I think it's a huge benefit to you if you're genuinely enthusiastic about the product and would want to use it yourself. In our case in particular, I definitely want to grow a large, profitable, kick-ass company... but even if we fail at that, or even if we weren't even trying to do that, I'd still enjoy the process of creating this stuff, because working on it is genuinely enjoyable to me. And when the hours start getting long, and things aren't going your way, and some doubt creeps in, I believe it's a lot easier to keep moving forward when that's true. If you do something that's only about the money, I would question if you'll be motivate enough to power through the inevitable tough periods.