How niche can a company be in the software industry? Examples are appreciated :)
what industry will this be in?
As long as profit > 0, it doesn't matter how big the niche is.
If you have only a single client with a custom solution, that could work. (I wouldn't advise that personally, though). If you're asking which niches could be profitable, I think the answer depends mostly on the "Expenses" part of the equation, provided income is not 0.
You didn't ask about how to judge the value of different ideas, you asked how big a niche market had to be for SaaS to work. The answer is that there is no pre-defined minimum niche size, so long as you can be profitable. The key part of being profitable, assuming you're bringing in any income at all, is managing expenses.
It is completely irrelevant and next to impossible to gauge whether one idea is more or less profitable than another. Some comparisons may be obvious, but spending too long figuring out what idea stands the chance of making you the most money is wasting your time.
Find the one you think might be the best, and built the thing. Rinse and repeat until profit > 0.