The book How Innovation Works by Matt Ridley changed my perspective on this. Originally I thought all innovation was like the dotcom era. Ridley attempts to show that there are times when it is not always clear what the final innovation will be.