That is just one of the problems right there, patent holder isn't necessarily the Inventor. It just gets worse from that point.
The problem with patent trolls is that they are too often working with patents that shouldn't have ever been granted in the first place, and that it's so expensive to defend against a lawsuit, it makes sense to just settle silently.
There are many things wrong with the current implementation that can be fixed without throwing out the whole notion of intellectual property protection for inventions.
Concept of prior-art still exists in first-to-file system.