Excepting instances of extremely small storms, Category is in fact the second most important factor when determining the destructiveness of a storm.
First most important is geography - where the storm hits.
Size of the storm is the least important factor. A large but weak storm (such as Sandy) will always be less destructive than smaller but more intense storm - assuming that they make landfall at the same location.
Sandy killed only 131 individuals, despite making landfall in an extremely densely populated area. Within the US, far smaller - but more intense - storms have killed far more people in areas far less densely populated.