I think it's better to drop them if they're holding you from getting to your target. The best example is how Mark Zuck, Dropped Savrin when he was not letting mark to get funds from VCS
At this point, it probably doesn't make sense to get in a legal fight. I think worst case scenario is you break off and continue on your own, and jettison the minimal contribution your partner has made to the project. It doesn't sound like the business is at the point where he is going to lawyer up and sue you. It would cost you both more than its worth at this stage.