If the finding of frivolity was separate from the verdict, though, it could work. Set the standard as, on a unanimous verdict for the defendant, the jury may decide the case was frivolous on the standard that "no reasonable jury would find the facts as applied to the law to render a verdict in favor of the plantif". It would cause greater scrutiny but cases that have merit would still go forward.

Edit: Also the lawyer would not being charged money to work, they would be getting fined for wasting the court and jury's time. That's like saying a contingency lawyer is working without pay: well yeah, but it's factored into the cost of business. There are still chances for high payout but now there is also a much needed downside.