"If you don’t pay me X we’ll report you under criminal law and you’ll have to pay much more."

"If you don’t pay me X we’ll report you under environmental protection law and you’ll have to pay much more."

"If you don’t pay me X we’ll report you under labour regulations law and you’ll have to pay much more."

How would GDPR be special?

They’d have to report it themselves. This is a worse value proposition than regular blackmail, where you take an existing violation, which the target already knows is illegal and has already shown willingness to conceal from authorities (or is not illegal at all, but e.g. just embarrassing), and threaten leaking it. In the proposed scheme, the mere reception of the threat itself creates a new situation for the receiver which must be reported. Paying off the blackmailer puts you in a worse spot than before. Not so with “ordinary”, Hollywood movie blackmail.

If he blackmailer can figure out, others can. Takes away the leverage.

Best strategy: Ignore the blackmailer and improve the security of your system and take privacy serious. (Doing such a cleanup also helps to reduce fines, thus also reduces leverage of blackmailer)

and if that's ever discovered you've just added a hefty multiplier to your fine. Risky.