But even with the current implementation it is possible to validate the assertion in your server without contacting browserid.org, but AFAIK nobody did it yet.
(I tried to find a reference for that, but they took it down from their Wiki. It used to say "You may choose to validate assertions on your own server": https://github.com/mozilla/browserid/wiki/How-to-Use-Browser... - the verification code is still online, though)