Open-Sourcing Subzero: Square’s Bitcoin Cold Storage Solution | Hacker News Reader

Open-Sourcing Subzero: Square’s Bitcoin Cold Storage Solution

medium.com

165 pointsmcpherrinm31 comments

>One specific customization we implemented is the ability to enforce that cold wallets can only send funds to a Square-owned hot wallet.

This actually seems like a significant risk. Cold storage is based on the idea that hot wallets are more easily compromised.

If someone does manage to compromise Square's hot wallet private keys, they can basically hold all cold storage funds hostage. Perhaps Square has some way to modify the allowed addresses, but that somewhat defeats the purpose of the restriction.

The code which implements the restrictions on destination is just code. We can add new destinations if needed, but that requires a quorum of codesigners and a more arduous process than simply withdrawing funds.

The goal here is the normal operation, cold withdrawal, is very low risk.

If we ever did suspect our hot wallet keys were breached, we'd have to do that or else we would be stuck as you suggested.

Makes sense, thanks for the explanation.

So what's the material difference between the cold wallet and the hot wallet if funds from the cold wallet can be moved into the hot wallet freely?

Moving funds from multisig cold storage is not “freely” when it at the minimum requires two people to travel to two different key storage/signing locations. You’re generally looking at a 12-48h process at a minimum.

“Cold” refers to it being entirely offline.

In this case, the people are really just part of the system.

If an attacker can fool the people into getting funds out of the cold wallet, then they are available for an attacker to steal.

The only way to avoid the people being fooled is to have full auditing of every customer account balance. One needs to ensure that hot wallet + cold wallet >= all customers account balances.

Doing that auditing in a secure way is really hard.

Square tried to minimize the impact that a human can have on the system:

* Several people would have to be compromised * The people are geographically diverse * Subzero will refuse to redirect funds to anything but a Square hotwallet. * And, they specifically built the Beancounter system for auditing of these accounts.

> If an attacker can fool the people into getting funds out of the cold wallet, then they are available for an attacker to steal.

Sure but then you still have 12-48h to discover that they are being fooled.

There's no foolproof method, there will always be people involved and anything you could make will only make it better because it give you more people/time to realize the mistake. They decided that 2 persons and 12-48h is enough and I agree with that.

I'm glad that exchanges are starting to take security more seriously. But the security of cold wallet funds is much easier to maintain. The issue here with centralization and hot wallets is that funds have to sit on publicly accessible servers. So really, any improvements made to cold wallet storage are nothing more than a red herring that distracts from the real issue that centralized exchanges were never designed to handle cryptocurrencies.

There are far more problems to this approach than just wallet security too. With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly? The exchange still has an edge over every trader in that market and can skim profits off of anyone. If we observe Mtgox it was the result of what can happen when everything goes wrong (hot and cold wallets mismanaged and order books manipulated.)

So call me a skeptic but I don't see how this solves anything. I did enjoy the hand waving though. Very cool, and I'm sure Squares customers will be happy with it when their convoluted security protocols are inevitability broken by a rubber mallet.

> With an inherent lack of transparency that comes from centralized exchanges how is the user to know that trades are being executed fairly?

How do we know Nasdaq isn't screwing the buyers and sellers? It's only a problem with some exchanges for cryptocurrency. The trust issue is with cryptocurrency exchanges in general and not centralization.

Check out the Unchained podcast. Laura Shin interviews the CEO of Binance. His answer on the question of whether tokens are securities is that some people say these are securities and some people say they aren't. Does that sound like someone you want to trust with your money?

Even if you believe in cryptocurrency why would you trust a CEO that's taking huge risks by throwing up his hands and saying that there isn't regulatory clarity when the SEC says the opposite?

You can always trade on a decentralized exchange like https://www.radarrelay.com

anyone who works at radar relay - could we chat about listing 0xBitcoin? thanks!

This is neat, but I think it'd be interesting to see a writeup that talked more about why they needed a new solution and what they did differently that solved their particular problem.

In particular, they mention an onion model - what does that look like for them? Hot/warm/cold can take a variety of forms...

The main motivation to build vs use an existing solution was a solid backup/DR story and the desire to be able to write custom business logic which is enforced inside the HSM.

It’s things like controlling where funds can go, or requiring specific authorization based on amount being move around, etc.

Being able to leverage hardware we are already familiar with or which has various certifications (such as Fips) is a neat bonus.

If you look at the code, you’ll see that the funds flow from “anywhere -> hot -> cold -> warm -> anywhere”. This is the minimal setup and you can add more layers if needed.

With HD wallets you can generate receiving addresses based on the master public key of a wallet, without needing any private key.

Is there a reason for not receiving funds straight into the cold wallet?

Edit: in the docs it seems that you indeed receive funds directly in the cold wallet

You are right. You can go from external entity to cold wallet directly. The important piece is how the funds leave the cold wallet.

Speaking of hardware security modules, are there any that can be easily and cheaply acquired by hobbyists?

As far as HSMs go the Yubico YubiHSM 2 is the cheapest and most approachable one I've found. [0] I don't have a good feeling for how it compares to some of the other products out there but it is also the only one that has a list price you don't have to call a sales person to get a quote on.

[0] https://www.yubico.com/product/yubihsm-2/

I've used both Thales nShield and Bull Proteccio models at my previous work.

Both are priced in the five figure euros/dollars range for any reasonable operational setup.

(In the sense that you need at least a production HSM and an additional machine to be able to react quickly to incidents and experiment with).

The YubiHSM v2 seems very promising to me too, but the thing you have to understand with HSMs is that their certifications are often as important, or even more important, than their actual security.

(Usually you use HSMs because you legally have to and not just for fun)

In that regard, both the nShield and the Proteccio are qualified at Level 3 of FIPS 140-2. This is a very expensive qualification to pass and the investment is reflected back in the price of those products.

Yubico does not advertise any qualification level for the YubiHSM 2, therefore I infer they did not pass any certification process. Hence I feel they will target a different market than Thales and Bull.

(People wanting more security but without needing certifications)

A 650$ FIPS level 3, eIDAS qualified HSM would be groundbreaking. As certification processes are quite political, I'm not even sure it would be possible as it would basically decimate current HSMs vendors.

Let's hope the YubiHSM v3 could be that though ? :)

Yes, if you want FIPS 140-2 you are going to pay the big bucks. Yubico's Yubikey 4 has a FIPS certified version so they have been through the process with other products but you are right that their HSM is not certified. If you care about FIPS certification you can look up if a product is certified or not in NIST's database. [0]

[0] https://csrc.nist.gov/projects/cryptographic-module-validati...

Here's a D.I.Y. one that looks like a fun project:

https://cryptosense.com/blog/building-a-raspberry-pi-hsm-for...

The Nitrokey HSM is what the guardian project recommends and is super affordable actually:

https://shop.nitrokey.com/shop/product/nitrokey-hsm-7

Edit: Changed from "what cloudflare uses" to "what the guardian project recommends"[1]. I was reading too many tabs at once and confused things.

https://guardianproject.info/2017/12/18/building-a-signing-s...

Sorry to jgrahamc for the incorrect endorsement.

We do?

Edited to fix my very incorrect assertion. Sorry about that!

Ah, thanks. I was worried there... I figured someone was up to something I didn't know about with HSMs!

Pieces of paper work pretty well.

Edit: literally any text data storage medium that can be powered down and locked in a safe will work perfectly well for any of this. It doesn't matter if it's a flash drive, floppy disk, CD ROM, piece of paper, or sheet of gold with characters scratched into it. Literally anything will work. You do not have to buy these $50 glorified flash drives to store your bitcoin.

That takes care of offline key storage, but you'll need a lot more than just a piece of paper (and a safe) to actually use the key securely: a secure offline computer, for a start, and a process for transferring the key and transaction data into that computer, signing the transaction, and getting the signed transaction back to a connected system for upload. At which point you've basically re-invented an ad-hoc, inefficient, and quite likely insecure hardware wallet / HSM.

Or were you just planning to scan the private key QR code with your (compromised) smartphone?

HSMs are a lot more than "glorified flash drives". The most important difference is that they keep the key data securely stored on the HSM and only allow it to be used in specific ways, such as signing individual transactions. Depending on the HSM you may even be able to program it to only sign transactions which meet specific requirements, such as transfers from cold storage to known hot wallet addresses.

Some HSMs allow you to run custom code (usually a co-processor within the HSM’s security boundary but distinct from where the keys are loaded / raw crypto primitives happen).

Other HSMs don’t have this ability.

Finally some vendors will tell you that their hardware is extensible by them implementing custom logic/algorithms.

If you want to play around with HSMs (or any enterprise-grade hardware) I would recommend looking for used stuff on eBay or talking nicely to sales rep who might lend you a demo/eval unit.

This project looks pretty interesting:

https://github.com/square/beancounter/

Allows mapping a date to blocks or see what the wallet balance was at any given time.

If it's taught us nothing else, I'm grateful for the whole bitcoin mania for teaching us some humility about computer security. It seems the consensus is forming that a computer that's connected to the network is pretty much insecure by definition.

I'd love to see a writeup on why they even went with the central storage model at all – I'm sure there were many considerations but it seems as though CashApp was perfectly positioned to become a mainstream Bitcoin wallet with the private keys in the hands of the users.

Users lose their keys all the time. The cost of support calls alone would probably be higher than the cost of developing a cold wallet solution, especially since it sounds like they already had HSMs and the expertise to use them.

Reply on news.ycombinator.com