HNHacker News
TopNewBestAskShowJobs

p10jkle

393 karma · joined October 23, 2017

submissionscomments
p10jkle··on We built network isolation for 1500 services
1. We have a lot of code paths, there isn't an integration test for everything. And for runtime, just because something is rarely called doesn't mean it never called; a bank can have yearly processes 2. We'll do service mesh too but this was probably an easier first step. We have rolled our own mesh with envoy sidecars and moving it towards istio style behaviour isn't trivial
p10jkle··on We built network isolation for 1500 services
I don't really mind it
p10jkle··on We built network isolation for 1500 services
I feel like we could do all of that by consuming the GitHub API. We essentially have directed graph stored as files
p10jkle··on We built network isolation for 1500 services
Monitoring can only access metrics, it's scoped to port. We'd like endpoint level controls and that's the next step
p10jkle··on We built network isolation for 1500 services
I think we do effectively have a graph, but its not stored in a centralised way. If it was, then we'd have to somehow gatekeep that state, which when you consider that we do hundreds of deployments a day, becomes tricky
p10jkle··on We built network isolation for 1500 services
The common idea is that if you have to regularly roll out all your services, then you still basically have a monolith. So if you have a lot of shared code in libaries, there are no benefits to microservices. In our case, maybe we do have core libaries like to read from Cassandra, but you don't necessarily have to roll out everything if you improve them. Some pods could be even a year old

All the services have their own keyspace in Cassandra, there is near-zero cross service data access

p10jkle··on We built network isolation for 1500 services
Yeah, our (mostly) ban on shared code is a major reason we end up with a lot of services; there's even service.business-day which tells you if a day is a working day
p10jkle··on Monzo: How our security team handle secrets
Vault is a lot more fully featured and generally has more granular access control than equivalent AWS products, but is tricky to run in a way that is available and secure. Obviously outsourcing that has huge value
p10jkle··on Monzo: How our security team handle secrets
This is on our radar and I think they can now be used directly with the kubernetes Auth plugin, although I've not heard much about it. This is a very recent change. We could have possibly got the same functionality with the jwt plugin, with some added complexity (and no tokenreview)

We don't allow read or list of secrets by any human, although of course that's not a perfect control.

p10jkle··on Monzo: How our security team handle secrets
We are using your database credential generation features! With thousands of clients, too. Very cool, we'll write a blog post soon. The root credential rotation is genius.
p10jkle··on Monzo: How our security team handle secrets
Fair enough, Vault has a ton of features I guess most don't need. It's been cool for us, we are now using it for secrets, database credentials, and certificate management (more blog posts coming!). Next up is hopefully AWS creds management
p10jkle··on Monzo: How our security team handle secrets
I'm a little confused; the system mentioned in this article allows services to retrieve secrets in plaintext; they're just stored encrypted at rest. You can do what you like with the retrieved secret.
p10jkle··on Monzo: How our security team handle secrets
Microservices generally have 0-10 RPC handlers (we use https://github.com/monzo/typhon). We have a lot of different object types at Monzo, it goes way beyond 'transaction' or 'user' and each object type is going to be at least one service, but can be several. Services that mostly just consume off queues are often separated from services that handle requests so they can scale separately.
p10jkle··on Monzo: How our security team handle secrets
This sounds interesting. I think with bound service account tokens (https://github.com/kubernetes/community/pull/1460) things may improve; pods will be able to prove their identity without it being equivalent to giving away your k8s api access.

I don't think there is anything with the same guarantees as the Task ID, sadly.

p10jkle··on Monzo: How our security team handle secrets
I don't think its particularly common for a company of our size. They are mostly very small and handle very specific things. It's one approach, with some benefits, although it makes projects that affect all services quite tricky!
p10jkle··on Monzo: How our security team handle secrets
This exact link is mentioned in the article! We use it!
p10jkle··on Monzo: How our security team handle secrets
> The bit where a secret gets pasted into the Very Secure system is a clear problem. Because before it was pasted, and while it was being pasted, it's not in that secure system it's on some dev's laptop.

Yep, so we have a different system for securely generated random keys. If its a Twilio API key, it realistically has to pass through a dev laptop, and its not that much of a big deal. If its eg an RSA key, we will generate it on an airgapped laptop, encrypt to a public key, and then we have a Vault plugin that decrypts, and writes it into Vault. So the unencrypted data is never anywhere but the airgapped laptop or Vault.

We also try to generate keys inside of Vault where possible, and we generate a lot of certificates this way.

p10jkle··on Monzo: How our security team handle secrets
> Ha I love how they explain what "cryptography" means, as if anyone reading this wouldn't know.

We try to make the posts accessible to our customers as well as engineers :)

p10jkle··on Monzo: How our security team handle secrets
We were pretty concerned about this. It's hard to get in a situation where the data is unrecoverable, because the root keys that you distribute can be used (with a bit of reading Vault code) to directly decrypt the internal storage, which is easy to navigate.

On a more general note, its taken us a long time to get comfortable with using Vault for increasingly critical things, and we are now at the point where it being down is extremely critical. But there are several components like this, and we are able to tolerate node failure, so its somewhat acceptable.

p10jkle··on Hyperledger Fabric: A Distributed Operating System for Permissioned Blockchains
I have done significant development work using hyperledger platform and aside from the fact that private blockchains are essentially pointless, this is (or at least recently was) pretty unusable, with very bad tooling and support.
← PreviousPage 4 of 4