Comment by lmm | Hacker News Reader

Comment by lmm | Hacker News Reader

Parent Full thread

I'm seeing a lot of people speculating about which bank this might be; I think the point is that it's all of them. I could loosely describe a previous job as implementing Morgan Stanley's Walpole and integrating more source code management into Minerva (even though that system wasn't actually Python-based).

Having a global view on everything is large banks' value-add, it's why they haven't been outcompeted by their more nimble competitors. Being able to calculate the risk of the whole bank isn't just a cool feature, it's the core value proposition of this platform.

Being able to just upload your code and run it is really cool, and if you squint it looks a bit like what the outside world is trying to set up with serverless/lambda-style platforms - just write a function, submit it, and there, it's running. (But it's worth remembering that Python is not a typical programming language; python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages). Obviously there's a tension between this and having good version control, diffs, easy rollbacks etc. - but because Minevera is already designed to do all that for data (because you need that kind of functionality for modifications to your bonds or whatever), doing it this way strikes a much better compromise than something like editing PHP files directly on your live server.

What this article calls data-first design has a lot in common with functional programming. I hope that as the outside world adopts more functional programming and non-relational datastores, Minerva-style programming will get more popular. It really is a much better way to write code in many ways. The difficulty of integrating with outside libraries is a shame though.

Does everyone use a giant Pickle dump ? I mean - how big is that ? Petabytes ?

I'm kind of surprised nobody monkey patched python serialisation to use a database (much like GitHub did with ssh key lookup in MySQL).

What does the devops there look like ? Snapshot every minute ?

It's not a single giant pickle dump; each individual object gets pickled and stored in Minerva (which works more or less like Cassandra or something). It's a pretty similar high level design to what the likes of Google or Facebook do do where you store everything as protobufs in BigTable - the bank uses pickle rather than protobuf because they put a higher priority on being able to store arbitrary objects and deal with robustness/compatibility later, rather than having to write a proto definition and a bunch of mapping code up front. You wouldn't want to use a relational database because they're not properly distributed (and, frankly, kind of bad and overrated).

The Minerva I worked on was temporal and append-only, like a HBase that never did compactions (so "delete" actually just writes a tombstone row at a particular timestamp - there was an "obliterate" command but you needed special authorization to use that), and it was distributed (with availability zones even) so you didn't really worry about losing data; loading data as-of a particular timestamp was part of every query (and implemented efficiently). There were probably regular dumps somewhere too but I never needed to encounter those.

So Minerva is like a distributed datastore, specifically for python object storage ?

Interesting. Do you think you would do this today with a Cassandra/Hbase? Can it be done - let's say take python 3.10 and the latest Cassandra (or even better - something like Firebase or Cloud Spanner).

Just curious that in a post AWS/Firebase world, can something like Minerva be built, without investing in writing the db store ground up.

The incarnation of Minerva I worked on actually used Cassandra as its storage backend. But it's something that's not particularly useful piecemeal; the great value of Minerva is that all the bank's data is there and it's all temporal, all access-controlled and all the rest. The most fragile and cumbersome parts of Minerva are the parts where it integrates with an external/legacy datastore - but if you tried to introduce a Minerva-style datastore as a small piece in a system that was otherwise using a "normal" technology stack, those integrations would be most of what you made.

ZODB is the object oriented database as a giant pickle dump. Surprisingly, it works and scales wuite well. The downside is that non-Python tools cannot access it all.

https://zodb.org/en/latest/

I learnt Python via Zope in 2000, and attended the Zope Conference in Python that year.

Joined JPMorgan in 2010 to work on Athena, and immediately had a real sense of deja vu... Athena's Hydra object db (essentially an append-only KV store of pickles) felt like a great grandchild of Zope's ZODB.

I remember explaining our tech stack (Python and Zope) to clients.

“Where is the code for that page?”

“It’s in the database”

“Oh… Like MySQL?”

“No. It’s an object database”

“???”

I called it “Martian Technology Syndrome”. But it worked. At later stages we paid the price and had to serialize the datastore for migrations, but that’s what you get for relying on pickles.

I use Pickle quite a lot for caching, a file read is almost always faster than a DB query.

For long-term persistent data ? Seems very dangerous to me, even reading a pickle from say PyPy vs a Cython intepreter corrupts the damn thing.

This works until...

Specifically, until you realize that pickle changes based on python version, so updating from py3.x to py3.x+1 will prevent your application from reading previously stored data.

This is wrong. pickle can read old files just fine and lets you generate files in old pickle format versions if you require backwards compatibility further than when the current protocol was introduced (it does not get increased with each python version).

I think it’s mostly true. The complaint with this kind of “industrial global Python code base”, be it at banks or elsewhere, is that often they are hastily cobbled together and depend on extreme user care to not flop over all the time.

I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt. When something does break in the end, someone senior just shouts at the poor devs until it works again - usually fixed with a hasty patch again.

Similarly, documentation? Access control? Sanity? These seem to be left behind.

You could not be more wrong. Quartz has first class documentation, solid tooling, very well thought out and rigorous code review and access controls. Banks are regulated up to the eyeballs, so everything has to be audited and justified in detail.

It's not nirvana, these are real working systems built by humans with human failings. There are tradeoffs. Not every application is suited to these sorts of platforms, but the people building these things are top notch technologists and know what they're doing.

For some reason people think everything is as good as it can be at FAANG and other big name tech companies, and everyone else just walks around with their pants down at their ankles bumping into walls until 5:00. It’s just not true.

I mean, it's not just FAANG. Literally everyone except banks use the standard Python environment. simonh is right about the reason for the forked tech stack.

Actually, it's probably not FAANG folks at all. I'd expect ex-FAANG folks to be more sympathetic to the forked python situation... FAANGs have an abundance of non-standard and frustrating infra (wasn't 5TB just posted yesterday?), and maybe even on steroids compared to banks (do any of the FAANGs not have at least one custom linux kernel?) Hell, both As roll a shitload of their own silicon.

Exactly.

Facebook literally maintains a python fork: https://github.com/facebookincubator/cinder

Google invented "NoSQL" before anyone else knew what it was, and all those "cloud" tools they used internally were obviously proprietary (except the ones they open sourced). Ex-Googlers I work with typically had to spend quite a bit of time re-adjusting to the "inferior" tools and processes in other companies.

Microsoft invented their own development ecosystem, and the only reason it's "common" or "standard" in the tech community is because they sell it as a product. This is the same for Apple at least for iOS development, and Amazon for their cloud service offerings.

When companies have millions of dollars to spend on maintaining a custom development environment that they think will give them a competitive edge, they will do it. It's the smaller shops that can't afford not to go with the flow, so to speak.

I mean Google has Bazel which has reliable hermetic distributed builds with modern statically typed languages. I don't think this weird custom Python system is really in the same league. Barely even playing the same game.

Well, I don’t know about Quartz. I worked with DB systems and they were awful in that regard. They worked, but largely because people stuck to convention.

For example, changing scheduler jobs required submitting a change in Excel and having it approved (twice...) by someone. Except the table was world-writable and changes not logged. So in principle only your appropriate superior could approve change, in practice anyone could, and you’d never even know.

Youch, that's nasty. I can completely believe it though, banks are huge unwieldy organisations. I spent a fair bit of time working with auditors though, so I know a huge amount of effort goes into rooting out things like that.

The thing is just because a team in a bank did this thing, that doesn't mean "The Bank" thinks that's a good idea. Like any company, banks are communities. I'm not making excuses, the fact this system wasn't properly architected is a failure of governance, but I've been on the other side of this trying to get teams to fix their problems and adopt resilient processes and procedures. Every offender thinks their service is special and their violation of the standards is justified.

> Quartz has ... solid tooling

Is that how you'd describe the IDE and (integrated) VCS?

Yes, it's fine IMHO. Ok it's not your favourite Java IDE, but it's way, way better than some of the crap I've had to use at various places. But then I wrote a 5Kloc PyQt desktop app almost entirely in IDLE so yeah, maybe I'm not the best judge.

Hmm, I found the opposite - the fact that there was this global framework that managed all the data and code meant that access control was actually pretty good, better than most tech companies I've worked for. You had a single source of truth for what your access rights were, there was integrated Kerberos any time you needed to access a system outside Minerva. And having all the code in a managed place meant good deprecation cycles - not instant deprecation like the Google monorepo, but tracking and policy for which old versions of libraries were in use and how much that was tolerated. Documentation was at least attempted, and while platform stability/enhancement work did have to be coupled to business initiatives to a certain extent (e.g. "we're doing this performance work to enable us to run risk estimation more often to meet MIFID requirements at low cost") there was leadership that put a value on maintaining high quality code and this paid dividends.

This was 100% my experience too.

The biggest productivity gains were:

- having a single source of truth for both data and code (in a closely coupled environment)

- strong, battle-tested libraries to take care of all infrastructure concerns.

- enforced code dev/test/review/deployment workflows

This let the front-office devs be highly productive on adding real business value for their trading desks.

Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010. The infra we all take for granted today at AWS/GCP/Azure simply did not exist back then, and banks' data security policies at the time did not allow cloud processing.

> Remember also that these systems at GS, JPMorgan and BAML started around 2007-2010.

GS had „these systems“ well before 2000 (via J Aron). I think around the time you mentioned they spread to other firms (in their Python reincarnation).

> I guess banks are the archetypal places that care only about feature creation and not about maintenance or technical debt.

It depends which department you are in, but in general: absolutely not. Actually the reverse is true. Banks have huge risks to manage: just imagine for instance what damage a hack of their account system could cause. Or a crash of their payment system. Therefore it is of the utmost importance that systems are stable and bug free. In most departments, feature development is only in second place: stability and reliability have priority one.

This concern is so important that it is not just left to the responsibility of the banks themselves: for many systems, banks have to comply with external standards and are audited for that by external agencies.

And that's why many old banks still use COBOL.

Well it s repeated a lot but I work in a big bank and I have yet to see tech older than Java 6 :(

> python build, dependency and deployment management is exceptionally awful in every respect, this isn't as big a pain point in other languages

I'm not sure how to react to that, but these features in Python are miles ahead of what many other languages have (or actually don't have).

If you compare Python's deployment and dependency management to those of statically compiled languages like Go, Rust, Zig, or Nim, you quickly see the experience with Python is quite poor.

In all the above languages, you simply ship a statically compiled binary (often just 1 file), and the user needs nothing else.

With any sufficiently complex Python project, the user will need:

1. virtualenv 2. possibly a C compiler 3. recent versions of Python (and that keeps changing. 3.0->3.4 are "ancient", and 3.6 seems to be the absolute minimum version these days --- due primarily to f-strings) 4. Or you ship a dockerfile and then the users need 600mb of Docker installed

I sometimes joke that in the future every Python script will require a K8s deployment and people will call it "easy".

Python is a great language, but deployment is a massive pain point for the language.

When I know I am writing something that has to "just work" on a wide range of systems that I don't necessarily control, well I don't write the solution in Python. I pick Go, Nim, or Rust (Zig would be a good choice too).

Or you just provide your own Python package. Most of the time that will be less than 100 MB if you don't include huge libraries. You can test and build automatically. For deployment you then have an installer or rpm that is probably smaller than most of the other enterprise software your customer's infrastructure admins are handling.

The problem is "less than 100mb" is unacceptable (for my use cases).

There are use cases where 100-300mb is no big deal and customer can handle this.

But single binary deployments with a statically compiled language where a fully-featured binary can weigh in from 5-30mb are what I'm after.

And honestly, with upx I can take even those fat Go binaries down from ~35mb to 7-8mb. That's an order of magnitude less than 100mb of Python and all it's dependencies. Not to mention with all those languages I mentioned (Go, Rust, Nim, Zig), I get multi-threading and high-performance as well.

I think there are too many options, or not enough direction for busy people. Once you understand how it all works and pick / build the right tools it all works pretty well.

I would agree for all but deployment. I know my way around python reasonably well, but pyinstaller and friends still make me have bad days pretty regularly.

Four Python projects, same customer, five different deployment systems. Docker, a Capistano look-a-like I coded in bash, git pull (their former standard), git format-patch plus scp, zip archives. Yes, python file.zip works if it contains the right files. Probably the latter is the easiest way, except it doesn't address the dependencies.

> except it doesn't address the dependencies

It does if you put them all in the zip :)

(and build for exactly the platform your customer is going to deploy on)

Python is literally decades behind. Dependency resolution is nondeterministic by default. The way you run a build is still not remotely standard (e.g.: I've downloaded the source of one of the top 20 packages on PyPI. How do I run the tests? Perl had a standard way to do that back in the '90s). Deployment is so bad that people recommend using containers as a substitute for something like fat jars.

That's just untrue. Maybe compared to C++ Python isn't too bad. But try something modern like Go, Rust or Deno. Python is light-years behind.

Am I the only one who has never had enough headache over the years to say its awful? I do think dependency management is a somewhat difficult problem to solve and a lot of systems have pros/cons but I never have had huge issues with Python's.

it's been 15 years and i'm still a bit traumatized by aurora

Reply on news.ycombinator.com