I built this for Morgan Stanley (the same place where kdb originally came from) and put it to use in some major high-volume/low-latency trading systems where a significant fraction of US equity trades execute every day:
187 karma · joined February 8, 2014
I built this for Morgan Stanley (the same place where kdb originally came from) and put it to use in some major high-volume/low-latency trading systems where a significant fraction of US equity trades execute every day:
I'd like to grow it into a successor to kdb, though kdb is very entrenched in finance and the company is extremely litigious (they threatened to sue me).
F# is great too, and all of the other ML variants. We'll get there eventually, it's inevitable, but there's a lot of institutional inertia.
Anyway, it's generic and open source, so can be used for anything. I use it at home to make little video games for my son.
https://github.com/Morgan-Stanley/hobbes
It’s kind of a structurally-typed variant of Haskell, integrates closely with C++, produces very fast code.
Also w.r.t. reinventing the schema for databases, I have got a lot of mileage out of deriving the schema from code. Shameless plug, I have this little project (https://github.com/Morgan-Stanley/hobbes/) where we have a self-contained header-only library to push structured data out of process with minimal latency, query data concurrently with a Haskell-like PL. It works well for our kinds of systems (millions of orders per day, ~1 billion market data ticks, structured log data from multiple systems involved, ...).
https://github.com/Morgan-Stanley/hobbes/
The logging part is just a single self-contained header-only lib, so very easy to incorporate in existing systems.
I started this project several years ago to support live code updates in production systems without sacrificing performance, but I eventually hit the same issues with exchanging structured data: https://github.com/Morgan-Stanley/hobbes
I split out structured logging into a single header-only lib, structured RPC into a single header-only lib, etc. It's not quite the same as having native support in the OS, but because the mechanism is so lightweight (don't even need to link anything special), it's a pretty good substitute.
I agree, and in a Haskell-like setting we can have any structured types (rather than just primitives).
That’s why I built hobbes: https://github.com/Morgan-Stanley/hobbes
The compression method is pretty neat, has useful applications in other contexts as well. Basically the same way you can view types as propositions, you can view them as probabilities as well (which gives you a scheme for representing the bias in any structured data type). This idea works in other contexts too, like "time compression" for pattern-match translation (choosing an order for tests to match the bias in the data you're deconstructing).
It's hard to find time to write about this stuff with my day job though. :T
What "structure" means and how it works can have a lot of nuance. With hobbes we basically start with algebraic data types (which map to most C-style data structures and so can be shared without conversion with C/C++ code). It's been a while since I looked at R, but IIRC it's a lot like Scheme (e.g. maybe the data sharing/translation story is more complicated?).
We do have some things that are helpful for dealing with unstructured text data, like a built-in LALR parser generator and regex matching (integrated with general pattern matching), but it's not one of the main use-cases we've been focused on.
The first story is that it follows the tradition of naming programming languages after logicians/mathematicians (e.g. Haskell, Church, Russell, Pascal, ...). Because the Curry-Howard isomorphism (CH) links logic and programming languages, and ideas in programming languages often come from much older work in logic, it's pretty natural to go to logicians for names (IMHO). Thomas Hobbes was one of the first people to connect logic and computation (literally saying "by reasoning I understand computation") and the logical arithmetic he developed inspired other people to continue that work. I think that story is a reasonable justification, works well for a general audience.
The second story (the real story?) is that Morgan Stanley internally has a tradition of naming things after things in popular culture (and at least one or two cartoons), so I named the project after the comic book characters Calvin and Hobbes (CH). Actually the programming language was "Hobbes" and the compiler was "Calvin" because if you know the comic book, Hobbes is really just a figment of Calvin's imagination (which works well to explain how a compiler makes a programming language real). I am a fan of the comics and was reading them frequently with my son at the time. There's a "Calvin and Hobbes ismorphism" (because both characters are different views of the same mind) so it works really well as an extended metaphor for logic and programming languages, I think. :)
I made this project several years back, it's been very fun and interesting. I've been meaning to write about some parts of it that haven't had much attention.
Eager Haskell with a structural type system, deterministic garbage collection, and no overloading penalty, that would be a fair summary.
It'd be interesting to see how GHC would do in the same area, there would be some significant challenges. I haven't tried to use GHC there so I can't say.
https://github.com/Morgan-Stanley/hobbes
From the article, it sounds like the Kx folks are working on the issues that led to the development of hobbes in the first place.
I’d be surprised if they’ve changed their opinions about type systems though (or bullying people with lawyers — those guys get nasty when their ideas are challenged).
I made a similar library for writing data into memory mapped files, also a self-contained header-only lib:
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
This one also serializes a representation of the type structure of recorded data so that it can be safely concurrently mmapped and read either with the same code or with the generic PL/compiler that I've developed in this hobbes project.
Where unpredictable disk latency is a problem, we've got a similar header-only lib for logging into shared memory (then have another process to consume this shared memory ring buffer and dump it to disk for concurrent querying):
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
This pipeline works well for having lightweight C++ processes feeding large volumes of data to generic query processes that we can run out of band to look at this data in various ways (with a Haskell-like query language).
We did hit a slight problem doing things this way that the straightforward representation of data (as in memory) for some cases just used too much space and too much time wasted in I/O. Basically for complex market data, where data structures aren't trivial and recording ~100GB/day makes it very awkward to keep around a few weeks of data for random querying.
So I also made this header-only lib to write data into these mmapped files with a simple compression method (I like to describe it as generalizing Curry-Howard to probabilities) that gives us much better throughput, much smaller files, faster query times, and still support concurrent constant time random access queries:
https://github.com/Morgan-Stanley/hobbes/blob/master/include...
It gives us compression ratios about the same as EOD gzip, but much faster and importantly works online and with these query use-cases we have with hobbes.
Anyway, maybe I should write up those details somewhere else, I just mean to say that this is a useful technique and you can push it very far and do many things with it in a very straightforward way.
I had an interview there recently in NYC. I solved the guy's puzzle too quickly and I think that he thought that I'd prepped (I hadn't). Then we got started on a different problem and I tried to explain how it was a good example of "boolean blindness" and that using a specific form of dependent type was an ideal solution. He looked at me like I had two heads and smugly told me that it's impractical to solve problems this way (but it's a bog-standard type checking problem).
So yes, smart people, but you should know the bias that they have before you decide that you want to work there. It could very well be bad for your career if you _do_ get in.
Referential transparency, mutation, non-termination, coinduction, memory pointers, linear types, coroutines, continuations, blah blah blah. All of these things exist, and more.
Most PLs are biased in one way or another, many don't even let you reason formally about those things (or allow you to reason informally).
But at least if you're a working programmer, especially if you work to map an existing human domain to software, there's always the fixed point of Curry-Howard to hang on to.
Unfortunately there are some places where aspects of the standard infrastructure around Haskell block production application.
In particular, we have applications with very hard latency requirements and also with complex type/binding requirements (where a direct structural type system is needed):
Obviously it's ultimately the same thing either way, but that's the lineage (SETL -> ABC -> Python).
False dichotomy, these things you distinguish are just two sides of the same coin:
https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
It's much easier to make sense of logs when we don't discard that type structure.
I would argue that this is one of the most overlooked cases _for_ schemas, because it's where schemas are the most complex and also where traditionally we've had few options for really representing that complexity.
This is something I've tried to change with the hobbes project, where we have a small header-only lib to efficiently record "log data" with inferred schemas into shared memory:
https://github.com/Morgan-Stanley/hobbes#storage
Basically, in this view, the type for recorded log data is a variant over all possible log statements in an application. This is a very large and complex type, but in memory or on disk it's a simple representation -- a number for the constructor ID of the variant, and then the payload type that corresponds to that constructor.
Then there's quite a lot that we build on top of that, but as a basic primitive I think that logging is actually a great case for "extreme schemas" rather than throwing out schemas entirely.
I kind of agree with you on this. I have a bidirectional binding process in my hobbes project at Morgan Stanley (https://github.com/Morgan-Stanley/hobbes) such that C/C++ functions can be viewed as hobbes functions and hobbes functions can be viewed as C/C++ functions (without translation).
This works as far as C/C++ types are adequate for your programming language, which obviously does cover a huge space, but there are types that don't translate well and there are staging considerations that don't translate either (e.g. there is a logical type that can be decided, but it's not decided until what might be regarded as "run time").
There are interesting problems to be considered in this area though, happy to discuss further out of band if anyone is interested (discussions on hacker news tend to be short-lived and shallow IME).