HNHacker News
TopNewBestAskShowJobs

kthielen

187 karma · joined February 8, 2014

submissionscomments
kthielen··on InfluxDB is betting on Rust and Apache Arrow for next-gen data store
Re: no open source alternative to kdb

I built this for Morgan Stanley (the same place where kdb originally came from) and put it to use in some major high-volume/low-latency trading systems where a significant fraction of US equity trades execute every day:

https://github.com/Morgan-Stanley/hobbes/

kthielen··on Fast high-level programming languages
Yep still used (pre and post trade, processing millions of orders and billions of market data events per day).

I'd like to grow it into a successor to kdb, though kdb is very entrenched in finance and the company is extremely litigious (they threatened to sue me).

F# is great too, and all of the other ML variants. We'll get there eventually, it's inevitable, but there's a lot of institutional inertia.

kthielen··on Fast high-level programming languages
No it's used for a much more basic purpose than that, efficient order execution (which investors need as much as speculators).

Anyway, it's generic and open source, so can be used for anything. I use it at home to make little video games for my son.

kthielen··on Fast high-level programming languages
Shameless plug, I made hobbes and used it in high volume trading systems at Morgan Stanley:

https://github.com/Morgan-Stanley/hobbes

It’s kind of a structurally-typed variant of Haskell, integrates closely with C++, produces very fast code.

kthielen··on How to Evangelize a Programming Language
I think that part of the problem is to define data structures in a way that can be produced and consumed in multiple languages, algebraic data types are a good "least common denominator" for this.

Also w.r.t. reinventing the schema for databases, I have got a lot of mileage out of deriving the schema from code. Shameless plug, I have this little project (https://github.com/Morgan-Stanley/hobbes/) where we have a self-contained header-only library to push structured data out of process with minimal latency, query data concurrently with a Haskell-like PL. It works well for our kinds of systems (millions of orders per day, ~1 billion market data ticks, structured log data from multiple systems involved, ...).

kthielen··on Do not log
Another voice in favor of logging. At Morgan Stanley we wind up recording about 1TB/day of structured log data, which also needs an efficient query engine to process. We carved a basic type system out for structured data, maps to C++ data well, and built a Haskell-like PL (and JIT compiler built on LLVM) to process it:

https://github.com/Morgan-Stanley/hobbes/

The logging part is just a single self-contained header-only lib, so very easy to incorporate in existing systems.

kthielen··on Programmer's critique of missing structure of operating systems
I agree especially with the proposition that we should exchange structured data (via files, network connections, ...).

I started this project several years ago to support live code updates in production systems without sacrificing performance, but I eventually hit the same issues with exchanging structured data: https://github.com/Morgan-Stanley/hobbes

I split out structured logging into a single header-only lib, structured RPC into a single header-only lib, etc. It's not quite the same as having native support in the OS, but because the mechanism is so lightweight (don't even need to link anything special), it's a pretty good substitute.

kthielen··on Is a Dataframe Just a Table? (2019) [pdf]
> Anyway, I think the way out of this mess is to have a functional, Haskell-like language for processing tables

I agree, and in a Haskell-like setting we can have any structured types (rather than just primitives).

That’s why I built hobbes: https://github.com/Morgan-Stanley/hobbes

kthielen··on Haskell Problems for a New Decade
I can't answer his "so what", I was just trying to clarify what was left implicit in this conversation (if not for you, then maybe for readers who don't know).
kthielen··on Haskell Problems for a New Decade
He means to say that Phil Wadler works for IOHK.
kthielen··on Hobbes – A Language and an Embedded JIT Compiler
Well I've done some work to split small header-only libraries out of the project to make it useful in other places without having to link in everything. Low-latency structured logging, data files, a minimal x86_64 backend (LLVM alternative), structured data compression, network/RPC, message conversion, shared memory data structures, ...

The compression method is pretty neat, has useful applications in other contexts as well. Basically the same way you can view types as propositions, you can view them as probabilities as well (which gives you a scheme for representing the bias in any structured data type). This idea works in other contexts too, like "time compression" for pattern-match translation (choosing an order for tests to match the bias in the data you're deconstructing).

It's hard to find time to write about this stuff with my day job though. :T

kthielen··on Hobbes – A Language and an Embedded JIT Compiler
Well one way we deal with typical unstructured data is to prevent the unstructuring in the first place. The low-latency logging method (for example) uses something like a printf-style interface but stores all of the source data with exactly the types you intended to print -- you can always erase the structure by printing when you want to, but having the structure when you need it is very useful.

What "structure" means and how it works can have a lot of nuance. With hobbes we basically start with algebraic data types (which map to most C-style data structures and so can be shared without conversion with C/C++ code). It's been a while since I looked at R, but IIRC it's a lot like Scheme (e.g. maybe the data sharing/translation story is more complicated?).

We do have some things that are helpful for dealing with unstructured text data, like a built-in LALR parser generator and regex matching (integrated with general pattern matching), but it's not one of the main use-cases we've been focused on.

kthielen··on Hobbes – A Language and an Embedded JIT Compiler
Oh, well there are two stories actually.

The first story is that it follows the tradition of naming programming languages after logicians/mathematicians (e.g. Haskell, Church, Russell, Pascal, ...). Because the Curry-Howard isomorphism (CH) links logic and programming languages, and ideas in programming languages often come from much older work in logic, it's pretty natural to go to logicians for names (IMHO). Thomas Hobbes was one of the first people to connect logic and computation (literally saying "by reasoning I understand computation") and the logical arithmetic he developed inspired other people to continue that work. I think that story is a reasonable justification, works well for a general audience.

The second story (the real story?) is that Morgan Stanley internally has a tradition of naming things after things in popular culture (and at least one or two cartoons), so I named the project after the comic book characters Calvin and Hobbes (CH). Actually the programming language was "Hobbes" and the compiler was "Calvin" because if you know the comic book, Hobbes is really just a figment of Calvin's imagination (which works well to explain how a compiler makes a programming language real). I am a fan of the comics and was reading them frequently with my son at the time. There's a "Calvin and Hobbes ismorphism" (because both characters are different views of the same mind) so it works really well as an extended metaphor for logic and programming languages, I think. :)

kthielen··on Hobbes – A Language and an Embedded JIT Compiler
Nice to see this here again. :)

I made this project several years back, it's been very fun and interesting. I've been meaning to write about some parts of it that haven't had much attention.

kthielen··on Better Intuition for Information Theory
The paper is from 1991, when was your textbook written?
kthielen··on Shakti, the new data platform from Arthur Whitney
Yes I do.
kthielen··on Shakti, the new data platform from Arthur Whitney
Yes there are persisted data structures and databases built on top of them. The whole type system applies to persisted data structures so you can make your own data structures and fill out a few type class instances to query them with plain list comprehensions (e.g. btree indexes are defined and queried this way in some internal tools).

Eager Haskell with a structural type system, deterministic garbage collection, and no overloading penalty, that would be a fair summary.

It'd be interesting to see how GHC would do in the same area, there would be some significant challenges. I haven't tried to use GHC there so I can't say.

kthielen··on Making Swift code extensible through plugins
Agreed, from the title I was expecting Swift language extensions with compiler plugins.
kthielen··on Shakti, the new data platform from Arthur Whitney
Except at Morgan Stanley (where the same could be said for kdb 20 years ago).
kthielen··on Shakti, the new data platform from Arthur Whitney
Morgan Stanley in recent years has used this where they’d previously used kdb:

https://github.com/Morgan-Stanley/hobbes

From the article, it sounds like the Kx folks are working on the issues that led to the development of hobbes in the first place.

I’d be surprised if they’ve changed their opinions about type systems though (or bullying people with lawyers — those guys get nasty when their ideas are challenged).

kthielen··on The Next 50 Years of Databases (2015)
That way of thinking has definitely proven useful in the domain where it started (at Morgan Stanley). We’ve also been looking at how to map it more closely to C/C++ and native hardware, kind of a structurally typed Haskell variant with small header-only views from C/C++ (for native access to files, network IPC, shared mem structures, ...):

https://github.com/Morgan-Stanley/hobbes

kthielen··on Mio – Cross-platform header-only C++11 library for memory-mapped file IO
Pretty cool, must be something in the air. This is a useful technique, and having it in a self-contained header-only lib is handy too.

I made a similar library for writing data into memory mapped files, also a self-contained header-only lib:

https://github.com/Morgan-Stanley/hobbes/blob/master/include...

This one also serializes a representation of the type structure of recorded data so that it can be safely concurrently mmapped and read either with the same code or with the generic PL/compiler that I've developed in this hobbes project.

Where unpredictable disk latency is a problem, we've got a similar header-only lib for logging into shared memory (then have another process to consume this shared memory ring buffer and dump it to disk for concurrent querying):

https://github.com/Morgan-Stanley/hobbes/blob/master/include...

This pipeline works well for having lightweight C++ processes feeding large volumes of data to generic query processes that we can run out of band to look at this data in various ways (with a Haskell-like query language).

We did hit a slight problem doing things this way that the straightforward representation of data (as in memory) for some cases just used too much space and too much time wasted in I/O. Basically for complex market data, where data structures aren't trivial and recording ~100GB/day makes it very awkward to keep around a few weeks of data for random querying.

So I also made this header-only lib to write data into these mmapped files with a simple compression method (I like to describe it as generalizing Curry-Howard to probabilities) that gives us much better throughput, much smaller files, faster query times, and still support concurrent constant time random access queries:

https://github.com/Morgan-Stanley/hobbes/blob/master/include...

It gives us compression ratios about the same as EOD gzip, but much faster and importantly works online and with these query use-cases we have with hobbes.

Anyway, maybe I should write up those details somewhere else, I just mean to say that this is a useful technique and you can push it very far and do many things with it in a very straightforward way.

kthielen··on Dark Motives and Elective Use of Brainteaser Interview Questions
Google has a lot of good people, but in my experience the interviews are specifically looking for a particular bias -- not just knowledgeable and personable people but specifically people who think a certain way.

I had an interview there recently in NYC. I solved the guy's puzzle too quickly and I think that he thought that I'd prepped (I hadn't). Then we got started on a different problem and I tried to explain how it was a good example of "boolean blindness" and that using a specific form of dependent type was an ideal solution. He looked at me like I had two heads and smugly told me that it's impractical to solve problems this way (but it's a bog-standard type checking problem).

So yes, smart people, but you should know the bias that they have before you decide that you want to work there. It could very well be bad for your career if you _do_ get in.

kthielen··on Introduction to Functional Programming in OCaml
Yes, there is only Curry-Howard and all PLs are just projections (to greater and lesser degrees).

Referential transparency, mutation, non-termination, coinduction, memory pointers, linear types, coroutines, continuations, blah blah blah. All of these things exist, and more.

Most PLs are biased in one way or another, many don't even let you reason formally about those things (or allow you to reason informally).

But at least if you're a working programmer, especially if you work to map an existing human domain to software, there's always the fixed point of Curry-Howard to hang on to.

kthielen··on Learn You a Haskell for Great Good
> Instead of just taking Haskell features and porting them > into already adopted languages, I'd really like to see > more companies just go directly with Haskell, [...]

Unfortunately there are some places where aspects of the standard infrastructure around Haskell block production application.

In particular, we have applications with very hard latency requirements and also with complex type/binding requirements (where a direct structural type system is needed):

https://github.com/Morgan-Stanley/hobbes

kthielen··on Python post-Guido
I don't mean to be pedantic but IIRC Python's comprehensions actually came from the SETL language.

Obviously it's ultimately the same thing either way, but that's the lineage (SETL -> ABC -> Python).

https://en.wikipedia.org/wiki/SETL

kthielen··on The Meaning of July Fourth for the Negro (1852)
> Writing software is a trade-skill subject to market forces, literacy is a critical thinking skill more akin to logic itself.

False dichotomy, these things you distinguish are just two sides of the same coin:

https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...

kthielen··on DeepLog: Anomaly Detection and Diagnosis from System Logs (2017) [pdf]
Once you discard type structure, it's a fruitless task to try to reconstruct it.

It's much easier to make sense of logs when we don't discard that type structure.

https://github.com/Morgan-Stanley/hobbes#storage

kthielen··on Ask HN: Have you ever chosen Postgres over Mongo and regretted it?
> 2. you actually don't care about schema that much (e.g. logs)

I would argue that this is one of the most overlooked cases _for_ schemas, because it's where schemas are the most complex and also where traditionally we've had few options for really representing that complexity.

This is something I've tried to change with the hobbes project, where we have a small header-only lib to efficiently record "log data" with inferred schemas into shared memory:

https://github.com/Morgan-Stanley/hobbes#storage

Basically, in this view, the type for recorded log data is a variant over all possible log statements in an application. This is a very large and complex type, but in memory or on disk it's a simple representation -- a number for the constructor ID of the variant, and then the payload type that corresponds to that constructor.

Then there's quite a lot that we build on top of that, but as a basic primitive I think that logging is actually a great case for "extreme schemas" rather than throwing out schemas entirely.

kthielen··on Depressing and faintly terrifying days for the C standard [pdf]
> That said, I really wish proponents of other languages would get their > stuff together about creating libraries that can be used from other languages. > A library written in C can be used by anyone else. Many other languages are > avid consumers of this functionality, many advertise it as a key feature, but > very few return the favor by producing reusable code.

I kind of agree with you on this. I have a bidirectional binding process in my hobbes project at Morgan Stanley (https://github.com/Morgan-Stanley/hobbes) such that C/C++ functions can be viewed as hobbes functions and hobbes functions can be viewed as C/C++ functions (without translation).

This works as far as C/C++ types are adequate for your programming language, which obviously does cover a huge space, but there are types that don't translate well and there are staging considerations that don't translate either (e.g. there is a logical type that can be decided, but it's not decided until what might be regarded as "run time").

There are interesting problems to be considered in this area though, happy to discuss further out of band if anyone is interested (discussions on hacker news tend to be short-lived and shallow IME).

← PreviousPage 3 of 5Next →