radikalus

I've only skimmed very briefly each chapter, but I'm a pretty big fan of PyMC and am extremely impressed from what I saw regarding its use and how to think about optimization problems from a Bayesian framework. Chapter 5 and the Dark World example were particularly interesting.

Greenlight Trading (Chicago, ???) -- Full Time, Intern

We're one of the many, many HFT startups in Chicago; we're looking for C/C++ people who love sockets (TCP/UDP not wrenches). Hardware design skills are a plus. (Verilog/VHDL/OpenCL not hammers)

If you instead feel more comfortable talking about multivariate time series, subspaces, manifolds, volatility surfaces, or just general data analytics, we'd also love talking to you.

Financial industry and/or low latency experience are not musts. Drop me an email if you're interested: green at glt-llc.com

How's the tech meetup scene in Tokyo these days? I'm there 3-4 times a year, but in previous years, I haven't really found much to crash.

I'm definitely excited to play with this; I do a lot of manifold, subspace etc type stuff in finance and am slightly terrified of DBNs but see them as the necessary progression in what I'm working on.

Yup -- this is the most common stack for quant nerdotry

Beautiful presentation; really loving all the stuff built on top nbviewer.

As an aside, I think an enterprise-quality zipline solution could be a worth a pretty fat chunk of $$ especially considering that the budget for KX and its support is generally > 1M per year per firm. (Yes, there's complex logistics and legal issues)

I believe seeing on github some requests for h5 support -- this + support could potentially be a big inroad...

Interesting practice project; I think the distinction between algorithmic trading and HFT is largely one of semantics.

I think we just generally label the higher latency, lower frequency strategies old fashioned 'stat-arb' -- this better connotes that the 'edge' from these trades is derived out of superior mathematical modeling and not better execution.

I highly recommend decoupling the simulation/back-testing framework from your strategy work.

Building a reasonable simulator is no small task -- market data is often NOT an accurate depiction of what is executable and there's a fair number of corrections/assumptions that must be made to reflect this.

Either way, I've always liked PSO -- I highly recommend playing with DE (differential evolution) as well. I've used DE to tune parameters for many strategies with great success where SGD would have been, well, painful.

I use HDF5s for storage/analytics of tick data; my experience has been that the performance for storing large sparse matrices is both expensive in storage space and speed.

I know that a lot of my peers in HFT have an illicit love for column stores, but for a lot of work, there's the need for converting to 'wide' format, which quickly can take a 10 million row matrix with a few columns to one now with a few thousand columns. (And thus stuff like KX, FastBit, etc becomes sort of suboptimal)

The need for massive 'last-of' information for time series leads to basically abandoning python/pandas/numpy and using C primitives and doing a lot more than you'd typically like 'online' but really a lot of this could happen behind the scenes with intelligent out of memory ops.

So...I'm pretty excited for innovation in data stores -- I look forward to seeing more!

This would be awesome. (Time to bust out your DBNs)

Most everybody I know (in the field) is in disbelief concerning the length of time the Knight disaster unfolded over. I've had systems "loop" in BAD ways, but never for more than 1-3 seconds.

Safeties are paramount in the field. There's many layers of protections to avoid *ing yourself. Most of these are deployed in ways that are not terribly expensive on the 'critical path' -- that said, there are a LOT of people doing REALLY stupid things in the field. (As there are in any field, it's just more 'scandalous' when an error in a financial application occurs)

Algorithm implementation is largely trivial, and the high pricetags paid developers in the industry are usually for specific engineering skills along with the semi-requirement that developers be 'all-hands-on-deck' when the situation calls for it. (There's a bug in the new strategy...well you need to fix it before tomorrow's open etc) At least, that's how I see it.

It slightly terrifies me that all the most popular groups are variants of "No fap"

Until the pay is comparable to finance, good luck?

I'd love to work on (arguably) cooler problems, but the combination of lower pay and the constant need to use the "hot new thing" to solve problems doesn't make transitioning look remotely attractive.

Really, the second is the HUGE obstacle: - You don't know anything about aNNs? Sorry, no job. - Nobody uses aNNs anymore, SVMs are all that matters. Sorry, come back after you catch up. - SVMs? Man, we need someone who's got expertise in optimizing RFs and Bayesian Trees. We don't want "black box" machine learning. We need to "understand" the results. Sorry no job. - Decision trees? GTFO man. We're doing rNNs now. - I'm pretty impressed with your data mining knowledge, but we're looking for someone with a background in DLMs and GPs. Sorry, no job. - repeat until vomit/suicide

I kind of wonder about the need for "badass" math skills; I'm not terribly convinced that math wizards are extraordinarily high value relative to people with other types of data analysis skills.

This hits very close to home for me as well. (I constantly witness how the race to lowest latency is a race to lowest security and lowest safety)

To be fair, a lot of exchanges mandate specific order validation pre-checks and specific firewall hardware.

You need to make a lot of money to finance your spaceship startup.

One would think, with the story coming from Bloomberg, that they might mention that finance almost exclusively hires older programmers. Hedge funds, trading firms, and even the banks place a significant premium on expertise and are generally staffed by a much older crowd than what you'd find in Tech. (For many of the reasons you'd expect...)

A pretty small subset of HFTs are building their own hardware from my perspective; there's a lot of people claiming to be doing awesome stuff on FPGAs.

I think to be successful your team needs to be strong in four key areas:

- Trading (Strategy) - Software - IT - Research

There isn't necessarily one of the four that trumps the rest, and different players have different relative strengths across those four disciplines. (Which are as you would imagine, not terribly specific to HFT)

I imagine the question is developer-focused, so on the software side, I think I value most highly:

- Don't fuck up. This is obvious, and the trading adage of "Don't lose money" is overdone...but essential. Assume you will fuck up and have nets in place to catch you. Doing safeties well isn't sexy and doesn't add any edge, but I've witnessed "unlikely" safeties save trading systems.

- Understand the hardware. Undestand the strategy. You cannot write fast "generic" code. Have a sense for how fast you could optimally be on the hardware and network stack you're using. This requires interfacing with IT and, sometimes, trading, as some "strategies" traders come up with aren't easy to do FAST; you might be able to suggest something similar that is potentially much quicker on the critical path.

- If you really want to compete in low-latency, C++. If you just want to be pretty fast and your strategies aren't competing for specific orders with the other market makers, I don't think it matters. You can probably do a lot of strategies in 100us in any language. GUI can be done in anything, but it should be able to scale. Most of the GUI pieces are pre-existing. Sortable tables for fills. Ladders for displaying markets, etc. I've used Java/C#/C++ GUIs -- C++ has always worked best for me, but I wouldn't be opposed to doing the front-end in JS/HTML honestly.

- Understand the critical path. This and, to some degree, only this, has to be FAST. Obsessing about how to optimally build a theoretical value cache more quickly etc etc is often not that big of a deal. (It's not on the path)

- Linux kernel and TCP/UDP stack DETAILED understanding. (Or verilog I guess...)

- Locks. When does a mutex really need to be locked? Obviously depends on system architecture. And can often dictate architecture.

I think almost anything else I'd say would be either vague or generic. I think there's a lot of niches one can carve out in HFT and, while I personally value generalists who can holistically understand the goal (make $), there's a ton of people doing really well by just being the most badass at X/Y/Z.

I strongly advocate pace as well; I don't think it's THAT dependent however on hours/week. I prefer working longer hours when my pace is high and working less when I'm feeling not very effective.

I think it is dangerous to "teach" people that they should only do what they love. So much of character comes from the determination to "do the right thing" even if that isn't what you want to do at that particular moment.

I take a lot of personal pride in being able to persevere; lots of people do -- why downplay this or regard them as "suckers"?

Random "fact" 1. Random "fact" 2. Priorities.

Why did I read that as expert-sex-change?

It instead was "nerds are too nerdy"

This is slick...but:

How often do you want to generate random walks of this type where the variance of the process isn't dependent on its current level?

Observe that, as you increase the standard deviation of the random normal (to even .1), your "random walk" always walks to zero.

I don't mean to be a beady-eyed-pterodactyl but, as cool as clever one-liners sometimes are, often they solve toy problems. I say this as someone who loves R and uses it everyday and constantly is forced to brute-force with ugly inlined Rcpp code. (Which is fine)

One counterpoint is that at the bleeding edge of low-latency, pattern analysis and sig-int become once again extremely meaningful.

Can you execute your strategy faster than your opponents if you go to a slightly more aggressive (less arbitrage-y) signal?

Should you? (Why bother if you're faster?)

For what situations is it worth "thinking longer"? Some straight arbs require speed beyond what you can do if you want to use your HOT "smart" model.

If you work outwards from the fastest "stupidest" trades, there's a vast array of strategies/opportunities that intersect ML/AI, hardware design, network optimization, and so forth -- I do agree that if you're looking at bad, inaccurately sampled tick data, the opportunities aren't really there anymore. (Because there's increasingly more players correcting relative value mispricings)

This looks pretty excellent. (Based on my playing with it for 30 seconds)

I'm constantly underwhelmed by Bloomberg and would love to see more players in news aggregation/analysis space in finance.

Awesome. Perhaps allow for annualized salaries?

I forgot all about this

Yeah, it's an acutely self critical meme.

I might jokingly label myself a "finance douchebag" but it's more a dig at my peers than an affirmation of the industry's behavior patterns.

The culture was becoming frat-house/locker-room long before the term was coined...(Now is there a feedback loop from memes like this, who knows?)

Getting upset over a meme seems kind of pointless. Plus the internets will always win.

I'd imagine tanimoto or cosine similarity would get you most of the way there while being very off-the-shelf.

If you're going to go the route of binary classification, I'd personally do it via RFs as variable importance (product importance) is built in. (But that's just personal pref)

I think that it's a solid step-by-step thought process on tackling the problem -- I'd probably think of the false positive vs false negative in terms of the relative expected values of success/failure in those classifications. (And perhaps cost-sensitivity could even be added to your original classifier -- perhaps if you had a forest of 500 trees, and you get even 100 votes for pregnant, that's enough to decide to send a pregnancy-targeted mailer)

I'm not sure if I was starting fresh I'd go with R anymore -- it's so hard to leave R when you've got a toolkit of 50+ packages you need though. =(