Arms Race in Quantitative Trading? | Hacker News Reader

Arms Race in Quantitative Trading?

spokutta.wordpress.com

34 pointscellis27 comments

"An easy example illustrating this point is maybe the following. Consider the sequence 'MDMD' and suppose that you can only store, say these 4 letters. A 4-letter-strategy might predict something like 'MD' for the next two letters. If those letters though represent the initial of the weekdays, the next 3 letters will be 'FSS'. It is impossible though to predict this sequence solely using information about the past on the last 4 letters. The situation changes if we can store up to 7 letters 'FSSMDMD'. Then a prediction is possible."

I was confused for a while, then I realized the guy is german.

Montag (Monday) Dienstag Mittwoch Donnerstag Freitag Samstag Sonntag (Sunday)

I disagree, those milliseconds are important as the market is continuously moving. From the time a trade enters your system until it's booked and executed you're carrying the risk that a deal you think you've done might not have been done. When you're processing thousands of trades a second that's not a small risk.

Although that's true, it's an unnecessary problem they've created for themselves through their now over-simplified trade matching and naive insistence on time-priority right down to time-frames so small that nobody could possibly care outside of the lame system.

Think about it - even if you got the news the president was shot, or even if the unemployment numbers are significantly off, you'll only be able to take small bets initially due to the lack of confidence in the news until further confirmation.. And confirmation can only come from reading or hearing words, which even if a computer does that, somebody has to type or say them... which means seconds, not milliseconds, much less microseconds.

The 'news' events they're reacting to are financial events at the level of price changes, order book depths etc. They don't react to things like the president being shot, and I'd imagine that if that did happen then at least some of those traders would be switching their models off - the markets could well go crazy at that point and a model won't have been back-tested in those conditions, hence you have no idea what will happen.

I think we agree on what you said, but I take it further and claim it's evidence that the common trade matching rules are badly flawed.

As a simple example, consider that common limit order matching rules often let time take priority over anything else (price, quantity, market maker status). That fails their stated goals. They've got concurrency bugs. It initially came from naive implementations of pit trading rules, but it's now often touted as a feature.

As a result, customers often get screwed into trades they didn't want, or get worse prices than what's fair. PLUS as a result of all the jockeying, they pay higher fees to support the escalating infrastructure costs.

I don't see what's so unfair about effectively a first-come first-served system, or to put it another way, how a fairer system could work.

Perhaps more to the point, exchanges compete for business, if the rules used in an exchange aren't what customers want then customers will move to other exchanges. And obviously the more financial clout you have as a customer the more an exchange is going to care about your opinion of how things should work. So if the banks and hedge funds and the other big players are happy with the status quo, then that's the way it will probably stay.

First-come first-served is fine when there's consensus about the price. A lot of, if not most of, trading comes from people disagreeing about what a fair price is though - the buyer thinks it's worth more, the seller thinks it's worth less.

Consider 3 orders: Bob bids $9, Sam offers $8, and Sara offers $5. If they're entered about the same time, you may end up with a matched price of $5, $8, or $9 with either Sam or Sara getting to sell - all depending on which order the match engine processes them in.

Same orders, same time, huge variation in what is determined to be a fair price with no good reason. All those prices are wrong too. You can debate about what'd be right, but eBay's pricing system is at least an improvement here (one minimum price increment past the next-best order).

It gets worse - consider if one of them entered their order well ahead of time, it probably gave them the worst price - penalizing them for exactly what people want - a quote to see and trade against.

There's tons more examples, but they can get pretty complicated.. email me if you want to talk about it more.

If you're still having trouble imagining a fairer way, work out what would've happened with these three traders on the old NYSE floor system.

I wonder why the term 'market efficiency' crops up over and over again in the discussions like this?

In 1986 Andy Lo and Craig MacKinlay performed a rigorous statistical test of Efficient Market Hypothesis and decisively rejected it, simultaneously supporting previous findings by Benoit Mandelbrot. Unlike Mandelbrot’s their work has been broadly discussed and accepted by the scientific community.

I remember talking to a guy working in this field when I was job hunting earlier. For all the talk of CPU speed, there seemed to be little interest in nontraditional architectures (e.g., FPGAs, etc), for the reason that their incoming data rate was the primary bottleneck.

Then again, I may have been talking to the wrong one. E.g., maybe a few've gotten some of those fpga ethernet boards? Still, I'd expect a little more VHDL demand than I'm seeing.

HFT = http://en.wikipedia.org/wiki/Algorithmic_trading + http://en.wikipedia.org/wiki/Complex_Event_Processing

Is there anyone on HN who works as a programmer in this field who would be willing to share some of their experiences?

What sort of stuff do you want to know ?

Just any notable stories (anonymized, of course). I wouldn't really know where to start asking questions, but here are a few:

What sets the field apart from other areas of programming, in terms of your problem-solving approach? What kind of special qualifications do recruiters look for?

Also, what kind of platform (OS, language, etc.) do you use? Do you hand-optimize the code?

Generally C++ is the dominant language (although Java is growing), unix the platform (Solaris and Linux mainly). I know at least a couple of firms that are embedding code onto programmable network cards for equities trading.

If you go to the eFinancialCareers website and search for "low latency" you can see the kind of thing recruiters look for.

In terms of problem solving it's not that different from other areas. Concurrency is obviously very big (lock free algorithms, etc) as is being comfortable with low-level stuff. You don't need it all the time but sometimes you do need to be able to read through kernel code and get an intimate understanding of things like tcp/ip.

Generally C++ is the dominant language (although Java is growing)

Do you typically hand-optimize the assembled code?

No. Some places do hand-optimize particularly cpu intensive parts of code , but it's not the norm for most code.

I presume the guys who embed code on network cards probably do it more often, but I don't know.

Yaron Minsky talks a a bit about this: http://janestcapital.com/?q=node/61

I want to know what the ratio of "junior developers" to "MIT PhDs" is. As an economics+machine-learning+software-engineering autodidact, I firmly believe that I am qualified to provide insights in high-frequency trading; but, the impression I get is that, since it's such a cool and profitable field, the firms have their pick of the elite-of-the-elite.

They do. And they probably won't give you a second look.

I know programmers at one of these kinds of firms, without degrees and making $200-500k per year, including bonuses. It's all about talent.

Try Jane Street.

When I worked in the field (not HFT, but automated trading nonetheless), I was the only one in my cluster of desks with a lowly MSc. Even the QA girl was a maths PhD. It was funny, all the alpha-PhD male programmers were always competing over who was the second-smartest... The smartest guy was light years ahead of anyone else (and the nicest bloke you could hope to meet). I always used to say, the rest of them should pick up his laundry and wash his car so he could concentrate more (they really liked that, HAHA).

I didn't do anything beyond the simplest algorithms, most of my work was on interfaces between our code and other stuff (like databases, data feeds, etc). Just grunt work really. Someone's got to do it tho'...

A touch off topic, but is anyone else dismayed at seeing so many bright friends disappear into banking as soon as they graduate? Nothing wrong with making money, but I just wonder how many inventions and discoveries are lost this way.

I suppose maybe the counter-argument to that is that the technological advancements that come about as a result of finance pushing for them, propagate out to the wider world and can help speed up the rate of discoveries as new technologies become available.

I also have the feeling that quite often discoveries and advances are not really to do with the discoverer but to do with the fact that an idea's 'time has come'. Take Google for example, they revolutionized search, but supposing the founders had headed off to Goldman Sachs rather than doing their PhDs, what would have filled the gap? I think that the stage was set for new things to happen in the search area and that other researchers would have come up with similar ideas.

I wouldn't want to say that the inventor is completely incidental to the discovery; I appreciate that there are brilliant people out there, and moments of inspiration, I just think that the world will steam-roller on with or without a particular person and we'll often end up at roughly the same place.

Anybody have any pointers/books about into getting into Quantitative analysis/trading?

All these people hang out at: http://www.wilmott.com/

I think SEC must confine HFT between 9am - 10am

Reply on news.ycombinator.com