Comment by short_sells_poo | Hacker News Reader

Comment by short_sells_poo | Hacker News Reader

Parent Full thread

short_sells_poo

No. Most "advanced" ML methods are extremely difficult to get to work in finance. You have a limited dataset that is low signal to noise ratio, highly non-normal and heteroscedastic. Crucially, you can't easily make more data (and no, synthetizing new price data is not an option in most cases). This makes the upfront costs very high, and you then have to prove that you can beat the performance of simple models, and not just by a little. A complicated ML model that offers a 10% better risk/reward ratio (e.g. 2.2 Sharpe instead of 2) is a complete failure, because you traded a simple model that is easy to reason about intuitively for a total blackbox that is very difficult to understand.

Sure, you've got the quintessential marketing induced ML overlay that many firms do, but in all cases I've seen so far it's completely defanged and really there only so that it can serve as a marketing move.

Really depends on time horizon you're talking about. They actually work pretty well for HFT, there's plenty of data around, and most of the information is just in market data - no nasty low frequency stuff to deal with like news, earnings, alternative data, insider trading, butterflies flapping wings in china etc. But the problem is by the time your GPU spits out a datapoint somebody else can go in and trade a few thousand times in the meantime. State of the art on the most heavy competed exchanges is that your fpga (or even asic) with a fiber connected directly to the exchange needs to start sending ethernet/ip headers even before it made up its mind what it wants to send in the payload.

At lower frequencies when the data gets thin and noise/overfitting is a major problem, yeah it makes sense to use simpler models. Bias/variance tradeoff in action.

Fair points. I talked with a couple of HFTs and the general view was that their asymptotic backtests look promising on GPUs but that for most of the liquid markets the latency is way too high - basically confirming what you write.

On niche markets with low liquidity, one doesn't have such tight latency envelopes but those markets also offer more opportunities in general so again there's no real justification to use fancy ML models or GPUs in general.

GPUs are too slow for HFT trading, yes. Deep neural networks in general do work in this domain, people do use them, but the stuff that you can profitably deploy into production is not going to be your vanilla garden variety neural networks, there's a lot of extra engineering required to make it fast enough.

Reply on news.ycombinator.com