Ask HN: Why are so few developers with ML background interested in trading? | Hacker News Reader

Ask HN: Why are so few developers with ML background interested in trading?

As someone who has briefly worked in HFT, I am curious why so few people coming from am software development (and especially AI/ML) background seem to be interested in trading. I found it to be a really interesting and intellectually challenging problem.

Recent ML techniques are being applied to a lot of fields, but I rarely come across someone working on trading. When I compare beating pros in Starcraft or Dota 2 from Deepmind/OpenAI to the complexity you are dealing with in trading, I think there some relatively long-hanging fruit to be picked up in the latter, all while working on an interesting real-world problem.

Is it because there is a cultural gap? Do developers believe finance is bad and want to "make the world a better place" instead? Any other reasons?

View on HN

I worked in banking (retail credit risk modelling, then markets as a quant) for just under ten years and left to work in a software company and am now an engineer. The main reasons for me leaving were cultural:

1. Quant-types are generally not as highly recognized or valued.

2. The pace of work in banking can be glacial. It depends on the institution and team, but in general I found it very frustrating to push through changes and I am quite capable of being assertive when needed.

3. A lot of people working in banks are not particularly motivated or ambitious, let alone curious or interested in much outside their own concerns.

4. More than once, I was put in a position where I was asked to compromise my ethical convictions. Not cool.

5. The last bank I worked for was a toxic and abusive workplace. No thanks and never again.

I acknowledge that this is probably more characteristic of the larger banks and perhaps not hedge funds or HFT shops. 1 and 2 are probably true in any large company but I found them much less so in software. 3, 4 and 5 can also happen anywhere but in banking it seems endemic. Everyone I know from banking has a story, particularly markets.

To your particular questions around culture, I don't believe finance is bad and would return for the right company and role. I don't think software or tech has much of a claim on "make the world a better place".

Finally, from my experience in markets:

1. The bar to displace proven and market-tested models and strategies in trading is very high. Traders don't want their P/L messed with.

2. I found hedging and risk mitigation to be the most intellectually challenging part of being a quant. You can build a model to predict prices, but what happens if that model fails? Every model makes assumptions and markets are really good at demonstrating how wrong they are. Figuring out how to not get wrecked is hard and the meta-problem of model error is minimized to a degree by keeping models simple and interpretable.

Probably because many (most?) such people believe that markets are closer to a random walk than not. And beyond that, they probably believe that if you find "alpha", it will always be short-lived because over time the market will price in the effects of your trading decisions. If that's true (and I'm not saying it is) then you're looking at constantly spinning the gerbil cage, to not necessarily make much progress.

I think it's also believed (known?) that to do the real good stuff with HFT and what-not, you need very expensive server co-location and peering connections to minimize latency to the highest possible degree. A lot of people probably aren't interested in doing all that, or can't afford it, or just don't know how to get started in that world.

Here's my personal opinion. I'm mildly interested in HFT, in the same way I'm mildly interested in building a Chess engine, or a Poker AI, or trying to optimize a video codec.

Most of these things, AFAIK, require deep low-level skills in C/C++/ASM and would have enough potential to satisfy the tinkerer in my head.

However:

* If I had time to hone my C/C++/ASM skills, I'd rather join the Demoscene, which would equally satisfy my creative self, and generate a tangible artifact that is entertaining to watch and listen to. With HFT, the effect is ephemeral.

* I personally believe that AI/ML can't be applied to stock markets. No AI predicted the current pandemic, and in order to have it make complex decisions based on complex trading patterns, you'd have to constantly train/feed it with current (geo)-political, environmental and societal factoids that are interwoven and opaque that your job consequentially becomes being the Content Manager of your Ultra AI (assuming the AI is perfect and doesn't need tweaking). The complete opposite is true when you're working in a world with fixed constraits, like a chess game, or Startcraft: You know most of the variables and their mechanics, and how the world's constraints work enough that you can actually model an AI navigating it, and the amount of knowledge it has to be fed is constant.

* I've read somewhere that HFT, as a market niche, has a relatively tiny market volume, which doesn't make it that attractive. If I were to spend my time on something, wouldn't I chose something that has a bigger market?

* From an idealistic viewpoint, while I see nothing "wrong" or immoral with stock trading or HFT, I don't see any value either. When I was 19, when you feel you've got infinite amounts of time on this planet, I probably would have considered getting into HFT, but with each year I'm getting older, I ask myself: Does what I do have the potential to make people's lives easier? Does it make people smile? What does it do that I'll be proud of when retelling my life's battles? If it doesn't, is this really the thing I want to be spending time on right now? From this perspective, for me, working on HFT falls into the same category as working for any AdTech corp.

Reply on news.ycombinator.com