Whats the goal of this though?
- First is to actually evaluate whether these LLMs have any intelligence around investing. If you actually give them all the data, can they do well? Can they beat the market? I'm not sure, we're testing that.
- My thesis is that they will actually beat the market (I know a lot of you will disagree). If that's the case, how can we invest a lot of resources in building the best harness, tool calling, etc to enable these models to invest.
What does "all the data" mean here? I see you mentioned SEC posts. What about news articles, twitter / blog / other posts, general info on the industries, etc?
I assume these are simulated trades, not real trades being executed. How accurately do you take into account trading fees, time from order-decision to order-placement, and things like this?
I would be interested to see the same test run on some prediction market (kalshi / polymarket / etc). In the stock market, a rising tide lifts all boats, so it's easy to deceive yourself about how well you've done, vs how important initial timing was. I suspect that prediction markets will eliminate that source of noise, since it's truly a 0 sum game. That said, it also adds lots of complication, insider trading will eat into your performance more, etc.
LLMs get you to average.
LLMs are not good at decision making under uncertainty.