Hacker News Reader

Comment by hmate9 | Hacker News Reader

I do not wish to be negative, I just think it is very hard to really judge this system without knowing the technical details behind it. Since we are currently in a bull market it could just be a fluke.

I did not read anything about backtesting. Have you done that? If so, I take it back :)

View on HN

Backtesting does not mean much. It's easy to make an algo do well yesterday. Very hard to make it do well tomorrow.

I completely disagree. Backtesting matters, especially in HFT. What would you rather have: an algorithm that has performed well on 5 years worth of historical data or one that has performed well in the past 2 weeks.

Neither; these are techniques applied on observed data in an obviously dynamic open domain. The data is uncontrolled - so we have no idea if it is representative of the current state of the domain theory - does it cover the distributions properly; we don't know. We know the domain is dynamic in that the world economy moves like around alot and we can't predict these movements, in the sense that most actors did not call the last substantial negative market movement. The domain is open in that new features can appear that drive the value of the variable of interest; for example the availability of property in China, new battery technology, a patent on nuclear fusion.

We do not have techniques that account properly for any of these things.

You may as well draw lines on charts and sell that advice, you'd probably get as good a hearing in any Investment Bank in the world !

oh....

wait...

He's not wrong, backtesting really is not the same as testing on real live data. You are fitting your models on data that has already been done, without any of the risks factored in. Even if you do find a system that picks every ups and downs in the backtest, it will do poorly with live trades because the volatility and risk is something that changes based on fundamental data.

However, in this case, the fact that no backtest or any sort of risk/reward ratios were published suggests high level of skepticism.

tl;dr: past prices can't predict future prices.

In my experience? The later. Regime change is a killer in trading algos.

Further, accurate back testing is very difficult for lots of reasons, the most obvious being your own actions aren't reflected.

Most algo firms use historical data to develop a hypothesis & then many different ways to validate that hypothesis. Test trading being the one with the most weight.

Does backtesting work better if you, say, start with seven years of data, use the first five to judge the strategy, and then test it against the last two? Is that a good way to avoid fitting the data?

It very much "depends". I'll give you an example.

A friend had a technique that worked very well on the NASDAQ 100 with roughly those BT criteria. One problem was overlooked: The NASDAQ 100 changes over that time. So you aren't necessarily investing in the same securities in year 6 as you were in year 5, even though you're investing in the same "top 100". IOW, backtesting in this particular case had future knowledge that wasn't easy to see.

When he started (paper) trading in the real NASDAQ 100, the returns were negative. In that he was lucky: They could have been positive for some time, until the constituent stocks changed.

No.

When you trade in a live market, your actions cause responses. When backtesting your actions get no responses.

True, but it would at least test OP's algo in some bearish market conditions. If no backtesting was done then it has only been tested in the current bull market.

Completely agree with you. The amount of backtesting has been very limited and the time frame was pretty short, so there are definitely changes this is a fluke. Confirming this is my main goal at the moment and I don't take any credit for creating something revolutionary.

Good luck! I am rooting for you.

Reply on news.ycombinator.com