Hacker News Reader

Comment by BeetleB | Hacker News Reader

My question for everyone: Where do people get reliable data for back testing? Ideally I'd like data that goes back over 20 years. And with relatively few data integrity issues (e.g. does not exclude companies that no longer exist).

Even more important: How do I know my data is accurate?

View on HN

I get my data from quandl.com

If you do equities Quantopian gives you a free backtesting platform with tick data going back to 2003. Pretty useful to just start hacking some ideas on it.

There isn't an easy answer for this. Large hedge funds have entire teams whose only job is to collect, process, and clean data.

I know, which makes me wonder how/why people are doing algorithmic trading on the side when they don't have a reliable way to backtest.

I don't want minute by minute data. Average price for the day is fine with me. Or max/min/average.

There's never a very 'reliable way' to backtest, as any interaction you would have done with the market is not accounted for. If you intend to trade (very) low volume it might work decently (on longer timeframes). Existing (open source) and my own home-made backtester use tweaks like slippage to try and 'simulate' this market interaction.. A few I have seen actually use tick-by-tick L2 data to try and get closer to the 'truth'. IMHO, the only really reliable way to evaluate a trading algorithm is to trade it live. This will cost you money, unless you get everything perfect the first time, but doesn't any kind of passive income generation require an initial investment? And yes, I have written, and currently operate, my own (quite basic) trading bot. Yes, it's profitable.

If I ever get into it, I do want to do low volume, with a longer time frame (minimum would be 5 years) - which is why I don't need minute by minute data.

I don't mind paying for data if it's not too expensive.

I use a combination AlphaVantage, RobinHood and Yahoo Finance. None are perfect by themselves and I'd be willing to pay for 20 years of intraday data.

Reply on news.ycombinator.com