Hacker News Reader

Comment by dominotw | Hacker News Reader

> 10k different series

who has these though? i cannot imagine somoone making sense of 10k timeseries.

All the businesses with product catalogs for example, every product can have several time series associated with it. If a business has 10k employees, you have, at minimum, 10k time series to perform employee analyst operations over. There are several hundred companies in the US with 10k+ employees. There are many more than 10k postal zipcodes in the US to perform forecasts over.

sorry i meant 10k types of timeseries data not individual items. I assume thats what gp meant.

The difference is the same, whether you have 10k series representing the same variables or 10k series representing different variables decomposes into the same forecasting problem that you can either over-generalize across all time-series, producing worse accuracy at the tails, or train a model for each individual forecast, producing worse accuracy in general due to limited data input.

As an energy retailer, I have 10k+ timeseries.

Though you could perhaps classify them first and use much fewer models.

No reason to spend time classifying and worrying about cross-series validation when you use a TSFM.

I forecast 400k timeseries every 30minutes at work.

What kind of infrastructure do you use for that scale? Are these computationally light ARIMA models?

We're running 150k models per day and while I'm sure I could scale this to running them every 30 minutes, it would mostly be me throwing money at the problem to parallelize the compute.

Some HPC Clusters, xgboost, read pretrained model from NFSRDMA Server, read data, parallelise over MPI with one process per model. And the latest Zen5 Epycs. I could do it all on just one of the 192 cores EPYCs but we like to parallelise it over multiple machines to get the extra speed and reliablity.

It wasn't that expensive, the main problem was the storage bottleneck which got fixed with just some LVM cache and gen5 ssds.

Reply on news.ycombinator.com