Timestral: Mistral's time-series foundation models
mistral.ai
mistral.ai
> Known-future covariates. Some drivers of sensor behaviour are known in advance, such as production schedules, planned shutdowns, weather forecasts. Timestral can condition on them to improve forecast quality, so a scheduled load increase is expected rather than flagged as a surprise.
To me this should be the entire basis of the system, not a condition to improve forecast quality? Like if you're modelling electricity consumption, if all you have is a window of N previous samples, you're going to get completely beaten by a comparatively simple system that knows the time of day, that there's holidays coming up, the weather forecast, etc.
I would also think that these time-series forecasting models would gain from a sophisticated understanding at least the labels of the covariates? A 5M model has no idea what a "bank holiday" is, it just sees an arbitrary class go from 0 to 1, and won't have any useful inductive biases as to what it means or how to link them to other pieces of information.
E.g., wouldn't you predict €/MWh of electricity far more accurately with access to price of natural gas (itself influenced by many geopolitical factors), breakdown in composition of production of each energy source, predictions about roll-out (requiring a sophisticated LLM capable of understanding news and their implications), etc. What's the use in trying to predict it from 2 variables and a narrow window looking back at just 8192 or 32768 samples?
Or perhaps I misunderstand the use-case and this is more for extremely short term movements?
You're right but, if the time series looks like music then you can apply your music domain knowledge. You don't have to only apply knowledge that's general across every domain, part of the magic of large neural nets is that you can do recognition and prediction at the same time thereby using (statistically) things known from training about how to identify a sequence and then also how to continue it.
Sure adding outside info helps even more, but with enough data it should become unnecessary (or so goes the theory).
Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).
Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.
Won't you need to attribute a weight to the data coming from the model and the weight for your data?
Would love to know actual time series foundation model use cases!
Problem is you really don't even need ML for the curve fitting tools. Just ARIMA or Holt Winters is enough and likely faster. Heck, Kubernetes HPA doesn't even implement full PID, just P.
It seems like the main challenge is going to be finding domains where accuracy is important enough to spend inference token money on, but not so important that you'd hire a team to build a custom model. Or maybe that team borrows Timestral weights to fine tune to the domain, but they probably already have a different model customized to the problem?
That seems like a scenario where there wouldn't be external sources of information capable of improving your predictions. It also seemed to me like a scenario where good old fashioned statistical methods would outperform LSTMs, which was what they used.
(I didn't get the job - I was offered it, but they rescinded their offer at the last moment, supposedly because their investors had insisted they should hire a web developer instead.)
I'm sure someone who actually knows this stuff has a better easy to view it but my impression was that these time series foundation models aren't trained for a specific dataset but more they are trained on a procedure for how to look at datasets and make predictions. It doesn't seem to lump things into a bucket much more than any other model type (which also are just consistent procedures to look at datasets). It's not like you have radically different model types across different fields.
To improve the robustness of machine learning models, artificial noise is often introduced into actual training data. Something like Timestral could simplify the process of generating it.
That being said, more information is of course better, assuming it doesn't introduce bias, like focusing on specific events like holidays too much, when the information is already contained in the historical data, without having to make assumptions about what an upcoming holiday means.
The biggest challenge last time I worked with time series forecasting (which is ~5 years ago) was learning long-term shifts in the baseline of the data (e.g., the general use of electricity always goes up, a lot more solar panels and EVs influence the baseline, etc.).
I can only assume (hope) that there has been some substantial progress in that area though.
(Edited the formatting for a nicer reading experience)
Sure, you could train a XGboost model on a time series with holiday or event covariates, but what if you have 10k different series to forecast for? You could train 10k different models, one for each series, but then they have very little history to learn from. Alternatively, you could train a single generic model to cover all 10k, but then you have to deal with scale mismatch between series in addition to the feature dev overhead between series. A TSFM saves a ton of work in modeling each of these series and could potentially generalize better to produce more accurate forecasts.
who has these though? i cannot imagine somoone making sense of 10k timeseries.
We're running 150k models per day and while I'm sure I could scale this to running them every 30 minutes, it would mostly be me throwing money at the problem to parallelize the compute.
It wasn't that expensive, the main problem was the storage bottleneck which got fixed with just some LVM cache and gen5 ssds.
This is a pretty approachable overview: https://casrai.org/guides/bayesian-hierarchical-models
Andrew Gelman has several books covering this topic in great depth as well.
I would be interested in such guide. Do you have alternatives?
Speed looks nice too. I want an open weight model with this performance now.
Edit : working with GLM5.3 now, it's also slow. Maybe Mistral's under heavy load.
Disclaimer : Not Cofounder of The Forecasting Company here
Something is going on with the performance (as in speed of the service, not quality). Some days it's quite fast, some days it's so slow I interrupt it because I thought it was hanging.
I think they're shifting hosting capacity around, or otherwise messing with the model. Note that it's still a "preview" version, meaning they're still training it further for the final release and they are likely still figuring out the optimal hosting configuration as well.
It wouldn't surprise me to see further changes in both quality of the model and speed before the final release.
<mistral vibe pm here>
I'm looking forward to the weights release as well, has anything about the architecture and possibly quantization resilience been shared publicly so far?
I agree with you though. It's slow as hell, and so is their GLM offering.
<mistral vibe pm here>
You are not authorized to exercise any rights under this license if the global consolidated monthly revenue of your company (or that of your employer) exceeds $20 million (or its equivalent in another currency) for the preceding month.They're French so they can't say "it was implied the US Dollar". This is so weird and broken I can't believe a legal team worked on this.
Typically, in linear, non-linear, and NN models, we fit some noisy data to A) An underlying model with unknown parameters or B) A model free search which attempts to construct combination of "basis" functions or C) A purely black box model which we only care about Input/Output error.
A) is useful because often we know an underlying model we want to match, but don't know the parameter values. If we can determine the parameters, we get insights about the real system. E.g, how are these two variables coupled and how strongly -> We can make business decisions about what happens if we adjust some parameters.
So what exactly does this timeseries foundation model do and provide?
Maybe, the observability is that you have a forecast, and you adjust the covariates and see what happens. But one would want much more insights than a "executive summary" of the forecast.
The important parts for me as a signals processing R&D engineer.
> Classical models such as ARIMA and linear regression remain well-established tools for forecasting time series. These models are usually fitted to a single series and learn only from that series' history.
> Deep learning models such as PatchTST add the capacity to learn nonlinear patterns across many related series, but they are still typically trained from scratch on each dataset, so what they learn doesn't carry over to new domains.
> Time series foundation models bring the pre-training approach to time series. Trained on a large and diverse corpus, our model learns patterns that transfer across series, forecasting zero-shot on data it has never seen, even in domains where data is scarce.
Edit: Looks like it's a pure black box which you can run on premise.
> Output: Probabilistic forecasts, 21 quantiles
> Inputs: Multivariate targets, past-only and known-future covariates
"We have published a GitHub repository so you can run Timestral on your own data."
No need to pay for proprietary TimeGPT.
Calling it "Modified" MIT is just misleading.
Sounds like a great problem to have.
Spacetimeformer was imo a huge leap forward that got overshadowed by all the language model progress
Excited to see more effort put into time series forecasting
lol. so what?
We keep struggling with disk capacity on servers being filled, which will then need cleaning. Currently, we got an alert when it's reaching 80%. It's reactive, we need to work on an unplanned task, and it disrupt our sprint and delivery commitments.
Instead, we could pipe the disk capacity data points to Timestral, and we could approximate the date when we will need to work on it, way earlier, so that we can plan our work accordingly.
I guess it might be integrated later directly in tools like Datadog, or Cloudflare straight away, rather than being used by corporates as it is.
Which is why at the end of the day why nobody trusts or blames the forecasting models, specially in all or nothing types of decisions.
Or the poll-bashing that happened after the 2016 election when an outcome with a 29% chance per one model was observed.
Any interesting forecasting problem is underconstrained. Monte Carlo methods are the classic way to mitigate that one-universe problem.
Nobody can predict the future with certainty. All models are wrong, but some are useful.
I fully agree that all models are wrong.
Their main usage is just replaying the past or encoding the modelers assumptions for the future.
But there is no way to know if a model is good or bad.
There would be if we had infinite universes.
1. No guarantee that the universe tomorrow will behave as the distributions you used in your Monte Carlo today.
2. If the universe stats serving you 10 unlikely events in a row, is your model model wrong, or just 10 unlucky draws? If your model was correct and you start "correcting it" based on these 10 draws, you now made it wrong.
For example you can look at the calibration of the forecasts: Take all the outcomes where you estimated ~10% probability. How many of them actually happened? If you are an well-calibrated forecaster, the actual number should be fairly close to 10%.
Brier Score:
Honestly this might be very good for feature selection to refine a much more specialized model.
With such strong human capital, both in quantity and quality, Europe should in theory be competitive with US and China in AI.
"Mistral is so bad, it's the only AI my school doesn't bother to block"