hrzn

A few years back I made a modest attempt at writing a concise yet readable sudoku solver in Python - in about 29 lines: https://github.com/hrzn/sudoku/blob/master/sudoku.py

Could have been made shorter at the price of readability.

We likely overestimate AI's short-term impact, and there might even be a financial bubble about to pop. But I also think we underestimate the long-term impact. We're building absolutely amazing capabilities faster than many would have thought possible only a few years ago - I especially think applications to science and engineering will be huge and transformative.

> Python had nothing comparable

This is in part why we built Darts. Now I think we can say the situation is quite different. Darts offers many things offered by the R forecast package, and then some (for instance the ability to train ML models on large datasets made of multiple potentially high-dimensional series).

I would recommend Darts in Python [1]. It's easy to use (think fit()/predict()) and includes

* Statistical models (ETS, (V)ARIMA(X), etc)

* ML models (sklearn models, LGBM, etc)

* Many recent deep learning models (N-BEATS, TFT, etc)

* Seamlessly works on multi-dimensional series

* Models can be trained on multiple series

* Several models support taking in external data (covariates), known either in the past only, or also in the future

* Many models offer rich support for probabilistic forecasts

* Model evaluation is easy: Darts has many metrics, offers backtest etc

* Deep learning scales to large datasets, using GPUs, TPUs, etc

* You can do reconciliation of forecasts at different hierarchical levels

* There's even now an explainability module for some of the models - showing you what matters for computing the forecasts

* (coming soon): an anomaly detection module :)

* (also, it even include FB Prophet if you really want to use it)

Warning: I'm probably biased because I'm Darts creator.

[1] https://github.com/unit8co/darts

Note that the actual EV of a Cauchy random variable is undefined...

You can install darts with conda using

    conda install u8darts-all

We are working on improving our support of conda based installs.

You can try installing it using

    pip install 'u8darts[torch]'

This will give you everything besides Prophet and pmdarima.

I would say that compared to Greykite, Darts really attempts to unify a wide variety of forecasting models under a common simple and user-friendly API. There are many differences, but for instance, AFAIK there's no deep learning model in Greykite (it focuses on two algorithms: their built-in algorithm and Prophet), whereas Darts tries to lower the barrier for using deep learning models for forecasting. Crucially for ML-based models, it also means being able to train on multiple (possibly thousands or more) of possibly multi-dimensional time series.

In short: yes, at least anomaly detection. That's one of the main things on our to-do list.

Oh, nice, I didn't know that. I've edited my reply, thanks!

We do have ensemble models. Something like

    model = NaiveEnsembleModel([model1, model2, ...])
    model.fit(my_series)
    prediction = model.predict()

Will return an average prediction. Look at RegressionEnsembleModel for an ensemble model which uses a regression model to learn how to combine the individual forecasts.

At the moment Darts doesn't have hierarchical reconciliation methods (if that's what you meant), but it's on the backlog :)

Thanks for the feedback, I absolutely agree about the need for easy-to-use tools for dealing with time series. This is exactly the motivation that prompted us to work on Darts initially.

I like your suggestions of adding comparison to the few other libraries out there, as well as explaining the need for having our own TimeSeries data structure. We should try to do that sometime soon.

Concerning dependencies, we already have some dependencies as extras. "pip install darts" will install everything, but "pip install u8darts" will install only the core (without Prophet and pmdarima), or "pip install u8darts[torch]" only the core+pytorch models.

> Or is it just turning every column in my pandas dataframe into a series to pass into the covariates array?

Basically if you have a multivariate series represented as a pandas dataframe with several columns, the way to go is to create your TimeSeries by calling TimeSeries.from_dataframe(my_df). That will return a multivariate time series.

We don't yet have a discord channel, but I'm planning to open a Slack channel sometime soon. If you have other questions feel free to drop me an email: julien@unit8.co

You're welcome, glad you like it and thanks for the feedback :)

In some cases Darts is wrapping around existing models (like Prophet, or statsmodels-based models for instance); in other cases we wrote our own implementations, so it's really a mix.

Hi! I'm one core developer (and creator) of the library. Thanks for all the comments. I just wanted to highlight a couple of things that we think are quite cool about Darts:

* It makes using all sorts of forecasting models (from ARIMA to deep learning) easy, using fit() and predict(), similar to scikit-learn.

* It's easy to fit deep learning and other ML-based models on multiple time series, potentially on big datasets too. The time series can be multivariate.

* Darts is not only wrapping existing models. We also have our own implementations, for instance of TCN (Temporal Convolutional Networks), or adaptations N-BEATS (which we extended to handle multivariate series), DeepAR and others.

* Darts makes it very easy to include past and/or future covariates as inputs for the predictions.

* Some models offer probabilistic forecasts; sometimes with the possibility to configure your favourite likelihood function (e.g. Gaussian for continuous values or Poisson for discrete values).

* Everything uses the "TimeSeries" class, which makes the API consistent across tools and models, and make it harder to make mistakes. For instance it's easy to consume the output of one model by another model, and all models can be backtested the same way.

This is supported but only by neural-nets models, which are fit using SGD, hence naturally not requiring the whole dataset in memory. Other models like ARIMA do need the full series loaded in memory.

The models that work on multiple time series in Darts accept Sequence[TimeSeries] for their fit() method. These sequences can either be Lists (fully in memory, simplest option), or when needed it can be a custom Sequence which for example does lazy loading from disk (somewhat similar to what PyTorch Datasets are doing) with the __getitem__() method.

If you need even more control, for instance because you have only one very long series that doesn't fit in memory then you can implement your own Darts "TrainingDataset". In this case you can control how to slice your series exactly.

Edit: I realised this only answers the first sentence of your comment ;) For now there's no mechanism for scaling to multiple machines beyond what PyTorch is already offering. AFAIK it's reasonably easy to scale to multiple GPUs on a machine, but I'm not sure how it would scale on several machines. We never had to try this yet! (Note that actually a single CPU can handle training deep nets models on 10's of thousands of time series similar to the M4 competition in a fairly reasonable time).

Both - some models are wrapped (like ARIMA & ETS around statsmodels, Prophet around fbprophet) and we write others ourselves (RNNs, TCNs, N-Beats, ...). Basically we take a pragmatic approach here, we do whatever is best to use a given model in Darts.

For training forecasting models on multiple time series (and potentially large datasets), you can take a look at Darts [1] and the blog post [2].

[1]: https://github.com/unit8co/darts/

[2]: https://medium.com/unit8-machine-learning-publication/traini...

For those interested in time series library, we are developing Darts [1], which focuses on making it easy & straightforward to build and use forecasting models. Out of the box it contains traditional models (such as ARIMA) as well as recent deep learning ones (like N-Beats). It also allows to easily train models on multiple time series (potentially scaling to large datasets), as well as on multivariate series (i.e., series made of multiple dimensions). It will soon support probabilistic forecasts as well.

[1] https://github.com/unit8co/darts/

A lot of these issues also have to do with the publishing system.

There is no incentive in publishing stuff that don't work even though they were reasonable things to try. This indirectly pushes a lot of bad/flawed/incomplete papers and results to be published. Even if your methodology is flawed but you try hard enough, you have a >0 probability of getting your paper published somewhere.

As part of a solution, I think we would benefit from the existence of prestigious journals for negative results, which would incentivise also publishing what doesn't work. This way researchers wouldn't have to try massaging their data and experiments until it looks like it works. They could just publish that it doesn't work, and it would be good for them too.

Even if these "noticeable" sequences make up only a tiny fraction of all possible sequences, the probability that they show up at least once in a while among all of the world lotteries is actually high.

In general the probability that some unlikely events occur is high.

"Why We Sleep" by Matthew Walker is an excellent book for those interested in what science has to say about sleep (disclaimer: it's excessively good for you)

What they meant is that we evolved in a way that made teenagers having shifted cycles, going to sleep later and waking up later. We're not living in tribes anymore, but evolution is slow and we still cary these biological traits. It therefore hurts teenagers to wake up as early as adults. The point here is not about leaving them more time to socialize in the evening.

Thanks! Since then I actually took a bit of time to read the whitepaper of Raiblocks and bought a few more. The tech looks nice.

I'll give it a try as well :) xrb_1pjzyhtcjwg5u3ot88dfa53bu6tf31syixpz78t9mq9bcfmjo81j7sjysox1

I encourage you to consider EPFL or ETH in Switzerland. Both are top-level universities in CS, where Iranians are welcome. The next deadline to apply to EPFL Phd school (EDIC) is in April. I wish you the best!

I Absolutely agree. Overall it's probably a mix of many things like luck, talent, insider information, ability to go against dominant investing ideas, where you start in life (if you are wealthy at birth that's equivalent to many coin-flips), etc.

> what is more improbable? that he is flipping a coin 40 times and got head all the time or that billions of people cant see what he sees?

Actually, the probability of getting a head 40 times is 1/2^40, which is 1 in about 1.1 trillion. If we assume there are 1 billions investors overall, the probability that at least one wins 40 times (which is the probability that one or several Warren Buffet could appear by luck in this simple model) is about 0.00091.

So yeah, in this particular phrasing of the question, with this model, the less likely option is to flip a coin and get head 40 times and the more likely option is that there is something else, like talent.

Gain background knowledge first, it will make your life much easier. It will also make the difference between just running black box libraries and understanding what's happening. Make sure you're comfortable with linear algebra (matrix manipulation) and probability theory. You don't need advanced probability theory, but you should be comfortable with the notions of discrete and continuous random variables and probability distributions.

Khan Academy looks like a good beginning for linear algebra: https://www.khanacademy.org/math/linear-algebra

MIT 6.041SC seems like a good beginning for probability theory: https://www.youtube.com/playlist?list=PLUl4u3cNGP60A3XMwZ5se...

Then, for machine learning itself, pretty much everyone agrees that Andrew Ng's class on Coursera is a good introduction: https://www.coursera.org/learn/machine-learning

If you like books, "Pattern Recognition and Machine Learning" by Chris Bishop is an excellent reference of "traditional" machine learning (i.e., without deep learning).

"Machine Learning: a Probabilistic Perspective" book by Kevin Murphy is also an excellent (and heavy) book: https://www.cs.ubc.ca/~murphyk/MLbook/

This online book is a very good resource to gain intuitive and practical knowledge about neural networks and deep learning: http://neuralnetworksanddeeplearning.com/

Finally, I think it's very beneficial to spend time on probabilistic graphical models. Here is a good resource: https://www.coursera.org/learn/probabilistic-graphical-model...

Have fun!