Visualization of a dense grid search over neural network hyperparameters

AutoML and [Partially] automated feature engineering have hyperparameters too. Some algorithms have no hyperparameters. And, OT did a complete grid search instead of a PSO or gradient descent, for which there are also adversarial cases.

Featuretools supports Dask EntitySets for larger-than-RAM feature matrices, or pandas on multiple cores: https://featuretools.alteryx.com/en/stable/guides/using_dask...

"Hyperparameter optimization with Dask": https://examples.dask.org/machine-learning/hyperparam-opt.ht... :

> HyperbandSearchCV is Dask-ML’s meta-estimator to find the best hyperparameters. It can be used as an alternative to RandomizedSearchCV to find similar hyper-parameters in less time by not wasting time on hyper-parameters that are not promising. Specifically, it is almost guaranteed that it will find high performing models with minimal training.

Note that e.g. TabPFN is faster or converges more quickly than xgboost and other gradient boosting with hyperparameter methods: https://news.ycombinator.com/item?id=37269376#37274671

"Stochastic gradient descent written in SQL" (2023) https://news.ycombinator.com/item?id=35063522 :

> What are some adversarial cases for gradient descent, and/or what sort of e.g. DVC.org or W3C PROV provenance information should be tracked for a production ML workflow?

https://x.com/jaschasd/status/1756930247633825827 :

> So it shouldn't (post-hoc) be a surprise that hyperparameter landscapes are fractal. This is a general phenomenon: in these panes we see fractal hyperparameter landscapes for every neural network configuration I tried, including deep linear networks.