jiayaozhang

Thanks for the heads-up! Yes, we use vector graphics in pdf formats for all figures (outputted from matploblib). I overlooked the size of the arXiv version but my local complication is around 14.5MB (pdflatex under TexLive 2020). I will check what went wrong :)

Thanks for sharing the interesting IRM paper!

Will read and be back for discussion hopefully soon.

> I thought the main selling point of deep learning that it finds non-linear connections in the data.

Agreed. Yet there are unknown feature spaces in which the deep nets separate the data linearly; the mapping learnt by the deep nets from the data to this space is likely highly nonlinear.

For example, a binary logistic regression can be thought as mapping data to a 1D space ([0, 1]), and separates the two classes at say 1/2 linearly.

> does it have direct application for training nets faster?

That's a great question! Unfortunately not yet -- though we believe further studies may bring us there finally. We found (at least for simple classification tasks) the features seem to have a two-stage behavior: a de-randomization stage to identify the best direction in the feature space; and an amplification stage where features stretches along these directions.

We've been thinking to identify a bound on the exit time of the first stage, and examine how it depends on different hyper-parameters, dataset properties etc, so that one may pinpoint how to reduce the time spending in the first stage, effectively making training faster.

> i.e. can the ode be integrated faster than backprop?

Also a good question, at this stage we need to estimate model parameters (the drift matrix) from simulations on DNNs. As future works we hope to explore if we can pre-determine those parameters so a comparison between backprop might make more sense.

Hello HackerNews! Author here :)

TL;DR: We devise a linear SDE/ODE model to imitate per-class feature (thinking logits) dynamics of neural nets training based on local elasticity (LE) [1]. We found the emergence of LE implies linear separation of features from different classes as training progresses.

The drift matrix of our model has a relatively simple structure; with that estimated, we can simulate the SDE using the forward Euler method, whose results align reasonably well with genuine dynamics.

Local elasticity models the phenomenon observed in DNN training: the effect due to training on a sample is greater for samples from the same class, and smaller for samples from different classes. For example, training an image of cats facilitates the model better learns images of other cats while not so for images of, say, dogs.

Any comments/thoughts/questions are most welcome!

[0] https://arxiv.org/abs/2110.05960 [1] https://arxiv.org/abs/1910.06943

Most websites I visited then denied my visit; finally I settled by using an ad blocker to block their prompts.