tsurba

Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example.

I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al.

For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind.

There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.

That’s the difference between truly new approaches to modelling an existing problem, or coming up with a new problem. No set of a bit different results or missing exact hyperparameter settings really invalidates the value of the aforementioned research. If the math works, and is a nice new point of view, its good. It may not even help anyone with practical applications right now, but may inspire ideas further down the line that do make the work practicable, too.

In contrast, if the main value of a paper is a claim that they increase performance/accuracy in some task by x%, then its value can be completely dependent on whether it actually is reproduceable.

Sounds like you are complaining about the latter type of work?

If you are not even going to bother writing them up properly, no one is going to care. Seems fair to me.

You don’t have to make a ”paper” out of it, maybe make blog post or whatever if that is more your style. Maybe upload a pdf to arxiv.

Half the job in science is informing (or convincing) everyone else about what you made and why it is significant. That’s what conferences try to facilitate, but if you don’t want to do that, feel free to do the ”advertising” some other way.

Complaining about journals being selective is just a lazy excuse for not publishing anything to help others. Sure the system sucks, but then you can just publish some other way. For example, ask other people who understand your work to ”peer review” your blog posts.

My favorite quote in this topic:

”If intelligence lies in the process of acquiring new skills, there is no task X that solving X proves intelligence”

IMO it especially applies to things like solving a new IQ puzzle, especially when the model is pretrained for that particular task type, like was done with ARC-AGI.

For sure, it’s very good research to figure out what kind of tasks are easy for humans and difficult for ML, and then solve them. The jump in accuracy was surprising. But still in practice the models are unbeliavably stupid and lacking in common sense.

My personal (moving) goalpost for ”AGI” is now set to whether a robot can keep my house clean automatically. Its not general intelligence if it can’t do the dishes. And before physical robots, being less of a turd at making working code would be a nice start. I’m not yet convinced general purpose LLMs will lead to cost-effective solutions to either vs humans. A specifically built dish washer however…

The article kinda sucks as it does not really answer the question it poses. Why ”noise tends to be a an unwanted amplitude modulation, not a frequency modulation”?

Is it due to naturally occurring background noise being low frequency high amplitude, showing up as AM? Could the situation change if humans keep generating more high-frequency noise? Or is it just that high frequencies do not travel as far so there will always be relatively little?

Shadertoy also has nice ones https://www.shadertoy.com/view/Ms2SD1

Beating all other models in latest benchmarks by a wide margin with a completely new approach, while having the only properly working audio chat with transformer isn’t enough? Not to even mention SORA if actually comes out someday.

I’m not an OpenAI fanboy but come on, credit where it’s due, they have so far been one step ahead of the competition from the start on every front.

I do research in other ML topics but I do follow the qualitative leaps they are pulling off continuously so far.

GPL doesn’t, but as others have already said, this discussion is not about that. If they want to keep using someone else’s free server capacity, maybe they should be giving something in return.

The writer should stick with nontechnical points since they clearly don’t understand why the models currently used are not deterministic. It’s trivial to make them deterministic.

>Tennis court size is in feet

Lol.

Yet understanding is necessary but not sufficient when you read university math, especially advanced courses.

Proofs assume you have the elusive thing referred to as ”mathematical maturity”, which means many algebraic manipulation steps are skipped because it’s assumed you can just see the result straight away.

This ability to see the connection is not understanding but learning by rote, having done the same tricks with similar equations a thousand times.

This is what makes advanced math books/courses slow for me as a CS phd researcher. I can very slowly progress through, but it takes a massive amount of time to work through what just happened. If you take 60 instead of 20 courses on math the routine you have is just completely different. I guess you can call it fluency in the language.

(For example now I’m reading optimal control & variational calculus along with the functional analysis it needs, its heavy.)

In a sense even this is not true, as in any sufficiently complex (which turns out to be quite simple) formal system you can create proofs that are true and untrue at the same time creating a contradiction. In other words, mathematics works by setting up useful axioms and following up on the logical consequences, but they usually can be used to create contradictory proofs even if useful in many problems.

I recommend learning about Gödel’s incompleteness theorem behind it all.

For a pop science book that explains it nicely I recommend ”I am a strange loop”. The wiki intro is also quite good

https://en.m.wikipedia.org/wiki/G%C3%B6del%27s_incompletenes...

Unfortunately also the majority of scientific papers for eg. image generation have had completely cherry-picked examples for a long time now.

Nice! I would like to someday finish writing an OS for my IRC bot that is still running.

Maybe the most useless comment but: that non-linear mouse movement (aka acceleration) is the very first thing I turn off when I boot up a new OS. It literally hurts my hand somehow. For example linearmouse is free for Mac. For Windows you can just turn off acceleration. For Linux its easy, obviously.

Using mouse acceleration stops you from learning to map a distance traveled by the mouse to a distance on the screen. I do think its more efficient in the long run without it. Something I learned to do from gamers, and something I think all gamers still do for a good reason.

Very cool that the dataset and model weights are open right away! This paper also doesn't have a bunch of weird architectural choices pulled out of nowhere like the other TS foundation models recently. Looks like it will actually be useful, thank you! Maybe I will actually get to do representation learning for TS during my PhD.

As a sidenote/rant, it would be nice if all supervised TS benchmarks included "DLinear + RevIN" as the standard baseline, as in my experiments it tends to get about the same performance as all other new SOTA forecasting models. Most papers compare to the linear model without RevIN while they themselves use it, and only beat it because of that :) And in any case supervised training of transformers from scratch on datasets having less than 1M points is just stupid (so less raw data than a single image?). Less than 1B is still at least mildly stupid.

Here of course the angle is zero-shot so its somewhat excused from this, but it still would be interesting whether it can beat that supervised model combination.

Getting a job to make money and solve customer problems as fast as possible was not a stated goal by the OP.

Besides, you are also wrong, having good fundamentals in the maths will help you pick up new methods much faster as they pop up. And especially if you want to come up with new methods (as in research), there are no shortcuts.

Very nice read from today’s perspective. Many points still hold, while a few have clearly progressed by leaps from back then. In hindsight language is a pretty nice generalizer as so much of it is available.

It could just be that improving the models does not parallelize that well, so you need to wait for the massive months long training runs to finish to see what worked and what to try next.

OpenAI got started earlier going full on with scaling up the transformer architecture (even if Googlers came up with it first).

Of course if you are smarter or can run more experiments simultaneously, you can catch up at some point. But it could still take a while even with just one year head start.

What Brockman tweets is from technical standpoint the most mundane, boring, and obvious stuff I’ve read from a programmer. My reads of this guy have been he’s not working on any problems that are technically difficult (or interesting). It’s much easier to work long hours on easy problems. He also has a managerial vibe in all communications which supports my feeling.

Most programming work in any project and company is mundane, so I do agree someone taking care of all that without whining is actually extremely valuable. I couldn’t do it.

Still doesn’t really make sense to put him on such a pedestal like many in this thread. It seems like a cultural thing in the US to overvalue individuals, and downplay the importance of good teams.

Whisper is generally better than the one in youtube.

Like others said, subscriptions suck, and especially since there are apps like ”Habit List” (which I use) that have a one-time payment of only 6€, I’ll rather use that.

It should not be a complicated app to make and maintain. The yearly Apple developer payment sucks but that is not my problem.

I’m pretty sure what communities you are in are not actual research but some hype alarmist bullshit communities, since as a ML researcher absolutely zero of my peers think the things you say.

Essentially you are advocating against information being more efficiently available. Come on.

It’s true we are fucked if bioweapons become easy to make, but that is not a question of ”AI”.

In the text they say you need to cram all information needed to predict the next token into a single 6KB word embedding, but isn’t that wrong?

Rather, isn’t the autoregressively predicted single next token a combination (based on attention) of all 6KB word tokens in the attention window.

So the size of memory where all information for next token prediction needs to be ”crammed into” is more like window_size*6KB, right?

The Theory That Would Not Die, by McGrayne was fun, about the history of Bayesian statistics.

Also not exactly what was asked, but so good: James Gleick’s ”Information”. It is about the history of information theory or how to statistically argue about information. Really starts from the beginning, eg. how some tribes used drums to talk to each other over kilometers away.

The Finnish capital area gov bought the health system from Epic systems made with Mumps for ~500 million € around 5 years ago, despite every software professional saying they shouldn’t.

And just like predicted it’s utter dogpoo now that it’s being deployed, and has already caused a bunch of health hazards and people quitting due to its impossibly bad and confusing interface.

You can guess will it be cheap to fix it now, as it probably was delivered up to the waterfall spec, and there is that one company in the world that owns the code and can change the software.

Worst case of corruption I’ve seen, in a country where its a saying we don’t have corruption.

Today I spent 2 hours cursing at GPT-4 for not being able to fix a stupid indexing mistake in the code it wrote. Just like code before.

It’s helpful and I wouldn’t have the energy to work on this hobby project without GPT. But for now at least, at some point I have to understand every line of non-trivial code eventually.

Mj earlier versions were around before SD came out. Before dall-e 2 too, but after 1 IIRC. So I assume they have their own custom setup. Perhaps based on dall-e 1 paper originally (not weights as they were never published) and improved from there.