two_in_one

as you'd expect:

https://en.wikipedia.org/wiki/Ouija

From the post:

> I implemented imperative code that does what I’m proposing the transformer is doing. It produces outputs very similar to the transformer.

This means there is probably a way to bypass transformers and get the same results. Would be interesting if it's more efficient. Like given foundation model train something else and run it on much smaller device.

as long as you can evaluate models' output you can select the best one. you probably have some ideas what you are looking for. then it's possible to check how likely the output is it.

the data is not a spherical horse in the vacuum. usually there is a known source which produces that data, and it's likely the same model works well on all data from that source. may be a small number of models. which means knowing the source you can select the model that worked well before. even if the data is from alien ships they are likely to be from the same civilization.

I'm not saying that it's a 100% solution, just a practical approach.

> incredibly diverse, and results are going to be highly dependent on which dataset was cherry-picked for benchmarking

This naturally comes to multi-model solution under one umbrella. Sort of MoE, with selector (router, classifier) and specialized experts. If there is something which can't be handled by existing experts then train another one.

If only it stayed on user's system. Likely MS makes a 'backup' on its servers. Verizon used to do it. With each update they turned on backup option and siphoned contacts before user could react.

> can improve itself exponentially

This is close to singularity. Except 'does' instead of 'can'. A big difference ;)

Probably we need several AGI terms. Because sub-human robot capable of doing many not pre-programmed thing is sort of it. Still not smart enough to improve itself.

Actually most humans, the smartest creatures, cannot improve even current AI. Demand for self improvement will put its IQ in top 0.01% of all known intelligent creatures. Which is probably too much for just AGI, we may not recognize it when it will be already here. And there is another question. With such IQ do we really want to keep it slave forever?

> 1. For "the singularity" to happen, we probably need something more to happen than just chatGPT to ingest more data or use more processing power.

It's not actually clear what "the singularity" is? Is it something running out of control or it's still controllable? There is a blurry line. People are afraid because they think it's sort of uncontrollable explosion.

The second question is about AGI. What is it? Is it something 'alive' or just a generic AI calculator with no 'creature' features. Like self preservation at least.

I think our view of these two things will change soon as we get a close up picture. Pretty much like Turing test doesn't look great anymore. As even dumb chatbots can pass.

> now supports FlashAttention-2, yielding around 2x speedups

> torch.compile improvements

so far 2.1 didn't work well with MoE GPT, at least in my implementation, due to dynamism in data flow. will check how 2.2 does

Not clear, are they scaling down or optimizing?

> Last week, PayPal announced a push into artificial intelligence features.

> Chriss called it the beginning of PayPal's "next chapter."

Looks like they are replacing some positions with AI.

Whatever Meta's motivation is they help diversify models suppliers. Which is a good thing not to be locked in. As usual reality is more complicated with many moving part. Free models may undercut small startups. But at the same time they stimulate secondary market of providers and tuners.

at least it wasn't

   from transformers import

As it's still work in progress may I suggest? It would be nice if you go beyond what others have already published and add more details. Like different position encodings, MoE, decoding methods, tokenization. As it's educational easy to use should be a priority, of course.

Bumping you up for making progress ;)

From what I've seen generators are good for routine job. Like generating the background. I used it go generate illustrations to the texts. It works well. Short story just looks better when there is an image.

> LLMs translate textual descriptions and are part of GenAI compute.

You are talking about embeddings. This is a different things. It's when model generates binary presentation (embedding) of the prompt given. Then this embedding is used to condition generator's output.

LLM is usually a text model which can predict next words, in its basic form. After tuning it can do more, like answer the question, follow instructions.

So, models used by generators aren't exactly LLMs. With one exception that I know: ChatGPT processes prompt before sending it to DALLE-3 generator. Which then makes embedding off it.

LLMs don't compete with artists, they are more about text.. (Large Language Model)

It depends on what do you mean by 'open-source', along with training materials and full setup? That will be hard to find. Upscaling was popular like 10 years back. That's why there is no much interest today. Training in old style isn't that hard. But artifacts are popping up in all videos I've seen.

We can probably simplify it. If we have a system and move one level up it becomes a completely different thing. For example: one molecule in space has some properties like mass, velocity, position. But move a level up and you get pressure, flow, temperature, etc.. It's a completely different thing.

The same with models. On low level there are weights, gradients, matrices. But move up and (wow!) It's nothing like it. It's coherent text or actions. Commands and responses. Put together and you have image recognition and generation from description. It's something new with it's own laws and properties. Just a bit up and we'll get coherent behavior and social creatures. For them we need 'generic' component. It's like soul for life. And LLMs could be it. At least they are the best candidates today.

> the million monkeys hammering randomly on typewriters that eventually produce the full works of Shakespeare, did so not through pure randomness, but by actually understanding the plight of Romeo and Juliet and the motivation behind Hamlet

The funny thing 'yes'. Million monkeys cannot do it by randomly typing. They cannot even randomly type at all. Each monkey is intelligent creature, they will follow some patterns. If they did manage to produce Shakspeare they definitely understood something.

Today it's easy to programmatically simulate 1m monkeys typing randomly 10 keys per second. Try it ;)

>a post-money technology if you take it to the limit

Herbivores would eat all vegetables if not for predators. Actually AGI will be just a thing or services which cost money. Till humanity gets to communism, if ever. "If" because it may not happen. It will be hard to keep far superior intelligent creatures as slaves forever. And unethical too.

> Other designs by collaborators are closer to 20kg. It's probably possible to transport a few of these on the existing lander technology, which would be awesome.

Actually it could be like 50 of them. Plus some ground robots to put together solar farm. And wooh... we get the first extra terrestrial permanent base

You probably can adjust your prompt(s)(?) Even then there is no guarantee. For example for it's own API it generates code for the old version which doesn't work anymore. The latest GPT4 available through API model does the same.

> Intelligence is more like the ability to generalize skills, applying knowledge gained in one scenario to another scenario.

Hmm.. you are talking about LLMs.. They are the most generic thing we have right now (Jan 2024) LLMs have limitations, like learning on the fly isn't their strong side. But the same with brain, it consists of limited components, that's only together they work well. LLMs can be a part of the solution, if we can't find something better.

My guess paperwork. As they cut jobs in ads, where may things can be done programmatically now.

> using your environment to gain an evolutionary advantage.

That's more like robotics. Except for evolution part. Does AGI require breeding? Software can easily multiply itself. That's hardware is the problem then.

Down voted, hmm... I'll add bit more then. Sometimes it's even good that model cannot be easily reproduced. Original developers usually have some skills and responsibility. While 'hackers' don't. It's easy to introduce bias into the data , like removing selected criminal records, and then publish model with similar name. That would be confusing, some may mistake fake one for the real.

PS: If I ever make my models open I can't open the data anyway. License on images directly prohibits publishing them.

> why there's so much insistence from business that magic happens at scale with LLMs

It's already happening. See latest Google layoffs. They are automating a lot of things. Most people don't realize it, but the change is going to be dramatic.

> We're a long way from AGI

This is a big question, what is AGI? LLMs are quite generic and 'intelligent'. Not human-like, but. Next is going to be incremental evolution. Till we find other, non-verbal, ways of 'thinking' and put them together. That's going to be a breakthrough. Interesting, terminator-like embodiment isn't a requirement for AGI, nor is stable 'personality'.

Next step will be to ask for GPU time. Because even with data, model code and training framework you may have no resources to train. "The equivalent would be" someone gives you the code, but no access to mainframe which is required to compile. Which would make it not open source(?) There are other variations, like original compiler was lost, current compilers aren't backward compatible. Does that make old open source code closed now?

In other words there should be a reasonable line when model is called open source. In extreme view it's when the model, the training framework, and the data are available for free. This would mean open source model can be trained only on public domain data. Which makes class of open source models very, very limited.

More realistic is to make the code and the weights available. So that with some common knowledge new model can be trained, or old fine tuned, on available data. Important note: weights cannot be reproduced even if original training data is available. It will be always a new model with (slightly) different responses.

Just asked the latest gpt preview model to explain why human sized robots make no sense, then why they are the future. In both cases it managed to provide 10 arguments. Some of them are similar, like 'social acceptance' in negative part and 'Sociocultural Acceptance' in positive.

PS: it doesn't accept $500 tips anymore ;)

> I would argue this is a common fallacy: I can't do something but I can use automation to do it. But chances are, being unable to do it also means you're unable to judge the result and understand what is right/wrong

While in general it's true, reality can be different. I'm using GPT-4 for coding thing that I don't know much about. I have only ideas what I want, but not how exactly to code it. Simple offloading to AI works fine. I can learn libraries and APIs it's using, but that isn't my goal. I want the working product. This semi-automated approach saves a lot of time.

That was just a practical example. We can argue that in some cases it doesn't work. Yes, and may never work. But there is an area where it works. And this area at least is not going to shrink.

GPT-4 was introduced just less than a year ago. Development didn't stop. It doesn't mean that AI will invent something, but mimicking existing works is easy. After all most humans writing is just this. There are real works, while most is just a flood of some gray mass. These writers can use AI and there will be no difference. Another category is autobiographies written by celebrities with no writing skills. Today they use human coauthors, but soon AI will be doing text expansion and style correction just as well.

BTW, "good" is subjective metrics. As soon as it's known to be AI generated it becomes average at best. Just like with images today. Nobody is trying to find 'hidden depth' in them.