zingelshuher

> Google knowingly made their search results shittier and shittier for years

unfortunately this extends to youtube too. now they have a new shitty trick. you click on the link and they randomly give you a completely different video.

> It is unlikely people are going to switch en mass to open source models

It depends on the task at hands. For complex tasks no way personal computer can compete with giants data centers. But, as soon as software becomes available, users will gladly switch to local AI for personal data search / classification / summation, etc. This market is potentially huge, for private sensitive there is no other way.

> poorly designed government intervention due to misunderstanding of the dynamics behind the process (homelessness)

Major drive is easy to understand, just cross the border and you are homeless on full support. Millions did with the help of dems, future voters. No language, no jobs, no skills. Of course they will vote for free food if they get this option. I.e. for dems, which was the whole idea.

had to upvote this

Only if it does nothing. In fact Google is one of the major players in LLM field. The winner is hard to predict, chip makers likely ;) Everybody jumped on bandwagon, Amazon is jumping...

I often use ChatGPT4 for technical info. It's easier then scrolling through pages whet it works. But.. the accuracy is inconsistent, to put it mildly. Sometimes it gets stuck on wrong idea.

Interesting how far LLMs can get? Looks like we are close to scale-up limit. It's technically difficult to get bigger models. The way to go probably is to add assisting sub-modules. Examples would be web search, have it already. Database of facts, similar to search. Compilers, image analyzers, etc. With this approach LLM is only responsible for generic decisions and doesn't need to be that big. No need to memorize all data. Even logic can be partially outsourced to sub-module.

It's impossible. Meta itself cannot reproduce the model. Because training is randomized and that info is lost. First samples a coming at random. Second there are often drop-out layers, they generate random pattern which exists only on GPU during training for the duration of a single sample. Nobody saves them, it would take much more than training data. If someone tries to re-train the patterns will be different, which results in different weight and divergence from the beginning. Model will converge to something completely different. With close behavior if training was stable. LLMs are stable.

So, no way to reproduce the model. This requirement for 'open source' is absurd. It cannot be reliably done even for small models due to GPU internal randomness. Only the smallest trained on CPU in single thread. Only academia will be interested.

If we can keep unlimited memory, but use only a selected relevant subset in each chat session. This should help. Of course the key is 'selected', it's another big problem. Like short memory. Probably we can make summaries from different perspectives on idle or 'sleep' time. Training into model is very expensive, can be done only from time to time. Better to add only the most important, or most used fragments. It likely impossible to do on mobile robot, sort of 'thin agent'. If done on supercomputer we can aggregate new knowledge collected by all agents. Then push new model back to them. All this is sort of engineering approach.

It's a different animal. In general you cannot reproduce the model even having all the training data. There are too many random factors and nobody keeps track of them. Just pushing the training data is done at random from the dataset. This results in some interesting facts. Having the model and the data it's impossible to say if the model was trained on that exactly data. All we can say is that some pieces of that data were used in training, in some cases. Model can be 'watermarked' in hard to detect, stable to quantization and finetuning way.

So, you cannot have a reproducible, 'open source' in its strict interpretation, model.

It's a matter of opinion how much open model should be to be called 'open source'. Looks like some believe they have the right to define it for everybody else to use. Like for software. Have to disagree. Why don't we introduce a separate term: 'open source, training infrastructure and data included for free'?

"open weight model" is confusing, because actually the architecture is open too, only data is missing.

> I'll bet Adobe or similiar will buy this 0,5 mil

May be that was the business plan, or 'plan B'

Yes. And there are many forgotten accounts on youtube, their owners now have another way to monetarize. Not sure why Adobe didn't talk to Youtube directly.

As you mentioned models need _random_ videos. Just 'walk outside' will produce more or less the same. My guess Adobe is more interested in 'family' sort of videos, with humans. This is what most users will be asking for. Getting them in other ways will be either illegal or really expensive.

It doesn't affect creators life as they guarantee (likely) it will not be made public. So it's just side income. Likely creators make a lot of fragments which don't get into the final version. The problem for Adobe is most of these videos will be similar which, if not filtered, will make AI model biased. In other words they will pay and throw away significant part.

It reflects the fact that Amazon in serious about AI. If fact they are well positioned with their datacenters and a lot of ways to apply, starting with smarter Alexa.

Expect sh*t load of AI hallucinations. As if Wiki isn't bad enough with BS some intentionally posting.

"the bigger you make epsilon "... " thus slower the training progress will be"

Sounds like variable epsilon is optimal, that's instead of learning rate, or both together. Would be nice if this can somehow be algorithmically regulated in generic way.

Intuitively looks like models should be close enough, or sparse enough for merge to work. I wonder if MoE experts can be merged(?)

Good luck with that ;) Actually you _can_ learn juggling this way, just couple of minutes at a time.

There are unstable cases when static learning rate doesn't work. Solution starts wobbling too much after some time and explodes. Using too small LR from the beginning leads to local minima. Making it stable _is_ possible, but it's a different story.

Have to disagree with this. The major limiting factor is the software and lack of applications. If there was a killer application it would be much easier to sell. Then the numbers would drive the prices down.

Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.

Question, have you seen the improvement after adding the noise? I mean in practice. Asking because intuition sometimes doesn't work.

I run some tests. Single model of the same size is better than MoE. Single expert out of N is better than model of the same size (i.e. same as expert). 2 experts are better than one. That was on small LLM, not sure if it scales.

It was inspired by Mixtral 8x7B, of course. I think the same approach, soft to hard MoE, can be used in other domains. Like video/image processing. Would be interesting to take it to extreme, like 4 experts out of 100.

Similar MoE implementation was on GitHub for a while, since Jan 2024

https://github.com/zxaall/moegpt

We've seen how it (didn't) with net neutrality. No reasons to think it'll work this time. Especially when current administration is looking in any way to shift attention from migrants packing in US. They will 'show strength and responsibility' on something else, like leading role in AI regulations.

If grandmother had balls she would be a grandfather.

Agree, it's better to be rich and healthy than poor and sick.

Some were even payed for university. Those were times when almost nobody actually could afford to pay, socialism, <beep>