dartos

> because code and text share domain (text based) – large, generic models will always out-compete smaller, specialized ones – that's the lesson

All digital data is just 1s and 0s.

Do you think a model trained on raw bytes would perform coding tasks better than a model trained on code?

I have a strong hunch that there’s some Goldilocks zone of specificity for statistical model performance and I don’t think “all text” is in that zone.

Here is the article for “the bitter lesson.” [0]

It talks about general machine learning strategies which use more compute are better at learning a given data set then a strategy tailor made for that set.

This does not imply that training on a more general dataset will yield more performance than using a more specific dataset.

The lesson is about machine learning methods, not about end-model performance at a specific task.

Imagine a logistic regression model vs an expert system for determining real estate prices.

The lesson tells us that, given the more and more compute, the logistic regression model will perform better than the expert system.

The lesson does not imply that when given 2 logistic regression models, one trained on global real estate data and one trained on local, that the former would outperform.

I realize this is a fine distinction and that I may not be explaining it as well as I could if I were speaking, but it’s an important distinction nonetheless.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

> one of the biggest learning is that larger, generic models always win

You’re confusing several different ideas here.

The idea you’re talking about is called “the bitter lesson.” It (very basically) says that a model with more compute put behind it will perform better than a cleverer method which may use less compute. Has nothing to do with being “generic.” It’s also worth noting that, afaik, it’s an accurate observation, but not a law or a fact. It may not hold forever.

Either way, I’m not arguing against that. I’m saying that LLMs are too general to be useful in specific, specialized, domains.

Sure bigger generic models perform (increasingly marginally) better at the benchmarks we’ve cooked up, but they’re still too general to be that useful in any specific context. That’s the entire reason RAG exists in the first place.

I’m saying that a language model trained on a specific domain will perform better at tasks in that domain than a similar sized model (in terms of compute) trained on a lot of different, unrelated text.

For instance, a model trained specifically on code will produce better code than a similarly sized model trained on all available text.

I really hope that example makes what I’m saying self-evident.

“The Alogrithm” is just another idea being propagated by an algorithm anyway.

You can say the same about GPU clock speed as well…

Are there examples of non-spaghetti react out in the wild?

AI was a term that was around for decades as well, but it’s meaning dramatically changed over the past 3 years.

Prior to gpt-3 AI was rarely used in marketing or to talk about any number of ML methods.

Nowadays “AI” is just the new “smart” for marketing products.

Terms change. The current usage of AGI, especially in the context I was talking about, is specifically marketing from LLM providers.

I’d argue that the term AGI, when used in a non fiction context, has always been a meaningless marketing term of some kind.

> that tech has gotten to a point where user hostile moves don't have as much of an impact anymore

It’s because the small dozens of us that care about the freedom that the small internet afforded vs algorithmic content discovery is now the minority and no big platform cares about us.

In the early days of the internet, they had to care. They had to convince people to leave their beloved forums and IRCs and move to Facebook or Reddit.

Internet users were fairly niche and fickle. Now everyone uses the internet and most people don’t care about their own freedoms wrt how they interact with technology.

The issue is that none of that drives engagement.

Private discord servers for everything nowadays.

The open searchable internet is dead for people trying to find community, imo.

AGI is, and always was, marketing from LLM providers.

Real innovation is going in with task-specific models like AlphaFold.

LLMs are starting to become more task specific too, as we’ve seen with the performance of reasoning model on their specific tasks.

I imagine we’ll see LLMs trained specifically for medical purposes, legal purposes, code purposes, and maybe even editorial purposes.

All useful in their own way, but none of them even close to sci-fi.

Both times trump won, this was true. The DNC is just allergic to putting forward candidates that people want to vote for.

Kamala really ran a campaign like someone trying to lose.

Are there examples of LLMs fixing spaghetti code in the wild?

> Which people?

The 51% of voters that voted for him… (correction: 49.8%)

> I hear that it is the same people who voted Donald last time who voted Donald again this time. No others voted for him.

This is wrong. There were more republican voters this time around than last time.

> The issue seems to be that many people who are not trumpists did not vote at all. Maybe i'm wrong, but it seems that if everyone who could vote actually did vote, then Donald would have lost.

This is a constant issue in the US. Low voter turnout causes seemingly unpopular candidates to win.

Go out and vote, people. Please.

> So ... maybe he won, but i hardly think he represents the majority is US citizens.

Never said it did. But if you don’t voice your opinion when it matters, then your opinion literally doesn’t count.

So, the American people voted for trump and this administration. Either through their vote or their inaction.

If I’m exposing myself to legal risk, yes.

It’s really easy to get an LLC registered

> Some lazy physicians are gonna get burned by thinking they can just zone out during the interview and let this do all the work.

How many patients are going to that physician, they’re going to get burned too

It may not be what people want now, but it is what people voted for.

People voted for tariff and massive day 1 changes and federal cuts.

People wanted a loud strong man president who makes deals.

These actions ARE what people (who voted) wanted… people just didn’t think about the outcome

In about 50% of the worlds, there’s 100% chance, would you say?

Does that math still check out if math works differently in other worlds?

> certainly is on quantum ones

I don’t think that’s a certainty.

All of this is a fine thought experiment, but in practice there are physical limitations to digital processors that don’t seem to manifest in our brains (energy use in the ability to think vs running discrete commands)

It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.

> nearly no one would be interested.

I went to a public US high school with a nursing magnet program and an automotive program.

Both saw you with a phlebotomist license or a technician certification respectively.

These were wildly popular programs despite having academic requirements (for the nursing track)

I think high schoolers care about practicality, actually.

> also you need to keep budget in mind and the teacher shortage

crazy thought, though.

Maybe pay teachers more? Make it a more attractive career? Vote in superintendents with more education experience than corporate.

Idk… anything but throw your hands up and say “well nobody wants to teach so idk”

Uhh… cyberpunk 2077 literally had AI Wars and made a big wall around most of the internet to prevent AI from affecting anyone, except for one taxi company.

Does that sound optimistic?

Not judging you for using AI for a post like this!

Don’t feel bad. I just didn’t think AI generated bullet points were as impressive as the comment I was replying to did.

Notice the “IMO pretty” before the word “obvious”

IMO means “in my opinion.” I used that phrase to express how the following statement is my opinion and not a universal truth. My “audience” in this case is myself.

I do that because otherwise there’s always a comment saying how things like “obvious” can be subjective.

I also used the word “pretty” to, again, soften the word “obvious” so that readers don’t think that it’s a universal truth.

I very much felt like that entire portion of the article was ai generated, actually.

IMO pretty obvious, surface level, information and some prose on each bullet.

There’s nothing to buy.

Say I have 100 followers and each of them have 100 followers.

Then say I make a post and all of my followers repost it.

That’s an explosion of updates that need to be sent around for 101 interactions.

These major social media platforms handle millions of interactions per second with accounts that have millions of followers and posts that get hundreds of thousands of reposts.

When there’s a centralized system, it’s easier to make sure everyone gets all their posts. Very little need to propagate every single update across every node in your whole network. From a user standpoint it’s seamless as well

It’s nothing unsolvable, but it’s much harder than a centralized social media platform while providing no visible benefit for the average, non technical, user.

A great story of MS incompetence and Amazon’s vendor lock in.

Well that and people seem to really enjoy (or at least feel compelled to use) social networks.

Social networks tend to be pretty difficult in p2p systems since the amount of traffic usually scales exponentially with network size (as posts get published to followers and shared over and over by the users of the network), which you’d want to maximize

No gatekeeping. Anyone can write programs.

But to build something which could handle a customer’s credit card, password, or other PII and charge them for it, you better know what you’re doing.

It’s all fun and games until you’re the cause of someone’s identity or password getting stolen.

Anyone can use CAD software, but if you’re designing a public space, you better know something about safety.

any phishing email could just directly link to a fake Google login form. Oauth just adds extra steps for the attacker.