cbsmith

That's an amusing perspective. I really struggle with watching any video at double speed, but I've never had trouble listening to any of his talks at 1x. To me, he seems to speak at a perfectly reasonable pace.

It seems counter intuitive that the reader would no longer have that authorial instinct due to lack of writing. Like, maybe they never had it, in which case, yes. But being exposed to a lot of different "writing opinions" tends to hone your own.

Let's say you're right though, and you lose that authorial instinct. If you've got five different proposals/PRs from five different models, each one critiqued by the other four, the needs for authorial instinct diminish significantly.

Fair enough. ;-)

I'm actually curious about the "lose their skills" angle though. In the open source community it's well understood that if anything reviewing a lot of code tends to sharpen your skills.

There's an implied assumption here that code you write yourself doesn't need to be reviewed from a context different from the author's.

There's an old expression: "code as if your work will be read by a psychopath who knows where you live" followed by the joke "they know where you live because it is future you".

Generative AI coding just forces the mindset you should have had all along: start with acceptance criteria, figure out how you're going to rigorously validate correctness (ideally through regression tests more than code reviews), and use the review process to come up with consistent practices (which you then document so that the LLM can refer to it).

It's definitely not always faster, but waking up in the morning to a well documented PR, that's already been reviewed by multiple LLMs, with successfully passing test runs attached to it sure seems like I'm spending more of my time focused on what I should have been focused on all along.

...and it's close relative NNTP. There was a whole distribution structure built out of intermittent data transmission. We've had the tools for this stuff for a long time, we've just switched to centralized, always available services because that's easier to build a company around.

When I sold printers in the early 90's, HP Laserjets were broadly considered to be the gold standard.

> But then how come that is exactly what is happening with modern search engines?

...because there are systemic forces driving this outcome, and those forces don't much care whether the company is a public service or a privately held one.

> Don't all these effectively inject new knowledge?

If you mean new knowledge in the sense of "improved weights for a particular task", I guess yes, but the issuing about "new knowledge" is about learning something you didn't know if the first place, rather than being able to more accurately arrive at a conclusion.

1. "excel at a particular task" -> no. A lot of what gets in the way of excelling at a particular task is extraneous knowledge that leads to "thinking" about things that are not relevant about the task. If the job is "hot dog or not hot dog", knowing about the endless "hot dog is a sandwich" debate or the people with hot dog fingers in Everything, Everywhere, All At Once tends to just gets in the way of doing the job as accurately and efficiently as possible.

2. "train on proprietary or sensitive data" -> no. Training on proprietary or sensitive data might not give you any new knowledge, but it may allow for much more refined weights to drive probabilistic decisions. So, if I train on a model with thousands of examples of X-rays of potential cancer patients, it doesn't learn new ideas, but it does learn better weights for determining if it is seeing a tumour.

3. "Complex domain-specific tasks that require advanced reasoning" "Medical diagnosis based on history and diagnostic guidelines" "Determining relevant passages from legal case law"

If you fine tuned an engine to identify species of animals, ones that it is already aware of, you can produce a model that knows with high confidence that "jaguar" is a kind of cat, and not a car or a sports team. It has this high confidence because it knows that, after lots of examples, knowing about there's a car or sports team with that name just gets in the way of making good judgments.

"OpenAI's analogy of training a human in a particular subject until they excel even arguably excludes the possibility of destruction because we don't generally destroy existing knowledge in our minds to learn new things (but some of us may forget the older knowledge over time)."

That is a pretty broad statement about the workings of the human mind. We absolutely do lose sight of neural pathways as our brain learns. I can't remember most of what I learned when I was 2 years old.

> That said, fine tuning small models because you have to power through vast amounts of data where a larger model might be cost ineffective -- that's completely sensible, and not really mentioned in the article.

...which I thought was arguably the most popular use case for fine tuning these days.

Yeah, as soon as I read that I felt like the author was living in a very different context from mine. It's never even occurred to me that fine-tuning could be an effective method for injecting new knowledge.

If anything, I expect fine-tuning to destroy knowledge (and reasoning), which hopefully (if you did your fine-tuning right) is not relevant to the particular context you are fine-tuning for.

Some curious misconceptions here about how the search industry works.

There's a very strong financial incentive for ad-powered search engines to keep SEO spam out of search results, because that makes advertisers more willing to pay for search placement. A publicly run search engine would not have those incentives and would be at if anything graver risk of a "tragedy of the commons" type scenario where the engine is overtaken with spam.

Yes, there are perverse incentives to populate search engine results with paid placements, but the best corrective force I can think of is having more competition in the search space. As long as people are willing to try other search engines (spoiler: for the most part, they are currently not), this creates a strong incentive to ensure that paid placements that harm the search experience are kept to a minimum.

...and I think the concerns about profitability of the LLM space is completely missing the larger agenda. Even if public use of OpenAI and its competitors NEVER TURNS A PROFIT, there is tremendous economic opportunity that investors expect to realize from a company with intelligent/powerful LLMs. That is why they are pumping so much money into these companies.

I did not know that the training code is closed source. That is troubling.

I don't know that that is true: https://researchify.io/blog/comparing-pytesseract-paddleocr-...

Using Surya gets you significantly better results and makes almost all the work detailed in the article largely unnecessary.

That's an entirely different line of reasoning from the article though, and "just git gud" isn't really the solution here any more than it is to use Rust. There are facilities for avoiding these problems that you don't have to learn how to construct yourself in either language.

Yeah, I kept thinking, "doesn't mp-units basically address this entirely"?

So basically, you're adding formal processes to ensure accountability. ;-)

There's also the nice sidestepping of the whole issue of human judgement comes with the baggage of ignorance and prejudice. Yes, the article acknowledges that processes can be largely beneficial, but it focuses on the horrors of the processes, while glossing over the horrors of NOT having the processes.

> Charging companies will charge what they can get away with. FWIW it used to be more like $0.30/kWh.

Yup, and they can charge a premium for fast charging, because it is really convenient when you need it.

> Most of California pays a rate 2x what I'm paying, which is a pretty typical rate.

Yeah, all the private companies in California charge a fortune for power. It's ironic that the municipal services are so much more cost effective.

> This is rates at DCFC. You're paying for far more than just the price of electricity. Those chargers and dispensers aren't free, they're often empty, they need to recoup costs in a few year horizon.

I'm very aware you pay a premium for DCFC. Presumably Texas' regulatory structure would help with all of that.

> Charging at home it costs me about $0.135/kWh.

So you're paying more than I pay to charge at home with my municipally owned and operated power company in California.

I think I've been misled about the wonders of energy deregulation in Texas.

Sure, but in Texas?

$0.42/kwH is competitive? I think the highest I've ever paid was $0.43/kwH.

There are all kinds of terrestrial alternatives to GPS. The US used to have LORAN-C. The trick is to deploy them...

Yes, they do often have such clauses, which carry a lot of weight when they aren't ignored.

There's a presumption in the logic of your statement here that if there is any waste in a program, it must be cut.

It turns out that there is generally waste in all operations; it doesn't follow that all operations should be terminated.

One of the great things about arbitrarily breaking contracts is that you save money in the short term, and signal future contractors that they need to charge even more exorbitant rates in future government contracts because you can never be sure if/when you will get paid.

The US has found ways to get China to adhere to IP laws, but they're never going to agree to restrictions the US doesn't impose on itself. Presuming that there is no way China will respect IP laws is BS.

You can cat Parquet and other formats into grep just as easily.

> The reason I'm skeptical of that framing is it implies coal production is being replaced by clean energy, rather than total energy production being increased.

I wasn't intending to imply that. It's entirely possible China might get to 90% renewable energy production without shutting down a single coal plant.

> Lots of countries announce decarbonization goals, but I will remain skeptical until the data show progress.

Sure, but that doesn't conflict with having more and more of your energy production being renewable.

> China's share of of electricity production from coal is at 60% as of 2023[1] compared to 16% for the US[2]. That's down from 80% in 2005.

Yeah, but they're kind of leapfrogging from coal to clean energy. In 10 years, they are expected to be at 60% renewables.