i2km

"Medvi, the AI-powered telehealth company, is fueled by ads from doctors who don't appear to exist" (https://www.businessinsider.com/medvi-ai-weight-loss-million...)

"As of Monday, at least six purported doctor pages were marketing Medvi's weight-loss drugs and a product that claims to increase men's sexual performance. One profile, "Dr. Matthew Anderson MD," lists an Angolan phone number and appears to have previously belonged to a gospel musician. Another, "Dr. Spencer Langford MD," features older posts and contact information corresponding to a clothing store in the Republic of Congo."

=====

Behold the AI business poster-child!

Madness. Makes me think back to a coding horror article from 2006 (I think) where it was accepted that even 500k loc codebases were generally incomprehensible by a single engineer. Now we’re at the stage of almost 500k FILES changing on a single commit…

And people still seem to believe they know what’s going on…

Hmm wasn't Sam Altman one of the early proponents of single-person unicorns? He certainly believes it can be done

A field guide to PKCS#11 issues in practice, covering some of the complexities and gotchas of HSM applications in practice

What I don't understand is that for most SE jobs on mature codebases, writing code is maybe 5-10% of the work. And it's that 5-10% coding that keeps you 'grounded' in the code and keeps your tacit understanding up to date.

Get rid of that and you get a basically invisible productivity increase (call it 10% to be generous) but you've now lost tacit understanding. No amount of half-assed code review (or getting the LLM to generate "code quizzes" etc for you) is ever going to substitute for the tacit understanding you get from doing the 10% of coding yourself.

I really just don't get it. I mean, I can believe that using coding agents gives people a rush of power and fleeting illusions of increased productivity, but it's a mirage.

A lot of the comments here are trying to find some middle ground, i.e. trying to find some way to preserve tacit knowledge. But it seems to me that it's a waste of time. You either need to write the code yourself or completely give up on even reading or attempting to understand it. I just don't think there's any middle ground. Any well-intentioned attempt to preserve understanding of the code will fall afoul of human laziness and understanding drift

I agree in that code is a byproduct of the tacit understanding which arises circularly when you write the code.

But if you're trying to gain the same level of tacit understanding by writing quizzes and simulations, are you actually gaining anything from using a LLM? Isn't it just better, faster and quicker to write the code yourself? Not to mention more enjoyable?

Curiously fitting that the Pope is also a Mathematician by training. I'd imagine he agrees with Tao on the fundamental aims of mathematical research and how these proof dumps largely miss the point

Around a year ago, the org I was working for ditched jetbrains (including teamcity) due to alleged Russian links of some of the founders/owners of jetbrains.

Never looked into it further, but might be a factor...

It’s just surprising how few people understand this. It’s not like it’s new either. Polyani’s theory of tacit knowledge captured this back in 1958

But we can’t do away with the quasi-religious lip service to the canons of the AI creed now can we? /s

And in general, especially with something like C, the correspondence between the compiled code and source code isn’t very “lossy”. You don’t get one-to-one correspondence but it’s not too bad and you’re not introducing massive uncertainty by running it through a compiler. With LLMs though all bets are off… you’re not gonna know what you end up with

Along with putting estimated reading times (21 minutes in this case) can authors please start putting estimated writing times?

Did it take 10 seconds of prompting? Or hours of thought, trial error and revision? Especially when asking people to read for 20+ minutes...

But but according to Mr Altman, we've like entered the singularity. Right? Who cares about a billing issue?

Precisely. I think everyone has been affected by the fearmongering and gaslighting to some dfgree. But step back and try and see whether software's getting better as a whole or going into reverse? OpenAI has basically unlimited internal compute and talent yet they screw this up amongst many other things. Shouldn't it be a 5 minute job for someone at AI to spin up a team of agents annd make sure this sort of thing never happens?

At a high level, most uses of AI I've seen seem to be people building other AI tools, orchestrators, managers, agent managers etc. But these are all means to ends. I mean I guess it's nice to play aroud with harnesses and command agents to do this and that, but where are the tangible outputs?

I just see so many people boasting of their token burn and the complexity of their agentic setup, yet they rarely show the actual outputs

No. The particular context I’m referring to doesn’t have any legal implications so it’s not that. It’s just brain-deadness instead

That's a fair challenge. AI review can mean many different things though. We treat it like a slightly better CppCheck, for context. TBH the way we use it doesn't really add much beyond regular static analysis tools.

Re "the hard way", I disagree. While it may seem like the hard way, in the medium and long term this is turning out to be the easy way. We have basically no technical debt, we can fix issues rapidly and (intangible I know) we can stand behind our codebase with absolute confidence.

Exactly. It fascinates me when I'm emailed long AI transcripts of meetings with the disclaimer "generated by AI. Be sure to check for accuracy".

Like, did somebody seriously think through the meaning and implication of that disclaimer and still write it?

Exactly. It's quite scary that some people are reacting viciously to this type of question. There's real insecurity here...

I do. I deny the "gains". The "gains" will turn out to be a mirage in the medium to long term. The technical and cognitive debt incurred will be too extreme.

I'm running a startup and we deliberately don't use AI for development. We only use it for review.

We're working in a really deep area where having full understanding of our code is more important than speed.

Besides, in this area, clients getting one whiff of AI code would be an immediate deal-killer

Article predicts that we'll see tools to "roughen up" slop to pass as human. Surprised we aren't there already...

I'm already being more careless with spelling and grammar because it gives authenticity.

Maybe people will revert to Jeffrey Epstein style prose to avoid any suspicion of slop..., e.g:

“no , bed, i, m laying on thefloor so it can't move.”

"employees were told about one instance in which the company spent $1.8mn on matching author details with listings on the group’s ecommerce website using Anthropic’s Claude Sonnet AI model despite the deployment failing"

Also contains a number of mentions of cost overruns not being detected until weeks afterwards. Which seems to me, maybe naively, like it should be trivial to detect in real-time...

Well, it seems to be meta-marketing. Essentially they'd trained the model with knowledge of exploits and given it a goal. Of course the training would allow it to 'reason' that having the answers would be a good way to score highly. And of course it had been trained on the potential tools to try to get the answers etc.

And OpenAI deliberately removed the guardrails.

If they were honest about it, instead of being smeared across the internet with shocked pikachu reactions, they should have just corrected their sandbox and re-run the test. There's really nothing to see here...

The whole "oh no what have we done. Regulate us PLEASE because we're one step away from terminator" is so stale. It's been trained on every exploit known and then told to use its training to brute force its way to score highly on a test FFS.

It's so bizarre. I keep on wondering where the root of this psychosis lies. Is there some common sci-fi literature that sparks these fears among the readers? Or is it all just latent religiosity and spirituality finding an outlet?

I mean, what an appalling way to live. To take themselves so seriously and simultaneously be terrified of what they're building. If we are really seeing the collapse of the bubble now, I wonder what's going to happen to these clowns when their AI god fails... what will they move on to next?

Couldn't anthropic just use fable to find security holes in Alibaba's systems and poison their models?

Or maybe there's been a bit too much hype...

It's somewhat difficult to have any sympathy for Anthropic here. They're entirely responsible for selling tokens at below cost, with the age-old bait-and-switch tactic.

If they weren't doing so, then these Chinese resellers wouldn't be viable. Radical idea, but how about they actually charge a viable price, even on subscription plans?

Surely what's coming is them offering to fix your vulnerabilities via higher-margin professional services?

Shocking. Been open a week and AFAICT just silence from OpenAI. I just find it baffling. You'd think that these vendors would be very sensitive to this sort of issue. I mean, surely they have multiple agents hooked up to github monitoring potential issues and proposing fixes, right? ...right?

Surely it should be trivial for them to have their own tools spinning away trying to fix all the github issues in real time...

Ridiculous. Haven't you heard? All critical thinking skills have long since been sacrificed on the altars of the AI gods and it's inconceivable that we write any code the old way. If you actually understand your code it means you're a luddite and are going to be left behind. /s

My theory is that anthropic have hit the beginnings of model collapse and the whole "fable may silently downgrade with deliberately incorrect results" is a diabolical attempt to gas light and get ahead of the curve.

So when it fails, people will chalk it up to "oh. Must have been silently downgraded because it thought I was doing something tricky enough to count as a distillation attack. My bad. Lemme try again..."