flatline

The tooling around LLMs is still nascent and, in my opinion, quite poor. You may be able to shotgun easily-falsifiable mathematical proofs with raw compute, but get into the messy world of real data, uncertainty quantification, and a host of other things that go into building real world models, and I think that an LLM plus tooling dedicated to the task plus a human in the loop will open up capabilities not available to an LLM in a chat or development centric software environment.

I had a conversation about a month ago with a friend who had gone through menopause and I realized I had no idea what the ovaries did when they stopped producing eggs. She didn’t either! I can see why it isn’t the most obvious target for research when the primary focus is fertility, but I do find it fascinating that women’s bodies stop the reproductive cycle and keep on kicking for decades. So much complexity, it’s quite the evolutionary trait.

MaM actually has one of the higher bars, but they have the most regular signups and the community to support it. That’s a good point of reference from 10 years ago, though.

Seedpool has regular open registrations, no fee, lots of freeleech content. You can pay $10 for a VPS that gives you a TL membership and get 5 TB upload credit with a couple hours of effort. All you have to do is seed and sign in like once a month to show you are still active.

I went the route of handling epub first (really, anything pandoc can convert to epub) on ebookaloud, and it took quite a few iterations to get consistent parsing across a wide corpus of test documents.

My experience is that gathering and testing with that corpus was about 30% of the overall effort. I have not yet gathered the motivation to tackle pdf: it’s more of a layout format than a text format. Not just tables but multi-column layouts and similar oddities leave a pile of imperfect heuristics in their wake.

Multi-lingual support is even worse. I was working with arabic script in pdf years ago and they don’t even capture the encoded text, just rendering instructions due to the high number of ligatures. I don’t know if things have improved since then.

I have no problem tipping for service. Tipping when I pay in advance and get no service beyond being handed my food — what exactly am I tipping for? I don’t think people are opposed to tipping in general, it’s just been applied to everything.

One time I was forced to pay a tip for self checkout at the Austin airport. I was in a rush for my flight, it prompted, and I could not find any way to opt out.

We're sitting on a year of more or less capable coding models and I have not exactly seen a revolution in new software being released. AI is likely a force multiplier for specific subsets of individuals and workflows. Coding is not a bottleneck for a lot of systems.

I suspect that SaaS is silently being eaten alive as an industry.

There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.

I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?

Meta, ByteDance, ...

I don't think the discrepancy is in LLM capability improvements over the past year.

Correctness has never been a priority across an industry where rapid iteration and feature delivery drive sales. There's always some opportunity cost to doing things right, at the price of technical debt down the road. If AI is primarily used to produce fragile code, people will be wary of AI solutions. There's also ongoing public debate about AI safety and alignment. Deploying AI in safety critical applications feels riskier than ever in the current environment, even though it doesn't have to be.

SQLite FTS could have done this a decade ago. We've had good local search capabilities for two decades and they either been underused or abandoned (e.g. Google Desktop, Yahoo! Search). This may be a reasonable projection of where AI is headed. You can do a lot locally but there is too much incentive to centralize around cloud infrastructure, then the privacy concerns make that prohibitive and we end up with what feels like a false choice of cloud or bust.

Maybe it's LLM hype that will bring more powerful capabilities to the desktop?

I think that's a bit strong of an assertion. How many people use copilot daily under an enterprise agreement? I don't necessarily disagree in spirit, especially given the questionable data sanitization around the recent Navier-Stokes announcement, but most companies disclose huge amounts of data regularly to hopefully-trusted third parties. I think the internet -- and a good share of the world's commerce -- would grind to a halt if we suddenly stopped. Setting up, securing, and maintaining local models for even a small user base is non-trivial and there is way more demand than supply for that skillset right now.

Exactly. It’s still just software, which someone programmed and deployed to do specifically dangerous/malicious things. I feel like we already have legislation and case law surrounding this.

Does mathematics still have a practical impact without humans in the loop? I don't think there's one single answer to that question, but I think it's worth considering exactly what that impact may be.

Tech companies are as much the topic of this post as AI, I think that's the immediacy.

Isn’t the implication that the vulnerability had already been found and exploited instead of reported to Google, because its value to Google was so low? Whoever found it originally had more incentive to sell it on the grey/black market. Or is the incentive structure truly different for vulnerabilities discovered to be already in the wild?

It’s Foucault’s interpretation of the panopticon. If we know that we may be watched — and punished — we self-police into following the norm, even if we do not know when we are being watched or that someone is necessarily monitoring us individually.

When I was a kid in the 80s none of this was an issue and I was free to roam. Starting in the 90s this became an issue and not only will parents not let their kids roam but the kids themselves have less interest because the norm is to stay in or only go out when supervised. You know you are doing something slightly wrong. It only takes occasional reminders like this to reinforce the rule, once established.

Same, this is like blogspam in the days of old that doesn't add anything to the original content and is just there to generate revenue for this site. The linked posts are better and per HN guidelines the toplink should probably be changed to one of those.

General doesn’t mean it applies to literally everyone. If autism turns out to be a specific condition that is e.g. the result of abnormal neuron pruning during early development, it could have a wide array of symptoms, but that understanding may someday lead to genetic treatments for sever cases, better early childhood interventions, etc.

I do think that the current ASD umbrella as written in the DSM is so broad as to seem nearly meaningless when take in isolation. It’s not meant to be in isolation however, it’s defined first and foremost as a disorder, so the symptoms need to cross some threshold of tolerable. It’s also associated with a growing body of research as to how nervous system development can cause related clusters of symptoms.

Is autism as we understand it scientifically accurate? Probably about as much as anything hovering in the intersection of social sciences and medicine. It does have some predictive power. If you meet the diagnostic criteria for autism you are likely to have certain comorbidities for example. Having certain autistic traits are good predictors of certain outcomes later in life. And we do have better therapies especially for young people than we have in the past.

The fact that people can have non-overlapping clusters of traits and still be autistic does seem problematic. It may not be the right model. Just my $.02, I’m not the autism researcher in this convo, but I am autistic :)

This line of thinking diminishes the real struggles autistic people face. Not everyone is autistic, and it's not just a quirky set of traits. I do believe society could be better arranged for people who for whatever reason don't conform to the overculture norms, so we could talk about disability along the axis of how society privileges or disadvantages certain individuals.

The concern is that autism as a rigorous category may be useless, but that doesn't mean that autistic people are suddenly neurotypical -- a term that is more politically than scientifically oriented. People who are not autistic broadly are allistic, to be precise with language. They may be neurodivergent in other ways.

I’m not sure why you were downvoted. I have to use Windows for work. The barrier to entry is simply lower for many system administrators and users alike. The homogeneity of a Windows rollout is extremely attractive. You are going to have a hard sell getting your accounting team and receptionist onto Linux. Your developers will complain and make do. I love complaining about Windows!

Their sandbox just sucked, the agent found a route through artifactory which was also connected to the internet. After the agent breached it, they put it back in the patched sandbox and it found its way out with another artifactory exploit — which was still connected to the internet.

No attempt was made at an actual air gap as far as I can tell. OpenAI is dangerous and irresponsible and doing a lot of pearl clutching over how their cybersecurity agent is doing cybersecurity things.

I think I agree with you in spirit. I don’t like what these companies are doing, and Anthropic’s actions can for the most part stand on their own. Copyright law is just a special interest of mine, and I do think it’s important to recognize what external incentives exist and what they prioritize. Because other companies will act in similar manners under the same incentive structure, and the problem is going to cascade and magnify if it hasn’t already. There are active court rulings setting precedence for this behavior - take note!

Why would a company keep the hard-copy around at the risk of it being inadvertently given away, resold, etc.? It's a huge outstanding liability given that the illegal copying of works -- the other part of that case -- is what they settled out of court for some huge amount of money. Destruction is the only thing that makes sense.

I'm old enough to have been around when DCMA legislation was under discussion. Many people were dead-set against it and raised concerns over matters exactly like this. In Rainbows End (2006), Vernor Vinge wrote about a similar scenario where a robot went through the university library shredding books, and scanned the shredded pieces to recombined them into a digital archive.

Anthropic may be doing shady things and may have even done this on their own recognizance, we just don't know. As it stand, this is 100% a consequence of US copyright law, much of which was written by large corporations to protect their own assets.

They cannot scan the books then resell them or donate them under current US copyright law. It’s not clear to me that they could warehouse them if they wanted. In the recent Bartz v Anthropic case the judge ruled this destruction as legal, saying

> The print original was destroyed. One replaced the other.

So that there was still only one “copy” of the book. This is in compliance with the DMCA. You can make a personal digital copy of a work but then you cannot resell the hard copy and keep the digital one. Same principle applies here.

This is why we have hierarchies of abstraction. Pretty much every field of mathematics relies on constructing notations, models, and other tools to simplify things in a way that is verifiable. LLMs rely on the same basic technique, they can just pull from a wide variety of these abstractions at once. So far we've been able to understand their proofs just fine. Computer-assisted proofs in the past that relied on brute-force is where we have run into trouble. We cannot reason about millions of possibilities at once, and we had to trust that the computer program that analyzed them was correct, which is a really hard problem and leaves humans fairly unsatisfied. I think we are actually progressing in terms of understandability in computerized proofs.

You deploy an application that ostensibly has the desired capabilities but it fails at scale, or the architecture is not extensible to add new capabilities or fix bugs. Those are expensive mistakes and in safety-critical domains could be deadly.

The thing is I've worked on plenty of code like this written by humans. AI just accelerates the process and puts this type of code within reach of more people. It could be seen as a step backwards in good software engineering practices we've painfully learned over the last two decades. I see it as a cycle repeating itself in a new iteration, with the hope that we'll end up with still better tools and processes in another few years. But I agree it can and will result in real harm in the meantime.

The author kept acting in good faith till the other party stopped doing so, which I took as the actual resolution to this story. Maybe when you stop treating people like rational adults when they have demonstrated otherwise is where you find your real “no.” It’s a hard lesson I’ve learned many times, and it sometimes entails walking away from something to preserve your own sanity/integrity.

I agree about OpenRouter. The AI Gateway number [0] is likely the figure that was actually coming to mind. Moreover, Qwen models alone have overtaken the previously-dominant Llama models in hf downloads by quite a margin.

Real question, and a refinement to my previous statement: would you find it more surprising if over 25% of worldwide inference was running on Chinese open-weight models, or not? I personally would not be shocked.

[0] https://vercel.com/blog/ai-gateway-production-index-july-202...

I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn.

https://www.secondtalent.com/resources/ai-trends-in-china/

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.”

This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.