gdiamos

Some thoughts on this.

The Pile was created by high school students. I was impressed how good it was when it came out given the team and how many resources they had.

Including 200k pirated books seems like a problem. It should be removed. If I was an author, I would be concerned by a process that allows this much content to be effectively published with no filter on the internet.

I personally think there is a lower bar for researchers and high school students to clean datasets used for research or class projects given that there is no commercial harm and cleaning huge datasets takes engineering resources that not everyone has.

In this case, it looks like it was removed by https://huggingface.co/collections/common-pile/common-pile-v...

Companies that commercialize the data should have a higher bar because they have more resources. It seems like Anthropic paid $1.5B for not cleaning the data before releasing a model based on it.

>> Where I draw the line is asking Claude to label training data. I don’t do this.

> Why draw any line? Anthropic doesn't. They actively take the world's intellectual property, don't pay for it, and put it behind a paywall.

I know a lot of silicon valley founders think like this. I personally don't think that just because someone else violated a boundary makes it okay to do it.

It's right here - you can't see it due to being downvoted.

I asked Claude to research this one and it turned up an interesting point. Pirated books got into the Pile and Anthropic paid 3k per book. Worth noting if anyone still uses the Pile.

[Claude researched, not verified]

In October 2020, developer Shawn Presser converted the Bibliotik pirate tracker into Books3 (196,640 books) and hosted it on The Eye, aiming to give everyone what OpenAI had with its secret Books1 and Books2. EleutherAI then folded Books3 into The Pile, distributed via Hugging Face and GitHub, and it went on to train Meta's LLaMA, EleutherAI's Pythia, and BloombergGPT. In August 2023, Alex Reisner of The Atlantic exposed its contents, and Maria Fredenslund's Danish Rights Alliance forced takedowns at The Eye, Hugging Face, and EleutherAI. A year later, Andrea Bartz (The Lost Night), Charles Graeber (The Good Nurse), and Kirk Wallace Johnson (The Feather Thief) filed Bartz v. Anthropic before Judge William Alsup, accusing Anthropic of using Books3, LibGen, and PiLiMi to train Claude; the case settled for $1.5 billion, about $3,000 per work.

Downvoted after providing evidence.

That's very click baitey. Do you have an argument or evidence for why?

One of my points of my ethical framework is that IP is valuable and should not be stolen.

Another way of saying is that hard work that advances the state of the art should be rewarded. Patents should exist.

I’m sure they are watching what I’m doing and would retaliate via service cut off and legal action if I crossed it egregiously and they found out.

I would do the same thing if a user or employee did this to my IP.

I don’t completely buy the argument of web scraping being an IP violation.

I don’t send my core ip in an email to my main competitors and I don’t post it on the internet.

Where I draw the line is asking Claude to label training data. I don’t do this.

I ask it questions about large models like big MoEs all the time. I’ve never had it refuse a question, but I would expect it to refuse sometimes.

For training data from existing models I just get open models to spill their guts. It’s fine

In industry we train employees to not reveal proprietary information.

I think it would be easier to do this for a model than a lot of employees

This tech is about to come for software.

Why don’t you do as much of this as possible for the next six months and make the best product you can.

It sounds like it will go faster if you use coding agents.

Anyone copying you will have to do the same.

Perhaps spend a month with your agent making it harder for other agents to reverse engineer?

Good job typesafe.

You won the competition with VCs

I wouldn’t trust an AI company to honor this as far as I could throw them

How to steal ideas with AI.

step 1, identify high value users by net worth, citation count, or number of followers

step 2, select all prompts by high value users

step 3, invest 10 billion thinking tokens in modeling an objective for each user

step 4, build an RL environment for each user

step 5, rollout 10 billion tokens per environment

step 6, train on resulting traces

Training improvements are very easy to copy.

Data is doing more of the work than it used to. Every source in our mixture is a curated artifact built with large models

Training a model this small on them is distillation

When models of this size were last studied seriously such corpora did not exist

The loss does not saturate. Across a 4.91B-token run, smoothed training loss falls monotonically within each curriculum phase and is still descending at the end

blog: https://gregdiamos.com/2026/09/07/outrageously-small-neural-...

X discussion: https://x.com/GregoryDiamos/status/2096873745420075020?s=20

I added some of the main points to the thread so they are easier to read.

I liked how the article was aimed at Mark personally.

Not everyone gets super voting shares, but everyone gets a life and has to live on the same planet.

OxyContin, Enron, WorldCom, Super-size-me, Pets.com, Asbestos in the ceiling tiles...

It was always burning since the world's been turning

we will look back on it as the big tobacco of our generation

how's the battery life?

I think we should revisit outrageously small neural nets.

I needed a cheap model that runs at over 10k token/sec on a single CPU core for some data processing. So I gave Anthropic claude code a pile of tokens to build one.

It made three discoveries that I thought were interesting:

1) One Intel AMX core can train a 3M active parameter MoE foundation model at 6,616 tok/s on 4.91B NVIDIA Nemotron tokens in a few days.

2) That model shows emergent in-context copying, positional analogies, and basic arithmetic after about 250M tokens.

3) The foundation model gives large gains in downstream SFT, and the training & eval loss keep going down all the way through 4.91B (and likely beyond).

Claude is not as good as a great MLE at debugging MoE. It made a bunch of bone headed mistakes, but it got there in the end.

I asked it to write a paper about it's work, and it produced this.

Claude Co-Authored Paper: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blo...

I read through it and it sounds a bit LLMy, but the main points and experiment results are correct.

Some of the models are published on HF: https://huggingface.co/gdiamos/amx-reasoning-v1-instruct

Progress compared to SLMs and the early days of deep learning is real.

However, I know of no theoretical limits on scaling laws other than compute and data.

I think it means that we should be aiming further ahead

I’d like to see more of these models.

I’ve been using diffusion Gemma and it is very fast on GPUs in output token/sec.

In the diffusion Gemma whitepaper, they say they could have done better with more time and compute.

Even with those caveats, it is very uses-able as a local model.

Best case scenario

There's certainly a place for enterprise and not breaking what's working.

Shouldn't that be 0 innovation tokens though?

In hindsight I disagree.

Instead I like “only work on impossible problems”

Most of them turn out to be impossible, but some of them turn out to be possible.

I’ve never met anyone who could pick 3 and be confident in getting even one right. Tokens are a terrible analogy for innovation or research.

In hindsight I’ve had to sift through hundreds or more to fine one that worked.

I thought this post was helpful when I first started thinking about startups.

After more time, I think boring tech isn’t worth thinking about.

How big is the open model? 30B?

Christopher Nolan beat you to it

vLLM is originally marketed as paged attention, but in hindsight, separating the web server and GPU process, continuous batching, kv caching / chunking, and a huge model library including low precision mattered more.

I wonder how much it would cost to vibe code the whole thing from scatch?

I wonder how much better models need to get before such a thing wouldn't look like code vomit?