cjbarber

And in some sense, all of my claude code usage feels tok/s bottlenecked. There's never really a time where I'm glad to wait for the tokens, I'd always prefer faster.

ie intelligence per token, and then tokens per second

My current feel is that if Sonnet 4.6 was 5x faster than Opus 4.6, I'd be primarily using Sonnet 4.6. But that wasn't true for me with prior model generations, in those generations the Sonnet class models didn't feel good enough compared to the Opus class models. And it might shift again when I'm doing things that feel more intelligence bottlenecked.

But fast responses have an advantage of their own, they give you faster iteration. Kind of like how I used to like OpenAI Deep Research, but then switched to o3-thinking with web search enabled after that came out because it was 80% of the thoroughness with 20% of the time, which tended to be better overall.

https://github.com/eugene1g/agent-safehouse via CGamesPlay

https://multitui.com/ via davidcann

https://github.com/obra/packnplay

https://github.com/strongdm/leash

https://github.com/lynaghk/vibe

(I've been collecting different tools for sandboxing coding agents)

New paper on a long-shot I've been obsessed with for a year:

How much are AI reasoning gains confounded by expanding the training corpus 10000x? How much LLM performance is down to "local" generalisation (pattern-matching to hard-to-detect semantically equivalent training data)?

tl;dr

- OLMo 3 training corpus contains exact duplicates of 50% of the ZebraLogic test set.

- We embed the corpus to find semantic duplicates of test data in the wild. 78% of the CodeForces test set had >=1 semantic duplicate

- The semantic duplicate rate is maybe >4 in 10000

* at least 50% and at least 78% that is

arxiv.org/pdf/2602.12413

Imagine you're head of training at at OpenAI, and you want your benchmark scores to be meaningful (: to estimate OOD performance)

You have a hard task ahead of you! Your models have seen so much, memorisation is so easy - as is local generalisation (noisy pattern-matching).

What can you do? Well, obviously you take every benchmark you're going to test on and try to "decontaminate" your training corpus (remove test data from the training data).

By default this is just one level above string matching ("n-gram matching" - if sentences overlap in (say) a 13-token window, remove them from the training corpus).

But you're actually trying, so you also translate the test sets and delete translations of test from train.

But! every piece of test data has an arbitrary number of logical equivalents and neighbours (like how `x + y = 10` is the same problem as `2x + 2y = 20`). And LLMs are amazing at semantic search, so maybe this inflates benchmark scores.

The cutting-edge tech for detecting these "semantic" duplicates is... an LLM. But you simply can't do 100T x 1M calls. There's not enough compute in the world (yet).

So you do what you can - maybe you

- categorise the entire corpus & do intense search inside relevant partitions (e.g. maths > number theory > ...)

- embed the whole corpus & look for things really close to test data

- train a wee 300M filter model & do what you can with that

How much does this process catch? How many semantic duplicates of test data slip through? And what's the impact on final benchmark scores?

We don't know, This (finally) is where our paper comes in:

We experiment on OLMo 3, one of the only really good models with open training data. Since we have its entire training corpus, we can exhaustively check for real "natural" duplicates and finetune it to estimate their impact. We embed the entire Dolma Instruct corpus.

Firstly: we were surprised by how ineffective n-gram decontamination was at catching exact duplicates - 70% of harder tasks had a match. But the spurious performance gain wasn't so large, at most +4pp.

Secondly, every single MBPP test example and 78% of CodeForces have semantic duplicates

Thirdly we generated 10k synthetic duplicates for MuSR, Zebralogic, and MBPP problems and finetuned on them.

- MuSR +22pp. Semantic duplicates as strong as exact

- ZebraLogic +12pp. Exact much stronger

- MBPP +17pp. Exact stronger

Fourthly we guess that 4 in 10,000 training datapoints are a strong semantic duplicate for a given benchmark datapoint (where strong means just "obvious to Gemini")

So: n-gram decontamination is not enough even for the easy (exact) stuff, semantic duplicates are at least a moderately big deal, and this probably transfers to frontier models to some degree. The above are probably underestimates too (since our detection pipeline was cheapo).

Data contamination is a huge field. Here's how we're new

This is preliminary work on a shoestring - we didn't get at the big questions yet ("what share of benchmark gains come from interpolation over a hidden training corpus?", "does this even matter?")

And local generalisation across very different strings is anyway pretty miraculous

The grand aim of this research programme is to decompose benchmark gains / apparent AI progress into 4 estimates:

1. benchmaxxing (memorising exact duplicates)

2. usemaxxing (RLing narrow capabilities)

3. hidden interpolation / local generalisation

4. OOD generalisation

We have a lot of ideas! If you're interested in funding this, grab me at gavin@arbresearch.com

Nearly all of the real work done by Ari Spiesberger, Juan_VaGu, Nicky Pochinkov, Tomas Gavenciak, peligrietzer and NandiSchoots

And ofc this work wouldn't be possible without allen_ai and natolambert working in public and enabling actually scientific evals.

It's entirely possible that this is the first step and that they will also do faster better models, too.

> more than 1000 tokens per second

Perhaps, no more?

(Not to mention, if you're waiting for one LLM, sometimes it makes sense to multi-table. I think Boris from Anthropic says he runs 5 CC instances in his terminal and another 5-10 in his browser on CC web.)

https://github.com/obra/packnplay

https://github.com/strongdm/leash

https://github.com/lynaghk/vibe

(I've been collecting different tools for sandboxing coding agents)

In case helpful, I collected some startups I find interesting, see the startup lists section: https://chrisbarber.co/start-here/.

Though I think the ideal would be:

* What are the products you find most enjoyable to use?

* Of those, which ones do you have ideas for improvement based on your usage?

* Who are the best people you've ever worked with or collaborated with?

* Where do they work today?

Search for "cable protector ramps"

> And then there are still those six feet of stone that needs drilling through to get the cable outside and back in.

Thin cables designed to run under doors or windowsills are an option. Search for "flat ethernet cable"

It seems like you prefer your setup for good reasons, and these solutions above are both ugly, but I still wanted to note to others reading this that workarounds exist.

You can do ethernet cables outdoors from your router in your house to your router in your office. Either thin cables that go under doors, or outdoor rated ones, both can work fine.

This same approach can work inside a house as an alternative to mesh networking or running cables through walls. The cables don't have to be invisible (underground or in walls) when you have tough constraints, unless you want them to be.

> Show HN is for something you've made that other people can play with.

> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.

https://news.ycombinator.com/showhn.html

The solution is very different for someone who is within walking distance of a neighborhood coffee shop vs someone who isn't.

It seems like there's 3 levels of solutions recommended here:

1) Individual: join recurring activities, volunteer, join communities, get a dog, work on yourself, sports/physical hobbies

2) Founder: Create third spaces, host events, or just create and initiate activities that bring people together

3) Policy: Urban design reform, third spaces. Make it easier for more third spaces to exist and more walkable neighborhoods.

It's like capex vs opex. A lot of the fixes recommended here are very high ongoing daily effort for individuals. But this is such an important thing for humans! So it would be better if the built environment was better, and human interaction was easier and lower effort to get for more people. More walkable high trust places, more third places.

Should there be lots more affinity based master planned communities? Probably yes. More in person theme parks and activity places? Probably yes. More games like Pokemon Go? Probably yes. Better walkability in existing cities? Probably yes. etc.

tl;dr at an individual level, these suggestions are good, but the fact that so much individual level effort is needed imo points to more of a need for macro solutions so it's lower energy for most people to have nice local walkable communities and friends (like people have in university, cities post-university, and in retirement homes)

I hope that Apple does a better job of this too! I don't want Uber's ad notifications, but I do want their notifications about my vehicle status.

See also: https://en.wikipedia.org/wiki/Twelve_leverage_points

https://www.ycombinator.com/library/4A-a-guide-to-seed-fundr...

https://www.youtube.com/watch?v=gcevHkNGrWQ

And as an aside, if you can, just get customers first instead of raising. Raising is best done either when it's impossible to not raise, or when you already have multiple investors offering you money.

Then once you have nice revenue and customer references, cold email investors who've invested in similar companies to yours. Or, maybe you'll see that you don't need to raise because you're getting enough money from customers.

and perhaps even moreso 2) Figuring out how to get them built

It seems we mostly know the answers for 1, we just don't know how to get them built in a sea of development regulations and entrenched interests etc.

Lots of things come from this: shared resources (less income need, less work stress), shared emotional support, shared childcare (less income need), etc.

Instead of single family homes (one family is not the atomic unit of the human species!) it should've been single community developments with 15 homes and a big shared backyard but still private for all the houses. And the landlord + all the tenants can select the residents based on their personal preferences and anyone can veto.

I think many challenges stem from the lack of this.

I don't know what the fix is though, because housing regulations seem difficult to change.

The iOS / macOS 26 frustration I think is particularly felt by the HN type crowd. Don't want something that looks cool but is less effective/performant/usable. "We" can feel Apple's priorities drifting away from ours.

Side note: I wonder how much easier AI will make it to migrate between operating systems? Perhaps future AI systems that are good at computer-usage could manage migrations/installs well.

This is a long winded way of saying that I think creators need what you're making! People who have hours of awesome footage but have to spend dozens of hours cutting it down need this. Then also people who have awesome footage but aren't good at editing or hiring an editor, same thing. I'd love to see someone solve this so that 90th percentile editing is available to all, and then it can be more about who has the interesting content, rather than who has the interesting content and editing skills.

New AI co, Bezos as co-CEO (alongside Vik Bajaj, ex-Google X)

$6.2b in funding

Nearly 100 employees

AI + real world scientific experiments, for the engineering and manufacturing of computers, automobiles and spacecraft

Tl;dr is:

> When a writer asks you for feedback on their writing, what they really want is to know is:

> What’s going on in your head?

> What are you thinking?

> How do you feel?

> When you agree to give feedback on a piece of writing, your job is to notice what’s happening in your head and share it with the author.

> The best way to do this, in my opinion, is simple: leave notes on everything. In other words, try to give the writer your play-by-play reaction to every part of their piece.

> Scion bought roughly $187.6 million in puts on Nvidia and $912 million in puts on Palantir, according to Securities and Exchange Commission filings.

But 13F reports the market value of the underlying shares for options, not the premium paid for the options

All we know is at time of filing he has 10k NVDA puts and 50k PLTR puts. We don't know the strike price or the duration or how much he paid