4,266 karma · joined August 13, 2011
I've seen dozens of conversations about it in last 24 hours, and every major inference provided added in first 24 hours. I think it's gaining plenty of traction.
Private offline transcription and summary. Speaker identification, working on voice-prints for identification across the corpus.
Compared to something like VRAM it's slow.
Amazing he still codes. But likely more experiment than prod level.
But these labs distill off the larger models. Both officially at the labs with the big ones, and unofficially. We need the giant models to get the smaller models.
They don’t have a prefix cache like other providers, or at least don’t have a discount in their billing structure. Each message charges for the whole context window. It’s wildly more expensive for long multi turn scenarios with lots of tool calls (coding). It’s better for short few turn tasks.
Edit: I don’t know if they actually have a proper cache. This could just be a billing artifact.
Plus the case above: OSS could have had another company storing sensitive medical records, and this hands over those drives to a PBS station.
> Elliff ordered the immediate return of any physical devices that hold Nine PBS’ data once access to OSS’ storage system is granted…
> Once Nine PBS retrieves its data, the station must work with a third party to ensure that no data from other OSS customers is among those materials.
I don’t love this call. Handing over any disk has several problems. It could easily contain very sensitive data from another client. It’s inherently one-sided (another client can’t make the same request if the disk is already gone). And it’s being handed over to a PBS station that doesn’t seem to have the technical chops to manage data backups/security.
I want them to get their data back, but this needs a technical intermediary that handles all client data equally.
2 it doesn't say either way. From owning systems like this, I'd assume there's some degree of "we have no idea what's in this bucket and can't just hand it over to random person asking for it".
Context: I owned a photo backup startup. 8.6 billion photos. Some might be yours, most are not. If you went to AWS and asked for a copy of the bucket, they rightfully wouldn't have complied.
Maybe they mixed data from many companies on the same volume like S3 does. Maybe there were ACLs on that. Maybe it includes PII. Maybe it's a bunch of shards and needs an index on OSS servers to reassemble. Maybe it's encrypted by keys OSS had.
Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).
Plus a size you can genuinely run at home: Unsloth lossless Q8 at 162GB.
In a technical sense it's absolutely possible. Owning the servers != transferring keys to the servers. Most E2E apps run both client and servers, it's about if they ever had key access.
That said, a pure c framework is probably a cleaner way to go.