4,278 karma · joined August 20, 2012
https://aidnn.ai
.NET Bytecode -> PHP Bytecode -> Flash Video games -> Big Data -> SQL warehouses -> AI Agents
The real reason rewrites to fix tech debt used to fail was that it took too long to get to feature partity & the goal posts moved constantly while you were on the rewrite team.
The sunk cost of existing code is no longer considered in a rewrite.
Maybe it was hard to write in the first place, but it is not hard to rewrite anymore, as long you've got decent tests.
My team hasn't finished writing their blog posts yet, but the customer demos [2] went out the moment TypeSafe took our call and sorted all the paperwork out.
There's a tiny section at the bottom of the demo which explains how it actually works (or the github repo is the code).
We have a single decision API[1] over all similar models, so we run evals across all the "System One" api products & definitely don't trust any benchmark that you haven't run. All standardized benchmarks have been benchmaxxed already.
PS: just in case it wasn't clear, the demo is a game where we play to calibrate the human vs machine decision making (or just do it in auto-pilot mode).
The management decisions to scale up production take years to materialize on your food.
Even if you hate what you got out of a bottle you bought this week, that taste is not what made the sauce popular years ago. Popularity seems to cause a quality fade in everything that scales out of a single factory.
For example, the Sriracha from 7 years ago tasted totally different than today & it is the peppers that changed, not the recipe. Palo Alto fire fighters has gotten more cumin than chillis as the spicy stuff goes to the XX version. The Athabascan sauces I've loved from Alaska have slowly morphed sweeter and maple.
Tabasco for all its faults hasn't changed that much in the 20 years I've been having it.
{ "model": "strands-decider-2B-hobson-v19", "answers": { "is_urgent": { "type": "noul", "noul": 0.8287 } }, "usage": { "input_tokens": 86, "output_tokens": 1 }, "latency_ms": 1732.17 }
This is how I got it running - https://gist.github.com/2891eb0db9ea92c1a4e860d44f556292
There's a lot more to be done if we optimize for MLX & let it run on a Mac mini instead of the docker wrapper.
Competing models is table stakes for decision making.
While doing the ablations for the "Team of Rivals" paper, I found that the models were very happy to ignore errors[1] when the data it is classifying looks like its own output.
Jev slots into the middle critique in the paper. The only reason we haven't flipped to Jev last week is because we're a HIPAA shop and Typesafe hasn't responded to our emails.
> don't key on the ANN address. That is precisely the change turbopuffer v3 makes. As you can imagine, it is not a trivial change.
This is a direct parallel to how Postgres and Mysql built indexes.
Your design choice went from a Postgres design pattern to a Mysql one. The difference is the reindexing cost vs the lookup cost - Postgres optimized for lookup and Mysql does for indexing on writes. Or more accurately, Postgres was better with good schema design using joins & mysql was optimized for a bad design with less normalization where many indexes exist for the same table.
Postgres always points an index to a row-id within postgres which is an arbitrary value which changes on each update.
Mysql, always assuming the storage engine is pluggable, points to the primary index entry and adds an extra indirection to the lookup.
This means that you point the mysql index to a stable id, so unless you go update the primary key for a row, you won't have to update the indexes for all the attribute lookups you might have made to data.
I don't do databases any more that much, but the design for NIMBLE file format has a lot of quirks which are relevant to this specific idea (wide tables).
But the old Uber post about switching from Postgres to Mysql to prevent index amplification[1] is a direct mirror to this post.
[1] - https://www.uber.com/us/en/blog/postgres-to-mysql-migration/
This is good, but they're the slow mover due to this exact thing.
Google is getting punished for not letting the models enter an echo chamber and go faster than humanly possible.
Until I had tailscale serve generating valid certs, I had a good reason to use Cloudflare tunnels.
But in general I don't want to put everything on the internet side of things.
Mostly, I don't want something open, but more like a "share with" for people who are in the same office (virtually over tailnet, not physically on the same LAN).
This still works great for a demo instead of a product pitch, to send an link out to see something.
I'd still use a real host over a laptop for those.
My emacs days are 1999 during my "functional or get out" phase, but my .emacs had close to 4000 lines in it, including xterm mouse support, a clone of midnight commander + sftp to a wehbost and deep integration with ctags/cscope/c++filt. There was even a line by line debugger with custom .gdbinit commands - look at the php project's gdbinit[1] if you want an idea of how to walk your own super special data structures.
You can't put Emacs in an IDE comparison and say anything about features without at least one person saying "well, actually".
I eventually switched to vim but that had to do with needing to ssh into 100k+ machines on support rotation & never having emacs on any of them.
Every person's biggest problem is their biggest problem and it eats up their life in the biggest way possible.
Maslow's hierarchy is not a "smaller and smaller" problems list, it is a definition of what your biggest problem is & the solutions just scarcer as you climb up the ladder.
It's close to 20 years since I wrote about the same "problem" [1] in my life and the intervening years have brought much more grief, toil and resilience. Death of several loved ones causes you to consider what a virtue sounds like an eulogy and what yours will sound like in the end.
As an aside, I've felt like literature & art was backwards in the way it was introduced to me - where the first came "man vs nature", then man vs fellow man & then a "man vs self" battle. After fighting self and others, most of us will succumb to nature instead.
So, whenever someone vents or complains, just remember that this is not a contest or at least rage bait, but to be seen as a genuine struggle within yourself that you have put to words.
There's an episode of "Person of Interest" where the AI singularity struggles to play chess as each move cuts the possible moves and the first move is paralyzingly hard for it. Some people are like that, but regret is a form of hope without feathers. It comes from a strong belief that if you could wind the clock back and start over, it would be all different.
I nearly fell into the trap of Nihilism there ("Consider the Total Perspective Vortex")
The ChatGPT app downloads a copy of LibreOffice when you ask it to make pptx files.
So I've downloaded LibreOffice 3 times this week without really paying attention to it.
Multiple model vendors is key here, the cascade pattern doesn't need it, but the critique pattern does.
Last Nov, my team wrote a paper ("Team of Rivals") on the difference between using an OpenAI model to Critique an Anthropic model's output vs running a self-review agent loop on the same vendor.
The ablations [1] proved that neither company alone was better than using both.
The paper was a general response to "What does your company do that Anthropic can't?" but more so a demonstration of how to make something 90%+ good with models which eval at 60% or so (& Gas Town post unblocked our "this is a trade secret" argument about the paper).
Zstd has a seekable format for frames, similar to pigz --independent works.
[1] - https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...
For example, I built a skew handling model which needed low overhead cross-thread counters, where Ampere and Graviton was different from the M1 mac in benchmark - even down to the same assembly on different systems (cmov specifically).
Someone's spending money and they can take a small % to move it from your account to an inference vendor's account. Except for this case you have to also ship tokens across.
If you see Ramp's counter point with launching router.com today, this move makes sense.
Because the next problem is cap budgets and block runaway spending.
I don't quite like how the same CTE lives in 60 different places in my codebase, but at least the WITH clause changed things for me.
Also really liked Snowflake's result_scan for composing chains, mostly because I don't rerun expensive parts again and again. You can use ->> as a shortcut, but I don't think it uses results caching internally to skip waiting for them to all re-run & actually optimizes the whole thing.
This is a labelling law, not a "You can't buy the bumpy tires" law.
This improves the quality of my choice here, because I already look for the little snowflake on tires when buying them for the 3 weekends I drive in the snow.
I'd be revolting too if they banned all season tires over this, because I am not swapping winter tires for thanksgiving weekend, Xmas and ski week.
Also efficient tires don't mean driving becomes boring, a GT86 is on skinnies and that makes it more fun at 35mph in a tight corner than a BMW with pilot sports.
I think they're only calling out cost, because the skinny tires cost more money overall, but there is a net payoff period on bills.
This is roughly what multi-agent systems are built for.
This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.
There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.
Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.
Out of the gate, hard work improves success rates, but not the count of opportunities.
Connections improve opportunities, it doesn't do shit for success-rate.
Luck does for both sides of the mix.
As a startup founder in my 40s, the entire VP/SVP level in silicon valley is people one or two hops away for an intro, though admittedly a lot of the first hop there is through the other founders.
The original paper about importance of weak ties is under appreciated, because people you work with tend to go on to better & bigger things - especially in the beginning of your career, if you are a nice person.
Veritasium put out a very nice video[1] about knowing which sort of game we're playing. It is a power law game.
There are only two takeaways I had, "show up" and "get lucky". There is no winning without showing up.
The core problem of solitude is curiosity and sonder.
When I smell fennel on a hike, I want to know what it is - yeah, this is dog fennel.
When I see a strange symbol on a taxi car, I want to know why it is 3 arrows on bow and what that is.
Seeing a strange dog and wondering what kind of dog that is (a whippet?). A glimpse of a black bird with red shoulder patches.
Hearing someone's ringtone (not these days anymore) and wonder why it is familiar ("ah, it is spelled fa9la, who'd have guessed").
There is a twinge of "This is why we can't have nice things" about the whole article for someone who can't get enough of the world you can see through your eyes.
You could've built a similar system with dezebium and delta lake for quite some time but it would fail compactions, if you run it fast enough. I've seen Oracle GoldenGate 12c do this trick in 2014 or so, using Mysql as the cheap replica. But they are all fragile to schema updates in some direction.
The closest batteries-included equivalent to this is the Aurora -> Redshift bridge[1].
Also to leave Meta, Amazon, Microsoft and everywhere else.
There would be more people who wouldn't join Google, but would love to do this instead.
Netherlands was the poster child of the "Dutch Disease"[1].
They in some ways could've turned into Qatar (and Norway into Aramco) without the right people twisting a few dials at the right time.
I used to use the new hires to hold expertise so that I can spend a weekend at the beach without getting paged. That was the true selfish incentive in spending negative productivity where it takes me more time to teach than to do.
AI absorbs knowledge that the company can keep when I leave.
This used to be a capacity and bandwidth issue, as well as a timing issue with spare time.
I couldn't spend 9 PM to 11 PM on an odd Thu doing this with a human.
> It is compute-bound, and INT4 weights have to be expanded back out before the model can multiply with them, so that extra step makes prefill slower rather than faster, GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. As with the KV cache, the disaggregated design turns this into a choice rather than a compromise: we run INT4 for decode, where it wins, and FP8 for prefill, where it wins.
So each of these improvements are useful even if they have a narrow area of applicability, since the systems can be hybridized for performance.
[1] - https://blog.cloudflare.com/smaller-faster-safer-models/
The A in 26B-A4B is the active weights.
The problem is that this is a per-token load/unload at best, not for the whole prompt.
The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.
Similar thing happened when TPC came up with SQL benchmarks.
If you're not good at TPC, your engineering team is no good.
If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.
Winning on it is the price of admittance into the game, especially in a crowded market.
But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.
For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.
The single trit packing is a commonly optimized DBNULL structure for booleans.
Bits/trit approaches the 1.5 asymptote, because that is the fundamental packing limit.
The trick is to use it when you have a trit to start with, like when you have a set which is a tiny bit over a power of two.
There are places where you end up with odd numbers in set sizes, for example when storing a poker hand.
Read Cactus Kev's trick[1] which I think needs a 27 bit section & optimizing it was where I first ran into trit packing.
Douglas Adams said this beautifully in the Campaign for Real Time & the poet Lallalfa.
And I just found out h2g2.com is no longer online, so no reference.