Yeah that’s sort of how I understand the OP too: The CPU will execute speculatively on the assumption that the next element in the linked list is consecutive in memory, so it doesn’t have to wait for L1 cache. It needs to check the real value in L1 of course, but not synchronously.
Yes. But I don’t think the OP is suggesting this as an alternative to using an array. As I read / skimmed it the linked list is just a simplified example. You can use this trick in more complex situations too, eg if you’re searching a tree structure and you know that some paths through the tree are much more common than others.
But that works on a different level, right? At least as I understand it data speculation is about prefetching from memory into cache. This trick is about using the branch predictor as an ultra-fast ”L0” cache you could say. At least that’s how I understand it.
Yeah I think this is a general principle. Just look at the quality of US presidents over time, or generations of top physicists. I guess it’s just a numbers game: the number of genuinely interested people is relatively constant while the number of gamers grows with the compensation and perceived status of the activity. So when compensation and perceived status skyrockets the ratio between those numbers changes drastically.
Yeah the whole methodology depends on forgetting about state and treating it as a long-running program. If you need to look at the state then you connect a debugger, etc.
Sure. But that’s just compression, right? I guess you could argue that some information is stored outside the genome, in the structure of proteins etc. But the counter argument is that that information is quickly lost in cell divisions. Only DNA has the error correcting mechanisms needed to reliably store information, is my impression.
Swedish banks (even the Riksbank linked above) regularly refuse to turn cash into digital money unless you can ”prove” where you got it from. It’s not sufficient to say (with immense credibility) that you worked hard all your life and saved it. Entire inheritances are regularly wiped out due to this, when high denomination bills are obsoleted by the Riksbank. So I’d say it’s not only a common sentiment but also government policy.
You’d have to explain where that innate knowledge is stored though. The entire human genome is less than a GB if I remember correctly. Some of that being allocated to ”priors” for neural circuit development seems reasonable, but it can’t be very detailed across everything a brain does. The rest of the body needs some bytes too.
The challenges around idempotency remain to some extent, yes. But you have that problem even in non-workflow code, so the usual patterns will just work with no extra mental effort from the developer.
Sure you get more control with explicit state management. But it’s also more work, and more difficult work. You can do a lot of writes to NVMe for one developer salary.
"Exactly-Once Event Processing" is possible if (all!) the processing results go into a transactional database along with the stream position marker in a single transaction. That’s probably the mechanism they are relying on.
> Yes, in any durability framework there's still the possibility that a process crashes mid-step, in which case you have no choice but to restart the step.
Golem [1] is an interesting counterexample to this. They run your code in a WASM runtime and essentially checkpoint execution state at every interaction with the outside world.
But it seems they are having trouble selling into the workflow orchestration market. Perhaps due to the preconception above? Or are there other drawbacks with this model that I’m not aware of?
But couldn’t an LLM search for documents in that enterprise knowledge base just like humans do, using the same kind of queries and the same underlying search infrastructure?
I doubt it was due to network effects. I’m probably one of those top quartile potential H1-B holders that never applied. My main reason for not applying is that the random nature of the lottery reduces the RoI on finding a US job so much that it becomes rational to stay in Sweden and focus on local / remote consulting opportunities. As I understand it the way those Indian IT firms get around that is that they hire locally in India and send in applications for more staff than they want visas for. So they can get around the lottery dynamics through collective action, something I can’t do.
I’d say an LLM is a form of Markov chain, where the state is defined to consist of the entire context window, and each state transition adds one token to that context window.
> So for data centers, training is just as important as inference.
Sure, and I’m not saying buying Nvidia is a bad bet. It’s the most flexible and mature hardware out there, and the huge installed base also means you know future innovations will align with this hardware. But it’s not primarily a CUDA thing or even a software thing. The Nvidia moat is much broader than just CUDA.
The CUDA moat is extremely exaggerated for deep learning, especially for inference. It’s simply not hard to do matrix multiplication and a few activation functions here and there.
Good read. I’ve thought about this distinction myself, but this was definitely more clearly articulated.
> This view is kind of like saying that engineers are just people who weren’t smart enough to do physics, or physicists aren’t smart enough to pure mathematics.
To some extent that’s true though. :) Source: I have an MSc in Engineering Physics.
So how do you best store and iterate on the spec? One way I guess would be to work on a branch an modify Claude.md to reflect what the branch is for. Is that a good approach? Are there others?
Not saying they are wrong, but my feeling is that selection effects probably dwarf any causal effects here. You don’t need much skew in the health distribution of people who are drawn to an intervention like this to get this kind of correlation and effect size.
> You want predictable, linear pricing of course, but sometimes you want to simply be able to get a predictably better response by throwing a bit more money/compute time at it.
Through more thorough ANN vector search / higher recall, or would it also require different preprocessing?
> Moreover, GraphRag preprocessing is insanely expensive and precisely does not scale linearly with your dataset.
Sounds interesting. What exactly is the expensive computation?
On a separate note: I have a feeling RAG could benefit from a kind of ”simultaneous vector search” across several different embedding spaces, sort of like AND in an SQL database. Do you agree?