elendilm

> What industries is it common ... perform better than the ones built with coordination with the storage engine and everything else ...

Uber for one as documented here https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...

There are many more and most may not be documented online.

Our usecase is offline first mobile ERP (roll-out hopefully soon) with online-sync capabilities doing invoice/coupon processing, etc. in a network of regional shops - with a focus on minimizing latency and aggressive reduction of costs at scale.

Once we measured the reduction in latency due to parallel fanout (over index shards - not just data shards) for range scan, there is no going back ever again.

When you need the same database to be running on mobiles and servers alike, you need embedded database for mobile and standalone executable for server.

> they will perform better than the ones built with coordination with the storage engine and everything else?

No amount of coordination with the storage engine can override the physics of lock contention in distributed systems.

> This seems a very unusual set of circumstances so I'm very curious on the use case.

Not with my background. Its the norm :)

> What is extreme scale here?

Not just data, but concurrent operations per second for writes and reads involving seperate transactions. Eg: Scaling concurrent write transactions on different shards with minimal write lock contention.

> Who actually needs this scale?

Many do including Uber https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...

> You can just do this with postgres.

You seem to have missed the final sentence beginning with "Note:" in my original comment. Nobody said you can't do this with postgres.

> Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?

What most people get right is not my concern. If you find building a thin wrapper layer for your application as hard, clearly this may not be applicable to you.

Usually first principles architectures and systems engineering doesn't require you to be validated by big players. You can always spin up and do your own measurements.

Yet since you asked, assuming good faith, here is what you are looking for.

Following is an excerpt from https://www.uber.com/us/en/blog/schemaless-part-one-mysql-da...

< The indexes are eventually consistent; whenever we write a cell we also update the index entries, but it does not happen in the same transaction. The cells and the index entries typically do not belong to the same shard. So if we were to offer consistent indexes, we would need to introduce 2PC in the writes, which would incur significant overhead. With eventually consistent indexes we avoid the overhead, but Schemaless users may see stale data in the indexes. Most of the time the lag is well below 20ms between cell changes and the corresponding index changes >

> You also now have to manage your own ACID guarantees and what amounts to cache invalidation ...

Hint: Extreme scale. You anyway have no full ACID guarantees at scale and one has to implement their own relevant guarantees and cache invalidation at extreme scale. You either have to implement your guarantees or pay the cost of bottleneck and latency.

They are achieved via idempotency, durable execution, etc.

There is sufficient reason for me to think that either you may not have read my comments attentively enough or you may lack the experience of building reliable systems at scale.

A developer need not be hand-held with this reasoning on the topic of scaling. I have given ample reasoning to reconstruct.

I advise do your own research.

Additional indirection was always there for indexes, be it database managed or application managed.

If the database is running embedded in your application, then there is very little extra cost involved.

If one is using the database as a separate process, you incur additional latency for the network round-trip + the payload of the matched index items.

That is the cost you pay.

But in return you get natural parallel fanout, reduced locks/contention bottleneck, etc.

You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further.

You also get a simpler mental model, simpler queries, more deterministic and predictable latency. The list goes on.

Application-managed indexes are ideal for extreme scaling.

> Are you suggesting that one should recreate B+trees in code?

Why would you think that?

No, not at all. Application maintains indexes as separate normal data/row via CRUD operations into a dedicated table/collection with references to the original instead of letting the database do it on your behalf.

> Seems to me you did what any engineer worth his salt would do: sit down and look at the underlying mechanisms, to map the problem, and then highly efficient solutions present themselves.

Truer words were never spoken.

> It's amazing to me no one seems to have done this before.

You would be surprised how many valuable problems and solutions have not been solved by anyone before. Apparently, it happens to be the norm in the wild. My understanding is that it's about how far one is willing to strive for an ideal.

> It seems we are getting closer to the "final language", and languages become a solved problem.

A final language? I like the sound of it. No more picking complexity, latency, simplicity, expressiveness, portability and other tradeoffs.

Looked around your blog. Awesome work.

A lot of developer rituals can be avoided for obvious cases (of multiple writes to the same variable) leading to clean code. Your Valen looks promising.

I learnt, while reading your blogs, that rustc can run embedded - which might come in handy for one of our projects, Craft, for embedded compiling + uploading on mobile.

We have built arcc (architectural compiler/linter) which enforces architectural rules for correctness like strict CQRS separation and other 10+ architectural rules at compile time. Currently, arcc is for TypeScript/JS codebase. Future scope is on Rust codebases.

> Then that evolved into a couple more things until I finally figured out this reverse approach. We'll see if it sticks.

The way you phrased that sentence, something tells me that it sticks. Reading tea leaves: compressed in those sentences somewhere is an insight you have which you are damn sure of, while deliberately withholding it and thinking it through before you guarantee it yourself. Did my tea leaves fail me :).

Extreme scale and indexes are not at all a happy couple.

Doing away with database-managed indexes at scale makes for simpler queries and predictable latencies now and in the future. These indexes have costs associated with them which become detrimental as scale rises.

Application-managed indexes along with the required codebase discipline results in reduced bottlenecks, minimal latencies, and better parallelism. One can even parallelize the index-fetch phase itself, not to mention all the other goodies that come with it. This results in a different trade-off as cost moves to the application where it is controlled and therefore leveraged.

Note: I am talking generic database architecture and not just about Postgres.

> I think good products carry something of the people who make them. Their experience, what they care about, and how they think things should work.

Couldn't agree more. It does reflect many of the values of what an ideal product should be based on the developers' experience.

It takes effort and work to un-dumb ourselves atleast in that domain.

I would argue that its a function of raw effort in varying domains.

> The deepseek philosophy has always been to make a small profit or not do it at all.

That is a very respectable motto for any company.

Article mentions GETs being slower and round trip latency as a major villain multiple times.

Article even dedicates a section titled "Caching on disk and in memory" for latency reduction.

Latency is the whole point as that is the ultimate villain when you decide to build index search on S3.

Its really hard to miss.

But otherwise agree.

Appreciate the strategies that went into the work to optimize a high latency foundation to feasibly work as a search index.

Fundamentally, our work and stack goes against this specific architecture. Were I to make one, where scalability and efficiency for a search index is prioritized, I would approach it more from a regionally distributed hierarchical fractal architecture where caching is leveraged at every layer.

But architectures as presented in the original article put strong constraints on the solution space which is how exceptional insights and breakthroughs happen.

You have my respects.

> There's not really a difference here between S3 and your disk, aside from the literal amount of latency.

Latency is the whole point.

> She's not really 'fearful', the framing is a device that lets ...

The very first paragraph of the article ends with extreme concern for being labelled as AI in the future. The whole article builds on that.

Pretending it to be a framing device or that somehow she is magically exempt from that foundational concern on which she further elaborates is a bold claim to make. I simply don't buy your excuse. You cannot have an entire essay be built on an emotional premise and then pretend that the premise doesn't count.

> What is conspicuously missing from your comment is the technological refutation that makes her wrong.

Your skill to read things that are not there is remarkable. Read sentence 1 again. Her being non-technical only makes for being gullible (not flat out wrong) due to a lack of understanding the nuances.

> But that would require a nuanced refutation and not just cheap shots.

Read the article's paragraph talking about Jacob Coxen, then the subsequent paragraph about bunkers and Thiel's anti christ and then further down about Jacob Coxen again.

The context of the former Anthropic employee fear mongering about world-ending and pointing to the Hugging Face attack as the primary evidence as if we developers are gullible enough to buy that is well known. If buying into that isn't a lack of understanding the nuances (due to being gullible), you tell me what is?

If you had attentively read both the article and my original comment, you would have understood the context and the content.

Ironically, your cheap shots argument has now demonstrably applied to you.

Seeing as it is your first post, do elaborate on what really exists.

My first impression: beautiful and impressive.

The product managers and developers at Apple have a unique capability to gaslight an atrocious user/developer experience as excellent.

Folks who parrot the same are just a symptom of the underlying epidemic that permeates the ecosystem.

Some of their engineering deficiencies just happen to be masked by integration with better hardware.

For example, the iOS mmap limit is stupid and ridiculous. But it paved the way for us to work around the restriction by rebuilding Dip's architecture, resulting in a better, more simplified mental model along with increased performance and parallelism.

These benefits are applicable cross-platform and not just for Apple's OS. This subsequently also acted as an indirect but causal trigger to later build our own KV engine: Singularity.

Thus my experience with Apple has been bittersweet. The frustration with their ecosystem and their stupidity was beneficial in hindsight.

I am torn halfway between pitying and thanking them.

> So if you decide someone is stupid, it means you’re not thinking, which is not being smart.

What if, according to the author's definition, after thinking through, one has decided someone really is stupid. Then one's decision that the person is stupid is just true. Isn't it being smart according to author's own definition.

It appears the author didn't think it through essentially making the author "stupid" according to the author's own definition.

That brought a smile to my face. Nice one.

Even during replication, any unnecessary network payload beyond the absolute minimum required is still a cost that boils my blood :-).

"Unnecessary" is the keyword you missed. Refer to sentences 1, 2 and 3 in my response.

The argument of cost is fundamental and no amount of Kafka, etc. can override it.

The very thought of a single storage request hitting the network unnecessarily boils my blood.

Some latency increase must be traded for scalability. But at least we get the required utility in return.

But without that utility, one is just hiking up the cost. Any competitor who can deliver at reduced cost holds a potent weapon that you simply cannot survive.

A company that has efficient cost structure across its operational domains holds an advantage which is near impossible to out-compete. A sense of inevitability and resilience is what reduction in cost buys you.

Cost is a weapon. One of the most fundamental weapons in almost all domains.

It is entirely plausible that, as you say, they could be piggybacking off of human work internally and using AI like the punching bag it is - dumb but useful. I wouldn't bet against that, but usually it is difficult for people who did the hard intellectual work to have their credit taken away - which might be exactly what we saw with Navier-Stokes drama that unfolded.

> If the models are PhD smart, shouldn't be able to come up with results with minimal guidance?

Smart is born out of wrestling with the problem domain, not in virtue signaling via academic credentials. So anyone leading with an academic credential is a dead giveaway of the PR stunt underneath. Nobody says PhD smart Einstein - because work can lead the speaking and doesn't need a credential as a support crutch.

> AGI and the near super intelligence? It doesn't track.

True ideal intelligence must not hedge, shouldn't pander to popular opinion or consensus. True intelligence must state what is true as maximally true as is correct while being able to derive more true sentences coupled with instant acceptance when proved false. The current status quo with the LLM doesn't display anything even remotely resembling that. Current LLMs confuse truth with popularity, consensus, corporate speak, etc.

My own bet, like all our stack, is on a memory efficient deterministic AI, where the next token is derived instead of predicted. I don't know whether anybody is working on something like that.

The current AI is definitely useful and it would be false to say it has no utility. I just would like everyone and everything to be more honest and true and focus on the LLMs existing capabilities. On how it can be a useful utility.

This hype train of anthropomorphization and grandiose capabilities, which obviously is false, is what pisses me off.

Yes, the arguments we are handed don't line up.

AI is innovative but cannot string together true confident sentences.

AI is so innovative that nothing innovative or groundbreaking has so far appeared. Maybe it will, but the claims to ground-truth ratio is off the charts.

World ending is so funny it is usually only parroted and swallowed by gullible people.

> it can do the work of a PhD on almost every field of human knowledge

This is the most revealing and the most hilarious one. This is essentially the argument from authority wrapped in a PR stunt. Those who produce real innovations overwhelmingly does it outside curriculum and as part of the real work they are engaged in with very few historical exceptions. That statement reveals a lack of understanding of what innovation itself means let alone the virtue signalling they are trying to sell.

> machine specific math proofs

This one was initially intriguing, but as I earlier said, is a well defined and constrained bounded domain. The accusation that hours earlier other human researchers's work may have been used to plagiarize the proof has muddied the waters further.

I agree. Your experience mirrors mine. Yes, they are often competent discussion partners, although a lot of people I hang around with are also excellent discussion partners.

But unlike with real people, LLMs have this unique property that we can abuse them without consequences directly leading to our improved understanding of the subject material using nothing but pure frustration as the motivating drive.

So for me, they are productive precisely because they remain dumb discussion partners.

I don't buy this whole narrative that "AI is innovative".

If AI is so innovative then why am I seeing a dumb idiot every time I discuss anything even remotely innovative with it.

Most often it cannot even produce a true sentence when the claim in the sentence is modestly strong without injecting qualifiers and distancing itself from the claim.

Having to just accept that it produces relentless innovation is so detached from reality that it is not even funny.

It is true that it can write decent code once the domain bounds are well defined. What I have found is that it is good at scrutinizing already written code - but only with expert supervision. Even then it most often tests our patience with stupid suggestions.

In C, mostly you think about your code running on the machine with some thought dedicated to the compiler gymnastics involved during the compilation phase.

With Rust, you are front loaded with a myriad of compiler gymnastics you need to think through.

But once you get comfortable enough, it becomes natural. You also have to get accustomed to writing code that is more verbose than C which might look ugly at first but later you start to accommodate it as the necessary cost for the utility you are handed in return by the compiler.

For example, multiple variables in Rust doesn't necessarily mean multiple memory allocated variables at runtime like in C. The rust compiler will usually keep track of and ensure multiple variables (non Copy types such as String with Move semantics) map to one memory allocated variable at runtime (in normal single threaded use cases under normal circumstances without using RC, ARC, etc.). Eg: let a = String::from("hello"); let b = a; ... Note: The example is for illustration purposes only and not always true. In summary source code variables are abstractions and may not belong to distinct runtime memory location. Yes it is true even for C. But Rust's ownership model makes that distinction aggressively visible.

My own journey has been quite different.

When I was building my kernel in my late teens, I first wrote the bootloader by hand on paper in assembly language. I then referenced the x86 manual for the instruction set and converted my assembly code into the equivalent hexadecimal machine code values of the x86 machine instructions, which I also wrote by hand on paper. Then I used a hex editor on the desktop to manually write the hex values into a file and used it as the bootloader i.e as the first 512 bytes.

The whole exercise gave me a sense of hard grounded zero magic, raw, unfiltered experience. This is an experience that is hard to replicate in any other way. I deliberately did that so as to peel away as much magic/abstraction layers as I possibly can.

Later on when I started using C, I never had to learn C but merely just had to reference the equivalents of the assembly language. Like how the primitive "if" doesn't exist in the hardware but is a composition of cmp and jmp instructions. Seen this way, C becomes a glorified portable syntactic sugar over assembly language.

Then higher up the ladder to C++ for object oriented problem solving while retaining the spirit of functional programming. Rust was a breath of fresh air, where correctness across a myriad of use cases was a first class primitive.

Each abstraction layer can thus be evaluated for its utility in problem solving while its underlying mechanics remain understandable down to the hardware level.

More recently, our own arcc compiler extends correctness to our architecture and not just the types.

So if you are young and have time to spare, I suggest a little bit of Assembly => C => C++ => Rust.

This essentially makes you immune to hype train bullshit.