rescrv

Hi HN,

I'm Robert, a Member of Technical Staff at Chroma. I've been working on transaction protocols for at least 14 years now, starting with my PhD in distributed systems. During my tenure at Dropbox, where I worked on the metadata layer for storing non-filesystem data, we added transactional support to Edgestore, the storage system for the same.

Agents fascinate me. With agents, we have real reason to question the fundamental tradeoffs we've made with software. Coding agents build differently, communicate differently, and thus need different solutions than engineers have built in the past.

We've built a new context layer for organizations called Foundation. With Foundation, we treat the agents as a distributed system and use techniques from the literature to guide the swarm effectively.

At the heart of Foundation sits some novel distributed systems work we call the Fission protocol, documented in this post. In short, breaking atomicity of transactions is encouraged and necessary.

I'll be here to answer any questions you have about the post.

Happy hacking, Robert

I think I've discovered something new and useful. Looking for others to share with.

I’m not claiming forth should be used as is. I’ve opened the benchmark so others can reproduce the result I share in the post: https://github.com/rescrv/stack-bench

I haven’t tried the extremes. Context rot says it’ll likely degrade there anyway.

What I’m investigating is if more compact languages work for querying data.

What makes you think it’s going to clutch at straws more? What makes you think it won’t do better with a more compact, localized representation?

Looking to discuss with people about whether LLMs would do better if the language had properties similar to postfix-notation.

I appreciate the feedback. I can definitely see people thinking I'm crazy by the first few chapters and not getting to the good part.

I was trying (and failed) to be clever about improving your life.

What you say about exaggeration vs hyperbole vs analogy vs literal resonates with me. Have you considered they may not, themselves, have an answer to this? For example, I often throw out (hyperbolicly) really wild numbers at work that are off by orders of magnitude to illustrate what happens when stuff grows to an extreme; yet, I'm also very cautious to get precise numbers when building. My co-workers may be bothered by this (so I'll be more mindful now that I've probed for this insight), but I would like to think that hyperbole is easily detected.

Honestly, I don't know if I can help with perspective without witnessing you with your care-person. Everyone's different and the nature of the beast is you only interact with really ill people when you yourself are out of commission. I wouldn't want to generalize me onto someone else. Perhaps you just show them the post I'm replying to and ask what they think?

From where I sit today, I can say that I see things others don't. I have a hard time saying they aren't real, though, as experience has been a zero-knowledge proof the the commentary.

The front half of the book, up until a clear pivot chapter, is me trying to put to words a coherent schizophrenic vision; both to establish credentials and to give people a sense of what it can be like.

A common theme I've seen both in myself and in others who have spoken to me about their experience is that it's common to have a delusion that you are at the center of a religion.

Personally, I've been blessed with insights at the core of every major religion that I know more than a slight deal about. It just so happens that the episode I detail in this book has me at the center of a prophecy; It rings in my head as truth, but I don't have any proof of that truth to produce.

In the latter half of the book, I introduce a conversation with Master Yoshu that I feel is critical to the message. It would have been awkward to say, "I heard this from a voice" without the setup to give that.

Regarding your last paragraph, I think what you'll find is that life is rarely ever centered around something that you can ever adequately put to words. Even if someone is always 100% literal, their concepts will not match your concepts; labels are the source of much of our suffering, and trying to label something as example or metaphor or "not literal" already betrays that you're unable to falsify it; thus, there's something to it.

I don't want to dismiss you though: If I went through that burden of categorizing my own text, how would your life be improved?

I appreciate you sharing.

My understanding is schizoaffective disorder is one of four categories of schizophrenia. I think people often use them interchangeably.

Medication can be tricky because when it works it seems increasingly unnecessary and often has side effects of its own.

I hope no one reads my book as a substitute or prescription to avoid medication on their path. To each their own, but it's not what I'm advocating for.

Thank you again!

For my LLM/agents framework, I went with a virtual filesystem abstraction and implemented helpers for mounting and permissioning the virtual file systems: https://github.com/rescrv/claudius/blob/main/src/agent.rs#L1...

I'm sorry to hear you were heartbroken.

I assure you I've seen people who are worse than me turn out much better than me and I hold hope that I'm on a good trajectory.

I appreciate you saying something. As you'll read in the book, I've got a project going to primarily help myself, but to date it's been 100% open-source.

I haven't been able to be consistent enough to both pay the bills and volunteer and work on the open source, so I focus on the former.

Would you mind saying more about what you do every day? For me, I anchor myself for at least a set time; when I sit during that time, nothing can touch me and I can deconstruct symptoms to recover any deviation from baseline.

Thank you for the book recommendation! I will see if it's for me.

Hi Dave,

I'm surprised you can remember me. We once sat near each other on a bus for SOSP, and that's the extent we interacted in person, so it warms my heart to hear someone like you remembers someone like me.

In case you're looking for the technical, the book doesn't say such, but one innovation laid out for me was my work on lsmtk, a new compaction algorithm for LSM trees. I'm not sure if I'm off my rocker here or not, but I documented it when I released the crate: https://crates.io/crates/lsmtk. I know you're busy, but in case you revisit this thread and want a neat trick, I thought I'd bring it up.

Lastly (and most important as I'm prone to doing), I appreciate you sharing something personal like that. Hearing that others who have tried to make it work can often do so keeps me going on my worst days.

Thank you! This is the kind of comment I hoped to see.

I'm betting it was published in a hurry. I know I would hit "publish" within 24 hours of creating such a result, and would hope it would go wide. I'd publish to arxiv before getting clearance to release the code. I bet that's what happened here.

I appreciate your explanation of the Curry-Howard correspondance. I was familiar with it, but not with Lean in particular. I'd heard of Lean, but didn't know how it worked.

Thank you again!

Once upon a time, people published solo often. It's just harder to do things like that these days.

I'm not sure if this is real, but the abstract says machine-verified.

Most agent frameworks aim for chat; stigmergy aims to replace the Fortune 500 CEO's role, or be able to run a healthcare clinic. Something of that scale. This is a prototype. I’m looking for feedback before implementing the main loop.

Absolutely! We took care during the build to address chroma’s needs, but I believe the only requirement is chroma storage. Long term it’ll appear on crates.io as a standalone package.

Hi,

I'm Robert, one of the authors of wal3. Here to answer any questions or explain more about our engineering philosophy.

Happy hacking, Robert

Can you elaborate on what you mean by "nesting" setsums? It's an associative and commutative construction, so you can run e.g. map-reduce to map every parquet file to a setsum and then reduce to roll them up.

I think you're pointing out a good tradeoff here. The original copysets work allows you to explicitly tune the likelihood of data loss with the amount of work done for recovery. A cluster replicated to minimize the likelihood of losing a replica set under correlated failure will have a higher cost of recovery from failure. A cluster replicated to minimize recovery time (e.g., RAMcloud's random allocation) will likely lose entire replica sets upon a set of (even random) failures.

Chainsets were an attempt to add the properties of copysets to a system based upon chain replication.

Working with the original Copysets authors we refined the chainsets algorithm into a tiered replication algorithm that can enforce independence assumptions on the replica set (what you've termed anti-affinity). Here's the paper on the subject: https://www.usenix.org/conference/atc15/technical-session/pr...

Indeed pocdb is not intended to be a fully "ready" implementation. There are likely liveness bugs (that can be mitigated by periodically calling work_state_machine on every write outstanding), but it is unlikely to have safety bugs stemming from protocol misunderstandings (any safety bug is likely a small typo).

My main projects using Paxos are Replicant[1] and Consus[2], both of which have consumed significantly more time to get Paxos correct.

[1] https://github.com/rescrv/replicant [2] http://consus.io/

JS vs sparse C++? Cut comments and change coding style and you can easily get close to 500 lines.

One thing I think is worth touching on is the pipelining ability of multi-paxos that is missing when you have a single register with the Synod protocol. For key-value operations, it's not a problem. For a true replicated state machine, this can hinder performance.

In Replicant[1] I added pipelining to ensure Paxos was unlikely to be a bottleneck to replicated state machines. For complex state machines, the state machine becomes CPU bound.

[1] https://github.com/rescrv/replicant

If you mean, "The Part-time Parliament," it's indeed a little complex.

It's worth reading his description of the paper to understand why: http://lamport.azurewebsites.net/pubs/pubs.html#lamport-paxo...

In short, not everyone has the same sense of humor.

In order to get a linearizable read in Spanner, you do need to have a heartbeat from the leader of the Paxos tablet for the data you are reading in order to know that all data has been replicated up to the time at which you are taking your read. This is not free, but can be cheaper than active communication. (referred to as t_safe in the paper)

Paxos's complexity is often overstated. I made a simple implementation of Paxos for a key-value store as a proof of concept to demonstrate how you can simplify Paxos by removing leader election, multi-master management, and log garbage collection.

Here's my blog post on the issue: http://hack.systems/2017/03/13/pocdb/

The entire implementation is 1100 lines of code including comments.

I'd ask you to please read the discussion section of the paper on Consus before calling Aphyr.

Almost every single host SQL solution I know of is built on a B-Tree for at least some of its properties. A B-Tree is nothing more than a lexicographically sorted key-value store.

Consus is a lexicographically sorted key-value store.

Where you will hit performance bottlenecks is when your query planner/executer makes traditional assumptions about performance that do not hold across machines.

A good SQL implementation on top of a kvs should not force you to accommodate it any more than your current "normal" databases subconsciously train you to accommodate your disks (hint: they do).

"... In Modern times, Consus is very similar to the word consensus; not coincidentally the former uses the latter to serve as the protector of data."

Code here: https://github.com/rescrv/libmacaroons