losfair

Hi!

> what exactly is verified vs. verifiable concerning the hosted linux kernels

We don't really have a fully formally verified hypervisor - SMP verification is not done on seL4 and AFAIK it's hard. And userspace verification is another story :) Verifiable here just means that we have a very small TCB that passes hardware-based trust (e.g. TPM / SEV-SNP) to guest VMs, so that users can verify their VMs are not tampered with.

> how your HV looks at DMA capable devices

We manage VT-d / AMD-Vi IOMMU in the root process. The processes that handles both untrusted guest data and DMA (blkd/netd) only start after IOMMU initialization completes. Guest VMs themselves only see paravirtualized virtio-blk / virtio-net devices.

Hi - zeroserve Caddy compat works by compiling Caddyfile to C. Technically I can keep the compiler in a separate binary/repository - but it just feels unnecessary.

zeroserve only understand eBPF at runtime. It's always the source-of-truth.

Hi, thanks!

Will implement forking + SO_REUSEPORT. I've been using nftables for things like this so haven't needed it for myself yet :)

Hi, author here - a few critical pieces of this, like async-ebpf, were written long before those coding agents were released. I use AI assistance a lot when creating zeroserve itself, but I manually check AI output and take responsibility for it :)

> Firstly, can there be an easier way to stop a microVM mid execution in this single executable bottlefire format and then rerun that and it would start mid execution. (something akin to how criu does it?)

Not yet - Firecracker supports snapshotting so this should be doable though!

> if something like microvm could be run in normal cloud infrastructure?

Some cloud providers - like GCP and DigitalOcean - do support nested virtualization, and they work pretty well with Firecracker. Using VM migration to run stable workloads on spot instances sounds very interesting :)

You might be looking for CRIU (https://criu.org/) - it works perfectly on the current kernel.

Distributed consensus + fsync() determines the lower bound on commit latency. We have to wait for the data to be durable on a quorum of transaction logs before returning success for a transaction. That's usually 5-10ms, even within a single region.

(I work on Deno KV)

A read-modify-write retry loop causes a high number of commit conflicts, e.g. for atomic increments on integers. Getting higher than 1/RTT per-key throughput requires the "backend" to understand the semantics of the operations - apply a function to the current value, instead of just checking whether timestamp(value) < timestamp(txn.start) and aborting commit if not.

I wrote a self-hostable control plane for Nebula (a Tailscale-like overlay networking tool), and have been using it for about a year: https://github.com/losfair/supernova

Built this because existing solutions like ZeroTier and Tailscale are trying to be too "smart" (auto-selecting relays, auto-allocating IPs, etc.) and do not work well for complex network topologies.

I work on Deno KV. Happy to answer any question!

Engineer working on Deno KV here :)

> Is the data stored in all regions at all times?

Data is always replicated to three of our "big" regions currently. Extending the list of storage regions and providing more flexible data distribution configuration is one of the next things we want to do.

> what about pricing?

During the closed beta it's free with a 1 GiB per project limit.

Engineer working on Deno KV here. Building on FDB is mostly a pleasant experience since it solves the hard part of the problem for us (concurrency control and persisting mutations).

We sometimes run into its limitations - the way we are using FDB is a bit beyond what it was originally designed for. But when it works, it works great.

Online DDL is now a WIP feature. This will allow to convert DB into read-only mode & run arbitrarily large schema migration concurrently (first stage), and eventually fully concurrent DDL by replaying logs.

I found foundationdb-rs pretty good & robust. Haven't tried any FDB 7.x features (tenants, etc.) though.

What are you trying to achieve with WAL mode? Is it some kind of application compatibility issue?

The entire SQLite journaling mechanism is not used by mvSQLite (you can set journal_mode=off safely - although SQLite won't be happy to do explicit rollback in this case)

We found a way around FDB transaction time limit. A mvSQLite transaction is not directly mapped to a FDB transaction (details in readme and wiki)

Maybe the inherent overhead of synchronous replication is more on latency rather than throughput.

Shameless plug of my mvSQLite [1] project here! It's basically another distributed SQLite (that is API-compatible with libsqlite3), but with support for everything expected from a proper distributed database: synchronous replication, strictly serializable transactions, + scalable reads and writes w/ multiple concurrent writers.

[1] https://github.com/losfair/mvsqlite

Bun's SQLite implementation has correctness issues:

https://github.com/oven-sh/bun/issues/921

https://github.com/oven-sh/bun/pull/1056#pullrequestreview-1...

This makes SQLite transactions no longer serializable (in regard to the schema), and breaks the safety of any kind of external concurrency (e.g. mvSQLite and Litestream).

TrueTime is an efficient primitive to ensure external consistency across multiple consensus groups, not within one group. The current DO infrastructure does not give the TrueTime guarantees for free: you cannot do transactions across two or more durable objects, and the max throughput within a transaction domain is limited to what a single V8 isolate can handle sequentially.

The one feature I miss the most on Cloudflare Workers/Durable Object is TrueTime.

Durable Objects are fundamentally replicated state machines with a nice JavaScript API. You can build an automatically sharded (yet correct) Kafka API implementation, or even an entire Spanner-style distributed database on Workers, given the right primitives (DO + TrueTime).

This project looks really exciting!

I'm working on mvsqlite [1], a distributed SQLite based on FoundationDB. When doing the VFS integration I have always wanted to patch SQLite itself, but didn't because of uncertainty around correctness of the patched version...

A few features on my wishlist:

1. Asynchronous I/O. mvsqlite is currently doing its own prefetch prediction that is not very accurate. I assume higher layers in SQLite have more information that can help with better prediction.

2. Custom page allocator. SQLite internally uses a linked list to manage database pages - this causes contention on any two transactions that both allocate or free pages.

3. Random ROWID, without the `max(int64)` row trick. Sequentially increasing ROWIDs is a primary source of contention, and causes significant INSERT slowdown in my benchmark [2].

[1] https://github.com/losfair/mvsqlite

[2] https://univalence.me/posts/mvsqlite-bench-20220930

Unlike a regular filesystem, mvsqlite offloads both storage and transaction processing to FoundationDB. Fine-grained optimistic locking is implemented, and massive write concurrency across a cluster is possible.

Software that is not formally verified contains bugs.

All known correctness bugs are fixed (except experimental features) - this is just a reminder that you need more than one layers of defense for something that stores your critical data.

The idea that mvSQLite is based on (decoupled storage and compute) has been implemented for the other two most popular SQL databases. MySQL and PostgreSQL have Aurora and AlloyDB, and PostgreSQL additionally has Neon. This is something missing from the SQLite ecosystem (before mvSQLite).

And SQLite has a unique advantage when implemented on top of decoupled storage and compute. Decoupled MySQL and PostgreSQL can't really linearly scale from zero in a serverless use case: you need to spin up a compute instance when you want to do queries. This isn't necessary with mvSQLite, because the embedded SQLite library can just connect directly to the multi-tenant storage engine (mvstore).

FDB refuses to process writes (degraded write availability) when usable disk space goes below 10%. Nothing will be lost though.

Hi mrkurt!

Litestream/LiteFS are amazing projects. The FUSE-based approach is interesting (I'm implementing something similar in mvSQLite, thanks for the idea!)

> Graceful failure

mvSQLite is designed to continue to operate under degraded network (there is a fault-injection test specifically for checking this property: https://github.com/losfair/mvsqlite/blob/1dd1a80d2ff7263b07a...). Network errors and service unavailability are handled with idempotent retries and not exposed to the application.

> Good for caching

mvSQLite caches pages read and written, and does differential cache invalidation (only remotely modified pages are invalidated in the local page cache). The local cache is just a regular KV store with invalidation strategies, and can be moved onto the disk. So it essentially becomes a consistent local database snapshot.

Hi otoolep, thanks for the corrections! Actually I've just made a few changes to the wiki page before this comment :)

rqlite's readme seems to indicate that it uses Consul only for service discovery, and runs Raft internally?

I think the most important difference here is rqlite runs its "data plane" on a single consensus group/state machine while mvsqlite relies on FDB's distributed transaction system (well at the bottom there's a Paxos but it is only used for metadata coordination). Both approaches have advantages and disadvantages though.

FoundationDB handles the hard part! It provides monotonic "versionstamps", externally consistent transactions, along with other useful features. I recommend FDB's architecture docs: https://apple.github.io/foundationdb/architecture.html

I'm working on the benchmarks. There shouldn't be surprising results with read scalability, as it is basically guaranteed by FoundationDB. For writes it is indeed more complicated and a benchmark would help here.