penberg

We're aware of your woes with ClickBench. The first time you added it I even explicitly told you that it was way too early to even try that. The attempt you did recently was fair, but it's all in the ingestion before the benchmark, and does not show up anywhere, so we ended up spending time on this instead:

https://turso.tech/blog/turso-0.8.0

In any case, this is on our radar and we do intend to improve it but not the highest priority right now.

There's a reference manual in Git repository, but probably not what you're looking for. Will add something.

Yup, Ghostty build requirement.

You're not wrong about VC-funded database arc, but what history are you even talking about?

I am sure you understand that I have absolutely nothing to do with Scylla's licensing. I have not worked there for four years nor was I ever in a position there that I would even had that opportunity to influence such decisions.

I am also sure you understand that Scylla's development model was completely different: they had AGPL license and contributors had to sign a CLA, which is why they were able to relicense in the first place. Turso is MIT licensed and there's no barrier to contributing and, therefore, already a much bigger contributor base.

I fully understand the scepticism, but you're mistaken about the open source history of Turso's founders.

The project is MIT licensed with a growing community of contributors. It does not even matter how long the company lives, all that matters is that some of the core contributors live.

The actual reality is that I personally started the project because its synchronous architecture is holding back performance. You can read all about it in https://penberg.org/papers/penberg-edgesys24.pdf. The design is literally the next evolution of SQLite's architecture.

Yeah… Attempting to integrate MVCC and then doing vector search gave enough perspective to do this!

Remember, that argument to `findOne()` is an arrow function. You need a compiler to turn it into a efficient, deferred query; otherwise you need to _evaluate_ the function at runtime.

I don't know if SQLite + Raft is going to be the next Postgres, but it's certainly an interesting solution for a variety of use cases where you don't need your data _partitioned_ on multiple machines (but still want it to be replicated for availability and fault tolerance).

As others point out in this thread, there's already rqlite for Go, sqlite for C, and now we've open sourced our work on bringing SQLite and Raft to Rust: https://glaubercosta-11125.medium.com/winds-of-change-in-web...

User-level threads do not solve the problem of synchronization and data movement. That is, with a “thread-per-core” model, you eliminate most needs to synchronize between multiple CPU cores, and, therefore, eliminate the overhead of acquiring and releasing a lock and allow the out-of-order CPU cores run at full speed. Furthermore, “thread-per-core” model ensures that memory accesses are always CPU local, which eliminates the need to move data between CPU caches, which eliminates the expensive CPU cache consistency protocol (that makes scaling to multiple cores more difficult).

That said, I am not claiming thread-per-core is _the solution_ either, just saying that if you can partition data at application-level, you can make things run plenty fast. Of course, you’re also exposing yourself to other issues, like “hot shards” where some CPUs get disproportionately more work than others. However, as we scale to more and more cores, it seems inevitable that we must partition our systems at some level.

So thread-per-core is not just about eliminating context switches. It's also about partitioning application-level data to reduce inter-core synchronization to let speculative, out-of-order cores run independently as much as possible. Don't get me wrong, user-level threads are great, and arguably much simpler programming model than async/await. But they're not _the solution_ either.

Excellent question! I think VoltDB, for example, uses the same application-level data partitioning approach, but with processes instead of threads. One advantage of the "thread-per-core" approach is that it allows fast communication between shards because the underlying threads share the same address space. In contrast, processes need to use a more heavy-weight approach of inter-process communication (IPC). Of course, process-based systems will have a reliability advantage, because processes cannot easily mess with each others state.

AGPL treats "networked access" as "distribution", not "linking".

With the GPL, you are required to make your modifications available as GPL if you distribute the modified program. This creates a loophole for ASPs, which can modify GPL’d code, but because they offer it as a service (and don’t distribute the program), they don’t have to share the modifications. AGPL closes this loophole by explicitly saying that network access is considered as “distribution”.

So no, you don't need to make your web application AGPL. Just like you don't make all your applications running on Linux GPL just because they happen to communicate with the kernel over system calls. Copyleft licenses are relevant only if your code is a derivative work of the original.

AGPL is an open source license as per OSI definition which you yourself point to!

You don't have to take my word for it, just check out their own site:

https://opensource.org/licenses/AGPL-3.0

The licenses you are talking about, such as MIT or Apache, are called _permissive licenses_ whereas AGPL is a _copyleft license_. However, both types of licenses can be open source licenses, if they fill the OSI requirements.

Thanks for the pointer!

Please note that @glommer is talking about the classic secondary index implementation in Cassandra, which is very simple but also broken. I don't know the details but we probably did have "half-ready" code for that. We decided against going forward with it because Cassandra had already moved to SASI (which is also much more complex). As I said, we're currently focusing on materialized views, and tacking secondary indices after that.

Btw, I highly recommend subscribing to our user mailing list or engaging on Github for questions and comments about features. You'll get better and up-to-date answers there.

Update: reading @glommer's reply carefully, he explicitly says that he's unsure if we'll move forward with that specific implementation: "_if_ we do implement it, it should land in our main version in a couple of weeks" (emphasis mine).

For many users, that "super high performance" really just means lower request processing latency, which is a very desirable feature to have in various market segments. Look at the various benchmarks available (or run the benchmarks yourself) and you will see that Scylla has a very consistent, low latency that's a direct result of its non-blocking, shared-nothing architecture and implementation in a non-manage language, which gives us more control.

Where did you see that kind of estimate?

We are currently working on first finishing materialized view support, which will hopefully be completed in the upcoming months. Secondary indices will be implemented after that and we're hoping to reuse MV infrastructure for that. So I personally expect both features to land into a release later this year.

Yes, exactly. Scylla needs proper AIO/DIO support at the filesystem level and XFS has so far been the one that implements it best. There are fixes to Scylla to make it also run well on ext4 (by working around some of its limitations), but that's not part of any release yet.

JSON types are actually not very often requested feature so we have not prioritized it very high. I suspect that's because most people just store their JSON data as text and do processing in their applocation.

There's an open issue about it on Github:

https://github.com/scylladb/scylla/issues/2058

Please feel free to upvote and comment on the issue to voice your interest in the feature.

OSv is not a general purpose OS although it does support a subset of the Linux ABI for compatibility. Both application and OSv share the same address space and application code is linked directly to the kernel at start-up (there's an embedded ELF linker). The design is wildly different from MirageOS but I don't really see why you would not call it an unikernel.

If you already know C, you can start out by looking at the machine code generated by your compiler with "objdump -d" on Linux and "otool -tV" on Mac. Start experimenting by writing out C constructs like functions, loops, switch statements, etc., and just looking at what the generated code looks like.

Of course, to do that, you need to find the manual for your machine architecture. The x86 manuals are, for example, available here:

http://www.intel.com/content/www/us/en/processors/architectu...

You also then start to notice things like the operating system specific application binary interfaces (ABI):

http://www.x86-64.org/documentation/abi.pdf

and object file formats such as ELF that's used in Linux:

http://www.skyfree.org/linux/references/ELF_Format.pdf

or Mach-O used in Mac OS X:

https://developer.apple.com/library/mac/documentation/develo...

You can also do the same thing with the JVM and look at its JIT-generated machine code with the '-XX:+PrintCompilation' option:

http://stackoverflow.com/questions/13086690/understanding-th...

Shameless plug: I have also implemented something similar in Ruby using the Curses module:

https://github.com/penberg/btct

I have written a Bitcoin terminal using ncurses in Ruby:

https://github.com/penberg/btct

Yes, and to be more specific, OSv is written in C++11 which is a big improvement over earlier C++ standards.

The CLI is actually CRaSH which is written in Groovy:

http://www.crashub.org/

The JavaScript CLI is being phased out.

I only skimmed through the VirtualBox article but it seems to me it's simply explaining how QEMU works (VirtualBox is derived from QEMU).

The "switching dynamically between emulation and virtualization" is simply what I referred to as "hardware emulation" using dynamic binary translation. Of course, when the host and guest CPUs are the same, you end up running bulk of the original native code pretty much as-is. But because you're dynamically generating the code, you end up causing a trap to the hypervisor for the corner cases.

The VirtualBox article is not clear on what it does when hardware supports virtualization (Intel VT-x ("Vanderpool") and AMD-V (SVM) support) but I am guessing it also does what QEMU does, which is to use KVM (or whatever the equivalent is on other platforms than Linux).

What these extensions do is eliminate dynamic binary translation completely. You only trap to the hypervisor on PIO and MMIO accesses and few other minor cases. But all this is controlled by hardware and KVM kernel subsystem and the hypervisor itself doesn't touch the guest instruction stream.

[ Disclaimer: I'm not that familiar with KVM internals so the above explanation is probably a slight over-generalization. ]