vvern

What I’m hungry for is not so much this coherent application for exploring json data but rather a set of React components to integrate explorations like this into a web app.

Does anybody know of any good components or libraries for that?

Transactional workloads over datasets in the single digit petabytes.

Cockroach is also doing 3x replication of the data, so I don’t think that’s particularly relevant here. Cockroach serverless will dynamically scale up sql serving processes based on load. The storage and compute are separated in the cockroach architecture. My point is that if your query load is relatively low, cockroach serverless is definitely cheaper because the storage costs dominate. I think there’s ambiguity on which product is cheaper for a real-world application with meaningful load and data size.

I remain curious about the perception that cockroach is a meaningfully more expensive product. Where does that idea come from?

Last time I checked, the cockroach serverless pricing model and free tier were cheaper than planet scale for small projects. IIRC, the dedicated cloud product was also cheaper if you kept it utilized. What’s your evidence that planetscale is cheaper?

For example, planetscale charges 3x as much per gb of storage if I read the pricing correctly.

Maybe one day the buck2 ecosystem will evolve to be that bazel replacement. It has a much smaller core. Right now it’s lacking in ecosystem support, tooling, examples, and a local build sandboxing (which could be fine if there was an easy to use local implementation of remote build that felt natural). Also no go support is sort of painful, and it’s a bunch of work as i understand it to get something like rules_go to work for buck2.

Can you provide some links to some projects? You’ve piqued my interest

CockroachDB has a similar feature built in. It takes existing queries which are slow and then analyzes hypothetical indexes for better query plans and suggests them. You can add them in the console ui with one click. It’s pretty useful.

Online schema changes (and upgrades) are absolutely critical for high-availability applications. In cockroachdb all schema changes are online and work more or less like this. One thing you give up with more or less all of these things is the ability to perform the migrations in a transaction. Postgres is quite advanced relative to other databases in terms of its ability to perform DDLs in transactions. Cockroach still isn't there either -- though it's making real progress, at least in terms of transactions that exclusively consist of DDLs (i.e. don't mix DDLs with DMLs).

I think crossing the chasm resonates more for a later stage, rather than just for getting started. I read it and felt like it really applied a lot more to my experience at a multi-hundred person company trying to become sustainable and public.

Table driven tests are a lot better than a bunch of imperative tests but they rapidly become unwieldy to debug, maintain, and evolve. Their readability often isn’t great.

If you’re using go, check out https://github.com/cockroachdb/datadriven. It takes a little bit of effort to craft a testing dsl, but it is so worth it.

Also, snapshot style testing where the test writes out its expectations and you just inspect it and save it (part of datadriven) is wonderful.

I’ve been using insta in rust lately and it’s some of what I want but not quite datadriven.

As hot of a take is that macros and meta programming are going to make startup and code bloat better

Safety third

Any good resources/deeper dives into the details of what actually happens on modern computers when making these syscalls on, say, linux? I might have a reasonable idea about what goes on when performing a mode switch or context switch, but I’d love to have a reference or a nice walkthrough.

> yes I know it runs Linux binaries unmodified

Is it that it runs Linux binaries unmodified or that it runs vms and manages VMs which run Linux, and as an end-user, that's what you run your software in?

“Heat in the wilderness” must be so small in terms of global emissions footprint to be almost irrelevant, no? The wilderness implies ultra low density sort of by definition.

Go has a mechanism to spawn a new thread (m in ho runtime parlance) if it thinks one of its threads might be blocked in a cgo (go’s “native function” equivalent). That prevents stuff like this.

Is there a definition of “code forge” somewhere?

A place that volatile shows up in C today that is stunningly handy is when writing eBPF programs.

In eBPF you have to appease a verifier which is trying to prove safety and liveness properties of the compiled program. To prove safety properties you often need to ensure that some offset into a buffer (bpf map) will be in bounds. Even if you judiciously sprinkle such bound checks into your code, the compiler may eliminate them entirely or perform them on some different register or stack value it knows to be semantically sufficient. Unfortunately the verifier is not as smart as the compiler.

Using volatile to reload some offset just before bounds checking it and using it to index the map is a very reliable approach to getting code to verify.

I’ve found the datadriven[1] testing approach in go to be quite effective. The idea is that you leverage a standardized file structure to construct a little DSL for testing your code. This allows you to write expressive tests that print the state of the code and then look at it. Rewrite is also very powerful.

This is all inspired by the sqllite logic test framework.

[1]: https://github.com/cockroachdb/datadriven

I’d love to play with and explore this code. Please put a license on it.

Code without a license should be assumed toxic and if you’re paranoid (like me) shouldn’t be read.

This sort of thing feels like a good Apache or MIT fit.

One approach could be to compile the core or file loader to WebAssembly, load the file using the WebAssembly which will ensure there’s nothing nefarious in terms of reaching outside the desired buffer, and then marshal the whole object across.

This what I understand Mozilla does in Firefox: https://hacks.mozilla.org/2020/02/securing-firefox-with-weba...

I don't think it's fair to say it's a NoSQL database. It's very much a transactional, consistent SQL database.

CockroachDB uses a skip list for its “timestamp cache” data structure, used to implement serializable isolation. The data structure records reads and exists to ensure that writes cannot rewrite history.

This[1] library is the basis of the implementation in the database.

[1] https://github.com/andy-kimball/arenaskl

Can we just properly price carbon emissions and be done with it?

Can you say more about workloads which suck for either of the two databases at their default settings?

dragonflycache sounds reasonable.

> utilize transactions correctly

This seems hard to do in Vitess or PlanetScale given the technically READ UNCOMMITTED isolation [1] when cross shard, and, in scaled deployments, will still require experimental 2PC [2] cross shard transaction. So, like, yeah, if you had serializable isolation, then transactions might save you so long as your code isn't buggy, but, literally the reason the system doesn't implement them is because it doesn't have isolated transactions.

[1]: https://news.ycombinator.com/item?id=22170416#22177783 [2]: https://vitess.io/docs/13.0/reference/features/two-phase-com...

> They have to be on the same logical shard

This is a pretty major limitation. Part of the reason to use Vitess is to scale out. It is often very valuable to have a small number of root elements in a star schema and to have foreign keys which ladder up to them.

I’d love to see benchmarks.

To clarify, FoundationDB has some documented multi-region features, but it's not at all clear that anybody who runs it at scale relies on those multi-region features [1]. Even if they do, it's not obvious they run this Fearless DR mode at any appreciable latency.

Geo-replication is an area where Anna really shines, and foundation pays severe costs. All of the concurrency control problems have contention footprints on the order of round-trips. Fully optimistic concurrency control needs to expose backoffs quite a few round-trips to be live. Even pessimistic concurrency control requires some number of round trips, probably at least 1.5 optimally, but in practice in most tuned systems, probably 3. A heck of a lot of use cases make sense at 0 global RTTs and don't at 1+. The ability to tell the database how to manage concurrency, and then providing causal is ultimately the best you can do. That's awesome.

At the end of the day, I have to believe that there's a whole big mess of applications we'll build this, on systems in the portion of the design space Anna is choosing, one day. This is only recently not cutting edge research, but it's definitely still research. We don't know how to model these concepts at a high-level and in a composable way where the masses of software developers can engage with them.

It's interesting to think about how long ago we were graced with BAYOU [2], that thing was ahead of its time. I suspect it's going to take a little while longer before these sorts of techniques make their way into data storage primitives we think of as part of the standard vernacular, but I believe we'll get there eventually.

Databases are fun!

[1] https://github.com/apple/foundationdb/wiki/Multi-Region-Repl... A parallel line of reach in this area of the design space out of the EU was https://www.antidotedb.eu/. [2] https://people.eecs.berkeley.edu/~brewer/cs262b/update-confl...