vaughan

EdgeDB and EdgeQL are doing something in this direction. SQL with GraphQL for joining solves some problems. Built on top of PSQL.

I think PSQL can only take you so far though, I think we will move to something new eventually.

For most apps you want to watch a query for changes. SQL is a pain for that. If you can get inside the query tree though, you can do some caching and materialization to help. And I want to run my db in my browser too.

Everything is just building blocks or layers on top of other technologies/paradigms making different trade-offs. You just have to figure out which features of which products compete with each other.

That said, it involves a ton of reading, and reading Twitter daily to stay up-to-date.

I've been thinking of putting together a curated website to break down the differences between JS tools, explain what the metrics and trends mean. In the end, everyone wants an exhaustively researched short-list so they're not missing some obvious trend, and then they want recommendations to suit their personal preferences. Existing comparison sites lack the necessary curation component to tease apart the layers involved in each tool.

Damn, I heard it was fixed, then another beta unfixed it - maybe they found some issue with the fix. Fingers crossed on this one I guess.

This is why Apple cannot add the thermal pads themselves, there are some regulations around the temperature limits. I never use it on my lap though. And as mentioned in the other comment "erythema ab igne" is a real thing is seems.

> ...otherwise please clarify what you mean by "logical db schema" vs. "physical db schema"?

Logical schema is an ER diagram, physical schema is how that is translated to a DB-specific DDL, not to be confused with the physical data storage layer.

Think about all the different ways you can model a list in SQL. E.g. jsonb, array, hstore, 1-M FKs, M-M join table, and all the different levels of normalization.

> I attribute it to too many CS/SE grads who didn't take database-theory electives

Completely agree! But it also makes you realize how the relational model is a poor fit for a lot of data models too - which again many are unaware of. It's not perfect, but its a nice sweet spot for now.

I think we are at the other end of another pendulum swing of databases though. But it's hard to innovate when the prerequisite for success of any new database remains as SQL compatibility.

I think this is a reason NoSQL became so widespread. If we stored every piece of data relationally, databases would have an insane number of tables, resulting in huge numbers of joins, and coupled with the difficulty in running migrations, would become unmaintainable.

Every data model is shoehorned into the relational model via normalization which involves a ton of judgement calls about how the data will be used at the time the schema is defined. And then this becomes extremely difficult to change.

It’s a big trade-off between schema flexibility and performance.

I’ve always felt we would be better off querying at a higher level (say the logical db schema) and then the physical db schema is managed for us behind the scenes. But then there is also a further level up at arguably the purest representation of data as triplets, which is why Datalog is called out so much.

Look up VRM thermal pad mod.

In my experience, high kernel_task is mostly due to VRM throttling. Once I resolved that I never had any issues with it.

The VRM thermal pad mod will fix all your issues.

There are two kinds of throttling going on. CPU and VRM.

CPU is normal - prevents CPU temps going above 100C.

The issue is VRMs which have no cooling. And no public sensors.

The fix is to add thermal pads to your VRM chips to dissipate the heat to back plate. Entirely solves all throttling issues. Cleaning fans should help too.

> the only solution

Putting thermal pads on your VRMs and dissipating heat to the backplate works perfectly.

> root cause

There is a bug for 5500M GPU that causes 18W sustained power draw with one external monitor. It should normally be 5W or so.

The throttling is due to your VRM chips overheating. There is no temp sensor for this exposed to the user.

The VRM chips have no cooling. The high GPU watt draw pushes the temps of the VRMs above their thermal limits with no cooling.

The solution is to put thermal pads on your VRMs to dissipate the heat to the backplate. Completely resolved my issue.

Disabling TurboBoost can also help. Running fans at max speed at all times too because Mac is slow to ramp them up when load increases.

Apparently the 5600M doesn’t have this issue and the 5500M bug is fixed in Monterey. But the thermal pad mod is still amazing. Apple couldn’t do it stock, because there are regulations about hot a backplate can get related to risk of burns.

Couldn't agree more. Debugger support in modern codebases has become a huge after-thought which is such a shame.

It is an amazing way to discover how a codebase works. You pick a point of interest, and then you get the entire path from the beginning of the app's execution to that point as your stack trace, and every variable along the way too. Watches are great too for tracking a value changing over time.

Micro-services and Docker also took debugging many steps backwards - one advantage of a monolith is that you can easily step-through the entire execution, whereas if you have to cross process-boundaries it becomes a lot more complex to properly debug.

I'm working on a monorepo template at the moment where everything is debuggable with a single-click. This includes debugging native addons in C++ and Rust for Node.js projects. It's not easy - which is why people avoid debuggers so much.

I recently setup debugging in a Rust project for IntelliJ was the alternative was adding `dbg()!` statements which involved 10s recomplilation. The difficulty was implementing the right pretty-printers in lldb so you could see the value of various types because support is quite immature at the moment.

It's driven by library developers wanting to prevent access to internals. I was annoyed by it initially because I love poking around in internals and it makes it easy to add missing functionality or fix bugs, but I can kind of understand from popular library maintainers.

If Bar depends on a private internal of Baz, and Foo depends on Bar, then if Baz publishes a patch version that changes an internal to fix a bug, Bar breaks, and so does Foo.

Yep, I know the pain all too well. ESM is also currently a disaster as many popular packages migrate without transpiling to CJS and you cannot call ESM from CJS contexts.

As you said, it's good to separate the ecosystem from the language/runtime. Things will improve over time, but everything has just been in transition for so long. I dream of the day that transpilers are no longer needed and TypeScript is baked in to all browsers. Until then it is a mess.

> node-gyp might then depend on python2

`cmake-js` is the preferred option now. And Rust native deps don't have this issue.

But not relational...

It is interesting that we use declarative SQL to query from the DB, but then when we get it into the app, we are manually doing further filter, mapping and joining of data. And then denormalizing it into a cache in a separate data store. Losing some benefits of the declarative nature of our queries. And this also applies to the frontend as well as the backend.

JS and Rust are best friends. Deno is written in Rust. And many Node native libs are being written in Rust.

I'm not sure OC is asking for NoSQL - just a better experience with SQL migrations.

The relational model always requires a judgement call about how normalized to go. If I have a deep nested json object of data, you could argue to define all the relations and normalize. But then to query the data you now have an 6-way join and the query planner starts to struggle a bit. And then if you decide to change that structure, you now have a very tricky migration script to run.

Whereas if you just kept it as a json object, you have avoided a lot of unnecessary pain, and you can just write some quick re-mapping code in the application layer to handle the older version of data, and you can actually start storing data in the new schema immediately. Then you can scan over the collection and modify the existing data of the old schema, and remove your re-mapping layer.

So yes, your data has been relational the whole time, but the query complexity and migration complexity increased dramatically. It became very difficult to change the model.

The relational model is essentially a constraint on how you need to structure your data in order to allow relational algebra to be used to help optimize query plans. When you release these constraints, you find things like Datalog that provide purer ways to represent your data and relationships, in the sense that they can map closer to the real-world - but then this comes at the cost of automated query optimization.

So the fact that relational data models are usually less representative of the real-world because of adherence to the relational model, usually denormalized to optimize the underlying database engine, and difficult to migrate, create complexity and slow down shipping some features (although they can also speed up many others greatly - like analytics).

That being said, SQL dbs are probably the best dbs we have today for solving business problems, and I think the migration and modeling problems could be solved with better tooling.

Wanting to make changes to your schema easier, doesn't necessarily mean you need NoSQL.

I think it's a question of having the right tooling.

It would be cool if you could dump an unstructured blob of data in a psql table. And then later you can add a schema to this defining relationships. So like `post.comments` is an array of `Comment`. And then you just run a command that runs a migration that normalizes the data into a `comment` table. And then it would map `post.comments` to a join. Although psql's jsonb support and indexing is pretty good.

The difficulty though is that if you change your mind it becomes much harder to change because now you have multiple tables and relationships. So what would be nice is if you can go back from the relational model to the unstructured model with ease.

I think what is needed is a visual tool to design your db schema and migrations that just works, and is also aware of unstructured data and that it can have a json schema.

Ideally we want to be designing a logical schema (and any unstructured data would implicitly be given a logical schema too), and the physical schema is automatically created.

I could see it as taking a typical day's worth of transactions, creating new test db from this log, and then letting a tool try out different indexes based on some common heuristics and re-run the queries and benchmark. Because often it requires experimentation to get the right indexes.

Becomes too complex. You have to maintain mapping of old columns to new columns and when you are splitting tables it becomes too crazy.

There are so many times I wish a blog post was an app that just had a nice gui to do guide me through the steps they suggest, instead of manually modifying config files and running bash scripts. SQL GUI tooling is lacking.

> Blurring (safely!) the line between database and the app using it

What do you mean by this?

> Comprehensive auto tuning...automatic schema tuning

You should be able to maintain a logical database schema, and then flip a toggle for things you want to have denormalized (and eventually have the db just do it automatically). Or maybe even take a blob of unstructured data and automatically normalize it.

absurd-sql[1] is the coolest thing I've seen in this space so far. sqlite.js brings sqlite to the browser via wasm, and then absurd-sql implements the storage layer in IndexedDb. So now we have an sqlite compatible db in the browser to play with.

There is also alasql[3] which is implemented in js, and lovefield[2] from Google which seems like an experiment that is now abandoned.

First, you could implement a REST/GraphQL cache in SQL. This would require maintaining mappings of your API response fields to SQL tables.

Going further, you could implement your backend API on the client, and have requests to it go directly to your browser SQL db. The benefit of this is you write your API once, and you get full offline support. If you don't need "local-first" then it's just a matter of figuring out when a query becomes invalid on the server. I could show instant results from local db, and then send a query to the server to ask if the data I already have in my cache can fulfill this request. Could optimize this on the server by listening to DML queries via WAL/LISTEN. WebSockets would be used to maintain real-time updates to only the data you are viewing.

You could also just use SQL directly as your UI API and then trigger these queries on the backend (respecting security of course).

What's doesn't feel optimal though is subscribing to sqlite.js updates in the browser. This makes me feel like we need an SQL db written in JS.

Also, if our DB is running in the same execution environment as our frontend and has one consumer, we could store each row as a JS object, and then reference this directly in our views, and subscribe to updates on a row-by-row basis. So if you are rendering a large table, if a column in a single row changes, when this row is updated in the database, instead of re-rendering the entire table manually, (or smartly detecting if the query was invalidated), you just bind to updates on the row instance, which is what the db is actually storing. I think this would reduce huge amounts of code.

Local-first is a little more difficult. There is some good exploration here[4].

I think writing a db in JS on the backend is not such a bad idea either. The hot paths would be written as Rust addons, but the majority of stuff people want today is just implementing the right logic to automatically handle denormalizations and subscriptions which we already hack together in our application layer using JS et al.

    [1]: https://github.com/jlongster/absurd-sql
    [2]: https://github.com/google/lovefield
    [3]: https://github.com/agershun/alasql
    [4]: https://actualbudget.com/blog/

Hasura discusses why they chose polling here (https://github.com/hasura/graphql-engine/blob/master/archite...).

> WAL: Reliable stream, but LR slots are expensive which makes horizontal scaling hard, and are often not available on managed database vendors. Heavy write loads can pollute the WAL and will need throttling at the application layer.

Would like to hear Supabase's response to this.

I feel like Rethink would have done a lot better if it were started closer to today. Funnily enough, in 2009 it was originally an SSD-optimized storage engine for MySQL.

The dev experience and API were great. It was the "relational mongodb".

Though no one feels daring enough to part from PSQL as a backend these days. With all the cloud deployment options and optimizations in PSQL, it's hard to justify writing your own db engine, but for a great real-time experience, that's what needs to happen.

> and for that applications you should build something custom.

Given the spirit of the original post, I think avoiding "build something custom" is the point.

It should be a toggle in the database.

> you don't have a 1:1 mapping between the database schema and the API (and if you do, you shouldn't, since a change of the internal database representation will require a change of the API and will break clients)

I'd argue this is a huge problem. When DB and API get out of sync it creates so much complexity, especially when working in a relational model, and things become impossible to change, or make client caching really difficult.

> The application when updates the data can publish it to whatever subscription system it wants to notify clients.

This is so much complexity though, compared with the experience of Firebase. Good for keeping more devs employed though.

> 4. Support streaming

I feel the same. Streaming is so lacking in most dbs today. RethinkDB's `.changes()` was really cool. I wonder if PSQL will eventually do it, or whether a new DB will take over. Everyone went to Mongo then ran back to PSQL, but maybe after lessons-learned there is room for a new db optimized for in-memory usage and with great first-class streaming support.