erichanson

I cut out the five minutes of me trying to figure out Twitch so here's a better link:

https://www.twitch.tv/videos/496396478

Tooling on databases has been traditionally awful, and I think that has a lot to do with folks' negative experiences with the database taking up more architectural space.

Schema migration is hard. Skeema looks cool. Aquameta's idea with schema migration is to treat the schema as data, so that each table is a meta.table table, each view is a row in the meta.view table, each column... etc. So when version control checks out a new version of the project, if columns say were added, they're just inserted at runtime. Checkout prioritizes meta entities so they happen first and in a sensible order [1] so the migration happens first. This does technically work for all scenarios (I think!), but in the case of say a column rename, it would naively delete all the values and then have to re-set them per row, which could be a big inefficiency. It would still work just fine for smaller tables though. Alternately some kind of optional migration script per commit was another idea.

I don't disagree particularly with any of this, and I'd say an immature part of Aquameta is where to put the business logic. I think though that Aquameta's approach is somewhat agnostic to this whole problem. As a tool right now, it's good at CRUD and a free REST API (that reflects the schema) and rapid UI development, but the best pattern for what you would typically put in an ORM isn't figured out. But, the approach really doesn't constrain adding a middleware layer if one is so inclined, or alternately putting all that logic in the database. I think there's a place for that for sure. As long as that place is somewhere in the database. :)

Hadn't seen STOMP before, cool. I think making Tor a core part of the architecture is a little over-engineered in our case. The Internet is already a p2p network damnit! I like WebRTC because they seem to solve just the NAT-piercing problem specifically without a whole additional network layer.

Dang some cool ideas in here.

Yeah trying to figure out how to annotate the schema was a big motivator for inventing the meta identifier system. Thought about using COMMENT for documentation, but once you have meta-ids, I thought it was cleaner to just put schema annotations in a different table.

Yeah I think the LISTEN/NOTIFY system in PostgreSQL is a bit primitive. Still working on that part of the project. We got NOTIFYs to propagate up through nginx over a WebSocket and get them into web-world that way, but that section of the project is still fairly immature.

I'm going to Twitch stream the Aquameta install and give a little demo, and answer questions folks might have. 5pm CDT. https://www.twitch.tv/events/j5MGrQ91TwSzRpo28wqGdw

Ha :)

No. :) It is still in early stages. I use it for my projects but I end up dropping to the PostgreSQL prompt occasionally and doing things nobody but me would know how to do. Every time I do though I try to fix the source of the problem. Really need to start putting up some example projects though. I'd say it's ready for very early adopters.

Not surprising. Databases have been basically awful at being approachable from the command-line. Let's say I have some HTML in some field in the db and want to make a change to it from the terminal. How do I do this?

PostgreSQL says use an UPDATE statement, but that looks like "update template set content='<html><body>......</html>' where id = x", which means you have to paste the entire content of the page into the update statement. There's no CL-based field editor, just one of many reasons why the db feels like a black box. Awful.

Aquameta has a filesystem integration layer that lets you browse the database from the command line, grep database content, edit the content of a field using your preferred text editor. http://blog.aquameta.com/intro-chpater2-filesystem/

Generally just trying to ease these pain points has been a big goal of the project... still a lot to do.

The long-term idea with Aquameta is that users run their own local database and make connections directly to their friends/peers to basically "git pull" new content as it appears via pub/sub. Then users would do the "browsing" on localhost. I still have some work to do figuring out all that NAT piercing p2p stuff, but some combination of WebRTC and headless chrome is showing some promise.

Agree! Can't happen soon enough.

My project Aquameta is one such attempt, trying to rethink the programming stack, and by extension all user data, as relational data instead of files, with the filesystem still there under the hood, but basically as a bootstrapping mechanism to get the database running.

Here's an essay about the reasoning and approach:

http://blog.aquameta.com/intro-chpater2-filesystem/

The TLDR is that the filesystem lacks a sophisticated information model, and doesn't provide mechanisms for defining structures, and we really need to embrace this basic requirement at a systemic level.

Boot to PostgreSQL! :)

I agree the system you describe would be a massive step backwards. :)

No, I'm saying the OS needs an information model as a first-class citizen. But since you brought up data corruption: This hypothetical OS could also benefit from having transaction support up in application space -- to avoid data corruption -- something most "modern" programs don't even have, even though most file systems do.

To be fair, I do think the database needs a way to edit the contents of a field using your favorite text editor. But we've got a pgfs plugin that uses FUSE to make the database accessible as a filesystem as well.

Nice to see folks rethinking files, as they're a scourge on the planet and an antiquated anti-pattern that has been holding back the industry pretty much since its inception. I don't know how anyone could take a look at /etc for example, and consider it anything but archaic. The adduser command is some 1130 lines long, and all it does is do CRUD on files, to name just one example. Then there are countless config files that just have to be edited by hand and happily accept syntax errors and logical errors. No modern system would tolerate this.

The root of the problem with files is that they lack an information model, beyond just a sequence of bytes. They are unopinionated to a fault. All files have structure. Even if that structure is a "non-structure" like "all these files are just a random sequence of meaningless bytes", then that is their structure. But this information isn't present in the system, nor can it be enforced or constrained when that is desirable.

To me, the obvious alternative is the database, aka "everything is a row". We have used the database (relational or otherwise, but mostly relation) to successfully model many many domains, and bring coherence and clarity to them. The cool thing about the relational database is that it's based on an underlying relational algebra. The syntax of data in an RDBMS is really just one manifestation of a deeper layer of structure that is syntax-free, and these abstract structures can be (and are) manifested in multiple coexisting syntaxes.

I'm exploring this pattern ("datafication", headshake) with Aquameta (http://aquameta.org/) and written a lot more about why file-centric is holding us back (http://blog.aquameta.com/intro-chpater2-filesystem/). Boot to PostgreSQL! :)

Fixed, thanks.

For Aquameta's purposes, we're trying to make it easier for people to build visual interfaces that do programming. Higher up in the stack is a PostgreSQL admin GUI. Without meta, the backend of this interface would be the DDL grammar, so when the user took GUI actions, they would be converted to CREATE statements etc under the hood using string concatenation and other dreadfulness. With meta, we can instead just build against meta's data model, so that building this GUI just as simple as any traditional web data manip interface. It seems more elegant and simple.

It is a lot easier to shoot yourself in the foot. You can do very destructive operations through meta. It's more powerful for doing useful things too. Just be careful.