ShishKabab

How to talk about important tech questions with your team in ways everybody understands. Includes things like technical debt, choosing technologies and developer experience.

If that's your concern, part of the plan for later would be to make it easier to generate a REST/GraphQL API that you can easily consume on the client side, while not having any schema data in the front-end. Say you'd have a TodoStorage which has high level methods like markAsDone(), you could have that either write directly to IndexedDB in your dev workflow, or move that class to the backend, and replace the front-end version of TodoStorage talk to a GraphQL API instead. The point is to have a choice, and allow the user of the library to understand the risks of technologies involved and use and mix them according to their needs with minimal efforts.

You're absolutely right, and I'm coming from the same place. All the thought that has gone into this is a bit hard to condense into one article, but I've written this stuff with the same things in mind. (Also have been a Django, and South user for years.)

The first idea is to have the common denominator implemented, and then being able to access or translate things to lower-level operations. Because even though there may be very complex queries happening, a lot of queries are also quite simple, especially when you're starting out. The goal therefore is to minimize, not completely eliminate database-specific code. And even though best practices for data modeling may differ between databases, in the end you want to express and work on relationships between data, whether in SQL, MongoDB, Firebase or Neo4j. If you express your intentions correctly, your database should provide you with options to translate it to common best practices. (Like the decision to embed objects in MongoDB, or do manual cross-relationship queries for example.)

About schema management, this is the first tool I've seen that actually does begin to support you with the more complex issues of migrations (although it's definitely not finished.) Most framework come with a package deal of how you should run your migrations, meaning from their codebase. Storex generates an intermediate structure of what it's going to do during a migration, which you can then decide what to do with (generate an SQL script, run it directly from code, etc.) This structure is divided into 3 phases for now: preparation, data migration and finalizing (which is something I've never seen before.) Running the prepare phase adds new columns and tables, so the application servers can still access the database. The data migration stage defines what needs to be done, both forward and backwards, which you can also execute in multiple ways (e.g. set up a process that migrates both old and new requests to the new data format.) Then after upgrading your application servers to new code, you can execute the finalize stage to drop old columns, etc. (All of this should happen with monitoring tools and good operation procedures in place, so I'd recommend doing nothing totally automatically.)

Feel free to contact me or open an issue here [1] if you can think of a better way to structure migrations.

Of course you need to do a lot of work yourself still, but at least Storex doesn't fight you, but instead gives you the flexibility to implement your operations exactly the way you want.

Also for the client-server interactions, additional packages will be written to help with this. You're more than welcome to contribute your experience in issues or by contacting me directly to design this in way that accounts for the real-world challenges involved.

What you're saying in the end is also what I'm saying: creating a real-world application is more than just one mega abstraction, but a collection of problems that we need tools for that we can flexibly recombine to our highly individual needs.

[1] https://github.com/WorldBrain/storex-schema-migrations

Thank you for your feedback, I got a bit confused there :) Indeed I think it's nice to be able to mix and match pieces of existing stuff, which I think more software should do. The multi-device sync I'm designing right now will also not be coupled tightly to Storex, so you can use it easily inside existing applications.

See comments below.

Sequelize is only for SQL databases. This is also for the front end and in the future also for Firebase and others like it. And there's more in the article about adjacent functionality planned, which is outside of the scope of an ORM. And yes, got my terminology a bit wrong: meant to say it's not ActiveRecord.

Ah, thanks for pointing that out. Will read up on my terminology ;)

Very nice, curious to see how we could learn from each other :)

You're free to implement an ORM on top if you feel that style of access suits you more. My mail is in the article, will endorse your package and help you implement it in a clean way if you want :)

There is one DAL which doesn't take relations into account, nor has a plug-in based approach, but I don't remember it's name. Also didn't have the goal of being a collection of packages that solve problems 'around' your data, like migrations and throwing it back and forth between client and server.

Hey, Vincent (author of article) here! Indeed it's a fine line, but ORMs give you objects with methods that allow you to modify or further query the database, like User.save() or User.objects.find(). This is for me the distinction, and in Storex you get data objects back, but all the manipulation happens through one object, the StorageManager.

[MEMEX TEAM]

TL;DR; AMA about Backup/Sync functionality, ways you use Memex and we can make your life easier, the Storex layer described in the second paragraph, design methodologies and decentralization/Blockchain!

Hey there! I'm Vincent, a weird mix of software engineer slowly going service- and UX-designer. Since joining WorldBrain in February, I've been architecting and implementing parts of Memex's Cloud functionality, including Memex.Links and the Backup and Sync functionality that is currently in the works. On the technical side, I try to make sure that we can move fast and make the best trade-offs between ideal and doable solutions, so that we can create stuff that is as useful and easy to use as possible. But being in the field for 13 years building software, my main interest is shifting from programming, to the whole processes from reseach to designing functionality and communication strategies to make things that are as useful as possible, which I can then help technically architect. And as such, in my spare time I'm currently thinking about how to rethink web development to lower the barrier between designers and developer, reducing the write-test-modify cycle while taking a wholly new approach to QA that is driven by designers instead of programmers churning out unit tests.

Lately the bulk of my work has been battling AWS and Google docs, trying to get the Backup functionality to work as nicely as possible. But, in the process I've taken Jon's work creating the fastest in-browser full-text search engine currently available, and made it into a low-level library acting as an abstraction layer on top of different kind of data sources (IndexedDB, SQL, NoSQL, soon to come REST/GraphQL.) It still needs some work to be in a presentable state though (docs, more tests, etc.) The goal is, we aim to create many loosely coopled, tightly cohesive layers on top of it that solve common storage problems, like schema migrations, access rights, content feeds, live collaboration and moving data between front- and back-end. Due to it's origins in Memex, we called it Storex. Feel free to brainstorm with us on how to make this useful to lots of people and especially, what tedious problems you keep solving time after time hovering just above the storage layer.