Mutagen – Cloud-based development using your local tools
mutagen.io
mutagen.io
It really is a fantastic piece of technology and completely transformed the whole experience (we were using good 'ol rsync before). In particular it works seamlessly across platforms.
If anyone's interested in how we use it, it's here: https://github.com/garden-io/garden/blob/master/core/src/plu...
I used Garden in the rsync days and it was already pretty good. The code behind Garden is some of the best TypeScript + Node I've ever seen, so if this was better and more performant than what you already had I'm sold.
Tell Jon I said hi =)
Basically, I’m working on a bare metal kernel in C/C++ on ARM, which only has tooling on Windows or Linux. As I work on a Mac system, I setup a Docker container that makes this trivial for me, including GDB debugging of QEMU on the host machine.
Mutagen was easy to get going, well documented, and has worked perfectly for this. I’m able to make my edits in my Mac IDE, then literally by the time I’ve switched to compilation, the edits are synced to the container, and vice-versa for the build product.
Prior to Mutagen, I was getting 8-10x longer build times since Docker file synchronization was slowing things down. Mutagen syncs slower than Docker, but faster than I can work, so all is well.
[0]: https://github.com/mutagen-io/mutagen-compose [1]: https://mutagen.io/documentation/orchestration/compose
Its synchronization algorithm in particular (which is essentially a (repeated) three-way merge with rsync-style differential file transfers) is designed for safety and robust tracking of the changes that it has propagated. It's also capable of resuming file transfers once it reconnects.
What are the plans for funding the project? It looks like everything is OSS now, will there be any closed source elements in the future?
Mutagen was actually part of YC's S19 batch, and it still has a significant amount of runway from that, but also some revenue from contracting work.
I really want to keep as much of Mutagen as FOSS as possible, ideally MIT licensed. I did just add a small portion of code under the SSPL, which I might experiment with dual licensing to SaaS embedders of Mutagen (since it's really only useful in cases where Mutagen is being embedded in other tooling), but even that I wanted to keep open source for other FOSS projects embedding Mutagen.
In the near term, I have some ideas about plugins and tooling that I want to build on top of Mutagen that will probably be closed source, but those will be separate entities from Mutagen itself and the aim will be to avoid compromising on any functionality that belongs in the core of Mutagen.
Development-oriented: Mutagen's sync configuration is primarily focused on development, so it adds more granular controls for things like uni-/bi-directionality, conflict resolution, ignores, symbolic link handling, etc. It also has a permission propagation model that's focused on things like cross-platform executability propagation and preservation (i.e. between Windows and POSIX), as well as operating in multi-service environments where many different process UIDs/GIDs might be in play. It also handles weird filesystem quirks (like macOS Unicode decomposition).
Low-latency: Mutagen's goal is to reduce the latency of sync cycles (i.e. time from local edit to change reflection on the remote) to an imperceptible level. It uses a lot of tricks to try to do this, but the goal is really to use the absolute best filesystem watching mechanism for each platform and to integrate that tightly with the sync loop. On Linux, for example, Mutagen is now starting to experiment with the recently revamped fanotify[0] API to get highly scalable but low-latency watching (as opposed to the emulated and janky recursive watching emulation with inotify that most tools use). It also uses tricks like rsync-diffing of the metadata snapshots that it transfers to get latency as low as possible. The eventual goal is to reach sub-100ms sync cycles for multi-GB codebases, and I think that's pretty close.
Git-like sync: Conceptually, Mutagen's sync algorithm is like a filesystem watcher + repetitive three-way Git merge (with the difference being that file transfers are deltified and the merging (potentially) affects both endpoints). This means it tracks content in a manner very similar to Git's CAS and branches, which is a little different than the way that Syncthing does it. This affords (in my opinion) more precise identification of conflicts.
Distrust: Mutagen takes a more aggressive approach to mutual distrust between endpoints, working hard to ensure that a malicious endpoint can't read outside the synchronization root on the other endpoint via symbolic links or maliciously crafted paths. It does this by using POSIX *at functions to traverse the filesystem and perform operations. This avoids issues like CVE-2017-1000420. You can harden this further by using unidirectional sync and other configuration options. This makes it well-suited to cases where multiple users might be syncing to file storage on a shared system (say on a SaaS platform) (though, at least in that case, you can protect yourself and users with the filesystem namespacing afforded by containers).
One-sided installation and flexible topology: Mutagen's primary M.O. is injecting small "agent" binaries to remote systems via a copy mechanism (such as `scp` or `docker cp`), so you don't have to manually install it on both endpoints. This is less important to "full stack" cases like Okteto, where your tooling can handle the setup of Syncthing on the remote, but it makes working directly over SSH or in ephemeral containers significantly more convenient. And Mutagen's architecture is also really flexible, allowing it to sync files and forward traffic between any combination of local and remote endpoints (including remote-to-remote, proxied via the local Mutagen daemon).
Command-based transports: Mutagen uses the standard I/O streams of commands like `ssh` and `docker exec` as its transport (similar to tools like Git or rsync), making it easier to target remote environments with your existing tooling and configuration. Again, this is less of an issue for a case like Okteto's, but is useful in the standalone case.
Network forwarding: This is outside the scope of sync, but Mutagen offers OpenSSH-style TCP/UDS forwarding (with the difference from OpenSSH being that Mutagen's forwarding is persistent and managed by a background daemon). This offers support for doing things like forwarding a local socket to a remote Docker daemon over SSH, and then forwarding web application traffic over that underlying forwarding by using Mutagen over `docker exec` (or reverse forwarding, or forwarding between two remotes and bridging them via your laptop, or loads of other crazy shenanigans).
I hope that clarifies things a bit. Ping me via email if you want an expanded comparison.
I've been looking at something like Lando[0] as a result, and I realize they're not the same thing, it seems like it is more fitting with the role for the development environment, at least.
Lando is great for getting up and running quickly,outside of the recipes, it's easier to just create your own docker-compose file.
Is there any thought - given this is a development tool - to exposing container aliases as commands? Kinda like with lando, where you can `lando composer install` or `lando yarn turbo` or something like that. One of the other issues I have with these container setups in general is that for anything that has a heavy CLI component or interactivity there's no good abstraction around interacting with the CLI.
I think framework-specific stuff like that is better handled by higher-level tooling, typically something that's just embedding Mutagen. DDEV, for example, adds that type of functionality for PHP-based projects, while embedding Mutagen underneath.
The goal with Mutagen itself is (ideally) to be invisible - only providing synchronization and forwarding. Even Mutagen Compose (being just tweaked version of Docker Compose) probably falls too low in the stack to know anything about the specific frameworks being used atop it.
That being said, I think this is something that Compose wants to address, and that Mutagen Compose could inherit once it's added to the Compose Spec and implemented into Docker Compose. Things like profile specifications in Compose files are designed to encapsulate these types of operations (among other things).
The project is way more snappy than using NFS mounts on the Mac (which was already WAY faster than bind mounts), and DDEV has file ignores set up correctly so the project's media files aren't synced and disk usage isn't doubled.
We work quite hard to maintain unified production/dev config for our docker-compose setups. Mutagen looks like it could simplify some of these since a whole layer of complexity comes from trying to optionally map ports and volumes in such a way that it works for development, while not being inappropriate in test/production. If instead, mutagen can virtually connect / synchronize these it'll be a lot simpler.
I do wonder a bit how debugging works .... if something is running in a container, it's great I can hot sync my code there, but what happens when I put a breakpoint in the code in my IDE?
In general Docker for Mac is far from ideal - particularly considering you must allocate a set block of memory to the VM. If only it was the year of the Linux Desktop :grin:
[0] https://gist.github.com/vschoener/b0b5ae08625744a1f4ad414027...
https://github.com/stephenh/mirror
But dressed up to work with docker + "just" :-) being a professional/polished product, which is impressive. Neat!
Having the webserver in the cloud and doing the heavy lifting is quite nice, especially if you work with phpstorm, which can be a resource monster for bigger projects, and working on remote projects with phpstorm built in sftp / file sync is a huge pain in the ass, really slow.
With mutagen it feels almost like I work locally. Thanks to the developer for this cool tool!
(not associated)
For internal code or slow to build code I use a remote VM hooked up to the internal network. That allows me to build quickly and test in an integration environment.
I use a file synchronization tool like mutagen at work (for ninjas) and I use mutagen on my personal laptop for the same workflow.
I’m thinking of trying remote development from jetbrains soon though: https://www.jetbrains.com/remote-development/
I generally prefer using a remote vm for development and only using my laptop for git.
Also you can use spot VMs for development so it can be cheap to get a high CPU vm for building and debugging or a high mem VM for running data science scripts and debugging them.
For web dev: https://www.jetbrains.com/help/webstorm/debugging-javascript...
I use jetbrains IDEs cause you get this stuff out the box without much fiddling with configs and it’s all integrated together.
For lower ram when working on multiple projects they did build a new ide for lightweight remote dev but I havent tried it: https://www.jetbrains.com/fleet/
Could be worth a shot.
The idea behind Mutagen is to use your local tools (e.g. editor, IDE, browser, etc.) to work on remote hardware (where "remote" can mean anything you like, e.g. your local system, a Docker container (local or remote), a Raspberry Pi, a cloud server, etc. - or some combination of all of those).
The architecture is designed to be maximally flexible, so you can work between your local system and a remote system, between two remote systems, or in any other topology you want.
What specifically is missing in the current ecosystem of electron state moving technologies that this solves? Why this instead of git and a Wireguard based mesh of devices (to offer my current setup as an example; there are others).
Basically, it's like VS Code's remote development extensions, but editor-agnostic and more flexible.