maxpert

On similar lines, I've put together a library for myself that is framework agnostic:

https://github.com/maxpert/htmxido

Hopefully will be useful for people as well

Maybe a good DSL language. But I will avoid adding such complexity to my daily programming language at all costs.

For the folks who have been writing these inline HTML views and been wishing to have a more composable way of generating HTMX like in this blogpost try https://github.com/maxpert/htmxido

Would absolutely love to explore it deeper. Feel free to drop by community discord if you start experimenting with Marmot.

Depends on use-case. What you are building, here is one use-case for example where it made sense over postgres https://www.youtube.com/watch?v=HycGtLjlikI

I have got an issue discussing this in place and exploring couple of proposals. Turns out folks using SQLite use plethora of techniques to maintain their schemas, so enforcing a framework that works with the rest of their ecosystem is easier said than done.

Glad you asked the question. I never recommend use auto-incrementing IDs in production always generate one (e.g. Twitter Snowflake). With ID generators you get rid of collisions.

I completely agree with your analysis. Understanding the complexities of achieving convergence with basic auto-increment counters without advanced CRDT types is 101 IMO. Those familiar with these issues inherently comprehend the challenges involved. While it's plausible for someone to leverage a library atop Marmot to construct and synchronize such types, it's important to note that this tool isn't tailored for junior developers grappling with transactional intricacies. I've witnessed instances where inexperienced developers initiate transactions and make HTTP calls while holding locks, resulting in system outages. Marmot isn't intended for individuals lacking a solid understanding of distributed systems. My recommendation aligns with advising entry-level individuals to explore these tools only when they reach a scale where such complexities become pertinent.

Author of Marmot here. Marmot was born out of my own use-case (was building replicated SQLite based cache). While working on various problems I realized how well suited it might be for read heavy sites/workloads. If I take a typical CMS site 90% of the time it's just reading and SQLite is perfect for that, but then how I get independently deployed nodes to replicate data. The philosophy I am sticking to so far:

- Sidecar! I would avoid any kind of in process library at any cost. Call me biased but I don't trust someone's code in my process space causing it to crash.

- No master - each node should be able to make progress on its own, if these processes go down your own process will keep functioning. They will converge once everything is back.

- Easy to start, yet hard to master - You can get up and running pretty quickly, but make no mistake this tool is not for rookie who doesn't understand how incremental primary keys are bad, and how to they can keep things conflict free.

I am far from getting everything I need in there, and again my philosophy might evolve over time as well. Talking to people on Discord has helped me think through use-cases a lot, so keep the good feedback coming. Would love to answer any questions people might have here.

I've watched all the Redis goodness and potential slowly getting locked behind paid license. Specially after departure of the author of Redis. It's kinda sad to see such vibrant ecosystem now feeling stale.

Clearly killing open source and wrapping it in $h1t sandwich! I wonder if they were not making enough money, and somebody advised them to do this stupid move.

Finally virtual threads I can use in production with generational ZGC! I've been using Kotlin coroutines for a while now, and want to see how that evolves with JVM now supporting virtual threads out of box.

I've written a whole SQLite replication system that works on top of RAFT ( https://github.com/maxpert/marmot ). Best part is RAFT has a well understood and strong library ecosystem as well. I started of with libraries and when I noticed I am reimplementing distributed streams, I just took off the shelf implementation (https://docs.nats.io/nats-concepts/jetstream) and embedded it in system. I love the simplicity and reasoning that comes with RAFT. However I am playing with epaxos these days (https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf), because then I can truly decentralize the implementation for truly masterless implementation. Right now I've added sharding mechanism on various streams so that in high load cases masters can be distributed across nodes too.

Very interesting! I have question ( out of my experience in https://github.com/maxpert/marmot ) how do get around the boot time, specially when a change log of table is pretty large in Postgres? I've implemented snapshotting mechanism in Marmot as part of quickly getting up to speed. At some level I wonder if we can just feed this PG replication log into NATS cluster and Marmot can just replicate it across the board.

One of the reasons I started writing Marmot (https://maxpert.github.io/marmot/) was for replicating bunch of tables across regions that were read heavy. I even used it for cache replication (because who cares if it’s a cache miss, but a hit will save me time and money). It’s hard to make such blue prints in early days of product, and by the time you hit a true growth almost everyone builds a custom solution for multi-region IAM.

Exactly my question as well? Its like saying Tabasco the sweetest honey in the market!

I am excited for something other than v8 getting some attention. But at the same time I know over time it's gonna get bloated like Node.js

Amazing I was just looking for a good mail server to configure for my demo. Which reminds me since you folks have mentioned LiteStream, have you tried Marmot (https://github.com/maxpert/marmot); I recently configured Isso with Marmot to scale it out horizontally (https://maxpert.github.io/marmot/demo). I am super curious what kind of write workload on a sub thousand people organization will have and if Marmot can help scale it horizontally without Foundation DB. I always find the the convenience of SQLite amazing.

Great that you brought it up. I will fill in the perspective of what I am doing for solving this in Marmot (https://github.com/maxpert/marmot). Today Marmot already records changes via installing triggers to record changes of a table, hence all the offline changes (while Marmot is not running) are never lost. Today when Marmot comes up after a long off-time (depending upon max_log_size configuration), it realizes that and tries to catch up changes via restoring a snapshot and then applying rest of logs from NATS (JetStream) change logs. I am working on change that will be publishing those change logs to NATS before it restores snapshots, and once it reapplies those changes after restoring snapshot everyone will have your changes + your DB will be up to date. Now in this case one of the things that bothers people is the fact that if two nodes coming up with conflicting rows the last writer wins.

For that I am also exploring on SQLite-Y-CRDT (https://github.com/maxpert/sqlite-y-crdt) which can help me treat each row as document, and then try to merge them. I personally think CRDT gets harder to reason sometimes, and might not be explainable to an entry level developers. Usually when something is hard to reason and explain, I prefer sticking to simplicity. People IMO will be much more comfortable knowing they can't use auto incrementing IDs for particular tables (because two independent nodes can increment counter to same values) vs here is a magical way to merge that will mess up your data.

I am going to sound like broken record throwing in shameless plugs. But for people who might want to look for alternate options, Marmot (https://github.com/maxpert/marmot) is another option that can let you both replicate, and backup your database. I have recently embedded NATS into Marmot and based on community feedback introduced example script that gets you up and running with example DB within seconds.

Great addition of SQLite backend. I would however love prebuilt binaries for Mac, that way I can test it with Marmot (https://github.com/maxpert/marmot sorry for shameless plug). I wanted to build myself but then the instructions in README discouraged it, not sure why, it's not like I am going to train a LLM so it's pretty complex.

So while working on Marmot (https://github.com/maxpert/marmot) I considered something similar, and the thing that stood obvious with my benchmarks was readers can go really fast if I let them do things in process vs over some protocol (be HTTP or Postgres). I had the same problem with rqlite and dqlite, so I made a very conscious design choice to build Marmot into a side-car and not a layer on top. This idea is so tempting and goes against ethos of SQLite IMO.

Wait! Can I store unicode escaped binary strings in JSON now?

4800mAh battery for bigger screen, and extra processing? What was feature team smoking when putting specs together? 7A has more battery and it hardly is gonna do a day, and you are telling me I am gonna spend $1800 for a bar trick phone? No thanks!

I am not installing Edge just to use it. Microsoft must stop shoving down their ecosystem for one tool they want me to try out.

I will be testing it with Marmot (https://github.com/maxpert/marmot) to help scale Wordpress when needed! IMO Wordpress + SQLite is a no brainer for 95% of its sites. Hopefully do a demo video. Marmot will help folks scale their sites without central MySQLs.

Exactly no cache no nothing required. Your only bottle-neck can be your disk and code execution speed.

Perfect! This will be a perfect use-case for Marmot (https://github.com/maxpert/marmot) to help scale Wordpress when needed! People can just publish (low writes + high reads), and Marmot will take care of replicating it as sidecar. Beauty of sidecar will pay off here, 0 modifications to Wordpress codebase itself to horizontally scale it.

IMO Wordpress + SQLite is a no brainer for 90% of sites on internet. I was actually surprised when I discovered the most famous CMS doesn't support SQLite.

Just in time! I've been planning a SQLite extension written in Rust for Marmot (https://github.com/maxpert/marmot) for evaluation purposes. I didn't wanted to touch C/C++ because I feel more comfortable with rust tooling. Thank you for doing this!

Hey, I am author of Marmot (https://github.com/maxpert/marmot) would love to get in touch and explore a few things with you. I’ve been using Marmot in production in a while scaling couple of services, and since you mentioned LiteFS I would love to pick your brain, and explore couple of ideas with you. Multiple ways to get in touch including Github discussions or discord channel. Hope to see you around!