diggs

I did, and the fixation on React’s age struck me. Was it the core premise? No. Was it a central theme? Yes.

I don’t care that React isn’t new or innovative anymore, that’s a good thing in my book. Give me old and boring any day.

An infinite timeout works in theory, but it is impractical at scale.

Every time this topic comes up, it seems people invariably divide into one of two groups. In the first group are those who trust in the odds and play the probability game. In the other, those who demand absolute guarantees. Neither group seems fully capable of understanding the other's standpoint.

Let me add my own perspective: Distributed locks are a fallacy. They can be beneficial in decreasing contention, under the assumption that "most of the time, only one actor will be active". However, by themselves, they offer no solid guarantees. The blog post addresses this point by introducing the concept of fencing tokens. These tokens have the potential to provide concrete guarantees, but they require the cooperation of downstream systems for enforcement, which isn't always possible.

I was really surprised to see Antirez argue for the probability approach.

A use case that I wish authorization service providers would talk more about is support for "list" queries e.g. What resources of type foo can the user read?

In really simple cases you may model this as a one-shot check on a logical collection resource e.g. If I have an organization, and an organization owns many repositories, I may check for action:read on resource:/organizations/:id/repositories. It's very limiting though. What if I want to list all repositories across all organizations I have access to? What if I have multiple levels I want to cut across? Do I need to do all the sub-queries and aggregations myself? Do I need to do a one-shot check on every potential resource? What if the answer is "no" for every one, and I end up doing a table scan of the entire DB just to produce an empty result set? etc.

This is a misleading and dangerous service.

You provide a distributed lease, not a lock. A distributed lease by itself doesn’t provide mutual exclusion. Distributed leases are typically accompanied with a fencing token (which your service cannot provide out of band) or an optimistic lock on the underlying "exclusive" resources (which could be implemented by the consumer of your service). I think of distributed leases as an optimization e.g. they provide soft exclusion in the happy path, which may reduce thrashing on the underlying real mutual exclusion mechanism (like an optimistic lock) under general use.

Your docs and terminology guide users into building incorrect systems e.g “ensure only a single instance of a process runs at any given time” is simply not true.

edit: I had a quick look at your Python library, and while Python isn't my forte, it looks like it can be trivially induced to break the mutual exclusion mirage: the "requests" lib's default request timeout is None e.g infinite, so what happens when `client.try_heartbeat` goes out to lunch indefinitely? Looks like `locks._run_heartbeat_loop` will hang, the lease will never be renewed, and within 60 seconds a competing lease holder will claim it. Boom. So you fix the timeout issue and problem solved right? Still no dice... what happens when you hit a pathological hang in your non-real-time runtime or OS? Boom.

There's so much more to CI/CD than the build definitions (e.g. dashboarding, log viewing/retention, access control, manual interventions, secret management, test reporting, etc.) and while some of your points resonate very strongly with me (e.g. local builds), I can't help but wonder what the endgame is here?

You've raised $27m to what? Give away a free tool for engineers to define their builds in? While they continue to use and pay for another service to actually run the builds? I assume you intend to replace the CI service at some point and move up the stack to monetize?

Without more transparency its easy to imagine something like...

Step 1. Drive uptake of your tool by selling people on the pitfalls of "CI lock-in" Step 2. Introduce your own CI solution, which people can now easily switch to Step 3. Lock people in

I had a little chuckle at this. One day you will too :)

I am yet to work on a team with a simple, reliable and repeatable build process.

Most build processes (from basic scripts or Makefiles all the way up to CI/CD product-specific pipeline configurations) are complex, brittle, opaque and poorly understood by most of the team. I've seen time and time again one or two people emerge as the "build" people because most team members want to run as far away as they can, and it's easier to let someone else gatekeep the mysterious black box that sometimes does something useful. These processes typically grow organically on an as-needed basis and are rarely designed under a unifying vision.

I think it's a combination of lack of experience on the engineers part, bad off the shelf tooling, lack of recognition of the cost and as a result under investment by the business.

Agreed - I’ve spent months working on a deep tech problem before I could even show a functional core, let alone putting it into production. Some problems take time, experience and foresight to solve and the solution only dovetails at the end.

Recognition and trust have never been a problem for me. Have you hired at this level before, or merely assuming? As always there is nuance, as other commenters have touched on.

“Someone that avoids talking about what they did, and focuses too much on "we"”

This is very interesting. I naturally use “we” when discussing past products and achievements. I do this in recognition of the fact that things are very rarely designed and built in true isolation. Even principal engineers socialize their designs and thinking with colleagues, making small tweaks here and there or gaining additional confidence to move forward.

“We” is absolutely not a red flag for me.

My first experience with event sourcing was having a new hire at a previous company try to shill it. It was a smaller company and he had previously built a career consulting on ES. Every problem we had he would come up with an argument for why ES would solve it. I soon learned he had zero technical ability, but was great at sounding credible to management. I liked to imagine him as a "seagull ES" practitioner. He flies in, sh*ts ES on everything, and flies out without having to deal with the mess.

Ironically I went on to use a lot of ES prinicpals in building a distributed deduplication engine for my previous startup after I left that company. We used a WAL which we were able to rebuild databases/projections from, ship to replica nodes for read replicas, push to S3 for resilient storage etc. it was an extremely powerful architecture for us. It was also complex and required a special skill set to work on and reason about.

I have to imagine anyone that advocates for "moving from CRUD to Event Sourcing" has recently been sucked into drinking the coolaid, and has no idea about the world of hurt they've signed up for, completely necessarily.

Object tags instead of named resources look like a neat and flexible concept, and is something I hadn't considered before. It is quite jarring transitioning from the filesystem example to the introduction of tags though, as something quite important appears to be lost in the translation.

In the filesystem example I can visit any node in thee tree and set an ACL on it, optionally recursively. This gives me a lot of flexibility to slice and dice permissions in a complex potentially-multi-tenant resource model. Tags appear to lead to a flattened view of resources - all resources with tag foo are treated the same, regardless of their location in the model. It would be nice to see the author acknowledge this and expand on their examples to see how they might incorporate relational data models.

The beauty isn't in the UX of the API per se, it's in the reduction of historically complex file system APIs down to a simple set of primitives and behaviours that enable massive throughput and scale while still being broadly applicable to most modern application needs.

There's also no deduplication with client-side encryption.

Deleting a bucket in an object store is surprisingly non-trivial. See Amazon’s UI - it works fine for small object counts but is very buggy for larger buckets.

I recommend using a static site generator and hosting on S3. It's the cheapest way to do it and will survive a hacker news storm. It's also easily managed in version control and zero maintenance. For extra points you could add a CDN in front of it but it's probably not worth it.

I'm a big golang user so I recommend Hugo for the site generator.

Sorry, I should have perhaps put a disclaimer in my original comment. I work for a company called StorReduce and built our replication feature* (an intelligent, continuous "sync" effectively). We currently have a patent pending for our method, so I'm not sure if I can offer any real insight unfortunately.

I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints.

* http://storreduce.com/blog/replication/

That depends on how you're storing the files. I was really just trying to highlight that for deduplication across files you need to deduplicate before you encrypt.

This approach works well enough for relatively small amounts of objects. Once you start getting in to the millions (and significantly higher) then it begins to break down. Every "sync" operation has to start from scratch, comparing source and target (possibly through an index) on a file by file basis. There are definitely faster ways of doing it that scale to much larger object counts, but then they have their own drawbacks.

It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.

How about if a single byte changes? Now the encrypted output looks 100% unique, and you have to treat it as an entirely new file. You lose the ability to do diffs on a file by file basis or proper deduplication across all files.

When flying, so quite a bit for me, actually :)

Our server software is typically deployed as close to the source data as possible. This lets us move only deduplicated data over the WAN. It also supports our encryption model where we encrypt using a pluggable key management service e.g. Amazon KMS or a on-premise HSM before any data leaves the customer site. This is essentially the same model as traditional tape and disk backup software within the datacentre.

Additionally we use HTTPS between clients and our server, and our server and the storage provider (e.g. S3), as well as being able to enable server-side encryption for S3.

If you're open to using something proprietary and commercially supported then depending on your scale you should check out http://storreduce.com. We're a growing startup in this space.

We do real-time compression, encryption (on the wire and at rest) and deduplication for object storage. We currently support an s3-compatible (including a full policy engine) api out the front end, and on the backend we can store to anything that exposes an s3 api (S3, Glacier etc.). Because of the s3-compatible interface we work with any existing client tools that work with s3.

We pride ourselves on our speed and scale. We can do 600mb/s sustained throughput and easily scale to multi-petabyte datasets. We typically see 95%-97% dedupe ratios on backup data. We support high availability clustering and replication (for example, replicate between regions for DR).

We don't currently support snapshotting but it's something we can implement relatively easily if people need it.

Our deployment model is based on a virtual appliance and can be deployed in the cloud or on premise. We can also do things like an on-premise writer (that only uploads unique, deduped data over the network), and a reader in the cloud to support cloud workloads or DR.

We have a real focus on backup to cloud in addition to supporting real time big-data use cases in the cloud.

Disclaimer: I work here - if you would like to contact me please feel free tpower@storreduce.com.

What a coincidence, I found myself writing a little Go lib for exponential back off yesterday, and was wondering what the purpose of introducing jitter was and what effects it would have. I need it for interacting with the Twitter stream api and they're pretty clear that when they say exponential they mean it. But jitter looks very useful in some situations after reading this, so I added that to my lib as well...

It's here if anyone's interested it https://github.com/Diggs/go-backoff