uroni

I implemented this (HS5). Etags match the AWS S3 ones as well.

Ceph is pretty much the standard, I think. Very complex to manage, though, but that might just be necessary complexity for a distributed storage system.

Not without snapshot currently. I guess I could collect all writes in the (optional) WAL file while a backup is running, then the files would be consistent.

My https://github.com/uroni/hs5 is designed for this use case.

One notable thing is that compared to MinIO (and others) it does not store the objects as individual files. I also have DuckDB directly integrated.

The readme has a comparison to Garage, seaweedfs, RustFS and Ceph.

That it keeps an infinite cache of malloc page allocations is annoying (the issue you referenced). I just removed that (after complaining on the mailing list about it). The performance advantage is probably negligible in many cases (since malloc implementations often already cache), while causing confusing memory usage behavior.

Idk, if it was your issue, but for long running write transactions it doesn't spill to disk. So you have all the changes being written to disk at the end of the transaction. One would think enabling write mapping fixes this, but it needs to mark all the pages as clean before commit, so same effect there. I fixed this for 0.9 here https://github.com/uroni/hs5/tree/main/external/lmdb . Will have to investigate if it is improved with 1.0, or if I need to redo the changes.

Edit: Just noticed that the issue is about free list in the file. Never had a problem with that, but I also had to replace that MIDL structure with something more scalable for the spilling.

In my similar project (s3 compatible single-node storage) https://github.com/uroni/hs5 I do use proper fsync for data and metadata durability. But it can be turned of via switch. It is a pet peeve of mine that the defaults should always be to fsync. I do have a section on this in my README of the project.

I also do have an optional WAL. Maybe I should add an additional mode that disables fsync only for the WAL. I don't think it would be a good idea. My WAL does use checksums and sequence numbers etc. to prevent committing wrong data.

I build https://github.com/uroni/hs5 as replacement for single node use with a focus on high performance. I list other alternatives in the README there. Some short version:

Ceph: Robust, widely used for multi-node deployments. Would recommend for serious use.

RustFS: As an in-place replacement using the same storage format. Though, to me it is a bit suspect, e.g. if it uses fsync for durability.

seaweedfs: Multi-node alternative, that keeps the object mapping in memory (so more RAM usage and startup cost compared to alternatives).

Garage: Multi-node alternative, web-interface available separately. To me seems unsuitable for single-node use at this point.

VersityGW: Single-node alternative, which uses the same object=file in file system tree storage as MinIO (with the same disadvantages/advantages). It uses extended attributes for S3 metadata, so less filesystem overhead than MinIO/RustFS at least. Cannot find any Sync() or fsync calls in the code, though.

I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss.

But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.

Yup, https://github.com/awslabs/git-remote-s3 (disclaimer: never used it)

For me it went into the multi-node direction, where I'd use Ceph anyway (or build on-top of an existing solid distributed database) if I needed it.

Also think there is an abstraction mismatch with the object stores that store the objects with a 1:1 mapping into the file system. Obvious issues are that you only get good listing performance with '/' as delimiter and things like "keys with length up to 1024 bytes" break.

I made https://github.com/uroni/hs5 -- focus is on single node and high performance. So plenty of alternatives available.

There is a lot of software that directly implements the HTTP S3 API. That API is also documented by Amazon.

E.g. the last implementation I saw was by DuckDB https://github.com/duckdb/duckdb-httpfs/blob/main/src/s3fs.c...

AGPL is "a plague" by design (viral). It has the explicit goal that any improvements flow back to the community project and the virality is a necessary building block for this. It is an elegant solution to a tragedy of the commons problem.

Companies like MinIO extending the virality beyond the single software/work, even though not intended by license, gives it a bad reputation. They have fixed https://min.io/compliance now, but I guess it does not matter anymore.

I never understood why one would use MinIO over Ceph for serious (multi-node) use. Sure, it might be easier to setup initially, but Ceph would be more likely to work.

For the single node use-case, I'm working on https://github.com/uroni/hs5 . The S3 API surface is large, but at this point it covers the basics. By limiting the goals I hope it will be maintainable.

I’ve used this technique in the past, and the problem is that the way some file systems perform the file‑offset‑to‑disk‑location mapping is not scalable. It might always be fine with 512 MB files, but I worked with large files and millions of extents, and it ran into issues, including out‑of‑memory errors on Linux with XFS.

The XFS issue has since been fixed (though you often have no control over which Linux version your program runs on), but in general I’d say it’s better to do such mapping in user space. In this case, there is a RocksDB present anyway, so this would come at no performance cost.

It uses LMDB, so if the object mapping fits in memory that should be pretty optimal for reading, while using the build-in Linux page cache and not a separate one (important for testing use cases). For write/deletes it has a bit of write-amplification due to the copy-on-write btree. I've implemented a separate, optional WAL for this and also a mode where writes/delete can be bundeled in a transaction, but in practice I think the performance difference should not matter.

W.r.t. filesystem: Yes, aware of this. Initially used minio and also implemented the use case directly on XFS as well and only had problems at larger scales (that still fit on a machine) with it. Ceph went into a similar direction with BlueStore (reimplementing the filesystem, but with RocksDB).

I'm not a contributor to Minio. This is its own separate thing.

I do have a separate AGPL project (see github) where I have nearly all of the copyright and have looked into how one would be able to enforce this in the US at some point and it did look pretty bleak -- it is a civil suit where you have to show damages etc. but IANAL.

I did not like the FUD they were spreading about AGPL at the time since it is a good license for end-user applications.

I've been working on https://github.com/uroni/hs5 as a replacement with similar goals to early minio.

The core is stable at this point, but the user/policy management and the web interface is still in the works.

I'm working on something that might be suited for this use-case at https://github.com/uroni/hs5 (not ready for production yet).

It would still need a resilience/cache layer like ZFS, though.