Hacker News Reader

Comment by user5994461 | Hacker News Reader

I feel like I have to do it and ask the stupidest yet most important question in this thread...

Why didn't you use S3?

I assume for three reasons:

  - small file performance (if they're not storing the git objects as packed)

  - POSIX compliance

  - Their on-prem installations for customers

For posix compliance, you can use s3fuse/gcsfuse or put Avere in front, but you're either paying more for it or you're not getting the best performance.

For small file performance though, no object storage system is particularly tuned for that (often the granularity is more like MBs, and at least with GCS you're talking ~100ms per write).

Finally, they need to be testing and supporting customers who want to run the whole thing "at home" wherever that is. So they would still need to run Ceph!

Disclosure: I work on Google Cloud.

I wouldn't say no distributed object storage system is particularly tuned for that. Facebook's haystack and systems patterned after it can work very well for tiny files.

Sorry, I meant for public cloud providers. You're right that Haystack would be fine for small files, but IIRC (and skimming the paper again [1]) it stores the entire photo inline in a single Needle. That's fine for photos, but not for people that want to say store a 5 TB single object (and for Facebook, they could just reject photos larger than say 100 MB). Still, good idea!

[1] https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...

S3 does not allow random access reads/writes to the files. "Editing" a file in S3 just means deleting the original blob and creating a new one.

Meaning, you can't run git on it, you'd need to make a full copy of the repo each time someone makes a commit.

That isn't true; a git repository isn't just a single file. It is one (or many) pack and object files, and it's possible to have a repository by using each push (which may be multiple commits) to store a new pack file rather than deleting the old one.

The only time you would need to delete/replace is during a GC operation which can be scheduled periodically to compact the representation.

The JGit driver supports pushing to S3 in this way, and I have implemented a custom backend to hosting git repositories in an S3 like service (which in turn is based on how Google hosts git repositories at scale)

Not all users may use Amazon or S3. This means that if we were to build something requiring S3 we'd cut off those users.

Building some kind of generic storage layer with multiple backends (S3/S4/etc) would result in us only maintaining whatever we happen to use for GitLab.com. It could also complicate maintenance too much.

That's a huge red flag to me.

You got an application designed to host a few repos on a single disk for a single company. And you wanna run it to store millions of repos for the entire internet... on a single volume.

I understand that you want to run the same thing as your customers. But you have different needs, you gonna run in permanent conflicts.

How much storage do you have now per service? How much growth do they experience? Do you even know if CephFS can scale linearly with servers/disks?

You don't solve storage performance problems by moving to S3. They're looking at response times in the hundreds of milliseconds - it's common to see multiple second response times from S3 (at least on AWS), sometimes they can take longer or just return an error.

> They're looking at response times in the hundreds of milliseconds

No they don't. Read their slides, they are running on 52 seconds latency at times ;)

I was referring to where the article compares 100ms latency to 8 years. Typical "enterprise" storage solutions have sub-millisecond response times. I'm assuming the 52 seconds is how long it's taking to flush the entire journal which wouldn't be a single write. In any case, when someone is having a problem with their storage performance I just find it funny that everyone suggests S3 which has terrible performance.

Their problem is not performance. Their problem is that they want high IOPS AND bandwidth AND storage size for a limited budget. There is no free lunch.

S3 is simple and it can ingest the volume. Of course, the performances are not comparable to a local disk.

Reply on news.ycombinator.com