HNHacker News
TopNewBestAskShowJobs

Eikon

1,270 karma · joined December 12, 2019

submissionscomments
Eikon··on Why did containers happen?

    make tinyconfig 
can get you pretty lean already.

https://archive.kernel.org/oldwiki/tiny.wiki.kernel.org/

Eikon··on NFS at 40 – Remembering the Sun Microsystems Network File System
ZeroFS uses NFS/9P instead of fuse!

https://github.com/Barre/ZeroFS

Eikon··on Bcachefs removed from the mainline kernel
> If you're using ZFS for a load which resembles random block I/O, you'll want to tune the recordsize to the app I/O.

You probably don't want to do that because that'll result in massive metadata overhead, and nothing tells you that the app's I/O operations will be nicely aligned, so this cannot be given as general advice.

Eikon··on I built Foyer: a Rust hybrid cache that slashes S3 latency
You don’t have to use AWS S3, any compatible implementation will work.
Eikon··on I built Foyer: a Rust hybrid cache that slashes S3 latency
The easy way to achieve that, is using ZeroFS NBD server with ZFS L2ARC (L2ARC with local storage and “main” pool on ZeroFS).
Eikon··on I built Foyer: a Rust hybrid cache that slashes S3 latency
If you have the network that matches, it should be perfectly fine.
Eikon··on I built Foyer: a Rust hybrid cache that slashes S3 latency
That should work!
Eikon··on I built Foyer: a Rust hybrid cache that slashes S3 latency
I use Foyer in-memory (not hybrid) in ZeroFS [0] and had a great experience with it.

The only quirk I’ve experienced is that in-memory and hybrid modes don’t share the same invalidation behavior. In hybrid mode, there’s no way to await a value being actually discarded after deletion, while in-memory mode shows immediate deletion.

[0] https://github.com/Barre/ZeroFS

Eikon··on TernFS – An exabyte scale, multi-region distributed filesystem
If SeaweedFS supports conditional PUTs, yes.
Eikon··on TernFS – An exabyte scale, multi-region distributed filesystem
Shameless plug: https://github.com/Barre/ZeroFS

I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.

Eikon··on AI coding
It’s interesting how science can become closer to pseudoscience than proper research through paper-milling.

It seems like that with such small groups and effects you could run the same “study” again and again until you get the result that you initially desired.

Eikon··on AI coding
Even though I don’t buy that LLMs are going to replace developers and quite agree with what is said, this is more of a critique of LLMs as English-to-code translators. LLMs are very useful for many other things.

Researching concepts, for one, has become so much easier, especially for things where you don’t know anything yet and would have a hard time to even formulate a search engine query.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
> For personal user, just don’t use things like S3

Why?

> For business they won’t use things like this either.

Again, why not?

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
Deku is great!
Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
Unless I am missing something, rclone will probably compare similarly to https://www.zerofs.net/zerofs-vs-mountpoint-s3
Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
> It still sucks that S3 is ~$20/mo/TB basically "in perpetuity", while random SSD drives are ~$80/TB and I'd feel comfortable effectively amortizing them out at ~$20/yr for local storage instead of $20/mo for S3. :-/

ZeroFS will work with any S3 implementation that supports conditional writes. Even something a bit premium-priced like CloudFlare R2 goes for around $15/TB and has no egress costs. If you want to go for as cheap as possible, I'd throw minio or similar on a few cheap Hetzner storage servers https://www.hetzner.com/dedicated-rootserver/matrix-sx/

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
> I'm referring to one client being able to do 1000s of operations a second

Again, I don't see any problems with "thousands of operations a second" if your single client is issuing parallel requests. Try any sequential workload instead.

> because of its minimal latency compared to raw object storage

That's not how it works. JuiceFS hits the object store for every file operation (except metadata, because it's backed by an external database). Latency cannot be lower, especially as JuiceFS doesn't seem to be implementing any buffering.

Above you also claim "allowing near instant synchronization and better performance", but near instant synchronization of what exactly? Metadata? Data blocks? And under what consistency model?

If you wish to run the benchmarks in bench/ on your setup and show better numbers, I'd happily update the JuiceFS benchmarks.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
There's an NFSv3 COMMIT operation, combined with a "durability" marker on writes. fsync could translate to COMMIT, but if writes are marked as "durable", COMMIT is not called by common clients, and if writes are marked as non-durable, COMMIT is called after every operation, which kind of defeats the point. When you use NFS with ZeroFS, you cannot really rely on "fsync".

I'd recommend using 9P when that matters, which has proper semantics there. One property of ZeroFS is that any file you fsync actually syncs everything else too.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
> Using something like ZeroFS could be a nice trick for getting decent price/performance on public cloud platforms. My only concern is integrity and durability: what testing have you done to validate that it doesn't lose data even in corner-cases?

There's always going to be uncertainty with software that young, but the CI is pretty expensive and we didn't yet receive corruption-related reports. https://github.com/Barre/ZeroFS/tree/main/.github/workflows

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
> You're also mounting it from Linux, which is just... weird. Nobody (for some values of nobody) does this. The sole purpose for this service is to provide backwards-compatible file shares for Windows servers and desktops.

That's not what Azure's marketing material says:

Serverless file shares

"Store your files on a distributed file system built from the ground up to be highly available and durable."

"Take advantage of fully managed file shares mounted concurrently by cloud or on-premises deployments of Windows, Linux, and macOS."

"Use integration with Azure Kubernetes Service (AKS) for easy cloud file storage and management of your data using NFS or SMB file shares."

Azure explicitly markets this for Linux and Kubernetes. If it performs terribly on Linux, that's Azure's problem, not mine.

https://azure.microsoft.com/en-us/products/storage/files

> Benchmarking cloud services can be tricky because they have odd quirks such as IOPS-per-GB ratios, so nearly empty storage can be much slower than you'd expect!

These managed services charge premium prices - they should deliver premium performance. At those prices, "quirks" and "you need to fill it up first" aren't acceptable excuses.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
What's so bizarre about it?

Are we prevented from implementing features when the platform itself provides adjacent tech?

Page blobs are 2x+ more expensive ($0.045/GB vs $0.02/GB) and would require Azure-specific code. We don't dream of maintaining separate implementations for each cloud provider.

The approach works identically on S3, GCS, Azure, and local storage. If anything, ZeroFS helps you distance from vendor lock-in, especially if you start to run "usually managed" components on top, say databases.

Speaking of Azure's "native" solutions - we benchmarked ZeroFS against Azure Files. ZeroFS is 35-41x faster for most operations and 38% cheaper. If Azure Files performs that poorly, I don't hold much hope for page blobs either: https://www.zerofs.net/zerofs-vs-azure-files

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
BuildBarn's nfsv4 server looks great!

I'm not too sure about NFSv4 for ZeroFS specifically, the benefits don't seem _that_ obvious to me, especially compared to something like 9P, where server-side locking in the ".L" version is native anyway. Especially considering that most ZeroFS users are mounting the fs on the same box as the server is running and so local locks are working.

MacOS is a bit of a blind spot for 9P though, and a client there would be great.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
There’s no on-disk write cache.
Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
I have no doubt that JuiceFS can perform “thousands of operations per second” across parallel clients. I don't think that's a useful benchmark because the use cases we are targeting are not embarrassingly parallel.

Using a bunch of clients on any system hides the latency profile. You could even get your "thousands of operations per second" on a system where any operation takes 10 seconds to complete.

Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
The durability profile before sync should be pretty close to a local filesystem. There’s (in-memory) buffering happening on writes, then when fsync is issued or when we exceed the in-memory threshold or we exceed a timeout, data is sync-ed.
Eikon··on Show HN: ZeroFS, the Filesystem That Makes S3 Your Primary Storage
The bench suite is published at the root of the repo in bench/. Stating “a bunch of claims” seems a bit dismissive when it’s that easy to reproduce.

JuiceFS maps operations 1:1 to s3 chunk wise, so the performance profile is entirely unsurprising to me, even untaring an archive is not going to be a pleasant experience.

Eikon··on Why you should delete WhatsApp and install Signal
I abandoned that idea as soon as they launched their weird crypto-coin stuff.

Also, can we backup our messages yet on iOS?

Eikon··on Google says "not a security vulnerability", quickly fixes without attribution
You accuse me of doing exactly the the thing you keep doing: deflecting with random stuff and ignoring the substance.

Meanwhile, if I was just a troll, you wouldn’t keep coming to this thread to prove me right.

I’ll leave it here.

Eikon··on Google says "not a security vulnerability", quickly fixes without attribution
The technical vulnerabilities I reported are factual regardless of organizational relationships. The concerning issue here is that my private security disclosure was forwarded to a public, Google moderated venue, without consent. Then, I was banned from the same venue to prevent me from being able to defend myself. That’s the actual breach of good faith practice and was actually intended to create drama.

You apply obviously double standards to the same situation.

Eikon··on Google says "not a security vulnerability", quickly fixes without attribution
There's a distinction between "Corrupted seed" and corrupted PK / password. The seed is provided without validation or checksuming, generating perfectly valid keys on the fly from unknown entropy quality.

If you have bad entropy (partially or fully corrupted/weak seed), you'll generate valid-looking keys that are actually insecure.

There's a reason there's not a single "end-user" crypto-related app / cli tool or server that takes a user-specified arbitrary seed as input. That's dangerous, broken design.

Why would you even do that?

← PreviousPage 3 of 13Next →