Hacker News Reader

Comment by whalesalad | Hacker News Reader

Just yesterday I was thinking how badly I want an obscenely fast local datastore for Docker. I am managing a handful of clients right now who are all using Docker containers for their microservice environments and my local machine runs out of disk daily.

What I really want is lightning quick network storage, but I don't think it would be feasible to roll out 10g networking in my current rental home.

View on HN

10gbe can be done over RJ45 with a direct cable connection and is something I've done before in a pinch between my personal NAS and a client machine (just be careful about your storage protocol because NFS has its pros and cons just like iSCSI and FCoE, and with macOS as a client you're likely in for a Bad Time like I did).

Out of curiosity, what issues did you run into with NFS on macOS? I'm looking into building a NAS for a network of macOS, Linux, and BSD machines and figured NFS would be my first choice.

I've had weird issues with some applications and NFS on macOS (notably, Final Cut Pro X and Photoshop). And I've also had weird issues with SMB and some apps (notably, Photoshop again!).

But at least SMB has been pretty stable for years. NFS support has had some hiccups. A few 10.14 macOS releases broke NFSv3 and NFSv4 for me, but it was fixed in later releases.

I'm not the author nor am I an apple user anymore, though i was at one point.

NFS doesn't have a lot of metadata, so you loose a lot of functionality. iPhoto for example could cause major issues if you persisted its library on a nfs, potentially making the library irreparable.

NFSv3 and NFSv4 clients were fine when local machines over TCP and always-on like with a previous Mac Mini I had but resuming from a laptop was pulling teeth and may or may not have been the reason for some of the crashes I got. iSCSI in theory should be better but I had a lot of problems getting the iSCSI initiator I found to work for myself despite being a pretty simple use case I thought, and I gave up on the project around then as I found macOS simply required more budget for myself to buy an appropriate iSCSI initiator. This was over 5 years ago and things may have changed between then and now.

I've had stability problems with nfs on MacOS. I ended up using smb instead. Smb ended up being CPU bound on my NAS due to encryption I haven't taken the time to figure out how to disable. It's close to line speed for me though. It's also reliable.

Just curious what size is your hard drive?

I had this pain with my last macbook. I ALWAYS ran out of disk space. Never again! I went with a 1 TB drive for my M1.

I offload docker to another machine and I'm still using about 400GiB of storage on my relatively new MacBook; and I basically never use this machine outside of work (and even then I didn't use it for like 4 months because I preferred linux). it has no local media or anything.

I'm shocked by how much disk I've used actually.

https://imgur.com/a/iHV4IKn

I got the 2TB drive so I can run out of even more disk space! ;-)

A larger common image + saner node_modules should save a lot

Would running something like the docker proxy below on an old laptop help?

  TLDR: A caching proxy for Docker; allows centralized management of (multiple) registries and their authentication; caches images from any registry. Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting.

https://github.com/rpardini/docker-registry-proxy

At some point you need to build images locally or store images locally to run a local docker-compose cluster.

I need a fat block store for that.

That is painful for sure.

What about separate Linux box laying around just to run docker? 8GB M1 Mac just don't have enough RAM for docker VM.

Looks like local docker server should just work exactly same way as local docker while you at same location.

Then you get into the problem of low latency directory sync between your box and that host. I’m still trying to crack that nut.

why don't you simply use one or many nvme thunderbolt/usb disks and move the caches there?

Not the op, but if the nas was pre-existing, and already serving other network clients, no reason to add a second one.

Author here: I actually used a thunderbolt nvme and adapter for a bit. Then, for kicks I got another one and combined them in a raid-0 on macos. Realizing I didn’t want to run nvme’s, in raid-0 over two cables in any sort of production (homelab - I use production loosely here), is what sent me down the Thunderbolt rabbit hole that precipitated this blog post.

In case a single-cable solution would work, I've been using an external TB NVMe RAID 0 in this enclosure for several months and it's worked flawlessly: https://www.amazon.com/gp/product/B08S5JPWR6 (The only drawback is that the power supply is bizarrely-large, and the power connector feels a bit janky. In practice it hasn't been an issue.)

Not OP, but my files are shared with many hosts. Having local files are a PITA to manage coherent state.

Reply on news.ycombinator.com