Hacker News Reader

Comment by bsder | Hacker News Reader

> What I learned though was that NFS was great until it wasn't. If the server hung, all work stopped.

Sheds a tear for AFS (Andrew File System).

We had a nice, distributed file system that even had solid security and didn't fail in these silly ways--everybody ignored it.

View on HN

Morgan Stanley was a heavy user of AFS for deploying software and might still be for all I know.

"Most Production Applications run from AFS"

"Most UNIX hosts are dataless AFS clients"

https://web.archive.org/web/20170709042700/http://www-conf.s...

That was still in place at least when I left, and I'd be amazed if it got replaced. It was one of those wonderful pieces of infrastructure that you rarely even notice because it just quietly works the whole time.

https://workshop.openafs.org/afsbpw08/wed_keynote.html

https://www.usenix.org/legacy/publications/library/proceedin...

https://workshop.openafs.org/afsbpw08/talks/wed_1/OpenAFS_an...

NCSA also used it for some data archival and I believe for hosting the website files.

I looked up at one point whatever happened to AFS and it turns out that it has some Amdahl’s Law glass ceiling that ultimately limits the aggregate bandwidth to something around 1 GBps, which was fine when it was young but not fine when 100Mb Ethernet was ubiquitous and gigabit was obtainable with deep enough pockets. If adding more hardware can’t make the filesystem faster you’re dead.

I don’t know if or how openAFS has avoided these issues.

The Amdahl's Law limitations are specific to the implementation and not at all tied to the protocols. The 1990 AFS 3.0 server design was built upon a cooperative threading system ("Light Weight Processes") designed by James Gosling as part of the Andrew Project. Cooperative processing influences the design of the locking model since there isn't any simultaneous between tasks. When the AFS fileserver was converted to pthreads for AFS 3.5, the global state of each library was protected by wrapping it with a global mutex. Each mutex was acquired when entering the library and dropped when exiting it. To complete any fileserver RPC required acquisition of at least six or seven global mutexes depending upon the type of vnode being be accessed. In practice, the global mutexes restricted the fileserver process to 1.7 cores regardless of how many cores were present in the system.

AuriStor's RX and UBIK protocol and implementation improvements would be worthless if the application services couldn't scale. To accomplish this required converting each subsystem so it could operate with minimal lock contention.

This 2023 presentation by Simon Wilkinson describes the improvements that were made to AuriStor's RX implementation up to that point.

https://www.auristor.com/downloads/auristor-rx-hare-and-the-...

The RX tortoise is catching up with the TCP hare.

  Connecting to [10.0.2.15]:2345
  RECV: threads   1, times        1, bytes        2000000000:          881 msec   [18.15 Gbit/s]

Wow that’s a lot of info.

> In practice, the global mutexes restricted the fileserver process to 1.7 cores regardless of how many cores were present in the system.

So in theory the bandwidth could scale with single CPU and/or point to point bandwidth but cannot scale horizontally at all. Except on the new implementations.

Correct, and the point-to-point bandwidth is limited by the maximum RX window size because of the bandwidth delay product. As round-trip latency increases, at some point the window size becomes insufficient to keep the pipe full, at which point data transfers stall.

One site which recently lifted and shifted their AFS cell to a cloud made the following observations:

We observed the following performance while copying a 1g file from local disk into AFS.

  AuriStor Client (2021.05-65) -> OpenAFS server (1.6.24): 3m.11s

  AuriStor Client (2021.05-65) -> AuriStor Server (2021.05-65): 1m

  AuriStor Client (2025.00.11) -> AuriStor Server (2025.00.11): 30s

All of the above tests were performed from clients located on campus to fileservers located in in the cloud.

There are many RX implementation differences between the three versions. It is important to note that the window size grows from 32 -> 128 -> 512.

I know quite a few AFS systems that moved to AuriStor's YFS: https://www.auristor.com/openafs/migrate-to-auristor/auristo...

As I understand it, it mitigated many of those issues, but is still very "90s" in operation.

I've been flirting with the idea of writing a replacement for years, about time I had a go at it!

I may be confusing two systems but I believe that AFS system was also encompassed the first iteration of “AWS Glacier” I encountered in the wild. A big storage that required queuing a job to a tape array or pinging an undergrad to manually load something for retrieval.

AFS implements weak consistency, which may be a bit surprising. It also seems to share objects, not block devices. Judging by its features, it seems to make most sense when there is a cluster of servers. It looks cool though, a bit more like S3 than like NFS.

The cephfs model of a file system logically constructed from an object store closely mirrors the AFS architecture. The AFS fileserver is horribly misnamed. Whereas AFS 1.0 fileserver exported the contents of local filesystems much as NFS and CIFS do, AFS 2.x/3.x/OpenAFS/AuriStorFS fileservers export objects (aka vnodes) which are stored in an object store. Each AFS vice partition stored zero or more object stores each consisting of the objects belonging to a single volume group. A volume group consists of one or more of the RWVOL, ROVOL and/or BACKVOL instances.

The AFS consistency model is fairly strong. Each client (aka cache manager) is only permitted to access the data/metadata of a vnode if it has been issued a callback promise from the AFS fileserver. File lock transitions, metdata modifications, and data modifications as well as volume transactions cause the fileserver to break the promise. At which point the client is required to fetch updated status information before it can decide it is safe to reuse the locally cached data.

Unlike optimistic locking models, the AFS model permits cached data to be validated after an extended period of time by requesting up to date metadata and a new callback promise.

An AFS fileserver will not permit a client to perform a state changing operation as long as there exist broken callback promises which have yet to be successfully delivered to the client.

I'm still in a place where AFS is chugging along for many years after it's much exaggarated demise.

It never ceases to amaze me how well it does what it does and how well it handles being misused.

Looks like these guys are still truckin' along?

https://www.openafs.org/

And heavily used by top universities for the past ~35 years.

Interesting. I think I had casually come across the term AFS (and its full form) back in the day, but never look into it deeply.

Why did everybody ignore it, do you know?

Not everyone ignored it but unlike nfs it didn't come in the box with the operating system, and you had to pay for it. In addition, AFS provided strong cryptographic authentication and wire privacy which meant that it couldn't be licensed in many countries because the U.S. government did not grant appropriate export licenses.

I often wonder how the world would be different if AFS 3.0 could have been freely distributed world wide in 1989 precluding the need for HTTP to be developed at CERN.

There were a few technical obstacles which other people mentioned, but I think timing was biggest issue (remember--AFS dates to something like 1983-ish).

1) AFS, IIRC, required more than one machine in its original configuration. That meant hardware and sysadmins which were expensive--until, suddenly they weren't.

2) Disk, memory and bandwidth were scarce--and then they weren't. AFS made a bunch of solid architectural decisions and then wasted a bunch of time backing some of them down in deference to the hardware of the day and then all that work was wasted when Moore's Law overran everything, anyhow.

3) Everybody was super happy to be running everything locally to escape the tyranny of the "Mainframe Operator" (meaning no NFS or AFS or the like)--until they weren't. Once enough non-technical people appeared who didn't want to do system administration, like, ever, that flipped.

We lost the VMS filesystem in this timeframe, too. Which was also a distributed, remote filesystem.

But those x86 processors sure are cheap ... sigh.

Reply on news.ycombinator.com