Hacker News Reader

Comment by ggm | Hacker News Reader

If technical goes to scaling and effects related to scaling, maybe they just decided to let the people with >5m inodes incur the technical penalty, which is almost always speed?

it was always a cost issue, in some sense. If they designed the entire FS model behind drive to have inode limits per entity, then there's a small technology issue bumping that number maybe. Quota systems can be quite painful, in days gone past the Robert Elz Quota model on UFS demanded reboots for some changes, which at scale in a google sized entity is a rolling surface of state change.

View on HN

Their explanation of the 5M limit is because of inode limitations when they materialize the content via the companion client?

5m is an odd number. 24 bit numberfield would get you 16m uniques. I can't see how they needed to steal bits to get to 23 or 22. So there must be some other factor at play here apart from 'we used 3 bytes in the gRPC model'

Perhaps the Windows exposed API assumed somewhere around 5m objects, and they care about the compatibility greatly.

I don't think the number is based on any technical limitation.

It could be a number based on usage. Say 99.9% of the users use less than 5M files. So my pruning 0.1% of the user base they are getting rid of half the number of files, saving X millions of dollars in cost.

Nobody wants to cater to the difficult users that are doing stuff like this.

After that pruning, you now have a new 0.1% of users using more files than the other 99.9%. Might as well prune that 0.1% of file hogs too. Rinse and repeat until you have no users left.

That's interesting, because if your argument is valid, they wouldn't have to roll back the change.

if filescount() > 5m then.. had to be removed. There is no quick release in a google scale monrepo and QA process.

Sure. But all that would mean is that you can only have 5M files materialized locally. I don’t see why drive itself would be limited at 5M. Very strange for this to have any real technical limit.

Maybe they just keep a safe margin so they can add their own metadata inodes or something. Or it's arbitrary.

Reply on news.ycombinator.com