Hacker News Reader

Comment by js2 | Hacker News Reader

In my 40+ years of computing dating back to an Apple II, I've used storage as varied as 140K floppy disks, to EMC RAID arrays attached via SANs to Sun Enterprise 4500s, to NetApp NFS clusters, to a TrueNAS box in my basement with terabytes of space.

I've been a sysadmin managing SunOS and Linux systems at scale.

I've had personal machines running a variety of OSs: FreeBSD, Linux, Windows, Mac OS, OS X, to say nothing off all the different filesystems involved across those OSes.

In all that time, I can't think of a single time I've ever bothered to de-duplicate files.

Once upon a time, you used to occasionally need to defragment some file systems. But it's been more than two decades since I've even thought about defragmenting. Every modern filesystem just handles that automatically.

What even is the use case where folks end up with enough duplicate files that it's worth worrying about?

View on HN

One time I had to consolidate a late family members personal files spread across several types of media which ended up totalling several TBs. None of it was organized, so there were duplicates everywhere. Probably only 20% of the data was non-duplicated stuff I genuinely wanted to keep.

I'll give you two use cases, one for non-technical and one for technical people.

Non-technical person will copy-paste a folder of documents to make a backup before editing a file. A crude form of version control. I've seen this method lead to the consumption of hundreds of gigabytes for duplicate content, because the folders contain a lot of auxiliary data like PDFs and JPEGs.

Technical person will run npm install and get a massive node_modules directory that is almost identical to the node_modules directory in another project.

For the second one I would love to have some kind of centralized node_modules used across my projects. I am pretty sure that a large part is duplicated, just for the fact that I use the same framework everywhere

I can think of exactly one case!

In college there was, for a glorious, brief, time, a program called mytunes. It would scan the network for iTunes instances that had the network sharing turned on. Then it allowed you to copy over entire iTunes libraries (iTunes only let you play songs from other users).

I would log on to the dorm network which normally had dozens of iTunes instances running at a go, and just copy everything to my, at the time, massive 160gb drive.

Then you would dedupe. That’s the last time I thought about deduping

I never really bothered myself, but when one of my SSDs was running low on storage, I decided to run a deduplicator over some folders.

It turns out the node_modules folders I've collected over the years had several gigabytes of duplication. The same was true to a lesser extend for Python code (because the dependency folders and venvs were a lot smaller).

> Every modern filesystem just handles that automatically.

Do they? I always assumed they just try to avoid fragments but ignore them once created, and rely on bigger faster disks to hide any problems this may cause.

Reply on news.ycombinator.com