As you might know ext* and some other FSs store filenames right in the directory file. Means the more files you have in the directory, the bigger directory size gets. In majority of cases nothing unusual happens, cause people have maybe a few dozens of dirs / files.
However if you'll put millions of files, then directory size grows up to a few megabytes in size. If you decide to clean up later, you'd probably expect the directory size to shrink. But it never happens. Unless you run fsck or re-create a directory.
That's because nobody believes the implementation effort really worth it. Here's a link to lkml discussion: https://lkml.org/lkml/2009/5/15/146
PS. Here's a previous discussion of the very same article posted in this submission. It's been 10 years already :) https://news.ycombinator.com/item?id=2888820
upd. Here's a code example:
$ mkdir niceDir && cd niceDir
# this might take a few moments
$ for ((i=1;i<133700;i++)); do touch long_long_looong_man_sakeru_$i ; done
$ ls -lhd .
drwxr-xr-x 2 user user 8.1M Aug 2 13:37 .
$ find . -type f -delete
$ ls -l
total 0
$ ls -lhd .
drwxr-xr-x 2 user user 8.1M Aug 2 13:37 .