It's a question of jargon. What does the layperson think of when they hear "directory", a telephone directory? Whereas if you call them "folders" or even better, as on the Amiga, "drawers", then it all becomes very obvious.
But apart from bad metaphors, I thing non-geeks have a problem with organizing things in arbitrary deep hierarchies. You don't really do that a lot outside of computing.
Inside drawers you have shelves. On the shelves you have boxes. In the boxes you have folders, envelopes, files...
There's more: in the files you have pictures, frames, tables