BabuDB (related to extreemfs.org): http://dl.acm.org/citation.cfm?id=1849822
Spyglass: http://www.ssrc.ucsc.edu/pub/leung09-fast.html
And more from the same researchers: "Scalable File System Indexing": http://www.ssrc.ucsc.edu/proj/fsindexing.html
Perhaps I'm thinking of reiser4?
See also:
TokuFS: https://www.usenix.org/conference/hotstorage12/workshop-prog... https://github.com/esmet/tokufs
I'm still certain I've seen talk of indexing metadata, and I thought it was in an actual, open, system...
This reminded me of BFS: https://en.wikipedia.org/wiki/Be_File_System
An index can be anywhere, and the data you want to keep is somewhat arbitrary - so why not just use a file on the filesystem anyway?
There's also an important difference: metadata indices can be considered disposable in a lot of cases. File data isn't which means the constraints are different: with metadata you want to pack it all into the tiniest, most local part of the disk you can.
With file-data (and the actual filesystem) you want to distribute and replicate that data as widely as possible to minimize the chances that a cluster of failures wipes out important structures.
Indexing on other meta-data, like tags for images and music files -- can be considered a filesystem level problem -- if one considers approaches like Mac (or Amiga) resource forks/info-files.
Perhaps I should have stated "file metadata" as opposed to "just" metadata... One could of course claim that the only thing the filesystem does is take an exact path, and return the data at that path. In such a case, you could replace the path+name with a guiid, and store the filename and path-name info in a file... and update file whenever you accessed a file... and then you end up building some of that into the filesystem interface. So the question really is where the filesystem ends and the "system" starts...