Ask Google about that.
Ask Google about that.
I genuinely am trying to understand.
So yeah, I'm not sure why you'd have to "crawl" through physical media when a search index ought to tell you right where your match is located. Is that not the entire point of indexing and searching?
Say, a file reference within a storage hierarchy or record within a database.
(The concepts are fundamentally identical.)
1. Provide an independent storage tier for the actual content. Say, "stacks". This would store various formats of documents, uniquely identified (say, by a corresponding hash), and maintained independently of access.
2. I'd have an interface tier that would, simply put, be a search interface. The concept of "filesystem as document management" could present this in a familiar, apparently hierarchical fashion, but the hierarchy would actually be searches against various metadata. Say: author, publisher, title, publication date (as range), subject, or keywords.
3. These metadata would be comprised of one or more indices of the actual contents. The indices are what you want on fast storage, in memory if at all possible. Various forms of caching and distribution might provide for this.
4. In order to associate metadata with works and their attributes, you'd need an intake process. This means that as works are onboarded, the would be, effectively, catalogued and classified, with metadata fields supplied. It turns out that there are available metadata repositories for a large class of works (the US Library of Congress and OCLC are two organisations providing same), and it's possible that various distributed, crowdsourced, automated, AI, or similar mechanisms could be used to fill in metadata for works not otherwise categorised.
5. Further capabilities, access, reports, summaries, groupings, workflows, workgroups, projects, security layers, publication, editing, revision, editing, reference, exploration, etc., might be provided by filesystem analogues as possible, other means where not.
If the overal system sounds a lot like a library and cataloguing system, that might be because it rather much is.
Google's web index operates in much the same way. Primary storage is the origin server of a given URL. The index is maintained and accessed by Google within its own systems. Google typically returns a response in a few hundredths of a second. Retrieval of the referenced source typically takes a number of seconds, or roughly 2-3 orders of magnitude slower.
________________________________
Notes:
1. See: https://redd.it/6bgowu