tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only the required parts and not the whole dictionary.

SStable does not support Regex queries, it would require a full load and scan, which would be very expensive.

Your best bet currently would be to make it work with tokenizing, which is way more efficient anyways.

prefix queries are supported btw