This got me thinking about the possibility of building a series of simple context/token probability tables in SQLite and running the show that way. Assuming we don't require massive context windows, what would prevent this from working?
It's not like we need to touch every row in the database all at the same time or load everything into RAM. Prediction is just an iterative query over a basic table - You could have a simple key-value pair of context & the next most likely token for the given context. All manner of normalization and database trickery available for abuse here. Clearly a shitload of rows, but I've seen some 10TB+ databases still satisfy queries in seconds. You could even store additional statistics per token/context for online learning scenarios (aka query-time calculation of token probabilities). You could keep multiple tokenization schemes online at the same time and combine them with various weightings.
What would be more efficient/cheaper than this if we could make it fit? Wouldn't it be easier to iterate basic tables of data and some SQL queries than to trip over python ML toolchains and GPU drivers all day?