Within reads the most expensive part was the re-construction of models from the rows returned by SQLite.
While deserialization is not the responsibility of the database, the overall throughput with SQLite was too low for it to serve our usecase.
Within reads the most expensive part was the re-construction of models from the rows returned by SQLite.
While deserialization is not the responsibility of the database, the overall throughput with SQLite was too low for it to serve our usecase.
If this is how you are doing reads, this is your problem. Limit/offset reads are slow.
You'll get much better results if you do a proper pagination implementation.
This article describes the problem and solution that'd give you much better read results.
https://use-the-index-luke.com/sql/partial-results/fetch-nex...
(don't forget page 2 https://use-the-index-luke.com/sql/partial-results/window-fu... )
Removing the offsets has unfortunately not sped things up, since it's the step post reading the entries (deserialization) that is the bottleneck[1].
Since SQLite does not support Lists, Protobufs came across as the best way to serialize the data at hand.
If there's more native way to solve this problem within SQLite, please let me know!