Hacker News Reader

Comment by cushychicken | Hacker News Reader

You're not the first person in this thread to suggest grabbing the whole page text. I've never tried, just because I assumed it was so much space as to be impractical, but I don't see the harm in trying!

View on HN

My current favorite cache for whole pages is a single sqlite file with the page source stored with brotli compression. Additional columns for any metadata you might need (URL, scraping sessionid, age). The resulting file is big (but brotli for this is even better than zstd), and having a single file is very convenient.

It is not advised to use SQLite with several threads, are you running only a single thread?

You can't use SQLite from multiple threads if you are in single thread mode [0]. If you are in multi-thread mode you can use multiple connections to the DB from different threads, or if you want to share a connection then you have to serialize over it. Or you can use SERIALIZED mode.

This is if you do not care writing any logic. If you do, there are other possibilities, for example using a queue and a a dedicated thread for handling database access, but I personally do not think there are many advantages to this more complicated approach.

[0] https://www.sqlite.org/threadsafe.html

Thanks, I wasn't aware of this. I'm not sure if it was old info or I "imagined" it but in any case it's good to know.

I hope it wasn't true in 2019, since I painstakingly wrote a multi-threaded app that was accessing SQLite on only 1 thread!

Reply on news.ycombinator.com