Hacker News Reader

Can anybody speak to their experience using sqlite for data analysis purposes?

Am I wrong in thinking it's just plain old row oriented storage and not something more aggregate oriented?

I use sqlite to store all of my small datasets. Minimally, I think of it as a replacement for zipped CSV files. But it also has the added benefit of a relational structure and SQL.

It is super easy to access from julia, R, python, etc, so instead of importing a CSV and manipulating the data, I find it a lot easier to connect to the sqlite database and use SQL for the a lot of the joining and manipulating.

I see. So the benefit is mainly in having more relational structure than in actually crunching numbers. Makes sense.

I use it quite a bit for data analysis, in particular for user-defined functions with python. Being able to explore your data (via SQL and it's powerful syntax) in addition to functions and aggregators that I define, is REALLY useful. You could do the whole thing in python (and data imports), but adding the SQL part in is so much easier than building dictionaries and filtering, sorting etc...

Reply on news.ycombinator.com