Hey!

OctoSQL[0] author here, this is really impressive! I like this much more than the approach taken by other sqlite-based tools which first load stuff into SQLite and then let you query it.

On the other hand, it does have a cons that it doesn't automatically infer the schema of the input JSON and you still have to manually parse the raw lines. Maybe it would be possible to surmount this by exposing a json-dedicated file reading function which also does SQL inference (I'm not knowledgable about SQLite internals)?

One piece of feedback is with regard to the benchmarks: I think it would be worth it to add additional benchmarks which work on slightly more complex datasets than the one used here. I did a comparison of this vs OctoSQL on the Brazil dataset, and - as expected - sqlite-lines wiped the floor with it. However, then I ran the following queries on a slightly more complex dataset (the Amazon review dataset in this case, from SPyQL's benchmark notebook[1]):

  ~> time OCTOSQL_NO_TELEMETRY=1 ./octosql "SELECT COUNT(*), AVG(overall) 
                                            FROM books.json
                                            WHERE reviewerName = 'James'"
  +-------+-------------------+
  | count |    avg_overall    |
  +-------+-------------------+
  |  3010 | 4.402325581395349 |
  +-------+-------------------+
  real 0m49.805s
  user 0m32.169s
  sys 0m9.163s
  
  ~> time ./lines0-linux-amd64-sqlite3 :memory: "SELECT COUNT(*), AVG(json_extract(line, '$.overall'))
                                                 FROM lines_read('books.json') 
                                                 WHERE json_extract(line, '$.reviewerName') = 'James'"
  3010|4.40232558139535
  real 1m47.933s
  user 1m27.024s
  sys 0m11.559s

and as you can see, the results go in a very different direction.

But anyhow, congrats on the project, and I'm pumped to see what you come up with next!

[0]: https://github.com/cube2222/octosql

[1]: https://github.com/dcmoura/spyql/blob/master/notebooks/json_...