This is really cool! Appreciate sharing this work and the explanation.

You mentioned that massaging the data into shape as one of the problems, which I think is possibly one of the best applications of LLMs in my opinion. Creating a pipeline of a data feed (still hard) into an LLM which outputs JSON with the fields of interest would be amazing.