Comparing Open-Source LLMs for NL-to-SQL

Open source LLMs are catching up with OpenAI. They are also considerably cheaper in the long run. We took a look at how they performed in terms of accuracy vs. OpenAI in an NL-to-SQL workload.

tldr -> For zero-shot prompting, the closed source LLMs are still far ahead in terms of generating accurate SQL. However the gap is closing. With the right dataset for fine-tuning the open-source LLMs might be able to perform at the same level.