and
> It seems like there’s almost no limit to how good GPT could get at this.
I don't see how that's a valid conclusion given the results. 2 simple things right, moderate to difficult things wrong? Hardly a ringing endorsement.
Feels extremely overblown. Very impressive, don't get me wrong, and is yet another bridge closer to letting business side do plain text queries.
But that said, stuff like this has existed for awhile, and it's certainly not replacing analysts. For a start, this assumes a VERY clean model, which is of course what you strive for, but the moment you step out of FAANG you'll find that many places have a decent, but not ideal, model/warehouse/db which only makes things like this worse.
Getting the simple examples right is interesting, but hardly replacing an analyst. A basic background around tech and some skill navigating w3schools would replace an analyst if this was the bar.
The mildly complex queries being "almost" right is frankly horrifying. Others have mentioned how much of a "oh fuck" it is to send out bad data, and having some AI generate your answers an be SLIGHTLY off is basically worse than it being obviously off. You now have to check every query it generates for errors, because you won't be able to glance check if its sane or not.
Finally, at least where I work, these probably wouldn't even qualify as medium complexity queries? I don't really know how the industry at large would qualify these, but "using 2 inner joins, a where, a group, and a order" all falls under beginner skills in my eyes?
Its still fascinating, and I'm glad they showed HOW they did it, but this isn't replacing anyone, at least if you're paying taxes off the data.
Even in FAANG, most data is going to be extremely messy.
* There's usually very little incentive for good documentation, so you have to guess what the table contains based on column names and exploring patterns in the data.
* Enforcing a single pattern/process for data across the entire org is a pipe dream.
* Logging tables are written to by hundreds of engineers across dozens of teams, with no standardized naming scheme.
* The product contains multiple edge cases/special overrides for very specific circumstances, so there's very few simple queries that don't come with 100 footnotes attached.
FAANG is not immune to large-organization problems, and data quality is no exception.
But with that in mind, thats because theres a major pipeline of people and processes to get the data to that point, and it meant that there could sometimes be significant delay on new KPI's as they had to be cleanly worked into the model, and it of course didn't represent everything.
The first car was hardly ready to replace the horse. But look where we are now.
And this is something people have been saying about things like AI and crypto for years now. They've both made some interesting developments, and they both face serious hurdles (some fundamental).
AI obviously has a ton of real world use cases, but overhyping things like this doesnt actually help.
Modern cars also have a lot of computing power in the form of fairly dumb systems that are good at sensing things and following rules e.g. "based on conditions X, Y, and Z the flow of the fuel injector nozzle should be N". Got a solid track record here too albiet one that's decades long and not millennia.
A tool that replaces a human data analyst needs to do a lot of inference. When I'm translating stakeholder requests into code and reports, it's about 10% coding and 90% human inference. Remains to be seen how good machines can get there.