> Out of curiosity, are you sure that GPT did it correctly? If yes, is it because you were able to "spot check" it in a few places? Or have you used GPT enough that you trust it with a task like this? Or is this for some kind of internal use where a few small errors is unimportant, and you only need the broad strokes to be correct?
> Yes, yes, and yes. (There were some obvious checksums, for instance by computing the total number of speakers several different ways.)
Still cool, but the circumstances above aren’t always true. In fact, in a lot of meaningful work, none of them probably are.
If GPT is like an assistant whose work you always have to double check in order to be sure of the validity of the results, that seems to be a pretty big caveat to me. In that case it’s probably better to just write a script whose output you know to be deterministic, even if it takes slightly longer (and the bigger your dataset to verify, the more likely it is that it’ll take less time to write a script than to validate results one by one).
The scary part is if/when companies/bureaucracies/governments/etc start using GPT for all sorts of Important Tasks and skip the validation part because they assume the machine will always get it right.