This will probably work as long as the material being annotated is similar to the material the LLM was trained on. When it encounters novel data (value) it will likely perform poorly.
From benchmarking, we've been positively surprised by how effective few-shot learning and PEFT are, at closing the domain gap.
"When it encounters novel data (value) it will likely perform poorly" -- is that not true of human annotators too? :)
Some humans have intelligence and reasoning abilities. No LLMs do :)