So, if I understand the approach correctly: we're essentially doing very advanced feature engineering with LLMs. We find that direct classification by LLMs performs worse than LLM feature engineering followed by decision trees. Am I right?

The finding surprises me. I would expect modern LLMs to be powerful enough to do well at the task. Given how much the data is processed before the decision trees, I wouldn't expect decision trees to add much. I can see value in this approach if you're unable to optimize the LLM. But, if you can, I think end-to-end training with a pre-trained LLM is likely to work better.