Yes. The fact that literally no one does this makes me continue to wonder wtf people are actually doing w/ LLMs, since I feel this need so acutely and the solution is so obvious.
I assume that this sort of functionality will become more commonplace.