This can certainly be done. Here's one example from 2021 demonstrating training an LLM to use a scratchpad ("talking to itself") to greatly improve accuracy on arithmetic problems:

https://arxiv.org/pdf/2112.00114.pdf