There are many different classes of problems that are affected by tokenization. Some of them can be tackled by code.