I think in a real parser your instincts would likely serve you well.

For example, if a string data type was added, our tokenize method could not be this simple.

To the actual question, I think one good approach when cutting corners or doing a quicker sub-optimal approach, is to document the limitations and assumptions in some way. You could then evaluate and prioritise you efforts and time from this, rather than getting stuck in the weeds early in the process.