presumably at each step they mask out all tokens that would be invalid at that step according to the grammar