The problem is if I delete all the tests for the lexer then any bugs in the lexer will only get exposed through the parser's tests.
This makes no sense to me.
The lexer has a clear boundary from the parser.
The issue that takes experience here is how to determine what's a unit. "The whole program" is obviously too big. "every public method or function" is obviously too small.
Just be pragmatic.
Of course.
> "every public method or function" is obviously too small.
Why "obviously"? If it's public, someone outside the class can call it. That's an external behavior.
The "feel" for good code that comes with experience is not reducible in practice to a set of black-and-white rules.
Public doesn't necessarily mean "not you".
For any token kind, it should be easy to come up with a minimal piece of syntax which includes that token.
If there is a lexical analysis case (whether a successful token extraction or an error) that is somehow not testable through the parser, then that is dead code.
The division of the processing of a language into "parser" and "lexer" is arbitrary; it's an implementation detail which has to do with the fact that lexing requires lookahead and backtracking over multiple characters (and that is easily done with buffering techniques), whereas the simplest and fastest parsing algorithms like LALR(1) have only one symbol of lookahead.
Parsers and lexers sometimes end up integrated, in that the lexer may not know what to do without information from the parser. For instance a lex-generated lexer can have states in the form of start conditions. The parser may trigger these. That means that to get into certain states of the lexer, either the parser is required, or you need a mock up of that situation: some test-only method that gets into that state.
Basically, treating the lexer part of a lexer/parser combo as public interface is rarely going to be a good idea.
There is the problem, any tests that fail in the lexer now reach down through the parser to the lexer. The test is too far away from the point of failure. I'll now spend my time trying to understand a problem that would have been obvious when the lexer was being tested directly.
>Basically, treating the lexer part of a lexer/parser combo as public interface is rarely going to be a good idea.
This is part of the original point, the parser is the public interface which is why the OP was suggesting it should be the only contact point for the tests.
If you keep code changes small, and keep tests working, you're good.
From that experience, I would say that in some contexts, tests shouldn't be removed unless what it's testing is no longer being used.
If you have a good answer to that, then the lexer is separate (as others said). If you don't then wirte parser tests for the lexer so that you can more easily refractor the interface between them.
There is no on right answer, only trade-offs. You need to make the right decision for you. (though I will note that there is probably a good reason parse and lex are generally separated and that probably means that the best tradeoffs for you is they are separate. But if you decide different you are not necessarily wrong)
Well, I was responding to "you should delete tests for everything that isn't a required external behavior".
Or, as @VHRanger pointed out, the lexer can be considered a unit and be tested independently.