Ideally, the lexer should be a system in of itself, exposing a public interface that is consumed by its client, the parser.

Public doesn't necessarily mean "not you".