Drawing the Line Between Lexer and Parser
Because ANTLR lexer rules can use recursion, lexers are technically as powerful as parsers. That means we could match even grammatical structure in the lexer. Or, at the opposite extreme, we could treat characters as tokens and use a parser to apply grammatical structure to a character stream. (These are called scannerless parsers. See code/extras/CSQL.g4 for a grammar matching a small mix of C + SQL.)
Where to draw the line between the lexer and the parser is partially a function of the language but also a function of the intended application. Fortunately, a few rules of thumb will get us pretty far.
-
Match and discard anything in the lexer that the parser does not need to see at all. Recognize and ...
Get The Definitive ANTLR 4 Reference, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
