Skip to content

Fix lexer_callbacks silently dropped for keyword terminals (#1618)#1619

Open
apoorvdarshan wants to merge 1 commit into
lark-parser:masterfrom
apoorvdarshan:fix-1618-lexer-callback-closure
Open

Fix lexer_callbacks silently dropped for keyword terminals (#1618)#1619
apoorvdarshan wants to merge 1 commit into
lark-parser:masterfrom
apoorvdarshan:fix-1618-lexer-callback-closure

Conversation

@apoorvdarshan

Copy link
Copy Markdown

Summary

Fixes #1618. When two or more terminals each absorb a same-priority keyword literal (so they get an UnlessCallback) and have a lexer_callbacks entry, every user callback except the last silently never fires.

fired = []
p = Lark(r'''
    start: (A | B | AKW | BKW)+
    A: /[a-z]+/
    B: /[0-9]+/
    AKW: "foo"
    BKW: "123"
    %ignore " "
''', parser='lalr', lexer='basic',
    lexer_callbacks={'A': lambda t: fired.append('A') or t,
                     'B': lambda t: fired.append('B') or t})
list(p.lex("abc 456"))
# before: fired == ['B']   (A dropped)
# after:  fired == ['A', 'B']

Root cause

BasicLexer chains an existing UnlessCallback with the user callback:

self.callback[type_] = CallChain(self.callback[type_], f, lambda t: t.type == type_)

The lambda t: t.type == type_ closes over the loop variable type_, so after the loop every CallChain condition compares against the last terminal's name. For all but the last terminal the condition is never true, so the user callback is skipped.

Fix

Bind type_ per iteration via a default argument (lambda t, type_=type_: ...).

Verification

  • Added TestGrammar.test_lexer_callbacks_with_keyword_terminals; it fails on master (only the last callback fires) and passes here.
  • Full test suite: 1107 passed, 167 skipped (no regressions).
  • Keyword absorption still works (the UnlessCallback continues to convert foo/123 to AKW/BKW, which correctly do not trigger the A/B callbacks).

When two or more terminals each absorb a same-priority keyword (producing
an UnlessCallback) and also have a lexer_callbacks entry, every user
callback except the last silently never fired.

BasicLexer built each CallChain with 'lambda t: t.type == type_', which
closes over the loop variable, so all conditions ended up comparing
against the last terminal's name. Bind type_ per iteration.

Fixes lark-parser#1618.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug in lexer_callbacks, causing some callbacks to be skipped in some grammars

1 participant