AssertionError when calling UnexpectedInput.get_context

[nchammas]

Quick question before I try to boil this down to a minimal reproduction.

I am trying to gracefully handle certain errors when parsing my language, and I hit a case where UnexpectedInput.get_context is itself failing one of its internal checks.

lark.exceptions.UnexpectedToken: Unexpected token Token('_DEDENT', '') at line None, column None.
Expected one of: 
        * _RPAR


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../app/home.py", line 150, in query_details
    print(e.get_context(query))
          ^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/lark/exceptions.py", line 63, in get_context
    assert self.pos_in_stream is not None, self
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Unexpected token Token('_DEDENT', '') at line None, column None.
Expected one of: 
        * _RPAR

Are there any cases where UnexpectedInput.get_context "should" fail in this manner?

[erezsh]

I don't think there's a good reason that get_context should fail like that.

If I had to guess, it's probably that the postlexer isn't assigning the pos_in_stream to the dedent tokens.Maybe it should instead assign the positions of the previous token.

[nchammas]

Here's a reproduction derived from the indented tree example:

from lark import Lark
from lark.indenter import Indenter
from lark.exceptions import UnexpectedInput

tree_grammar = r"""
    %import common.CNAME -> NAME
    %import common.WS_INLINE
    %import common.SH_COMMENT
    %import common.ESCAPED_STRING
    %ignore WS_INLINE
    %ignore SH_COMMENT
    %declare _INDENT _DEDENT

    ?start: _NL* tree
    tree: NAME _LPAR ESCAPED_STRING _RPAR _NL [_INDENT tree+ _DEDENT]
    _NL: (/\r?\n[\t ]*/ | SH_COMMENT)+
    _LPAR: "("
    _RPAR: ")"
"""

class TreeIndenter(Indenter):
    NL_type = '_NL'
    OPEN_PAREN_types = ['_LPAR']
    CLOSE_PAREN_types = ['_RPAR']
    INDENT_type = '_INDENT'
    DEDENT_type = '_DEDENT'
    tab_len = 8

parser = Lark(tree_grammar, parser='lalr', postlex=TreeIndenter())

test_tree = """
a("hello")
    b("poop"
"""

def test():
    try:
        print(parser.parse(test_tree).pretty())
    except UnexpectedInput as e:
        print(e.get_context(test_tree))
        raise

if __name__ == '__main__':
    test()

Running this yields:

Traceback (most recent call last):
  File ".../lark/lark/parsers/lalr_parser_state.py", line 77, in feed_token
    action, arg = states[state][token.type]
                  ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '_DEDENT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lark/examples/indented_tree.py", line 38, in test
    print(parser.parse(test_tree).pretty())
          ^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/lark.py", line 659, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/parser_frontends.py", line 123, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/parsers/lalr_parser.py", line 42, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/parsers/lalr_parser.py", line 88, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/parsers/lalr_parser.py", line 111, in parse_from_state
    raise e
  File ".../lark/lark/parsers/lalr_parser.py", line 102, in parse_from_state
    state.feed_token(token)
  File ".../lark/lark/parsers/lalr_parser_state.py", line 80, in feed_token
    raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
lark.exceptions.UnexpectedToken: Unexpected token Token('_DEDENT', '') at line None, column None.
Expected one of: 
        * _RPAR


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../lark/examples/indented_tree.py", line 44, in <module>
    test()
  File ".../lark/examples/indented_tree.py", line 40, in test
    print(e.get_context(test_tree))
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lark/lark/exceptions.py", line 63, in get_context
    assert self.pos_in_stream is not None, self
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Unexpected token Token('_DEDENT', '') at line None, column None.
Expected one of: 
        * _RPAR

Assuming this is a bug, should we somehow convert this discussion into an issue?

[nchammas]

@erezsh - Does it make sense to convert this discussion into an issue?

[erezsh]

Yes, it makes sense.

But anyway I just created a PR that fixes it: #1547