mako: Sporadic SyntaxException w/ Python3.11

We use mako for rendering CICD Pipeline Definitions (YAML files). It is a rather complex template w/ a lot of Mako-Function-Definitions and nesting + multithreading. The resulting documents vary in size (depending on input-parameters), and are typically between ~300 kiB and ~1.2 MiB.

The template was used w/ Python-versions 3.6 … 3.10. When upgrading to 3.11, we saw (and still see) sporadic SyntaxExceptions, which roughly occur in about 5% of the time (w/ unchanged template-parameters, of course!). I started working on a minimalised reproducer. If instantiating the same template w/ same parameters 64 times using 2 threads, I almost always see at least one exception stacktrace. The incriminated lines vary, whereas the Mako-part of the stacktrace seems to be always the same.

The error does not seem to occur when limiting concurrency to just one thread. Thus, I suspect a race-condition, probably within Mako’s codebase.

The error occurs for latest versions of python3 alpine (3.11.3-r11) when running inside a virtualisation container, and archlinux (3.11.3-1) when running natively.

Example Stacktrace

Traceback (most recent call last):
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/pyparser.py", line 36, in parse
    return _ast_util.parse(code, "<unknown>", mode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/_ast_util.py", line 91, in parse
    return compile(expr, filename, mode, PyCF_ONLY_AST)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SystemError: AST constructor recursion depth mismatch (before=63, after=65)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 140, in render
    definition_descriptor = self._render(definition_descriptor)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/shared_profile/src/sap/makobug-reproducer/concourse/replicator.py", line 211, in _render
    t = mako.template.Template(template_contents, lookup=self.lookup)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 300, in __init__
    (code, module) = _compile_text(self, text, filename)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 677, in _compile_text
    source, lexer = _compile(
                    ^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/template.py", line 657, in _compile
    node = lexer.parse()
           ^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 248, in parse
    if self.match_python_block():
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 392, in match_python_block
    self.append_node(
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/lexer.py", line 129, in append_node
    node = nodecls(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/parsetree.py", line 158, in __init__
    self.code = ast.PythonCode(text, **self.exception_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/ast.py", line 42, in __init__
    expr = pyparser.parse(code.lstrip(), "exec", **exception_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/redacted/.local/lib/python3.11/site-packages/mako/pyparser.py", line 38, in parse
    raise exceptions.SyntaxException(
mako.exceptions.SyntaxException: (SystemError) AST constructor recursion depth mismatch (before=63, after=65) ('import os\n\nimport oci.auth as oa\nimport model.cont') at line: 2 char: 1

It is probably worth mentioning that by decreasing the template’s output size, the likelihood of this error seems to become smaller.

I could share a copy of my somewhat slimmed-down reproducer; it still contains most of the code from the repository I referenced above, if this is considered helpful.