Skip to content

Conversation

@eb8680
Copy link
Contributor

@eb8680 eb8680 commented Jun 17, 2025

Supercedes #242

As the first (and primary) step toward full and robust support for comprehension syntax overloading, this PR adds a self-contained module effectful.internals.disassembly that performs symbolic execution of the CPython VM bytecode of a compiled generator expression object in order to extract an equivalent Python ast.AST.

It has one public-facing function disassemble which takes a generator expression and returns an AST:

def disassemble(genexpr: Generator[T, None, None]) -> ast.Expression: ...

This function should satisfy the following:

genexpr: Generator[T, None, None] = ...

genexpr_ast: ast.Expression = disassemble(genexpr)
assert isinstance(genexpr_ast.body, ast.GeneratorExp)

genexpr_ast_code: types.CodeType = compile(genexpr_ast, "<>", "eval")
reconstructed_genexpr: Generator[T, None, None] = eval(genexpr_ast_code, genexpr.gi_frame.f_globals, genexpr.gi_frame.f_locals)

assert materialize(genexpr) == materialize(reconstructed_genexpr)  # materialize recursively calls list()

It currently supports a large subset of valid comprehension syntax in Python 3.12 and 3.13, including filter expressions (e.g. if predicate(x) in (f(x) for x in xs if predicate(x))), nested loops, nested comprehensions, unpacking, structured data, mixed comprehension types, inline lambdas and more.

The main features missing right now from the symbolic executor are conditional expressions (a if b else c), multiple and/or-based conditions and chained comparisons (e.g. a < b < c), which require handling jumps with a little more care.

Remaining tasks:

  • Add Python 3.12 support
  • Drop Python 3.10 support
  • Add Python 3.13 CI build Add parallel CI builds for Python 3.12 and 3.13 #326
  • Add Python 3.14 support
  • Support fstring expressions
  • Support conditional expressions
  • Support lazy conditions
  • Support chained comparisons
  • Add more test cases, both passing and xfailing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request status:WIP

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants