Metaprogramming in Python

Metaprogramming in Python involves the programmatic creation, modification, transformation, or control of program structure and behavior through mechanisms such as runtime structural changes, dynamic compilation and execution, abstract syntax tree (AST) transformation, dynamic class construction, class decorators, and metaclasses. These techniques enable programs to manipulate their own code or behavior during execution, enhancing abstraction and flexibility.

Foundations of Metaprogramming in Python

Metaprogramming is characterized by code that operates on the structure or behavior of other code and runtime program objects, going beyond ordinary computation which typically processes only application data. It enables programs to generate, modify, or adapt code dynamically, influencing how the program behaves or is structured while it runs.

Introspection is related but distinct: it involves observing program objects or execution state without altering them. Metaprogramming, on the other hand, actively creates, modifies, transforms, wraps, or controls program structures or behaviors.

Representative stages and techniques of metaprogramming include:

• Modifying existing runtime objects by adding or replacing attributes or methods.
• Compiling generated source code into executable objects.
• Transforming syntax trees to alter program structure before execution.
• Constructing classes programmatically using dynamic APIs.
• Decorating classes to add or modify behavior after creation.
• Participating in class creation through metaclasses, which control how classes are constructed.

While metaprogramming increases abstraction power and expressiveness, it can reduce transparency, complicate debugging, obscure interfaces, and introduce security risks if executable input is derived from untrusted sources.

Runtime Structural Modification in Python

Runtime structural modification means changing attributes, methods, bindings, or other supported structural properties of existing Python objects while the program is running. This is done dynamically and can affect object behavior immediately.

Python provides dynamic attribute operations through built-in functions:

• setattr(object, name, value): sets the attribute named name on object to value.
• getattr(object, name[, default]): retrieves the value of the attribute named name from object, or returns default if absent.
• delattr(object, name): deletes the attribute named name from object.

Unlike ordinary dotted attribute syntax (e.g., obj.attr), these functions allow the attribute name to be constructed programmatically as a string, enabling dynamic and flexible attribute manipulation.

Example of dynamically adding, reading, replacing, and deleting attributes on a user-defined object and class:

class MyClass:
    pass

obj = MyClass()

# Add attribute dynamically
setattr(obj, 'dynamic_attr', 42)
print(getattr(obj, 'dynamic_attr'))  # Output: 42

# Replace attribute
setattr(obj, 'dynamic_attr', 'replaced')
print(getattr(obj, 'dynamic_attr'))  # Output: replaced

# Delete attribute
delattr(obj, 'dynamic_attr')
print(hasattr(obj, 'dynamic_attr'))  # Output: False

# Add attribute to class
setattr(MyClass, 'class_attr', 99)
print(obj.class_attr)  # Output: 99

Class methods and attributes can also be added or replaced at runtime. Instances resolve attributes dynamically, so changes at the class level are visible to existing instances unless they have an overriding instance attribute.

Example demonstrating method addition and replacement after instance creation:

class Greeter:
    def greet(self):
        return "Hello"

g = Greeter()
print(g.greet())  # Output: Hello

# Replace method at runtime
def new_greet(self):
    return "Hi there"

setattr(Greeter, 'greet', new_greet)

print(g.greet())  # Output: Hi there

Monkey patching refers to runtime replacement or extension of existing attributes, often used to fix or adapt behavior temporarily. It is a deliberate, controlled form of modification and should be distinguished from uncontrolled changes that may affect unrelated implementation details.

Example of controlled monkey patching with restoration:

class Calculator:
    def add(self, x, y):
        return x + y

calc = Calculator()
print(calc.add(2, 3))  # Output: 5

# Save original method
original_add = Calculator.add

# Monkey patch method
def patched_add(self, x, y):
    return x + y + 1

Calculator.add = patched_add
print(calc.add(2, 3))  # Output: 6

# Restore original method
Calculator.add = original_add
print(calc.add(2, 3))  # Output: 5

Dynamic method binding differs depending on where the function is stored:

• A function assigned to a class becomes an unbound method; when accessed through an instance, it is bound automatically.
• A function stored directly on an instance is a plain attribute and is not automatically bound.
• Explicit binding can be done using types.MethodType.

Examples illustrating this:

import types

class MyClass:
    pass

def func(self):
    return f"Called func on {self}"

# Assign to class
MyClass.method = func
obj = MyClass()
print(obj.method())  # Output: Called func on <__main__.MyClass object at ...>

# Assign to instance
obj.inst_func = func
try:
    print(obj.inst_func())
except TypeError as e:
    print("Error:", e)  # Missing self argument

# Explicitly bind function to instance
obj.bound_func = types.MethodType(func, obj)
print(obj.bound_func())  # Output: Called func on <__main__.MyClass object at ...>

Class namespace mappings exposed for introspection (e.g., via __dict__) may not be directly mutable. Mutation should normally be performed through class attribute operations rather than by assuming that the namespace view itself can be altered.

Runtime registration patterns involve classes, functions, or handlers registering themselves or being registered into explicit mappings. This is a form of metaprogramming distinct from modifying the registered object itself; it organizes behavior dynamically.

Example of a simple runtime registry associating symbolic names with callable objects:

registry = {}

def register(name):
    def decorator(obj):
        registry[name] = obj
        return obj
    return decorator

@register('handler_one')
def handler_one():
    return "Handled by one"

@register('handler_two')
def handler_two():
    return "Handled by two"

# Dispatching
for name in ['handler_one', 'handler_two']:
    print(registry[name]())

Practical limits of runtime structural modification include:

• Immutable or restricted built-in types that do not allow attribute changes.
• Descriptor protocols that mediate attribute access and may prevent certain modifications.
• Inherited attributes that may shadow or complicate resolution.
• Cached assumptions by other code or the interpreter about object structure.
• Risks of maintenance challenges when changing structures expected to be stable by other components.

Dynamic Code Compilation and Execution in Python

Dynamic compilation and execution refer to converting source text or compatible syntax representations into executable code objects, followed optionally by evaluation or execution within specified namespace environments.

Dynamic Code Compilation in Python

The built-in compile function translates source text or an abstract syntax tree into a code object but does not execute it immediately.

compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1)

• source: Python source code as a string or an AST.
• filename: Used for error messages and debugging metadata.
• mode: Specifies the kind of code to compile:
  • 'exec': Compiles a sequence of statements.
  • 'eval': Compiles a single expression.
  • 'single': Compiles a single interactive statement.

Examples:

expr_code = compile('2 + 3', '<string>', 'eval')
print(type(expr_code))  # <class 'code'>

stmt_code = compile('a = 5\nb = 10\nc = a + b', '<string>', 'exec')
print(type(stmt_code))  # <class 'code'>

filename metadata is used in tracebacks and debugging tools to identify the origin of the compiled code. Compilation flags can enable features or optimizations, but these are advanced uses.

Syntax errors are detected at compilation time and raise SyntaxError. This differs from exceptions raised during execution of successfully compiled code.

Example contrasting invalid source rejected at compile time and valid source raising an exception at execution:

try:
    compile('if True print("missing colon")', '<string>', 'exec')
except SyntaxError as e:
    print("SyntaxError at compile:", e)

code = compile('1 / 0', '<string>', 'eval')
try:
    eval(code)
except ZeroDivisionError as e:
    print("Error at execution:", e)

Compiled code objects can be reused to evaluate the same code multiple times in different namespaces without recompilation, improving efficiency in some workloads. However, this does not guarantee safety or performance in all dynamic execution scenarios.

Dynamic Expression Evaluation in Python

The eval function evaluates an expression or compatible code object and returns its result.

eval(expression, globals=None, locals=None)

• expression: A string or code object compiled in 'eval' mode.
• globals and locals: Optional dictionaries defining execution environment namespaces.

Name resolution during evaluation depends on these mappings.

Examples:

expr = 'x + y'
code = compile(expr, '<string>', 'eval')

namespace1 = {'x': 1, 'y': 2}
print(eval(code, namespace1))  # Output: 3

namespace2 = {'x': 10, 'y': 20}
print(eval(code, namespace2))  # Output: 30

Restricting the namespace dictionaries passed to eval does not by itself establish a secure sandbox. Executable input from untrusted sources must never be trusted regardless of such restrictions.

ast.literal_eval is a safer alternative restricted to parsing and evaluating Python literals and container displays. It rejects arbitrary expressions.

Example:

import ast

print(ast.literal_eval('[1, 2, 3]'))  # Output: [1, 2, 3]

try:
    ast.literal_eval('__import__("os").system("echo hack")')
except ValueError as e:
    print("Rejected by literal_eval:", e)

Dynamic evaluation is appropriate only when behavior genuinely depends on expression evaluation and cannot be represented clearly by explicit data structures, mappings, functions, or control flow.

Dynamic Code Execution in Python

The exec function executes dynamically supplied statements or compatible code objects in a specified namespace environment.

exec(object, globals=None, locals=None)

• object: Source code string or code object compiled in 'exec' mode.
• globals and locals: Namespace dictionaries controlling variable bindings.

Unlike eval, which returns an expression value, exec creates, replaces, or uses bindings in supplied namespaces.

Example:

namespace = {}
exec('def greet(name): return "Hello, " + name', namespace)
print(namespace['greet']('Alice'))  # Output: Hello, Alice

exec accepts either a single namespace dictionary (used for both globals and locals) or separate globals and locals mappings. The difference affects variable scope and resolution.

Example contrasting one shared namespace vs separate globals and locals:

code = 'x = 5\ny = 10\nz = x + y'

# One shared namespace
ns = {}
exec(code, ns)
print(ns['z'])  # Output: 15

# Separate globals and locals
globals_ns = {}
locals_ns = {}
exec(code, globals_ns, locals_ns)
print('z' in globals_ns)  # False
print(locals_ns['z'])     # 15

Dynamic code execution should not substitute parsing structured data, configuration files, dispatch tables, or templates that can be represented without executing arbitrary code.

Security and auditability risks of eval and exec include:

• Execution of arbitrary operations.
• Access to supplied capabilities and environment.
• Hidden dependencies and side effects.
• Difficult static analysis.
• No general secure sandbox just by restricting namespaces.

Python Abstract Syntax Tree Metaprogramming

Python abstract syntax trees (ASTs) are structured representations of parsed source code syntax that can be inspected, constructed, transformed, and compiled without treating source as unstructured text.

The ast.parse function parses source code into an AST node tree, separating parsing success from later execution.

Example parsing a small function and inspecting its tree:

import ast

source = '''
def square(x):
    return x * x
'''

tree = ast.parse(source)
print(ast.dump(tree, indent=4))

AST nodes consist of types representing syntax elements, each with fields and child nodes. Source-location metadata such as line and column numbers help relate nodes to original source.

Traversal can be done broadly with ast.walk, or with specialized visitation using ast.NodeVisitor, which calls type-specific methods.

Example NodeVisitor that counts function definitions:

class FuncCounter(ast.NodeVisitor):
    def __init__(self):
        self.count = 0

    def visit_FunctionDef(self, node):
        self.count += 1
        self.generic_visit(node)

source = 'def f(): pass\ndef g(): pass'
tree = ast.parse(source)
counter = FuncCounter()
counter.visit(tree)
print(counter.count)  # Output: 2

ast.NodeTransformer is a visitor class specialized for replacing, removing, or rewriting nodes by returning new nodes or None.

Example AST transformation replacing calls to old_func with new_func:

class ReplaceOldFunc(ast.NodeTransformer):
    def visit_Call(self, node):
        if isinstance(node.func, ast.Name) and node.func.id == 'old_func':
            node.func.id = 'new_func'
        return self.generic_visit(node)

source = 'old_func(1)'
tree = ast.parse(source)
transformer = ReplaceOldFunc()
new_tree = transformer.visit(tree)
ast.fix_missing_locations(new_tree)
compiled = compile(new_tree, '<string>', 'exec')
exec(compiled)

Preserving source-location metadata is important for debugging and error reporting. Utilities like ast.copy_location and ast.fix_missing_locations help maintain or repair metadata when constructing new nodes.

Extending the above example with source location handling:

class ReplaceOldFunc(ast.NodeTransformer):
    def visit_Call(self, node):
        if isinstance(node.func, ast.Name) and node.func.id == 'old_func':
            new_node = ast.Call(
                func=ast.Name(id='new_func', ctx=ast.Load()),
                args=node.args,
                keywords=node.keywords)
            return ast.copy_location(new_node, node)
        return self.generic_visit(node)

source = 'def test(): return old_func(42)'
tree = ast.parse(source)
transformer = ReplaceOldFunc()
new_tree = transformer.visit(tree)
ast.fix_missing_locations(new_tree)

namespace = {}
exec(compile(new_tree, '<string>', 'exec'), namespace)

# Assuming new_func is defined
def new_func(x):
    return x + 1

namespace['new_func'] = new_func
print(namespace['test']())  # Output: 43

Direct AST construction involves creating syntax nodes programmatically, but the resulting trees must satisfy structural and semantic requirements expected by the compiler.

ast.unparse generates equivalent Python source text from a supported AST, though formatting, comments, and exact lexical details may differ.

Example parse-transform-unparse:

source = 'x = 1 + 2'
tree = ast.parse(source)

# Transform: replace 2 with 3
class ReplaceTwo(ast.NodeTransformer):
    def visit_Constant(self, node):
        if node.value == 2:
            return ast.copy_location(ast.Constant(value=3), node)
        return node

transformer = ReplaceTwo()
new_tree = transformer.visit(tree)
ast.fix_missing_locations(new_tree)

new_source = ast.unparse(new_tree)
print(new_source)  # Output: x = 1 + 3

AST transformation is generally more reliable than naive textual replacement for syntax-aware modifications but does not preserve every lexical or formatting detail.

AST validation concerns include:

• Invalid node contexts.
• Malformed node combinations.
• Missing required fields.
• Transformations that alter semantics unexpectedly while remaining syntactically valid.

Complete solved example parsing, transforming, compiling, and verifying:

source = '''
def increment(x):
    return x + 1
'''

import ast

class IncrementTransformer(ast.NodeTransformer):
    def visit_Return(self, node):
        # Replace "return x + 1" with "return x + 2"
        if (isinstance(node.value, ast.BinOp) and
            isinstance(node.value.op, ast.Add) and
            isinstance(node.value.right, ast.Constant) and
            node.value.right.value == 1):
            new_value = ast.BinOp(
                left=node.value.left,
                op=ast.Add(),
                right=ast.Constant(value=2))
            return ast.copy_location(ast.Return(value=new_value), node)
        return node

tree = ast.parse(source)
transformer = IncrementTransformer()
new_tree = transformer.visit(tree)
ast.fix_missing_locations(new_tree)

namespace = {}
exec(compile(new_tree, '<string>', 'exec'), namespace)

print(namespace['increment'](3))  # Output: 5

Class Metaprogramming in Python

Class metaprogramming involves programmatic creation, transformation, registration, validation, or control of class objects and class-definition behavior.

Dynamic Class Construction in Python

The three-argument form of type creates a class object dynamically:

MyDynamicClass = type('MyDynamicClass', (object,), {'attr': 42, 'method': lambda self: self.attr})

Example creating a dynamic class with attributes and methods:

MyClass = type('MyClass', (object,), {
    'greet': lambda self: "Hello"
})

obj = MyClass()
print(obj.greet())  # Output: Hello

Base classes and namespace entries can be selected dynamically. Classes created this way participate normally in inheritance, descriptors, methods, and metaclass selection.

types.new_class is a higher-level API that allows dynamic class creation and namespace population through a callback.

Example using types.new_class:

import types

def ns_populator(ns):
    ns['value'] = 100
    def method(self):
        return self.value
    ns['method'] = method

DynamicClass = types.new_class('DynamicClass', (), {}, ns_populator)
instance = DynamicClass()
print(instance.method())  # Output: 100

__init_subclass__ is a class-level hook called when subclasses are created. It can be used to validate, register, or configure subclasses without requiring a custom metaclass.

Example base class using __init_subclass__ to register subclasses:

class Base:
    subclasses = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.subclasses.append(cls)

class SubA(Base):
    pass

class SubB(Base):
    pass

print(Base.subclasses)  # Output: [<class '__main__.SubA'>, <class '__main__.SubB'>]

Class Decorators in Python

A class decorator is a callable applied to a newly created class object; the returned object replaces the original class binding.

Example decorator adding a class-level capability:

def add_metadata(cls):
    cls.metadata = {'version': 1.0}
    return cls

@add_metadata
class MyClass:
    pass

print(MyClass.metadata)  # Output: {'version': 1.0}

Parameterized class decorators are callable factories returning decorators capturing configuration.

Example parameterized decorator registering classes with a name:

registry = {}

def register_class(name):
    def decorator(cls):
        registry[name] = cls
        return cls
    return decorator

@register_class('special')
class SpecialClass:
    pass

print(registry)  # Output: {'special': <class '__main__.SpecialClass'>}

Stacked class decorators apply in bottom-up order (closest to the class first), transforming the class post-creation. This differs from participation in class creation itself.

Metaclass-Based Metaprogramming in Python

A metaclass is the class of a class; it participates in the process constructing class objects.

Metaclass selection depends on:

• Explicitly specified metaclass in the class definition.
• Metaclasses of base classes, which must be compatible with the selected metaclass.

The __prepare__ method of a metaclass optionally provides the namespace mapping used while the class body executes, distinct from creating the final class object.

__new__ and __init__ methods of a metaclass inspect or transform class-construction inputs and initialize the resulting class object. They differ from instance-level __new__ and __init__ which control instance creation.

Example custom metaclass validating a class definition and adding a class-level property:

class ValidatingMeta(type):
    def __new__(mcls, name, bases, namespace):
        if 'required_attr' not in namespace:
            raise TypeError(f"{name} must define 'required_attr'")
        cls = super().__new__(mcls, name, bases, namespace)
        cls.custom_property = 'added_by_metaclass'
        return cls

class Good(metaclass=ValidatingMeta):
    required_attr = 123

print(Good.custom_property)  # Output: added_by_metaclass

try:
    class Bad(metaclass=ValidatingMeta):
        pass
except TypeError as e:
    print(e)  # Output: Bad must define 'required_attr'

Metaclass __call__ can influence what happens when the class object is called to create instances, but it is often clearer to use ordinary constructors or factories for such behavior.

Choosing among ordinary class definitions, __init_subclass__, class decorators, dynamic class construction, and metaclasses depends on the required level of control, preferring the least invasive mechanism that expresses the needed behavior clearly.

Solved Metaprogramming Exercises in Python

The following example accepts a trusted declarative specification, dynamically constructs several related classes without using eval or exec, registers them, then applies a class decorator for validation and enrichment.

registry = {}

def register_class(cls):
    registry[cls.__name__] = cls
    return cls

def validate_has_method(method_name):
    def decorator(cls):
        if not callable(getattr(cls, method_name, None)):
            raise TypeError(f"Class {cls.__name__} must define method '{method_name}'")
        return cls
    return decorator

# Declarative specification for shapes
shapes_spec = {
    'Circle': {
        'radius': 0,
        'area': lambda self: 3.1415 * self.radius ** 2
    },
    'Square': {
        'side': 0,
        'area': lambda self: self.side ** 2
    }
}

created_classes = {}

for name, attrs in shapes_spec.items():
    # Separate methods and fields
    namespace = {}
    for key, value in attrs.items():
        namespace[key] = value

    # Dynamically create class
    cls = type(name, (object,), namespace)

    # Register and validate with decorators
    cls = register_class(cls)
    cls = validate_has_method('area')(cls)

    created_classes[name] = cls

# Use classes
circle = created_classes['Circle']()
circle.radius = 3
print(f"Circle area: {circle.area():.2f}")  # Output: Circle area: 28.27

square = created_classes['Square']()
square.side = 4
print(f"Square area: {square.area()}")      # Output: Square area: 16

print("Registered classes:", list(registry.keys()))

Step-by-step explanation:

• The declarative specification shapes_spec defines classes by name with attributes and methods.
• For each spec, a namespace mapping is constructed.
• Classes are created dynamically with type using the name, base classes, and namespace.
• Each class is registered in a global registry via a decorator.
• Another decorator validates that each class defines an area method.
• Instances are created normally, and the dynamic behavior is exercised.
• This approach uses direct structural APIs, avoiding the risks and complexity of generating and executing source code.

This exercise demonstrates the power, safety, and clarity of metaprogramming using Python’s runtime structural APIs and class decorators without resorting to eval or exec.