Python Grammar

Python grammar is the syntactic system that specifies how lexical tokens can be combined into valid statements, expressions, patterns, targets, and complete syntactic forms. It defines the structural rules governing token arrangement without conflating grammatical structure with the formation of lexical tokens or with the runtime meaning of constructs. Grammar describes the shape and relationships of language constructs independently from their lexical validity or execution semantics.

Foundations of Python Grammar

Grammar consists of a set of named production rules that describe the permitted structural relationships among tokens and subordinate grammatical forms. Each production defines how a higher-level syntactic unit can be constructed from lower-level units or tokens, ensuring that only structurally valid sequences are recognized.

It is important to distinguish among three related but distinct notions:

• Lexical validity: Recognition that a sequence of characters forms a legal token (such as a valid identifier, keyword, literal, or operator).
• Grammatical validity: Conformance of a sequence of tokens to the syntactic rules that combine tokens into statements, expressions, patterns, or targets.
• Runtime semantic validity: The correctness of a construct’s meaning or behavior during execution, which is not guaranteed by grammar alone.

Thus, lexical validity is necessary but not sufficient for grammatical validity, and grammatical validity is necessary but not sufficient for runtime semantic correctness.

Python grammar assigns major roles to statements, expressions, patterns, and targets:

• Statements represent executable instructions or declarations, arranged in sequences or blocks.
• Expressions produce values and include operations, function calls, and literals.
• Patterns describe structural forms used in pattern matching (e.g., in match statements).
• Targets specify syntactic positions that can bind names or receive assigned values.

While these categories reference one another (for example, expressions appear within statements and patterns), they remain distinct grammatical classes with unique constraints and are not interchangeable.

Python grammar recognizes different top-level parser entry forms for various input contexts:

• File input: Represents a full Python source file, parsed as a sequence of statements.
• Interactive input: Supports single or compound statements in an interactive REPL.
• Expression input: Parses a single expression, for use in evaluation contexts.

Each input type begins parsing with a different grammar rule to suit the expected syntactic structure.

Python grammar can encode precedence, grouping, allowed placement, repetition, alternatives, and structural restrictions within its production rules. However, grammar alone does not specify every runtime effect or semantic consequence of the resulting constructs; it only constrains what sequences of tokens are structurally valid.

Python Grammar Notation

Python grammar notation is a mixture of Extended Backus-Naur Form (EBNF)-like conventions combined with Parsing Expression Grammar (PEG) concepts. This hybrid notation precisely describes production rules and parser choices.

A named production consists of a rule name followed by a colon (:) and one or more alternatives separated by the ordered choice operator (|), specifying what the rule can match.

• Lowercase rule references refer to subordinate grammatical productions.
• Uppercase token references refer to lexical token categories (e.g., NAME, NUMBER).

Literal text in grammar notation represents fixed tokens and is distinguished as follows:

• Hard-keyword spellings: Reserved keywords that must appear exactly as spelled.
• Soft-keyword spellings: Context-dependent keywords that may be recognized as tokens only in specific grammar contexts.
• Punctuation tokens: Symbols such as :, ,, . treated as tokens by the lexer.

Juxtaposition represents a sequence of elements that must appear in order. The vertical bar | is PEG ordered choice, meaning alternatives are attempted in order from left to right and the first matching alternative is selected.

Grouping parentheses ( ... ) group elements to form subexpressions. Optional forms use square brackets [ ... ] or a trailing ?, indicating zero or one occurrence. Zero-or-more repetition is denoted with *, and one-or-more repetition with +.

Separator repetition notation, such as s.e+, means one or more occurrences of e separated by s. The separator s is omitted from the parse tree sequence generated by that shorthand.

Positive lookahead &e requires that the element e matches at the current input position without consuming input, while negative lookahead !e forbids a match of e at that position.

The PEG cut operator ~ commits the parser to the current alternative after the cut, preventing backtracking beyond that point. Eager parsing notation is used selectively and only to the depth necessary to read the current grammar fully.

Example invented productions illustrating these elements:

# A simple list of identifiers separated by commas
id_list: NAME (',' NAME)*

# An optional trailing semicolon
stmt_end: ';'?

# A pattern matching either 'foo' or 'bar' literal tokens
foo_bar_pattern: 'foo' | 'bar'

# A repeated sequence of digits with optional sign
signed_digits: ['+' | '-'] DIGIT+

# A lookahead that requires a 'def' keyword without consuming it
check_def: &('def') NAME

# A cut operator to commit after matching 'async'
async_func: 'async' ~ 'def' NAME '(' [param_list] ')'

Decoded into plain structural language:

• id_list matches one or more names separated by commas.
• stmt_end optionally matches a semicolon at the end of a statement.
• foo_bar_pattern matches either the literal word 'foo' or 'bar'.
• signed_digits matches an optional sign followed by one or more digits.
• check_def requires the next token to be 'def' but does not consume it.
• async_func matches the keyword 'async', commits parsing here, then matches 'def' followed by a function name and optional parameters.

Python Statement Grammar

A Python statement is grammatically either a simple-statement sequence or a compound statement. This classification is structural and distinct from the runtime action performed by the statement.

Python Simple Statement Grammar

Simple statements are forms that participate in a logical-line-oriented sequence and can be separated by semicolons.

A simplified grammar fragment:

simple_stmts: simple_stmt (';' simple_stmt)* ';'? NEWLINE

simple_stmt:
    assignment_stmt
  | expression_stmt
  | return_stmt
  | import_stmt
  | raise_stmt
  | pass_stmt
  | del_stmt
  | yield_stmt
  | assert_stmt
  | break_stmt
  | continue_stmt
  | global_stmt
  | nonlocal_stmt

• NEWLINE marks the end of the logical line.
• Optional trailing semicolons separate multiple simple statements on one line.
• Each alternative corresponds to a distinct simple-statement family.

Principal simple-statement grammatical families include:

• Assignment: Binding values to targets.
• Type alias: Annotated assignments.
• Expression-oriented: Standalone expressions.
• Return, import, raise, pass, deletion, yield, assert, break, continue, global, and nonlocal statements.

Multiple simple statements may occupy one logical line when separated by semicolons, which is distinct from suites containing nested compound statements.

Assignment-statement grammar includes:

• Annotated assignment forms with type hints.
• Chained target assignment (a = b = value).
• Augmented assignments (+=, *=).
• Target restrictions limiting valid syntactic target forms.
• Right-hand side expressions representing values to assign.

Valid examples:

a = 1
b = c = 2
x: int = 3
y += 4
pass; continue; break
del obj.attr

Invalid examples (grammatically):

1 = a              # Invalid target
a +=              # Missing right-hand side
x: int 3           # Missing '=' in annotated assignment
a = b =            # Missing right-hand side expression
pass; return 42    # Semicolon before a compound statement header

Return, yield, raise, assert, break, continue, pass, global, and nonlocal statements have defined grammatical shapes emphasizing optional or repeated syntactic components:

• return [expression_list]
• yield [expression_list]
• raise [expression [from expression]]
• assert expression [, expression]
• break
• continue
• pass
• global name_list
• nonlocal name_list

Import statement grammar includes:

• import followed by one or more dotted names optionally aliased.
• from import forms with module names, relative dots, parenthesized import targets, and optional trailing commas.

Python Compound Statement Grammar

Compound statements contain one or more clauses or blocks, including:

• Function definitions (def)
• Class definitions (class)
• Conditionals (if, elif, else)
• Loops (while, for)
• Context-management statements (with)
• Exception-handling statements (try, except, else, finally)
• Match statements (match, case)

Clauses and blocks are structured with:

• Keyword-led headers.
• Colons (:) marking header ends.
• Logical-line termination tokens.
• Indentation tokens marking the start and end of blocks.
• Subordinate statements forming the block body.

Valid examples:

if x > 0:
    print(x)
elif x == 0:
    print("zero")
else:
    print("negative")

while condition:
    do_something()

for item in iterable:
    process(item)

with open(file) as f:
    data = f.read()

try:
    risky_operation()
except ValueError:
    handle_error()
else:
    no_error()
finally:
    cleanup()

def func(a, b):
    return a + b

class MyClass(Base):
    pass

match command:
    case 'start':
        start()
    case 'stop' | 'exit':
        stop()

Conditional and loop grammar includes if and elif alternatives, optional else clauses, while blocks, for targets and iterable expressions, and asynchronous variants (async for, async with).

with, try, and match grammars cover repeated context items, exception or exception-group clauses, optional else and finally forms, match subjects, case blocks, patterns, and optional guards.

Function and class definition grammar includes names, optional type-parameter lists, parameter or argument-related forms, optional base or keyword arguments, return annotations, decorators, and blocks.

Async variants (async def, async for, async with) are grammatically represented as prefixed keywords and distinguished syntactically from asynchronous execution behavior.

Deliberately invalid examples:

if x > 0        # Missing colon
    print(x)

def func()
    pass       # Missing colon

a = 1; if b:    # Semicolon before compound statement header invalid
    do_something()

for x in        # Missing iterable expression
    print(x)

a +=            # Missing right-hand side expression

del 1           # Invalid deletion target

A grammatically valid statement may still be invalid under semantic or contextual restrictions not fully encoded in the most general production.

Python Expression Grammar

Expression grammar defines a hierarchy of productions forming values, operations, calls, indexing, displays, comprehensions, conditional forms, lambdas, assignment expressions, and related structures.

Expression lists include:

• Starred expressions (*expr)
• Named expressions (NAME := expr)
• Comma-separated sequences where commas are syntactically significant to distinguish single expressions from tuples.

Assignment-expression grammar uses the form NAME := expression with restrictions distinct from ordinary assignment targets.

Conditional expressions and lambda expressions form recursive productions within the broader expression hierarchy, with grammar encoding their position and grouping.

Boolean operators are encoded through separate productions for disjunction (or), conjunction (and), inversion (not), and comparison forms, structurally encoding precedence.

Other grammar layers include:

• Bitwise OR, XOR, AND.
• Shifts (<<, >>).
• Addition and subtraction.
• Multiplication, division, modulo.
• Unary factors and exponentiation.

Precedence emerges from nested productions, not from runtime evaluation order or semantics.

await expressions have a specific grammatical placement distinct from runtime contextual restrictions.

Primary expressions and atoms form the syntactic foundation for:

• Names
• Literals (strings, numbers, constants)
• Grouping parentheses
• Displays (lists, sets, dictionaries)
• Calls
• Attribute references
• Subscriptions
• Generator expressions

Recursive primary grammar allows repeated attribute access, calls, subscriptions, and generator-expression attachment, enabling chained structures like f(x)[y].attr().

Slice grammar includes optional lower, upper, and step expressions, distinguishing it from ordinary named expressions in subscription positions.

Display grammar covers tuple, list, set, and dictionary forms with syntactic significance given to commas, colons, starred items, and double-starred expansions.

Comprehension and generator-expression grammar involves an initial expression or key-value pair followed by one or more for or async for clauses and optional filtering clauses.

Call-argument grammar includes positional expressions, starred expressions, keyword arguments, and double-starred arguments with structural ordering constraints.

Atoms may be ordinary strings, formatted strings, template strings, numeric tokens, singleton constants, names, collection displays, or the ellipsis token.

Example expressions of increasing grammatical depth:

42
x + y * z
f(a, b=2, *args, **kwargs)
[a for x in iterable if x > 0]
(lambda x: x**2)(5)
result := (a if cond else b)
await coro()

Grouping parentheses can serve to group expressions, form tuples, or start generator expressions, distinguished by the presence of commas or comprehension clauses.

Invalid expression examples:

[x for in iterable]         # Missing variable after 'for'
f(a=1, 2)                  # Positional argument after keyword argument
x if else y                # Missing condition in conditional expression
a := b := 3                # Nested assignment expression invalid
(1, 2                   # Unmatched parenthesis

Precedence encoded by the grammar differs from runtime evaluation order, short-circuit behavior, operator method dispatch, numeric semantics, and side effects.

Python Pattern Grammar

Pattern grammar is the syntactic system used after a case soft keyword to describe structural pattern forms including literal, capture, wildcard, value, sequence, mapping, class, OR, AS, and related patterns.

Top-level pattern grammar includes productions named patterns, pattern, AS patterns, OR patterns, and closed patterns. Alternatives compose larger pattern structures without overlapping categories.

Principal closed-pattern grammatical families are:

• Literal patterns: Signed numeric literals, complex-number forms, supported string forms, singleton spellings.
• Capture patterns: Binding names to matched values.
• Wildcard patterns: The _ token, syntactically distinguished.
• Value patterns: Dotted names or attribute references representing values.
• Group patterns: Parenthesized grouping.
• Sequence patterns: Comma-sensitive sequences, bracketed lists with optional starred subpatterns.
• Mapping patterns: Key-pattern pairs with optional double-star capture for remaining mapping content.
• Class patterns: Class-like value reference with positional and keyword subpatterns.
• OR patterns: Alternatives separated by |.
• AS patterns: Patterns with optional capture binding.

Literal-pattern grammar includes signed numeric forms, complex-number source forms, supported string forms, and singleton constants, differing from ordinary expression syntax.

Capture, wildcard, and value-pattern grammar distinguishes the _ wildcard from ordinary capture names, and dotted value patterns from bare captures.

Group and sequence pattern grammar includes parenthesized grouping, comma-sensitive sequences, bracketed sequences, and optional starred subpatterns.

Mapping-pattern grammar includes key-pattern pairs, optional double-star capture, and restricted key syntax.

Class-pattern grammar includes a class-like value reference, positional subpatterns, keyword subpatterns, and ordering constraints.

OR and AS pattern grammar enforce syntactic restrictions on capture targets to avoid malformed or ambiguous binding.

Valid modern Python pattern examples:

case 42:                          # Literal pattern
case x:                           # Capture pattern
case _:                           # Wildcard pattern
case module.CONSTANT:             # Value pattern
case (x, y, *rest):               # Sequence pattern with starred subpattern
case {'key': value, **rest}:     # Mapping pattern with double-starred capture
case Point(x, y):                 # Class pattern with positional subpatterns
case A | B:                      # OR pattern
case pattern as binding:         # AS pattern

Invalid pattern examples:

case x | y | z | a | b:           # OR pattern too many alternatives without grouping
case *x, *y:                     # Multiple starred sequence patterns invalid
case {**x, 'key': value}:         # Mapping rest not last or duplicated
case Point(x, y=2, 3):            # Class pattern positional after keyword arg invalid
case (x as y) as z:               # Nested AS pattern invalid

literal_pattern:
    [sign] NUMBER
  | STRING
  | 'True' | 'False' | 'None'

capture_pattern:
    NAME
  | '_'

wildcard_pattern:
    '_'

value_pattern:
    dotted_name

group_pattern:
    '(' [pattern (',' pattern)* [',']] ')'

sequence_pattern:
    '[' [pattern (',' pattern)* [',']] ']'

mapping_pattern:
    '{' [key_pattern (',' key_pattern)* [',' [starred_key_pattern]] ']'}'

class_pattern:
    dotted_name '(' [pattern (',' pattern)*] [',' [keyword_pattern (',' keyword_pattern)*]] ')'

or_pattern:
    pattern '|' pattern

as_pattern:
    pattern ['as' NAME]

Match guards are syntactically associated with a case block but are expressions, not patterns, and are not folded into the pattern grammar.

Contextual soft keywords like case and match are recognized by the parser in specific contexts without being reserved keywords globally.

Patterns are distinct from expressions even when source forms resemble literals, names, attributes, calls, sequences, or mappings.

Python Target Grammar

Target grammar defines restricted syntactic forms allowed in positions that bind, assign, unpack, iterate into, context-bind, or delete names, attributes, subscriptions, and nested target structures.

Generic starred-target grammar includes:

• star_targets: sequences of star_target separated by commas.
• star_target: target forms that may include a starred atom.
• Target atoms: names, attributes, subscriptions, or parenthesized/nested tuple or list structures.

Names, attribute references, and subscription forms are principal non-starred target shapes. Target primaries differ from unrestricted general expressions by their syntactic restrictions.

Parenthesized, tuple-like, and list-like target grammar uses commas and nesting to form unpacking targets without encoding runtime unpacking mechanisms.

Starred-target syntax allows exactly one starred subtarget in an unpacking pattern, with placement restrictions to avoid ambiguity.

single_target and single attribute-or-subscript target grammars are narrower forms used where augmented or annotated assignment requires a single admissible target.

Deletion-target grammar differs from assignment-target grammar by excluding starred targets. Valid deletion targets include names, attributes, subscriptions, and parenthesized, tuple-like, or list-like sequences.

Target productions are reused syntactically by assignment statements, for clauses, comprehensions, and with ... as ... forms. However, this reuse does not imply semantic equivalence of those constructs.

Solved Python Grammar Exercises

# Statement: assignment with expression
a, b = 1, 2

# Statement: for loop with target and expression
for x in iterable:
    print(x)

# Statement: match with several patterns
match command:
    case 42:
        print("Answer")
    case x | y:
        print("Variable")
    case Point(x, y):
        print(f"Point at {x}, {y}")
    case _:
        print("Default")

# Statement: deletion target
del a.b[0]

# Invalid statement (invalid target deletion)
del 42

# Invalid pattern (multiple starred subpatterns)
case (x, *y, *z):
    pass

Step-by-step explanation:

• Tokens a, b, =, 1, ,, 2 form an assignment statement with a tuple-like target and comma-separated expressions.
• The for statement uses x as a target grammar form and iterable as an expression grammar form.
• The match statement introduces pattern grammar after case:
  • 42 is a literal pattern.
  • x | y is an OR pattern.
  • Point(x, y) is a class pattern with positional subpatterns.
  • _ is a wildcard pattern.
• The del statement deletes a target consisting of an attribute and subscription.
• The invalid del 42 fails because 42 is not a valid deletion target.
• The invalid pattern (x, *y, *z) fails because multiple starred subpatterns are not allowed in a sequence pattern.

Each invalid variant fails the grammar at the corresponding production: target forms for deletion, or pattern forms for sequence patterns.