Python Lexical Structure

Python lexical structure defines the rules by which encoded source text is decoded into Unicode source characters, organized into physical and logical lines, and transformed into tokens such as names, literals, operators, delimiters, indentation markers, and line markers for subsequent syntactic processing.

Foundations of Python Lexical Structure

Python source code begins as encoded bytes, which are decoded into Unicode source characters according to a specified source encoding. These source characters are organized into physical lines, which can be combined into logical lines through explicit or implicit line joining. The lexical analyzer then scans these logical lines to produce a stream of tokens—lexical units such as names, literals, operators, delimiters, indentation tokens, newline tokens, and an end marker—that feed into the parser.

It is crucial to distinguish these concepts:

• Source characters: Unicode code points obtained after decoding the encoded bytes.
• Physical lines: Sequences of source characters terminated by a line-ending character or end of input.
• Logical lines: Sequences of source characters that represent a single logical statement or expression; may span multiple physical lines joined explicitly or implicitly.
• Tokens: Atomic lexical units such as names, literals, operators, delimiters, indentation markers (INDENT, DEDENT), and line markers (NEWLINE, ENDMARKER).
• Grammatical structures: Higher-level constructs recognized by the parser from sequences of tokens, not part of lexical analysis.

The principal lexical token families include:

• Names (identifiers)
• Numeric literals (integers, floats, imaginary numbers)
• String-related literals (strings, bytes, formatted strings, template strings)
• Operators and delimiters (punctuation such as +, -, (, ), etc.)
• Indentation tokens (INDENT, DEDENT)
• Newline tokens (NEWLINE)
• End marker (ENDMARKER)

Whitespace and comments play important lexical roles:

• Ordinary separating whitespace (spaces, tabs, formfeeds) generally does not produce tokens.
• Leading whitespace at the start of physical lines determines indentation and generates indentation tokens.
• Comments, starting with # outside string literals, are ignored by the lexer for token generation but terminate logical lines.
• Logical line termination influences token generation but the whitespace itself is not a token.

A conceptual overview of the lexical processing pipeline is shown below:

Decoding or lexical processing can fail, producing errors before parsing or runtime evaluation occurs. For example, invalid byte sequences produce decoding errors, and invalid token sequences (e.g., malformed literals) produce lexical errors. Some syntax errors require later parsing stages and are not detected during lexical analysis.

Python Source Text

Python source text begins as a sequence of encoded bytes, which are decoded into Unicode source characters before lexical tokenization. UTF-8 is the default source encoding if no valid encoding declaration is present.

Source encoding declarations:

• Must appear on the first or second physical line.
• If on the second line, the first line must be a comment or blank.
• Specify the encoding for decoding the entire source file.

When the source encoding is UTF-8 (implicit or explicit), an initial UTF-8 byte-order mark (BOM) is removed during decoding and does not appear as a source character.

Source characters are Unicode code points obtained after decoding. Python source may contain any Unicode character except the NUL character (U+0000), subject to lexical context constraints.

Failure to decode source bytes correctly results in a decoding error that prevents lexical analysis from proceeding. Valid decoding does not guarantee valid Python tokens or syntax later.

The source encoding applies globally before lexical interpretation of identifiers, comments, and literal source text, not only to string values.

Examples:

# -*- coding: utf-8 -*-
π = 3.14159           # Unicode identifier (pi)
message = "café"      # Unicode string content
# Comment with emoji 😀

• The first line uses an encoding declaration.
• The variable name π is a Unicode identifier.
• The string "café" contains a non-ASCII character.
• The comment contains emoji, which is allowed in source comments.

Decoding applies before token recognition: the encoded bytes are decoded to source characters, then tokens such as the name π or the string literal "café" are recognized.

Python Line Structure

Python distinguishes physical lines and logical lines:

• A physical line is a sequence of source characters ending with a line-ending sequence (\n, \r\n, \r, \x0b, \x0c, or Unicode line separators) or end of input.
• A logical line is a sequence of source characters that form a single statement or expression, possibly spanning multiple physical lines through joining.

Common physical end-of-line sequences are recognized and normalized uniformly. End of input is treated as an implicit terminator for the final physical line.

Explicit line joining uses a backslash (\) immediately preceding a physical line ending. The backslash and the line ending are removed, joining the physical lines into one logical line. Restrictions include:

• A comment cannot appear after the backslash on the same line.
• The backslash cannot split a token, only join physical lines.

Examples:

Valid explicit line joining:

total = 1 + 2 + \
        3 + 4

Invalid usage with trailing comment after backslash:

total = 1 + 2 + \  # comment
        3 + 4
# Lexically invalid: comment after backslash is disallowed.

Invalid splitting of a token:

a = 12\
3  # Invalid: splits the token "123" into "12" and "3".

Implicit line joining occurs inside open parentheses (), square brackets [], and curly braces {}. Inside these, physical line endings do not generate NEWLINE tokens, and comments may appear on continuation lines.

Example:

items = [
    1,
    2,
    3
]

Here the logical line spans multiple physical lines without explicit backslashes.

Comments begin with # outside string literals and continue to the end of the physical line. Comments terminate logical lines but are ignored for token generation.

Blank logical lines contain only whitespace and optionally a comment. They do not generate ordinary NEWLINE tokens in non-interactive mode.

Leading whitespace determines indentation levels. Changes in indentation generate INDENT and DEDENT tokens, managed via a stack of recorded indentation levels.

Tabs in indentation are treated specially: mixing tabs and spaces inconsistently makes indentation interpretation tab-width dependent and causes a TabError.

Whitespace between tokens (spaces, tabs, formfeeds) separates tokens and is required to prevent concatenation of adjacent character sequences into a single token.

Python Token Formation

Lexical analysis scans decoded source characters to produce tokens:

• NAME: identifiers and keywords
• Numeric literals: integers, floats, imaginary numbers
• String-related literals: strings, bytes, formatted strings, template strings
• OP: operators and delimiters
• NEWLINE: logical line terminators
• INDENT / DEDENT: indentation control tokens
• ENDMARKER: end of input token

The longest-valid-token principle applies: when multiple token boundaries are possible, the lexer consumes the longest sequence that forms a valid token, reading left to right.

Token boundaries are established by characters such as whitespace, punctuation, quotes, digits, and name characters. Whitespace is necessary between tokens if concatenation would produce a different token (e.g., a and + must be separated from a+).

Token classification alone does not guarantee that the resulting tokens form syntactically valid Python code; grammar analysis is performed later.

The ENDMARKER token is emitted at the end of non-interactive input to mark the end of tokenization, distinct from an ordinary newline character in source.

Example tokenization of source snippet:

def f(x=1):
    return x + 2

Tokens include:

• NAME('def')
• NAME('f')
• OP('(')
• NAME('x')
• OP('=')
• NUMBER('1')
• OP(')')
• OP(':')
• NEWLINE
• INDENT
• NAME('return')
• NAME('x')
• OP('+')
• NUMBER('2')
• NEWLINE
• DEDENT
• ENDMARKER

Lexical errors occur when tokenization fails (e.g., invalid literal syntax). Token boundaries do not guarantee syntactic correctness, which is checked by the parser.

Python Identifiers

Identifiers are names used to identify variables, functions, classes, etc., formed from:

• Name-start characters: Unicode characters with identifier start properties, including ASCII letters (a-z, A-Z) and underscore (_).
• Name-continue characters: name-start characters plus digits (0-9) and other Unicode continuation characters.

Identifiers are case-sensitive, must contain at least one character, and have no upper length limit. Identifiers cannot start with a digit.

Unicode identifier eligibility follows Unicode Standard Annex #31-like rules:

• Characters with the XID_Start property can start identifiers.
• Characters with the XID_Continue property can continue identifiers.

NFKC normalization is applied during parsing to identifiers: visually distinct source spellings may normalize to the same identifier.

Examples:

Valid ASCII identifiers:

variable
_var123
X

Valid Unicode identifiers:

π = 3.14
straße = "street"

Invalid identifiers:

123abc   # starts with digit
var!name # contains invalid character '!'

Controlled normalization example:

• The Latin capital letter A with a combining ring above (Å) and the single character Å normalize to the same identifier after NFKC normalization.

Reserved forms involving underscores:

• Single underscore _: often used as a temporary or insignificant variable.
• Leading underscore _name: conventionally indicates internal use.
• Double leading underscore __name: triggers name mangling in classes.
• Double leading and trailing underscores __name__: reserved for special methods.

Lexical validity is independent of these conventions.

Note that lexical normalization of identifiers does not automatically apply to arbitrary strings used at runtime in name-based APIs.

Python Keywords

Hard keywords are reserved names that cannot serve as ordinary identifiers in contexts requiring identifiers, such as if, for, def, class, etc. The set of hard keywords depends on the Python language version.

Identifiers, hard keywords, and soft keywords all originate from name-like source forms. Lexical name formation is distinct from parser-level interpretation.

Soft keywords are names reserved only in particular grammatical contexts, allowing them to be used as ordinary identifiers elsewhere. Examples include:

• match, case, _ in structural pattern matching.
• type in the type statement.

Soft-keyword recognition is context-sensitive and occurs during parsing, not tokenization.

Examples:

match = 5  # valid: 'match' as identifier
match x:
    case 1:
        print("One")

The keyword module can detect hard keywords:

import keyword
print(keyword.kwlist)  # list of hard keywords

Soft keywords are not included in keyword.kwlist but may be enumerated by language version-specific tools or modules that support them.

Python Literals

Literals are source notations for constant values. The main lexical literal families are:

• Numeric literals: integers, floating-point numbers, imaginary numbers
• Ordinary string literals: text strings enclosed in quotes
• Bytes literals: sequences of bytes with a b prefix
• Formatted string literals (f-strings): strings with embedded expressions
• Template string literals (introduced in Python 3.14): strings with template interpolation

Numeric Literals in Python

• Integer literals may be decimal, binary (0b), octal (0o), or hexadecimal (0x) with respective digit restrictions.
• Underscores _ can be used for digit grouping but not at the start or end.
• Leading zeros in nonzero decimal integers are disallowed (e.g., 0123 is invalid).
• Floating-point literals include decimal points and/or exponents (e or E).
• Imaginary literals end with j or J.
• A sign (+ or -) is not part of the numeric literal token but a separate operator token.

Examples:

• Valid integers: 123, 0b1010, 0o755, 0x1a3f, 1_000_000
• Invalid integers: 0123 (leading zero), 0b102 (invalid digit)
• Valid floats: 3.14, 1., .5, 1e10, 1.5e-3
• Valid imaginaries: 3j, 4.5J
• Token boundaries: a+=1 is NAME('a'), OP('+='), NUMBER('1')

String Literals in Python

• Enclosed in single '...', double "...", or triple quotes '''...''' or """...""".
• Triple-quoted strings can span multiple physical lines, retaining embedded newlines.
• Prefixes include raw r (no escape processing), and legacy u (Unicode strings).
• Escape sequences apply unless raw prefix is used.
• Raw strings cannot end with an unmatched backslash before the closing quote.

Examples:

'Hello'
"World"
'''Multi
line'''
r'Raw\nString'
u'Unicode'

Literal prefixes must appear immediately before the opening quote, distinguishing them from identifiers followed by strings.

Bytes Literals in Python

• Prefixed with b or B, optionally combined with raw prefix (br, rb).
• Must contain only ASCII characters, using escapes for non-ASCII byte values.
• Different from text strings in lexical restrictions and runtime types.

Examples:

b'byte string'
br'raw bytes\n'
b'\xff'    # escaped byte value

Invalid example:

b'café'  # invalid: 'é' is non-ASCII character in bytes literal

Formatted String Literals in Python

• Prefixed with f or F, optionally combined with raw (fr, rf).
• Contain literal text and replacement fields enclosed in {}.
• Escaped braces {{ and }} represent literal braces.
• Replacement fields may include conversions and format specifications.
• Cross lexical and grammatical boundary: literal parts are lexed as string content, expressions inside braces parsed separately.

Examples:

f'Hello, {name}!'
f'{value:.2f}'
f'{{escaped braces}}'
fr'Raw {expr}'

Template String Literals in Python (Python 3.14+)

• Prefixed with t or T, optionally combined with raw (tr, rt).
• Use source syntax closely resembling formatted strings.
• Support template interpolation with fields, conversions, and format specifications.
• Differ from formatted strings semantically rather than lexically.

Examples:

t'Hello, ${name}!'
t'Value: ${value:.2f}'
tr'Raw template string ${expr}'

Literal token spelling and boundaries are lexical concerns; runtime construction, formatting, interpolation, and evaluation are semantic issues handled later.

Python Operators and Delimiters

Operators and delimiters are punctuation-oriented lexical tokens categorized as generic operator (OP) tokens. Their syntactic or semantic roles depend on parsing context.

Representative examples include:

• Arithmetic: +, -, *, /, //, %, **
• Comparison: <, >, <=, >=, ==, !=, is
• Bitwise: &, |, ^, ~, <<, >>
• Assignment: =, +=, -=, *=, /=, //=, %= etc.
• Enclosing delimiters: (, ), [, ], {, }
• Separators: ,, :, ., ;
• Annotation: ->
• Assignment expression: :=
• Ellipsis: ...

Longest-match behavior applies: multi-character operators like **, //, <<, >=, :=, -> take precedence over their shorter constituents.

Example source snippet:

a += 1
b = (x * y) ** 2
if x is not y:
    ...

Tokens include:

• NAME('a'), OP('+='), NUMBER('1')
• NAME('b'), OP('='), OP('('), NAME('x'), OP('*'), NAME('y'), OP(')'), OP('**'), NUMBER('2')
• NAME('if'), NAME('x'), NAME('is'), NAME('not'), NAME('y'), OP(':')
• OP('...')

The same punctuation can serve multiple grammatical roles. Lexical classification does not imply precedence or semantic meaning.

Solved Python Lexical Structure Exercises

Consider the following Python source snippet incorporating Unicode identifiers, line structure variations, indentation, keywords, literals, and operators:

# -*- coding: utf-8 -*-
def π_function(x, y=3):
    raw_str = r"Line1\nLine2"
    b_string = b"bytes\x20"
    f_str = f"Value: {x + y}"
    t_str = t"Template ${x}"
    if x > 0 and y != 0:
        result = x ** y
    else:
        result = None
    return result

Lexical analysis breakdown:

• Source characters: Unicode decoded from UTF-8, including π (U+03C0).
• Physical lines: Each line ending with \n.
• Logical lines: No explicit or implicit line joining here; each physical line is a logical line except the indented blocks.
• Indentation tokens: Indentation increases after the function header line, generating an INDENT token; dedented at the end with a DEDENT.
• Tokens:
  • NAME('def'), NAME('π_function'), OP('('), NAME('x'), OP(','), NAME('y'), OP('='), NUMBER('3'), OP(')'), OP(':'), NEWLINE
  • INDENT
  • NAME('raw_str'), OP('='), STRING(r'"Line1\nLine2"'), NEWLINE
  • NAME('b_string'), OP('='), STRING(b'"bytes\x20"'), NEWLINE
  • NAME('f_str'), OP('='), FSTRING(f'"Value: {x + y}"'), NEWLINE
  • NAME('t_str'), OP('='), TSTRING(t'"Template ${x}"'), NEWLINE
  • NAME('if'), NAME('x'), OP('>'), NUMBER('0'), NAME('and'), NAME('y'), OP('!='), NUMBER('0'), OP(':'), NEWLINE
  • INDENT
  • NAME('result'), OP('='), NAME('x'), OP('**'), NAME('y'), NEWLINE
  • DEDENT
  • NAME('else'), OP(':'), NEWLINE
  • INDENT
  • NAME('result'), OP('='), NAME('None'), NEWLINE
  • DEDENT
  • NAME('return'), NAME('result'), NEWLINE
  • DEDENT
  • ENDMARKER

Lexical errors in invalid variants:

• Using an invalid byte sequence in the file encoding causes decoding failure before lexical analysis.
• Introducing a backslash with a trailing comment causes explicit line joining failure.
• Mixing tabs and spaces inconsistently in indentation causes TabError.
• Invalid numeric literal spellings (e.g., 0123) cause lexical errors.
• Unterminated string literals cause lexical errors.

This exercise demonstrates how lexical analysis processes Unicode source, physical and logical line structure, indentation tokens, literal families, keyword recognition, and operator tokens, producing a token stream for parsing without implying syntactic correctness or runtime behavior.