Iteration in Python

Iteration in Python is the collection of protocols and language mechanisms through which objects expose successive values to consumers, enabling repeated retrieval of elements one at a time. This includes how custom objects define traversal behavior, how generators suspend and resume computation to produce values incrementally, and how asynchronous iterators extend iteration to operations that can await between values.

Foundations of Iteration in Python

An iterable is any object capable of providing an iterator. An iterator represents an active traversal state that produces successive values until exhaustion.

Conceptually, synchronous iteration follows this cycle:

1. Obtain an iterator from an iterable by calling the built-in iter function.
2. Repeatedly request the next value from the iterator by calling the built-in next function.
3. When there are no more values, the iterator raises the StopIteration exception to signal termination.

The for statement in Python automates this protocol by internally calling iter once and then repeatedly calling next until StopIteration is raised, at which point it stops.

Example demonstrating manual iteration and the equivalent for loop:

# Manual iteration with iter and next
lst = [10, 20, 30]
it = iter(lst)
try:
    while True:
        value = next(it)
        print(value)
except StopIteration:
    print("Iteration complete")

print("---")

# Equivalent iteration using for loop
for value in lst:
    print(value)

In this example, the for loop internally performs exactly the steps of obtaining an iterator with iter and calling next repeatedly until StopIteration is raised.

Python Iterable Objects

An iterable is any object from which iter(obj) can obtain an iterator. This is primarily achieved via the __iter__ method. Older or sequence-style iterables might also support iteration through successive integer indexing starting at zero via the __getitem__ method, though this is a fallback.

Some iterables are re-iterable containers, meaning they produce fresh, independent iterator objects on each call to iter. Others are one-shot iterable objects whose iteration state is the object itself, so repeated calls to iter do not create new traversal states but return the same iterator.

Repeated calls to iter need not always create a new traversal because some iterators are designed to be single-use and maintain their own state.

Example comparing a list (re-iterable container) with an iterator/generator (one-shot iterable):

lst = [1, 2, 3]
it1 = iter(lst)
it2 = iter(lst)
print(next(it1))  # 1
print(next(it2))  # 1 (independent iterator)

gen = (x for x in range(3))
gen1 = gen
gen2 = gen
print(next(gen1))  # 0
# gen1 and gen2 are the same iterator, so next yield continues
print(next(gen2))  # 1

# Attempting to restart iteration on a generator fails:
try:
    iter(gen)  # returns the generator itself; no new iterator created
except Exception as e:
    print(e)

Iterability only concerns the ability to produce successive values and does not imply:

• Indexing by position
• Knowing length in advance
• Reversibility
• Restartability of iteration
• Optimized membership testing
• Materializing all values in memory

Python Iterator Objects

An iterator implements the iterator protocol by defining:

• __iter__(), which returns the iterator object itself.
• __next__(), which returns the next value or raises StopIteration if no values remain.

Once an iterator is exhausted (i.e., __next__() raises StopIteration), subsequent calls should continue to raise StopIteration rather than implicitly restarting traversal.

Here is a custom iterator example:

class CountUpTo:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current < self.limit:
            result = self.current
            self.current += 1
            return result
        else:
            raise StopIteration

counter = CountUpTo(3)

# Manual consumption
print(next(counter))  # 0
print(next(counter))  # 1
print(next(counter))  # 2
try:
    print(next(counter))  # Raises StopIteration
except StopIteration:
    print("Exhausted")

print("---")

# Using for loop on a fresh iterator
counter2 = CountUpTo(3)
for num in counter2:
    print(num)

print("---")

# Using next with default to avoid StopIteration handling
counter3 = CountUpTo(2)
print(next(counter3, 'done'))  # 0
print(next(counter3, 'done'))  # 1
print(next(counter3, 'done'))  # 'done' (no exception raised)

Built-in Iteration Tools in Python

Python provides several built-in functions and adapters to support iteration:

• iter(obj) obtains an iterator from an iterable.
• next(iterator[, default]) retrieves the next value or returns default if provided instead of raising StopIteration.
• iter(callable, sentinel) creates an iterator that calls callable until it returns the sentinel value.

Common iteration adapters:

• enumerate(iterable) produces pairs of (index, value).
• zip(*iterables, strict=False) combines multiple iterables element-wise.
• map(function, iterable, ...) applies a function to each element.
• filter(function, iterable) produces elements where function returns true.
• reversed(sequence) produces elements in reverse order.

Common consumers that consume iterators and produce a result:

• any(iterable) returns True if any element is truthy.
• all(iterable) returns True if all elements are truthy.
• sum(iterable, start=0) sums elements.
• min(iterable, *[, key, default]) returns minimum element.
• max(iterable, *[, key, default]) returns maximum element.

Example combining tools:

data1 = ['a', 'b', 'c']
data2 = ['a', 'b', 'd']

# Using enumerate and zip with strict=True to compare elements with indices
for index, (x, y) in enumerate(zip(data1, data2, strict=True)):
    print(f"Index {index}: {x} vs {y}")

# Using map and filter lazily
numbers = range(10)
squares = map(lambda x: x**2, numbers)
even_squares = filter(lambda x: x % 2 == 0, squares)

print("Even squares:")
for val in even_squares:
    print(val)

# Using any, all, sum immediately consume iterator
print("Any even squares > 50?", any(x > 50 for x in map(lambda x: x**2, range(10))))
print("All squares >= 0?", all(x >= 0 for x in map(lambda x: x**2, range(10))))
print("Sum of squares:", sum(map(lambda x: x**2, range(5))))

In this example, map and filter produce lazy iterators that compute values as they are requested. Consumers like any, all, and sum immediately consume the iterator to produce a final value.

Custom Iteration in Python

A reusable custom iterable separates collection state from traversal state by implementing __iter__ to return a fresh iterator object each time. This allows multiple independent traversals.

A self-iterating object returns self from __iter__, meaning the object itself maintains traversal state and typically supports only one active iteration.

Additional optional customization includes:

• __reversed__ to support reversed iteration.
• __contains__ to optimize membership testing.
• Legacy support for iteration fallback via __getitem__, but this is not required for modern iterables.

Example reusable custom collection and separate iterator:

class MyCollection:
    def __init__(self, items):
        self.items = list(items)

    def __iter__(self):
        return MyCollectionIterator(self.items)

class MyCollectionIterator:
    def __init__(self, items):
        self.items = items
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index >= len(self.items):
            raise StopIteration
        val = self.items[self.index]
        self.index += 1
        return val

coll = MyCollection([10, 20, 30])
it1 = iter(coll)
it2 = iter(coll)

print(next(it1))  # 10
print(next(it2))  # 10 (independent iterators)

print("---")

# Self-iterating single-use example
class MyOneShot:
    def __init__(self, items):
        self.items = list(items)
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index >= len(self.items):
            raise StopIteration
        val = self.items[self.index]
        self.index += 1
        return val

oneshot = MyOneShot([1, 2, 3])
for v in oneshot:
    print(v)

# Attempting a second traversal will fail silently or yield no values
for v in oneshot:
    print(v)  # No output

Python Generators

Python Generator Functions

A generator function is defined by a function containing one or more yield expressions. Calling this function returns a generator object without executing the function body to completion at call time.

Python Generator Objects

A generator object implements the iterator protocol and supports suspension and resumption of execution around each yield. Local execution state, including local variables and the point of suspension, is preserved between resumptions.

When a generator terminates, it raises StopIteration. If the generator function returns a value explicitly, that value is attached as the value attribute of the terminating StopIteration exception.

Example generator function and usage:

def count_and_return(limit):
    for i in range(limit):
        yield i
    return "Done"

gen = count_and_return(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2

try:
    next(gen)
except StopIteration as e:
    print(f"Generator exhausted with value: {e.value}")

Here, calling count_and_return returns a generator object without running the loop yet. Values are produced only when next is called.

Python Generator Control

Generators support advanced control methods:

• send(value) resumes execution, supplying the suspended yield expression the given value.
• throw(type, value=None, traceback=None) raises an exception at the suspension point inside the generator.
• close() raises a GeneratorExit inside the generator to request termination and allows cleanup code to run.

Example demonstrating control:

def controlled_gen():
    print("Started")
    try:
        x = yield "First yield"
        print(f"Received: {x}")
        yield "Second yield"
    except ValueError:
        print("Caught ValueError inside generator")
    finally:
        print("Cleaning up")

gen = controlled_gen()
print(next(gen))      # Start and yield first value
print(gen.send(42))   # Send value, resume and yield second value

gen.throw(ValueError) # Exception injection

gen.close()           # Request generator cleanup

Note: A just-started generator cannot receive a non-None value with send before its first yield; the first send must be None or next() should be called.

Generator Delegation in Python

yield from iterable delegates part of a generator's operation to a subiterator, forwarding values produced and control interactions (send, throw, close) transparently. The final return value of the delegated generator is available as the value of the yield from expression.

Example:

def subgenerator():
    yield 1
    yield 2
    return "subgen done"

def outer():
    result = yield from subgenerator()
    yield f"Subgenerator returned: {result}"

gen = outer()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # "Subgenerator returned: subgen done"
try:
    next(gen)
except StopIteration:
    print("Outer generator exhausted")

Here, yield from delegates yielding to subgenerator. When subgenerator returns, its return value is assigned to result in outer.

Asynchronous Iteration in Python

The asynchronous iteration protocol involves:

• __aiter__() returning an asynchronous iterator.
• __anext__() returning an awaitable whose result is the next value.
• StopAsyncIteration signaling exhaustion.

The async for statement drives this protocol automatically by awaiting __anext__ repeatedly until StopAsyncIteration is raised.

Example custom asynchronous iterator:

import asyncio

class AsyncCounter:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.current >= self.limit:
            raise StopAsyncIteration
        await asyncio.sleep(0.1)  # Simulate async operation
        value = self.current
        self.current += 1
        return value

async def main():
    async for num in AsyncCounter(3):
        print(num)

# To run: asyncio.run(main())

Python Asynchronous Generators

An async def function containing yield is an asynchronous generator function. Calling it returns an asynchronous generator object that can await between yielding values.

These terminate by raising StopAsyncIteration on exhaustion and cannot use yield from.

Asynchronous generators support control methods:

• __anext__(): returns an awaitable producing the next value.
• asend(value): resumes with supplied value at suspended yield.
• athrow(exception): raises an exception at suspension point.
• aclose(): requests asynchronous finalization and cleanup.

Example asynchronous generator:

import asyncio

async def async_gen():
    try:
        for i in range(3):
            await asyncio.sleep(0.1)
            yield i
    finally:
        print("Async generator cleanup")

async def main():
    agen = async_gen()
    async for value in agen:
        print(value)

    # Controlled interaction example
    agen = async_gen()
    print(await agen.__anext__())  # 0
    await agen.aclose()

# To run: asyncio.run(main())

This example shows asynchronous generation with awaitable suspension, cleanup on close, and consumption with async for.