Python Generators: How yield Works with Examples

Generators in Python are a powerful mechanism for working with data sequences efficiently. They allow you to create iterable objects that compute values on demand, significantly saving system memory. The foundation of generators is the yield keyword, which differs fundamentally from the traditional return.

How Python Generators Work

Generators are special functions that return iterators instead of concrete values. The key difference lies in using the yield keyword, which preserves the function's state between calls.

Each time a generator function is called, execution resumes from where the last yield statement left off. This makes generators an ideal solution for processing large data streams without storing everything in memory.

def simple_generator():
    yield 1
    yield 2
    yield 3

gen = simple_generator()
for value in gen:
    print(value)

Output:

1
2
3

The Mechanics of the yield Keyword

The yield keyword works on the principle of lazy evaluation:

1. On the first next() call, execution starts from the beginning of the function until the first yield
2. After returning a value, the function "freezes" its state
3. The next next() call resumes execution from where it stopped
4. This process continues until the function completes

def counter():
    print("Generator started")
    yield 1
    print("After first yield")
    yield 2
    print("After second yield")

gen = counter()
next(gen)  # Prints: Generator started
next(gen)  # Prints: After first yield

Generators vs Lists: Key Advantages

Using generators instead of lists offers significant benefits when working with large datasets:

Memory Efficiency: Generators don't store all values at once; they create them on the fly.

# List - consumes a lot of memory
squares_list = [x * x for x in range(1000000)]

# Generator - minimal memory usage
squares_gen = (x * x for x in range(1000000))

Performance: Generators start returning values instantly, without waiting for the full sequence to be created.

Creating Infinite Sequences

Generators allow you to create infinite sequences without the risk of memory overflow:

def infinite_counter(start=0):
    while True:
        yield start
        start += 1

counter = infinite_counter()
print(next(counter))  # 0
print(next(counter))  # 1
print(next(counter))  # 2

Working with Generator Methods

Generators support several specialized methods:

next() — retrieves the next value from the generator
send(value) — sends a value back into the generator
throw() — raises an exception inside the generator
close() — forcefully stops the generator

def interactive_generator():
    value = yield "Start"
    while True:
        value = yield f"Received: {value}"

gen = interactive_generator()
print(next(gen))           # Start
print(gen.send(42))        # Received: 42
print(gen.send("Hello"))   # Received: Hello

Exception Handling in Generators

Generators support exception handling, including the special GeneratorExit exception:

def example_generator():
    try:
        yield 1
        yield 2
    except GeneratorExit:
        print("Generator closed!")

gen = example_generator()
print(next(gen))  # 1
gen.close()       # Prints: Generator closed!