Generators & yield

PythonMidpython

The Question

How do generators and the yield keyword work in Python?

What a Strong Answer Covers

• yield suspends state
• "lazy
• "__next__
• "memory efficient

Senior-Level Answer

A generator in Python is a special type of iterator. Unlike a regular function that computes its entire result and returns it, a generator produces values one at a time and suspends execution between each value. This lazy evaluation model enables processing of large or infinite sequences without materializing them in memory.

A generator function is defined like a regular function but contains one or more yield statements. When you call a generator function, it does not execute the function body -- it returns a generator object immediately. The body only starts running when you call next() on the generator. Execution proceeds until the next yield statement, at which point the yielded value is returned to the caller and the function's execution state -- local variables, instruction pointer, everything -- is suspended. The next call to next() resumes exactly where it left off. When the function body exhausts, a StopIteration exception is raised, signaling the iterator is exhausted.

Memory efficiency is the primary benefit. A generator that yields one million numbers uses O(1) memory -- it holds only the current execution frame. A list of one million numbers uses O(n) memory. For pipelines processing large datasets -- reading lines from a file, transforming records -- chaining generators keeps the entire pipeline at O(1) memory overhead.

Generator expressions provide a compact syntax analogous to list comprehensions: (x * 2 for x in range(10)) returns a generator rather than a list.

yield from is a delegation mechanism introduced in Python 3.3. It allows a generator to delegate to another iterable or generator, flattening the delegation without manually iterating in a loop. It also properly forwards send() values and exceptions, which matters for coroutine-style code.

Generators also support the send() method, which allows the caller to send a value back into the generator, making generators a building block for coroutines. Python's asyncio async/await syntax is built on generators at the lower level. Understanding generators is prerequisite to understanding how the event loop and coroutines work internally.

What Separates a 2/3 from a 3/3

Common Mistakes

• Saying generators return all values at once -- they produce one value at a time, suspending between each
• Forgetting that calling a generator function does not execute any code -- it returns a generator object
• Not knowing yield from -- manually iterating an inner generator in a loop loses send() and throw() forwarding
• Treating generators as just memory-efficient lists, missing the coroutine use case entirely

Follow-Up Questions

• What is the difference between a generator and an iterator? — Every generator is an iterator, but not every iterator is a generator. An iterator implements __iter__ and __next__; a generator function uses yield to implement these automatically.
• What does send() do? — send(value) resumes the generator and makes value the result of the current yield expression inside the generator -- enables two-way communication, the basis for coroutines.
• How does asyncio use generators under the hood? — Early asyncio coroutines were generators using yield from. async/await in Python 3.5 added native coroutines driven by the event loop similarly to how next() drives a generator.
• What is the difference between a generator expression and a list comprehension? — A list comprehension evaluates eagerly and stores all results in memory; a generator expression is lazy and generates each element on demand.

Related Questions

• GIL — What It Is and What It Protects
• ThreadPoolExecutor & ProcessPoolExecutor
• Decorators — Under the Hood
• Context Managers
• LEGB Rule