Python Generators

Generator Basics

Generators are a special type of iterator that generate values on-the-fly rather than storing them in memory. They're created using functions with yield statements.

                        # Simple generator function

                        def countdown(num):

                            while num > 0:

                                yield num

                                num -= 1

                        # Using the generator

                        for i in countdown(5):

                            print(i) # Prints 5, 4, 3, 2, 1

                        # Generator object

                        gen = countdown(3)

                        print(next(gen)) # 3

                        print(next(gen)) # 2

                        print(next(gen)) # 1

                        # print(next(gen)) # Raises StopIteration

Generators maintain their local state between calls, making them memory efficient for large datasets or infinite sequences.

Generator Expressions

Similar to list comprehensions but with parentheses, generator expressions create anonymous generator objects.

                        # List comprehension (creates full list in memory)

                        squares_list = [x**2 for x in range(10)]

                        # Generator expression (creates iterator)

                        squares_gen = (x**2 for x in range(10))

                        print(squares_list) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

                        print(squares_gen) # <generator object <genexpr> at 0x...>

                        # Using the generator expression

                        for num in squares_gen:

                            print(num) # Prints squares from 0 to 81

                        # Can be consumed only once

                        print(sum(squares_gen)) # 0 (already exhausted)

Generator expressions are memory efficient and perfect for working with large datasets where you don't need all values at once.

Yield From

The yield from expression (Python 3.3+) allows delegation to subgenerators, simplifying generator composition.

                        # Without yield from

                        def concat_old(*iterables):

                            for it in iterables:

                                for item in it:

                                    yield item

                        # With yield from

                        def concat(*iterables):

                            for it in iterables:

                                yield from it

                        # Both produce the same result

                        gen1 = concat_old('ABC', 'DEF')

                        gen2 = concat('ABC', 'DEF')

                        print(list(gen1)) # ['A', 'B', 'C', 'D', 'E', 'F']

                        print(list(gen2)) # ['A', 'B', 'C', 'D', 'E', 'F']

                        # Also works with generator expressions

                        def chain(*iterables):

                            yield from (item for it in iterables for item in it)

yield from not only simplifies code but also enables direct communication between the caller and subgenerator.

Stateful Generators

Generators can maintain state between calls, making them useful for implementing coroutines and pipelines.

                        # Averaging coroutine

                        def averager():

                            total = 0.0

                            count = 0

                            while True:

                                num = yield

                                total += num

                                count += 1

                                print(total / count)

                        # Using the coroutine

                        avg = averager()

                        next(avg) # Prime the coroutine

                        avg.send(10) # 10.0

                        avg.send(20) # 15.0

                        avg.send(30) # 20.0

                        # Pipeline example

                        def producer(data):

                            for item in data:

                                yield item

                        def filter(gen, condition):

                            for item in gen:

                                if condition(item):

                                    yield item

                        def consumer(gen):

                            return sum(gen)

                        # Build the pipeline

                        data = range(100)

                        result = consumer(

                            filter(

                                producer(data),

                                lambda x: x % 2 == 0

                            )

                        )

                        print(result) # Sum of even numbers from 0 to 99