Python Iterators and Generators

Introduction Reading Time: 10 min

Table of Contents

Description

Iterators are objects that allow sequential access to elements using the __iter__() and __next__() methods. Generators are a simpler way to create iterators using yield instead of return. Generators are memory efficient and lazy, meaning they generate items one at a time and only when required.

Prerequisites

  • Python loops (for, while)
  • Understanding of functions
  • Knowledge of classes (for custom iterators)
  • Exception handling (StopIteration)

Examples

Here's a simple program in Python:

✅ Iterator using __iter__() and __next__()
class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        # Returns the iterator object itself
        return self

    def __next__(self):
        # Returns the next value or raises StopIteration
        if self.current <= self.end:
            num = self.current
            self.current += 1
            return num
        else:
            raise StopIteration

# Using the iterator
c = Counter(1, 5)

for num in c:
    print(num)  # Output: 1 2 3 4 5
✅ Generator Function using yield
def countdown(n):
    while n > 0:
        yield n    # Pause and resume here
        n -= 1

# Using the generator
for i in countdown(5):
    print(i)  # Output: 5 4 3 2 1
✅ Generator Expression
# Similar to list comprehension but lazy
squares = (x*x for x in range(5))

for num in squares:
    print(num)  # Output: 0 1 4 9 16
✅ Generator vs List (Memory Comparison)
import sys

lst = [x for x in range(10000)]
gen = (x for x in range(10000))

print(sys.getsizeof(lst))  # Larger memory
print(sys.getsizeof(gen))  # Much smaller memory

      

Real-World Applications

File reading line by line (yield)

Streaming large datasets (e.g., CSV, logs)

Data pipelines (ETL systems)

Implementing infinite sequences (e.g., Fibonacci)

Memory optimization in ML and web scraping

Where topic Can Be Applied

Backend APIs for pagination and streaming

Machine learning model input pipelines (e.g., TensorFlow datasets)

Asynchronous programming and coroutines

Lazy evaluation and on-the-fly data generation

Custom collection classes

Resources

Topic video source

A comprehensive video

Watch

Python pdf

pdf on topic

Visit

Interview Questions

What is the difference between an iterable and an iterator?

What are the advantages of using generators?

How does yield differ from return?

Can we iterate multiple times over a generator?

What does StopIteration do?

How can we create an infinite generator?

Difference between generator expressions and list comprehensions?

How do __iter__() and __next__() work in a class?

How does memory usage differ between a generator and a list?

Can a generator be reused once it is exhausted?