The World of Python Programming

Python Iterators vs Generators: Differences & Examples

Functions & Classes
目次

1. Introduction

Python is a programming language with simple and intuitive syntax, but to handle data more efficiently, understanding the concept of an “iterator” is important. This article explains the basic concepts of iterators, how to use them in practice, and even covers advanced examples in detail.

2. Basics of Iterables and Iterators

In Python, when handling data, the important concepts are “iterable” and “iterator”. Understanding the difference between these two allows you to learn how for loops work internally and more advanced data processing techniques.

What is an Iterable?

Iterable refers to a object that can be iterated over. In Python, lists, tuples, dictionaries, sets, strings, and the like are considered iterable objects. Iterable objects can be traversed element by element using a for loop.
# Example of a list (iterable)
numbers = [1, 2, 3, 4, 5]

for num in numbers:
    print(num)
Thus, the list numbers can be easily processed with a for loop. Internally, Python’s for loop converts this iterable into an iterator to retrieve elements sequentially.

What is an Iterator?

Iterator is an object that provides a mechanism to retrieve elements one at a time from an iterable object. In Python, you can convert an iterable into an iterator using the iter() function.
numbers = [1, 2, 3, 4, 5]
iterator = iter(numbers)  # Create iterator

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
Here, calling next(iterator) shows that the list’s elements are being retrieved in order.

3. How Iterators Work in Python

Python iterators are implemented as objects that have specific methods.

Roles of __iter__() and __next__()

Iterator objects have the following two special methods.
  • __iter__(): returns the object itself
  • __next__(): returns the next element and raises StopIteration when the end is reached
For example, handling an iterator manually with a list looks like this.
class CustomIterator:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

# Create iterator
custom_iter = CustomIterator(1, 5)

for num in custom_iter:
    print(num)  # 1, 2, 3, 4 are printed in order
Thus, by creating a custom iterator, you can retrieve data sequentially using a for loop.

4. How to Implement Iterators

In Python, implementing custom iterators enables flexible data processing. This chapter provides a detailed explanation of how to create custom iterator classes, using generator functions, and utilizing generator expressions.

Creating a Custom Iterator Class

Python iterators can be implemented as classes that define the __iter__() and __next__() methods. The following code is an example of creating a custom iterator that returns values sequentially from 1 up to a specified number.
class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self  # Return self as the iterator

    def __next__(self):
        if self.current > self.end:
            raise StopIteration  # Raise exception when iteration ends
        value = self.current
        self.current += 1
        return value

# Create iterator and use it in a for loop
counter = Counter(1, 5)
for num in counter:
    print(num)  # 1, 2, 3, 4, 5
Key Points:
  • __iter__() method returns self (because the iterator itself is an iterator)
  • __next__() method returns the current value and updates to the next value
  • When there are no more elements, it raises StopIteration
Using this approach, you can define custom iterators and apply them to stream processing of data and more.

Using Generator Functions

In Python, you can use generator functions to create iterators. Generator functions allow you to easily create iterators by using yield.
def counter(start, end):
    current = start
    while current <= end:
        yield current  # Return the value while preserving execution state
        current += 1

# Use the generator function
for num in counter(1, 5):
    print(num)  # 1, 2, 3, 4, 5
Advantages of Generator Functions:
  • yield lets you return a value and resume execution on the next call
  • It reduces memory consumption while allowing you to retrieve needed data lazily (deferred evaluation)
Generators are especially useful for large-scale data processing and stream processing.

Using Generator Expressions

Python provides “generator expressions,” which use syntax similar to list comprehensions to create generators.
# List comprehension (stores all elements in memory)
numbers_list = [x * 2 for x in range(1, 6)]
print(numbers_list)  # [2, 4, 6, 8, 10]

# Generator expression (saves memory with lazy evaluation)
numbers_gen = (x * 2 for x in range(1, 6))

print(next(numbers_gen))  # 2
print(next(numbers_gen))  # 4
Advantages of Generator Expressions:
  • More memory-efficient than list comprehensions
  • Computes and yields only the needed elements (lazy evaluation)
Generator expressions contribute to improved data processing performance, making them valuable for handling large datasets.
RUNTEQ（ランテック）｜超実戦型エンジニア育成スクール

5. Iterator Use Cases

By using iterators, you can implement various processes efficiently. In this chapter, we introduce three practical use cases.

① Sequential File Reading

When handling large files, loading them entirely into memory is inefficient. By leveraging iterators, you can process a file line by line.
def read_large_file(file_path):
    with open(file_path, "r", encoding="utf-8") as file:
        for line in file:
            yield line.strip()  # Process each line sequentially

# Usage
for line in read_large_file("data.txt"):
    print(line)
Benefits:
  • Process large files while saving memory
  • Suitable for streaming data processing

② Generating Infinite Sequences

Using iterators, you can efficiently generate sequences that continue indefinitely.
def infinite_counter(start=0):
    while True:
        yield start
        start += 1

# Use with limits to avoid an infinite loop
counter = infinite_counter()
for _ in range(5):
    print(next(counter))  # 0, 1, 2, 3, 4
Use Cases:
  • Time-based data streams
  • Real-time processing of sensor data

③ Processing Data Streams

Iterators are also effective when processing data from APIs or databases sequentially.
import time

def api_simulation():
    for i in range(1, 6):
        time.sleep(1)  # Wait 1 second (simulate API response)
        yield f"Data {i}"

# Retrieve data from API sequentially
for data in api_simulation():
    print(data)
Key Points:
  • Data can be obtained while being processed in real time
  • Can handle data while reducing network load

6. Cautions When Using Iterators

Python iterators are convenient, but using them incorrectly can lead to unexpected behavior. This section explains the three key points to watch when handling iterators: “Handling of StopIteration”, “Reusing an Exhausted Iterator”, “Memory Efficiency and Lazy Evaluation”.

① Handling StopIteration

When operating an iterator with next(), a StopIteration exception is raised after all elements have been retrieved.
numbers = [1, 2, 3]
iterator = iter(numbers)

print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
print(next(iterator))  # StopIteration occurs
The best way to handle this exception is to use try-except.
numbers = [1, 2, 3]
iterator = iter(numbers)

while True:
    try:
        print(next(iterator))
    except StopIteration:
        print("Iterator has finished")
        break
Key Points
  • for loops automatically handle StopIteration, so you don’t need to address it explicitly
  • When using next() manually, handling it with try-except is safe

② Reusing an Exhausted Iterator

Python iterators cannot be reused once all elements have been retrieved.
numbers = [1, 2, 3]
iterator = iter(numbers)

for num in iterator:
    print(num)  # outputs 1, 2, 3

for num in iterator:
    print(num)  # no output (iterator is empty)
If you need to reuse an iterator, you must create a new one.
numbers = [1, 2, 3]

# create a new iterator
iterator1 = iter(numbers)
iterator2 = iter(numbers)

print(list(iterator1))  # [1, 2, 3]
(list(iterator2))  # [1, 2, 3]
Key Points
  • Iterables such as lists can be reused, but iterators are single-use
  • If you want to reuse, call iter() again to create a new one

③ Memory Efficiency and Lazy Evaluation

Using iterators and generators allows you to process data while keeping memory usage low. For example, let’s compare lists and generators.
# Use a list (stores all elements in memory)
numbers_list = [x * 2 for x in range(1000000)]  # 1,000,000 elements

# Use a generator (produces elements on demand)
numbers_gen = (x * 2 for x in range(1000000))

print(sum(numbers_list))  # after calculation, the list remains in memory
print(sum(numbers_gen))   # calculates while saving memory
Key Points
  • Lists store all elements in memory, making them unsuitable for processing large data sets
  • Generators are lazily evaluated, so they generate elements only when needed

7. Frequently Asked Questions (FAQ)

Finally, we have compiled a list of frequently asked questions about Python iterators.

Q1. What is the difference between iterators and generators?

A:
  • An iterator is a class that implements __iter__() and __next__()
  • A generator can easily create an iterator by using yield
  • Generators tend to result in simpler code

Q2. Why are the __iter__() and __next__() methods necessary?

A:
  • __iter__() is called by a for loop to handle the iterator
  • __next__() is used to retrieve the next element
  • Because Python’s for loop internally uses iter() and next()

Q3. What are the benefits of using iterators?

A:
  • Memory-efficient (especially when processing large data sets)
  • Supports lazy evaluation (generates only the data that is needed)
  • Ideal for data stream processing (API responses, file handling, etc.)

Q4. In what situations do you need to manually create an iterator?

A:
  • When performing custom data processing (e.g., retrieving only data that meets specific criteria)
  • Processing streaming data from an API
  • Generating data in an infinite-loop fashion

Q5. What is the relationship between iterators and for loops?

A:
  • for loops internally call iter() and retrieve data with next()
  • Using for automatically handles StopIteration exceptions, preventing errors

Summary

In this article, we explained Python iterators in detail, from basics to advanced topics. Reviewing the key points… ✅ An iterator is an object that retrieves data one item at a time ✅ If you implement __iter__() and __next__(), you can create custom iterators ✅ Using generators makes it even easier to create iterators ✅ Iterators leverage lazy evaluation to improve memory efficiency ✅ They are especially useful for API responses, stream data processing, file handling, and similar tasks By leveraging Python iterators, more efficient data processing becomes possible. Please try out the code yourself to deepen your understanding!
侍エンジニア塾