🐍 Day 16: Iterators & Generators — Memory-Efficient Data Processing

---

1. Learning Objectives

By the end of Day 16, you will be able to:

• Understand the iterator protocol (__iter__ and __next__)
• Create your own custom iterators using classes
• Use built‑in iterators like iter() and next()
• Write memory‑efficient generator functions with yield
• Build generator expressions for one‑liner lazy evaluation
• Know when to use generators vs. lists — the memory trade‑off
• Chain and compose generators for data pipelines

---

2. Concept Explanation

2.1 Why Iterators & Generators? — Lazy vs. Eager

So far, you've created lists eagerly: [x**2 for x in range(1000)] builds the entire list in memory at once. What if you have 100 million items? Memory explodes.

Iterators and generators produce items lazily — one at a time, on demand. They never hold the entire collection in memory.

Approach	Memory	When Available
List (eager)	Stores all items	All at once
Iterator/Generator (lazy)	Stores only current item	One at a time, on request

---

2.2 The Iterator Protocol — What Makes Something Iterable?

Any Python object that can be used in a for loop is iterable. Under the hood, for calls two methods:

Method	Purpose
__iter__()	Returns the iterator object itself
__next__()	Returns the next item; raises StopIteration when exhausted

Built‑in example:

nums = [1, 2, 3]
iterator = iter(nums)       # Calls nums.__iter__()

print(next(iterator))       # 1  (calls iterator.__next__())
print(next(iterator))       # 2
print(next(iterator))       # 3
# print(next(iterator))     # StopIteration!

What a for loop actually does:

# for item in iterable:
#     print(item)

# is equivalent to:
iterator = iter(iterable)
while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        break

---

2.3 Custom Iterators — Build Your Own

You can make any class iterable by implementing __iter__() and __next__():

class Countdown:
    """Iterator that counts down from n to 0."""
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self      # An iterator returns itself

    def __next__(self):
        if self.current < 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# Usage
for num in Countdown(5):
    print(num)   # 5, 4, 3, 2, 1, 0

---

2.4 Generators — Iterators Made Simple

A generator is a function that uses yield instead of return. It automatically creates an iterator — you don't need to write __iter__ or __next__.

def countdown(n):
    """Generator that yields numbers from n down to 0."""
    while n >= 0:
        yield n
        n -= 1

# Usage — identical to the custom iterator above
for num in countdown(5):
    print(num)   # 5, 4, 3, 2, 1, 0

# Or use next()
gen = countdown(3)
print(next(gen))  # 3
print(next(gen))  # 2

How yield works:

• When the function hits yield, it pauses and returns a value.
• On the next next() call, it resumes right after the yield.
• When the function ends, it raises StopIteration automatically.

💡 Golden rule: If your function uses yield, it's a generator. Calling it returns a generator object, not a value.

---

2.5 Generator Expressions — One‑Line Generators

Just like list comprehensions, but with parentheses () instead of brackets []:

# List comprehension (eager — builds the whole list in memory)
squares_list = [x**2 for x in range(1000)]   # List of 1000 items

# Generator expression (lazy — produces items on demand)
squares_gen = (x**2 for x in range(1000))     # Generator object

print(next(squares_gen))  # 0
print(next(squares_gen))  # 1
print(next(squares_gen))  # 4

Memory comparison:

import sys

big_list = [x for x in range(1_000_000)]
big_gen = (x for x in range(1_000_000))

print(sys.getsizeof(big_list))   # ~8 MB
print(sys.getsizeof(big_gen))    # ~200 bytes

The generator expression takes nearly the same tiny amount of memory regardless of range size.

---

2.6 Chaining Generators — Data Pipelines

Generators shine when you chain them together — each processes one item at a time:

# Read lines from a file (lazy)
def read_logs(filename):
    with open(filename) as f:
        for line in f:
            yield line.strip()

# Filter out empty lines
def filter_empty(lines):
    for line in lines:
        if line:
            yield line

# Extract only error lines
def filter_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

# Pipeline — no intermediate lists!
errors = filter_errors(filter_empty(read_logs("app.log")))
for error in errors:
    print(error)

Each function processes one item and passes it along. Memory usage is constant regardless of file size.

---

2.7 When to Use What

Scenario	Best Tool
Need random access or indexing	List
Need to iterate multiple times	List (generators are exhausted after one pass)
Processing huge datasets	Generator
Infinite sequence	Generator
Pipeline of transformations	Generator chain
Need the length upfront	List (generators don't know their length)

---

2.8 Common Mistakes

Mistake	Problem
Reusing an exhausted generator	gen = (x for x in range(3)); list(gen); list(gen) — second is empty
Using return in a generator	Returns StopIteration with a value (Python 3.3+) — don't use for normal return
Expecting generator to have length	len(gen) fails — generators don't know their size
Forgetting () for generator expression	(x for x in range(10)) not [x for x in range(10)] if you want a generator

---

3. Code Examples

Example 1: Infinite Fibonacci Generator

def fibonacci():
    """Generate infinite Fibonacci sequence."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Take first 10 Fibonacci numbers
fib = fibonacci()
for _ in range(10):
    print(next(fib), end=" ")   # 0 1 1 2 3 5 8 13 21 34

Example 2: File Line Filter Pipeline

def file_lines(filename):
    """Yield lines from a file, stripping whitespace."""
    with open(filename) as f:
        for line in f:
            yield line.strip()

def filter_keyword(lines, keyword):
    """Yield only lines containing the keyword (case-insensitive)."""
    keyword = keyword.lower()
    for line in lines:
        if keyword in line.lower():
            yield line

# Usage
lines = file_lines("server.log")
errors = filter_keyword(lines, "error")

for i, line in enumerate(errors, 1):
    print(f"Error {i}: {line}")

Example 3: yield from — Delegating to Sub-Generators

def numbers():
    yield from range(1, 4)    # Yields 1, 2, 3
    yield from [10, 20, 30]   # Yields 10, 20, 30
    yield from "AB"           # Yields 'A', 'B'

print(list(numbers()))  # [1, 2, 3, 10, 20, 30, 'A', 'B']

Example 4: Custom Range Iterator

class MyRange:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

for num in MyRange(5, 10):
    print(num)   # 5, 6, 7, 8, 9

---

4. Hands-On Exercises

Exercise 1: Custom Range Iterator

Create a class MyRange(start, end) that works like Python's range(). Implement __iter__() and __next__(). It should yield numbers from start (inclusive) to end (exclusive). Test with for num in MyRange(5, 10): print(num).

Exercise 2: Generator for Even Numbers

Write a generator function even_numbers(n) that yields all even numbers from 0 up to n (inclusive). Use it to print even numbers up to 20.

Exercise 3: Generator Expression Replacer

Take an existing list comprehension from your codebase and convert it to a generator expression. Use next() to verify it still produces the same values. Measure memory with sys.getsizeof().

Exercise 4: File Word Counter with Generators

Write a generator read_words(filename) that yields one word at a time from a text file (split by spaces). Then use it to count total words without loading the entire file into memory.

Exercise 5: Infinite Sequence Taker

Write a generator count_up() that yields 1, 2, 3, ... infinitely. Then write a function take(n, generator) that returns a list of the first n items from any generator. Combine them to get the first 15 numbers.

---

5. Applied Challenge Task 🏗️

Log File Analysis Pipeline

Build a memory‑efficient log analysis system using generators.

Scenario: You have a large server log file (server.log) where each line looks like:

2026-05-07 10:15:23 INFO User login: alice
2026-05-07 10:16:01 ERROR Database connection failed
2026-05-07 10:17:45 WARNING Disk usage at 85%
2026-05-07 10:18:02 ERROR Timeout on request /api/data
2026-05-07 10:18:30 INFO User logout: alice

Tasks:

1. Create a pipeline of generators:

   • read_logs(filename) — yields each stripped line.
   • parse_log(lines) — yields (timestamp, level, message) tuples from each line.
   • filter_level(parsed, level) — yields only entries of a given level (INFO, ERROR, WARNING).

2. Build a LogAnalyzer class:

   • __init__(self, filename) — sets up the pipeline.
   • count_by_level(self) — returns a dictionary {level: count} for all entries.
   • recent_errors(self, n=5) — returns the last n ERROR entries.
   • summary(self) — prints total lines, errors, warnings, and info counts.

3. Handle edge cases:

   • File not found
   • Malformed log lines (skip gracefully)
   • Empty file

Example output:

=== LOG ANALYSIS SUMMARY ===
Total lines:     15000
ERROR:           23
WARNING:         145
INFO:            14832
Recent errors:
  2026-05-07 10:18:02 - Timeout on request /api/data
  2026-05-07 10:16:01 - Database connection failed
============================

Stretch goals:

• Add a time_range filter that only yields entries between two timestamps.
• Calculate error rate (errors per hour).
• Export filtered results to a new log file.

---

6. Brief Review Summary

Concept	Key Points
Iterable	Object with __iter__() that returns an iterator
Iterator	Object with __next__() that raises StopIteration when done
iter() / next()	Built‑ins that call __iter__() and __next__()
Generator function	Uses yield instead of return; pausing + resuming
Generator expression	(expr for item in iterable) — lazy version of list comprehension
yield from	Delegates to another generator
Why generators	Constant memory, lazy evaluation, composable pipelines
When not to use	Need indexing, length, or multiple iterations

---

7. Preview of Next Topic — Day 17

Tomorrow we explore one of Python's most elegant features:

• Decorators — functions that modify other functions
• The @decorator syntax
• Writing your own decorators
• Common built‑in decorators: @staticmethod, @classmethod, @property (review)
• Decorators with arguments
• Real‑world uses: timing, logging, authentication checks

---

🎯 Your Action Items for Day 16:

1. ✅ Complete all 5 exercises
2. ✅ Build the Log File Analysis Pipeline
3. ✅ Convert an old list comprehension to a generator and observe memory difference
4. ✅ Write a custom iterator class for something creative (playing cards, calendar days, etc.)