Python Generators Tutorial – Complete Guide with Examples

Python - Generators 

When working with large data in Python, storing everything in memory can be inefficient. To solve this problem, Python provides Generators.

Generators allow you to produce values one at a time, only when needed, instead of storing everything in memory at once.

In this tutorial, you will learn what generators are, how they work, and how to use them in Python with practical examples.

---

What is a Generator?

A generator is:

A special type of iterator that generates values on the fly using the yield keyword.

Instead of returning all values at once, it produces them one by one.

---

Why Use Generators?

Generators are useful because they:

• Save memory
• Handle large datasets efficiently
• Improve performance
• Produce values lazily (on demand)
• Avoid storing entire data in memory

---

Generator vs Normal Function

Function	Generator
Uses return	Uses yield
Returns all values at once	Returns one value at a time
Memory heavy	Memory efficient

---

Creating a Simple Generator

Generators are created using the yield keyword.

---

Example

def my_generator():
    yield 1
    yield 2
    yield 3

gen = my_generator()

print(next(gen))
print(next(gen))
print(next(gen))

---

Output

1
2
3

---

How Generators Work

• yield pauses function execution
• Function resumes from last yield point
• Values are generated one at a time

---

Generator in For Loop

def numbers():
    yield 1
    yield 2
    yield 3

for num in numbers():
    print(num)

---

Why This Works

Python automatically calls next() internally.

---

Generator with Loop

def count(n):
    i = 1
    while i <= n:
        yield i
        i += 1

for value in count(5):
    print(value)

---

Output

1
2
3
4
5

---

Generator Expression

Like list comprehension, but uses parentheses.

---

Example

gen = (x * x for x in range(5))

print(next(gen))
print(next(gen))

---

Output

0
1

---

Memory Efficiency Example

List (High Memory)

numbers = [x for x in range(1000000)]

---

Generator (Low Memory)

numbers = (x for x in range(1000000))

---

Difference: List vs Generator

Feature	List	Generator
Memory	High	Low
Speed	Fast access	Lazy processing
Storage	Full data	One item at a time

---

Infinite Generator Example

def infinite_numbers():
    i = 1
    while True:
        yield i
        i += 1

gen = infinite_numbers()

print(next(gen))
print(next(gen))

---

Important Note

⚠️ Infinite generators must be handled carefully to avoid endless loops.

---

Real-World Example: Data Streaming

def data_stream():
    yield "packet1"
    yield "packet2"
    yield "packet3"

for data in data_stream():
    print("Processing:", data)

---

Real-World Example: File Reading

def read_lines(file):
    with open(file, "r") as f:
        for line in f:
            yield line.strip()

---

Why Generators Are Powerful

• Used in big data processing
• Used in web frameworks
• Used in pipelines
• Used in streaming systems

---

Advantages of Generators

• Memory efficient
• Faster execution for large data
• Easy to implement
• Lazy evaluation
• Ideal for streaming data

---

Disadvantages

• Cannot reuse once exhausted
• No indexing support
• Harder to debug than lists

---

Common Mistakes

1. Trying to reuse generator

gen = my_generator()
list(gen)
list(gen)  # empty

---

2. Confusing return and yield

---

3. Not handling StopIteration

---

Best Practices

1. Use generators for large data

---

2. Prefer yield over return for sequences

---

3. Use generator expressions for simple cases

---

4. Avoid unnecessary storage of data

---

Summary

Generators in Python are powerful tools for creating iterators that generate values on demand. They help reduce memory usage and improve performance, especially when working with large datasets or continuous data streams.

Key Takeaways

• Generators use yield
• They produce values one at a time
• Memory efficient compared to lists
• Ideal for large or infinite data
• Widely used in real-world applications

Mastering generators is essential for writing efficient and scalable Python programs.