LESSON 39 · DEEPER PYTHON30 min

Build Small Data Pipelines with itertools

Compose iterator tools to stream data, bound consumption, compute running totals, and understand consecutive grouping.

Before you start: Iterators and Generators in Python · Loops · Functions

Example check: Worked and guided code samples are checked against CPython 3.13. Practice uses Pyodide 0.29.5 (Python 3.13) in your browser. How we check lessons

WATCH

Video lesson: Build Small Data Pipelines with itertools

The preview is stored on this site. YouTube loads only when you press Play. Watch on YouTube

Jump to a chapter

Read the video transcript

Select a timestamp to watch that moment on YouTube.

00:00
A stream of readings can be processed as it arrives. For each new reading, we want the running total: after four, the total is four; after minus one, it is three; after three more, it is six. If the user asks for only the first three totals, our program should read only three source values. That second requirement matters when the source is a file, a network feed, or a generator with side effects. A tempting implementation builds every running total into a list and then slices off the first three. The returned list looks correct, but the source may have been completely consumed. In this lesson we will make that hidden cost visible, replace it with a lazy combination of `itertools.accumulate` and `itertools.islice`, and verify how many inputs were actually pulled.
00:49
We will also look briefly at `chain` and `groupby`, two related tools for stream processing. The companion PythonLessonLab page includes runnable examples, a checked task, a quiz, and common mistakes. Before running each snippet, predict not only the printed totals but also which source values had to be read. Thinking about consumption is essential when you work with iterators. Once a generator yields an item, that item is gone from that generator object unless you stored it elsewhere or can recreate the source. A correct-looking final value is not enough if obtaining it exhausted a stream that another stage expected to use.
01:31
The requested totals are four, three, and six, so the first line passes a superficial check. The second line shows the problem: the generator yielded all four readings. `list(accumulate(readings()))` consumes the entire accumulate iterator before the slice takes three elements from that completed list. If the source had a million readings, this would build a million running totals only to discard nearly all of them. If the source were infinite, converting it to a list would never finish. If reading each item triggered a database call or another side effect, those extra effects would happen even though we requested only three results.
02:11
This is a common pipeline mistake: moving a limiting operation to the end of a computation changes how much upstream work is performed. Slicing a list and limiting an iterator are not interchangeable when the amount of consumed input matters. The correction is to limit the iterator before converting the limited portion to a list. `islice(accumulate(readings()), 3)` asks accumulate for at most three values, and accumulate needs one source reading for each of those values. Then list collects only the requested portion. We will verify that with the same consumed log rather than assuming the helper is lazy.
02:52
This time the consumed log stops after the third reading. `accumulate` emits a new running total as each input arrives. `islice` requests only its first three outputs, so the fourth source value is never needed. The final list conversion is safe for this bounded slice. `chain` shows another common pipeline idea: iterate over one input and then another as a continuous sequence. Here we converted its short output to a list for display, but chain itself does not need to build the combined sequence up front. Lazy tools are most useful when the source is large, when we may stop early, or when we want small processing stages that compose.
03:33
Be precise about what laziness does not promise. It does not make expensive calculations free, and a later `list(...)` can still force full consumption if no limit is applied. Nor can we restart a generator after it has already been consumed. The lesson task asks for `first_running_totals(readings, count)` and should use accumulate followed by islice. The order is the key part: the limit belongs around the iterator, before list. Passing zero should request no values at all. If count exceeds the finite source length, the result should contain all available totals without inventing more. The function is short because the standard-library tools each have a clear job.
04:17
Accumulate produces running totals; islice takes a bounded prefix; list gives the caller a reusable result. The sample output matches our prediction and the consumed log confirms the fourth reading was untouched. That log is important because two implementations can return the same list while having different effects on a source stream. The function accepts any iterable of compatible numbers, including a list or a one-pass generator. The returned totals are stored in a list, but only up to count elements are materialized. If you needed to preserve laziness all the way to the caller, you could return an iterator instead, but that would be a different contract: the caller would then manage one-pass consumption.
05:03
For this lesson, a bounded list is straightforward to inspect and check. Use a nonnegative count; `islice` rejects a negative stop value, which is appropriate to surface rather than silently reinterpret as a list slice from the end. On PythonLessonLab, complete this exact function signature and run the practical checker. Try a much longer generator that records which inputs were requested. A correct solution should never pull a fourth reading when count is three. That is the central property of the exercise, not merely the numerical totals. The assertions cover the result and the work performed.
05:43
Only three source items were read for three totals. Empty input produces an empty list, a zero count requests no output, and a count larger than a finite input returns all available running totals. There is no padding or extra value. The `groupby` line demonstrates another iterator trap. It groups consecutive equal values, so the As before B form one group and the As after B form a second group. It does not aggregate all As into one global bucket. If that is what you need, sort by the grouping key first or use a dictionary-based accumulator, depending on whether order matters.
06:23
This is why pipeline tools must be understood through their exact iteration rules. On the companion lesson, run the broken full-conversion version and the corrected limited version with a generator that logs consumption. Then complete the task and quiz. As a follow-up, change count to one and predict how many readings are touched. Change it to zero and confirm that your generator body never executes. In real data processing, this discipline can save memory, avoid unnecessary network calls, and keep one-pass sources available for later work. A pipeline is not just a sequence of values; it is also a schedule of when each value is requested.
07:05
Testing that schedule is sometimes as important as testing the final list.

THE IDEA

Understand the concept

itertools contains building blocks for iterator pipelines. chain connects iterables without copying all their items into a new list first. islice requests only a selected portion, and accumulate computes running results as inputs arrive. The tools are lazy: constructing a pipeline does not pull every value immediately. A for loop, list(), or next() drives it. This is especially useful when the source is large or has an expensive read step.

The order of the pipeline matters. islice(accumulate(readings), count) asks for at most count running totals, so a one-pass readings iterator only supplies the needed inputs. In contrast, list(accumulate(readings)) computes the entire stream before slicing, losing the bounded-consumption benefit. count=0 should consume nothing, and an empty source naturally gives an empty result.

groupby groups consecutive items with the same key, not all matching items across the entire input. If statuses are ok, ok, fail, ok, groupby produces three runs. To group globally by status, sort by that key first or collect into a dictionary, depending on whether ordering and memory costs matter. Each group iterator shares the underlying source; consume a group before advancing to the next one.

The checked exercise takes numeric readings and returns at most count running totals. Use accumulate for the rolling sum, islice to bound the stream, and list to return the small requested result. Test positive and negative readings, a generator input, count=0, empty input, and an iterator whose later values must not be requested. In a real pipeline, choose your stage order deliberately so one stage does not drain a stream before another needs it.

chain joins iterables lazily
accumulate emits running results
islice bounds consumption
groupby groups consecutive matching keys

WORKED EXAMPLES

See it step by step

Read the code, predict the output, then compare it with the result.

01. Chain two batches in source order

PYTHON

from itertools import chain

morning = [3, 4]
evening = [5, 6]
stream = chain(morning, evening)
print(next(stream))
print(list(stream))

EXPECTED OUTPUT

3
[4, 5, 6]

chain yields one batch after the other; next consumes the first item, so list sees only the rest.

02. Bound a running-total stream

PYTHON

from itertools import accumulate, islice

readings = [4, -1, 3, 7]
print(list(islice(accumulate(readings), 3)))
print(readings)

EXPECTED OUTPUT

[4, 3, 6]
[4, -1, 3, 7]

accumulate produces 4, then 3, then 6; islice stops after three totals and leaves the source list unchanged.

03. Group consecutive status runs

PYTHON

from itertools import groupby

statuses = ['ok', 'ok', 'fail', 'ok']
runs = [(status, len(list(group))) for status, group in groupby(statuses)]
print(runs)

EXPECTED OUTPUT

[('ok', 2), ('fail', 1), ('ok', 1)]

The last ok is a new run because fail separated it from the first two ok values.

GO DEEPER

A closer look

Follow the reasoning, inspect each result, then try the suggested changes in the console below.

01 / 03

Join two sources without copying them first

itertools.chain presents multiple iterables as one iterator. It yields all items from the first source, then all from the second, without first building a combined list. This matters when a source is a generator or when the combined data would be large. Creating chain does not pull values from either source. A consumer such as next, a for loop, or list drives the actual reads.

In the demonstration, the generator announces each value only when chain requests it. The first next reads one morning value. Converting the remaining chain to a list then pulls the second morning value and both evening values. Once consumed, the same chain object is exhausted. Recreate the pipeline for another pass or keep the resulting list if repeated access is more important than memory.

PYTHON

from itertools import chain

def readings(label, values):
    for value in values:
        print('read', label, value)
        yield value

stream = chain(readings('am', [2, 3]), readings('pm', [4, 5]))
print('pipeline ready')
print(next(stream))
print(list(stream))

EXPECTED OUTPUT

pipeline ready
read am 2
2
read am 3
read pm 4
read pm 5
[3, 4, 5]

Follow the reasoning

Constructing chain prints no read lines.
next requests just the first morning value.
list consumes the remaining three values in source order.

02 / 03

Bound a running calculation before collecting it

accumulate produces a running value for each input. With numbers and no function argument, that value is the cumulative sum. islice can limit how much of that stream a downstream consumer asks for. The order matters: list(islice(accumulate(source), 3)) reads at most three source items, while list(accumulate(source))[:3] reads the whole source before discarding later totals.

The seen list below records every reading actually requested from a generator. After three cumulative totals, the fourth reading is still untouched. The lesson's checked function first_running_totals uses this arrangement and returns a list because the requested portion is deliberately small. For count zero, islice requests nothing. For a production pipeline, beware of hidden materialization inside helper functions; one eager stage can erase the savings of lazy stages before it.

PYTHON

from itertools import accumulate, islice

seen = []
def readings():
    for number in [4, -1, 3, 7]:
        seen.append(number)
        yield number

totals = list(islice(accumulate(readings()), 3))
print(totals)
print(seen)

EXPECTED OUTPUT

[4, 3, 6]
[4, -1, 3]

Follow the reasoning

accumulate adds each incoming reading to the running total.
islice stops the consumer after three totals.
The fourth source value, 7, has not been requested.

03 / 03

Recognize consecutive groups rather than global buckets

groupby starts a new group whenever the key of the next item differs from the current key. It is well suited to consecutive runs in log events or readings already sorted by the grouping key. It does not automatically gather every matching value from across an unsorted stream. In the example, two ok values form one run, fail forms another, and the final ok begins a third run. The underlying iterator is shared with each group, so consume a group before moving on.

If the question is how many events of each status occurred anywhere, use a counting dictionary or sort by status first and then group. Sorting may change the original order and requires collecting values, so choose it only when that tradeoff fits. For streaming run detection, the original consecutive order is the very thing you want. Try adding another fail at the end and predict whether it joins the earlier fail or starts a new run.

PYTHON

from itertools import groupby

statuses = ['ok', 'ok', 'fail', 'ok']
runs = [(status, len(list(group))) for status, group in groupby(statuses)]
print(runs)
print(sum(size for _, size in runs))

EXPECTED OUTPUT

[('ok', 2), ('fail', 1), ('ok', 1)]
4

Follow the reasoning

groupby begins a new run when the status changes.
The second ok run is separate because fail sits between the two ok stretches.
The run sizes still add up to all four source events.

HANDS-ON

Try it in Python

Edit the example and run it. Python starts in your browser the first time you click Run.

Python console

Ready to run

from itertools import accumulate, islice

def first_running_totals(readings, count):
    # Return up to count cumulative sums without reading later items.
    return []

print(first_running_totals([4, -1, 3, 7], 3))

Need input()? Add one value per line

Input lines

Ctrl / ⌘ + Enter to run

OUTPUT

Your output appears here.

Your turnComplete first_running_totals(readings, count). Return a list containing at most count cumulative sums, without consuming any later readings. Use itertools.accumulate and itertools.islice. Assume count is a nonnegative integer. The prepared example should print [4, 3, 6]. Empty input and count=0 should return [].

Need a hint?

Wrap accumulate(readings) in islice(..., count), then pass that bounded iterator to list(). Keep islice before list so extra readings are not consumed.

Complete the task and select Check task to verify your code.

DEBUG BETTER

Typical mistakes

Everyone meets these errors. See what causes them and how to fix them.

Materializing everything before taking a few values

COMMON MISTAKE

first = list(accumulate(readings))[:3]

FIX

first = list(islice(accumulate(readings), 3))

What happens: The entire readings iterator is consumed, even when only three totals are needed.
Place islice before list so upstream work stops at the requested count.

Treating groupby as global grouping

COMMON MISTAKE

from itertools import groupby
print([(key, list(group)) for key, group in groupby(['ok', 'fail', 'ok'])])

FIX

from itertools import groupby
statuses = ['ok', 'fail', 'ok']
ordered = sorted(statuses)
print([(key, list(group)) for key, group in groupby(ordered)])

What happens: There are two separate ok groups, not one combined group.
Sort first when you need one group per key, and consider whether sorting is acceptable for your data.

Trying to reuse an exhausted pipeline

COMMON MISTAKE

stream = chain([1], [2])
print(list(stream))
print(list(stream))

FIX

stream = chain([1], [2])
values = list(stream)
print(values)
print(values)

What happens: The second list is empty because the iterator has already been consumed.
Recreate an iterator for another pass or store the results if repeated access is required.

Browse this lesson’s problem fixes

VERIFY THE STORY

Sources and further reading