LESSON 62 · DEEPER PYTHON34 min

Read a JSON API with urllib

Decode a JSON response through an injected opener and validate its shape without requiring a live API.

Before you start: Read and Write JSON and CSV · Exceptions · Functions

Example check: Worked and guided code samples are checked against CPython 3.13. Practice uses Pyodide 0.29.5 (Python 3.13) in your browser. How we check lessons

WATCH

Video lesson: Read a JSON API with urllib

The preview is stored on this site. YouTube loads only when you press Play. Watch on YouTube

Jump to a chapter

Read the video transcript

Select a timestamp to watch that moment on YouTube.

00:00
A successful HTTP request does not prove that the response matches your application's contract. The server could return valid JSON with a different shape, an error document, or an empty body. A small Python program that extracts titles from an API needs to check more than the status of `json.load`. Today we will build that program with the standard library: `urllib.request.Request` for a URL and an Accept header, `urlopen` for the response, and `json.load` for decoding. The browser lesson must work without a live service, so the practical function accepts an opener argument.
00:40
In the video we pass a fake opener that returns a real `BytesIO` byte stream and records the request details. The code still executes; the response is simply controlled so every viewer sees the same behavior. You will watch a broken run fail on a valid but unexpected JSON object, inspect the decoded type, then repair the parser and verify malformed inputs. In production, the caller could pass `urllib.request.urlopen`. It should use an explicit timeout, handle HTTP and URL errors, and avoid putting credentials in a URL. Our teaching example never transmits data and uses the reserved example.test hostname only as a label in a Request object.
01:26
Predict the first failure before we run it: when Python loops over a dictionary, what values does the loop receive? The JSON text is syntactically valid. `json.load` returns a Python dictionary containing the key items. The list comprehension loops over that dictionary, so its first `item` is the string `items`, not a dictionary with a title field. Looking up `item["title"]` on a string raises TypeError. The traceback points at the comprehension, but the underlying cause is an unchecked assumption about the response shape. In a live application the same bug might appear only when a server changes its schema or sends an error object, making it hard to reproduce.
02:10
A broad `except Exception` around the whole request could hide the mismatch, but then a genuine defect might be converted into an empty result. We want a narrower fix: state the expected shape, inspect the decoded value, and reject anything else with a clear ValueError. We also need to inspect each list element and its title, because even a top-level list can contain numbers, missing keys, or blank text. The website's checked task requires those validations and preserves article order. Before coding the fix, let us run one diagnostic that prints the type and first value actually observed.
02:50
The type check explains the traceback: a dictionary iterates over keys, and the only key is items. We should not quietly use `payload["items"]` in this exercise because the chosen API contract is a top-level list. If a real API documents an object containing an items key, follow that documented contract instead. The Request object holds the URL and header we intend to send; it does not make a network call until an opener is invoked. That separation lets our fake opener inspect the Request and return controlled bytes. The response's `read()` method yields bytes, so `json.load` can decode the JSON from the file-like object.
03:34
A real network opener might raise an HTTPError for a non-success status or a URLError for a connection problem; those are different from a ValueError about valid JSON with the wrong shape. Keep those failure categories separate when adding error handling to a production client. We will now write a function that builds Request with an Accept header, calls the supplied opener with `timeout=5`, decodes in a `with` block, validates the top-level list, and checks every title. Our fake opener will also prove that the URL, header, and timeout are passed correctly. The function makes its dependency explicit.
04:15
It receives an opener rather than reaching for a global network call, so we can test the parsing behavior offline. The fake returns BytesIO, which supplies the same read and context-manager operations this code needs from a response. The returned list preserves the original order and strips surrounding title whitespace. The second printed line proves that the function constructed the expected Request and passed an explicit five-second timeout. In production, we could call `fetch_titles(url, urlopen)` after importing `urlopen`, with suitable handling for network failures in the caller. Here we do not need a live API or credentials to check the most important part of the program.
05:00
Notice the two layers of validation. `json.load` checks JSON syntax; our code checks the application structure. A list of numbers, a dictionary, or a record with a blank title is not accepted just because it is valid JSON. We could use a custom result type or schema library in a larger project, but the simple explicit checks make this lesson's contract visible. Do not put an API token in the sample URL; URLs can appear in logs and browser history. Follow the real API's documented authentication method instead. The first assertion proves that two valid titles remain in order after whitespace is removed.
05:44
The next four inputs are all legal JSON documents, but none satisfies our article-list contract. A dictionary is the wrong top-level type; a number is not an article; an empty dictionary has no title; and a title containing only spaces is blank after trimming. Each one must raise ValueError. The last assertion checks every recorded opener call for the URL, Accept header, and timeout. This test does not claim that a remote server is available or that a network connection succeeded. It proves the local parser's contract, leaving real HTTP errors to a separate integration test.
06:24
On the matching PythonLessonLab page, try the three solved examples before completing the browser task. Use the task checker to see which boundary condition you missed, then answer the quiz and inspect the common mistakes. When you build a real client, consult the service's documented response shape, handle the relevant HTTP and URL exceptions, and avoid logging full responses if they can contain private data. A trustworthy API client treats decoded JSON as untrusted input until its shape and fields have been checked. That extra validation turns an obscure downstream TypeError into a clear error near the request boundary.

THE IDEA

Understand the concept

urllib.request is part of Python's standard library and can send HTTP requests without a third-party package. A Request object holds a URL and headers; urlopen returns a response whose body is bytes. JSON can be decoded from that response with json.load, which accepts a binary file-like object containing UTF-8 JSON. Real network calls should use a timeout, handle HTTP and URL errors, and validate the response body before trusting it.

A browser console should not depend on an external API for an introductory exercise. This lesson injects an opener callable into fetch_titles(url, opener). Production code may pass urllib.request.urlopen; the lesson passes a fake opener returning io.BytesIO with a small JSON document. BytesIO is a context manager and supplies read(), so the same `with opener(request, timeout=5)` shape works without network access. The fake also records the Request URL, Accept header, and timeout for tests.

The expected API body is a JSON list of objects with nonblank string title fields. That is an application contract, not a guarantee from HTTP or JSON parsing. json.load can successfully decode a dictionary, a list of numbers, or null; each would be wrong for this function. Raise ValueError when the top-level list or an item is malformed. Let JSONDecodeError surface for invalid JSON, and let HTTPError or URLError surface from a real opener until the caller has a policy for retrying or explaining them.

The checked task builds a Request with Accept: application/json, opens it with a five-second timeout, decodes the response, validates its shape, and returns stripped titles in their original order. Hidden checks use a fake response and inspect the Request. Do not put API tokens in URL query strings or print response bodies containing personal data. For a real service, use its documented authentication scheme and terms.

Request carries URL and headers
urlopen returns a context-managed byte stream
Always set a timeout for real network calls
JSON syntax validation and application shape validation are separate

WORKED EXAMPLES

See it step by step

Read the code, predict the output, then compare it with the result.

01. Decode a binary JSON response locally

PYTHON

import json
from io import BytesIO

response = BytesIO(b'{"status":"ok","count":2}')
with response:
    data = json.load(response)
print(data["status"], data["count"])

EXPECTED OUTPUT

ok 2

json.load reads the file-like byte stream and produces Python values.

02. Build a Request without sending it

PYTHON

from urllib.request import Request

request = Request("https://example.test/items", headers={"Accept": "application/json"})
print(request.full_url)
print(request.get_header("Accept"))

EXPECTED OUTPUT

https://example.test/items
application/json

Request creation is local; opening it would perform the network operation.

03. Validate shape after valid JSON parses

PYTHON

import json

for text in ('{"title":"A"}', '[{"title":"A"}]'):
    value = json.loads(text)
    print(isinstance(value, list))

EXPECTED OUTPUT

False
True

Both inputs are valid JSON, but only the list matches this function's API contract.

GO DEEPER

A closer look

Follow the reasoning, inspect each result, then try the suggested changes in the console below.

01 / 03

Separate valid JSON from the required API shape

A response body can be syntactically valid JSON while still violating an application's contract. A dictionary with a title is valid JSON, but fetch_titles expects a list of article objects. Starting with a local byte stream removes the network from this observation. json.load decodes it, then the function must inspect the resulting Python type.

Treat the list rule as your application's rule, not as something urllib or json will enforce automatically. Another API may return an object with an items field instead. Read that service's documentation and code for its shape. In this tutorial, every list item needs a nonblank string title, and the returned list preserves server order after trimming whitespace.

PYTHON

import json
from io import BytesIO

for body in (b'{"title":"A"}', b'[{"title":"A"}]'):
    with BytesIO(body) as response:
        value = json.load(response)
    print(type(value).__name__, isinstance(value, list))

EXPECTED OUTPUT

dict False
list True

Follow the reasoning

Decode two syntactically valid byte streams.
Compare their Python shapes against the expected list contract.

02 / 03

Prepare a request and inject the opener

Request construction does not send data. It packages the URL and headers, including Accept: application/json. The network operation occurs when the opener is called. Passing that opener into the function lets a browser exercise supply a fake that returns BytesIO, while a local production script can supply urllib.request.urlopen. The function uses the same context-manager pattern with either one.

An explicit five-second timeout is part of the call contract. A real service can still fail with HTTPError, URLError, or malformed JSON; this small function leaves those exceptions visible for the caller to handle according to its use case. Hiding every error behind an empty list would make an outage look like a valid empty response.

PYTHON

import json
from io import BytesIO
from urllib.request import Request

seen = []
def fake_open(request, timeout):
    seen.append((request.full_url, request.get_header("Accept"), timeout))
    return BytesIO(b'[{"title":" First "}]')

request = Request("https://example.test/articles", headers={"Accept": "application/json"})
with fake_open(request, timeout=5) as response:
    payload = json.load(response)
print(payload[0]["title"].strip())
print(seen)

EXPECTED OUTPUT

First
[('https://example.test/articles', 'application/json', 5)]

Follow the reasoning

Inspect the recorded URL, header, and timeout without making a network call.
Decode and trim the title from the controlled response.

03 / 03

Validate every item before returning titles

The completed function decodes once, rejects a non-list top-level result, then examines each item. A title must be a string that remains nonblank after trimming. Building a new list preserves item order and avoids mutating the decoded response. The fake opener provides an offline success case and an offline malformed case, so the behavior is stable across machines.

For real use, pass urlopen as the opener with an HTTPS URL and keep the timeout. Follow the service's authentication rules, avoid putting credentials in URLs, and do not log private response bodies. Those concerns are separate from the parsing exercise. A test with an injected opener cannot prove a remote service is available; it proves this function sends the intended request and handles the promised response shape.

PYTHON

import json
from io import BytesIO
from urllib.request import Request

def fetch_titles(url, opener):
    request = Request(url, headers={"Accept": "application/json"})
    with opener(request, timeout=5) as response:
        payload = json.load(response)
    if not isinstance(payload, list):
        raise ValueError("expected a list")
    titles = []
    for item in payload:
        if not isinstance(item, dict) or not isinstance(item.get("title"), str) or not item["title"].strip():
            raise ValueError("invalid title")
        titles.append(item["title"].strip())
    return titles

print(fetch_titles("https://example.test/articles", lambda request, timeout: BytesIO(b'[{"title":" A "},{"title":"B"}]')))
try:
    fetch_titles("https://example.test/articles", lambda request, timeout: BytesIO(b'{}'))
except ValueError as error:
    print(error)

EXPECTED OUTPUT

['A', 'B']
expected a list

Follow the reasoning

Return stripped titles in their original list order.
Reject a valid JSON object that does not match this API contract.

HANDS-ON

Try it in Python

Edit the example and run it. Python starts in your browser the first time you click Run.

Python console

Ready to run

import json
from io import BytesIO
from urllib.request import Request

def fetch_titles(url, opener):
    # Send Accept: application/json and timeout=5.
    # Validate a list of dictionaries with nonblank title strings.
    return []

def fake_open(request, timeout):
    return BytesIO(b'[{"title": " First "}, {"title": "Second"}]')

print(fetch_titles("https://example.test/articles", fake_open))

Need input()? Add one value per line

Input lines

Ctrl / ⌘ + Enter to run

OUTPUT

Your output appears here.

Your turnComplete fetch_titles(url, opener). Build Request(url, headers={'Accept': 'application/json'}); open with `opener(request, timeout=5)` in a with block and decode with json.load(response). Require a list of dictionaries, each with a nonblank string title. Return stripped titles in order. Raise ValueError for a wrong shape. The supplied fake makes the prepared call print ['First', 'Second'] without network access.

Need a hint?

Use `with opener(Request(url, headers={'Accept':'application/json'}), timeout=5) as response: payload = json.load(response)`. Check `isinstance(payload, list)` and each item's title before adding `title.strip()` to a result list.

Complete the task and select Check task to verify your code.

DEBUG BETTER

Typical mistakes

Everyone meets these errors. See what causes them and how to fix them.

Making an unchecked network call

COMMON MISTAKE

with urlopen(url) as response: data = json.load(response)

FIX

with urlopen(request, timeout=5) as response:
    data = json.load(response)
if not isinstance(data, list):
    raise ValueError('expected list')

What happens: The operation can wait indefinitely or return an unexpected shape.
The timeout bounds waiting; the shape check rejects a non-list response. Validate individual items before using them, as in the next fix.

Assuming valid JSON always has a title

COMMON MISTAKE

return [item['title'] for item in data]

FIX

if not isinstance(data, list):
    raise ValueError('expected list')
titles = []
for item in data:
    if not isinstance(item, dict):
        raise ValueError('expected object')
    title = item.get('title')
    if not isinstance(title, str) or not title.strip():
        raise ValueError('expected nonblank title')
    titles.append(title.strip())
return titles

What happens: A dictionary, numeric item, or missing title can cause an unrelated error or misleading result.
Validate the list and each object before extracting a stripped, nonblank title.

Embedding a credential in the URL

COMMON MISTAKE

url = 'https://api.example.test/items?token=' + token

FIX

request = Request(url, headers={'Authorization': 'Bearer ' + token})

What happens: URLs may be retained in logs and browser histories.
Follow the service's documented authentication method and avoid logging secrets.

Browse this lesson’s problem fixes

VERIFY THE STORY

Sources and further reading