LESSON 85 · ADVANCED PROJECT43 min

Project: Offline Web Crawler on a Fixture Graph

Traverse an in-memory graph of HTML page fixtures in stable breadth-first order without network requests.

Before you start: Dictionaries · Functions · Build Queues and Stacks with deque · Build Query Strings with urllib.parse

Example check: Worked and guided code samples are checked against CPython 3.13. Practice uses Pyodide 0.29.5 (Python 3.13) in your browser. How we check lessons

WATCH

Video lesson: Project: Offline Web Crawler on a Fixture Graph

The preview is stored on this site. YouTube loads only when you press Play. Watch on YouTube

Jump to a chapter

Read the video transcript

Select a timestamp to watch that moment on YouTube.

00:00
The word crawler often suggests a program roaming the open web. This project is narrower and safer to reason about: every page is a string in a Python dictionary supplied to the function. The dictionary maps absolute URL labels to fixture HTML. There are no HTTP requests, no login, and no claim that the code can visit arbitrary websites. We will start at one known URL label and follow only links whose normalized target is another key in that fixture mapping. The first implementation will use depth- first recursion. It visits all four supplied pages, but it reaches C before B, violating the breadth-
00:40
first order we want. Then we will parse anchor href attributes using HTMLParser, resolve relative URLs with urljoin, remove fragments, and use a queue to visit the closest linked pages first. We will mark a URL as seen when it is enqueued, so two parents cannot add the same page twice. A max_pages limit will stop the traversal predictably. The matching PythonLessonLab mini-project has a checked task and a related local GitHub example you can inspect after the lesson. First, let's make the wrong order visible on this tiny graph. The depth-first function follows the first root link to A and then immediately
01:22
follows A's link to C. Only after that branch returns does it visit B. The cycle from C back to A does not loop forever because seen prevents revisits. So the code is not completely broken; it finds the reachable fixture labels. It is broken relative to our traversal contract, which says breadth first. B is one link away from the root, while C is two links away, so B should appear first. This matters if max_pages is small: with a limit of three, a depth-first walk would spend one of its slots on C while missing B, even though B was a direct root link. We will use a deque queue and enqueue the root's children in source order.
02:04
A second issue is hidden in this first demo: it already has a tidy links dictionary. Real fixture HTML contains href attributes, including relative paths and fragments. We need to extract and normalize those labels without fetching anything from the network. HTMLParser calls handle_starttag as it encounters anchors, so the href list retains source order. urljoin turns a relative path such as slash-a into an absolute URL label based on the current fixture page. urldefrag removes the fragment identifier. A link to C number-sign-part is still a link to the C page; the fragment refers to
02:45
a location within that page, not a separate page for our traversal. We are not opening these URLs. They are keys in an in-memory mapping. After normalization, the traversal must check whether each target actually exists in the supplied pages dictionary. An external link, a missing fixture URL, or a mailto label has no matching key and is not enqueued. This is an important boundary: a function called crawl_fixture should never silently start making network requests just because an href looks like a URL. We also need to avoid adding the same page twice when A and B both link to C.
03:25
Marking a target seen at enqueue time prevents duplicate queue entries. Marking only after dequeue would let both parents add C, wasting slots and making a max_pages rule harder to reason about. The result now follows breadth-first order. The root is visited first. Its anchors add A and B to the queue in the order they appear in the HTML. Visiting A discovers C and adds it after B. Visiting B also points to C, but C is already in seen, so it is not queued again. C points back to A, which is also already seen. The cycle ends naturally. The queue's order and enqueue-time seen marking explain the output; we are not
04:08
relying on dictionary iteration order to decide what to visit next. The function rejects a missing start label and invalid limits before traversal. Its parser only reads strings supplied in pages, and the membership check ensures a link to a URL outside that mapping is ignored. There is no network library and no request call anywhere in this implementation. That is why I call it an offline fixture crawler, not a general web crawler. A networked version would need permission to fetch, robots rules, timeouts, rate limits, content limits, and careful URL scope; those are separate design problems.
04:48
This project isolates the graph traversal and normalization logic so we can test it fully. With max_pages equal to two, the traversal visits only the root and A, the first direct link. With a larger limit, the full order is root, A, B, C. The fragment on A's C link is removed so C is found by its page key, and the repeated C link from B does not duplicate the visit. The off-fixture outside URL is parsed as a link but ignored because it is not a key in pages. No request is sent there. The final loop checks a missing start, zero limit, and boolean limit. Booleans need special care in Python because True is an int subclass, but the
05:31
contract calls for an ordinary positive integer. In the browser task, implement the same queue, normalization, and validation yourself. If C appears before B, inspect whether you used recursion or a stack instead of a queue. If C appears twice, mark seen when you enqueue. If a fragment creates a phantom page, remove it before the membership check. Once the exercise passes, inspect the related local GitHub project as a separate next step. Keep the boundary clear: this lesson teaches traversal over supplied HTML fixtures; connecting to real websites requires additional policies and is not demonstrated here.

THE IDEA

Understand the concept

A crawler discovers pages by following links, but the browser console cannot safely fetch arbitrary external sites. This project models the traversal algorithm with a dictionary whose keys are page URLs and whose values are small HTML strings. The fixture graph behaves like a miniature site and makes every visit deterministic. No code in this lesson sends an HTTP request. A separate local GitHub project can later adapt the same ideas to real network access under robots rules and rate limits.

HTMLParser extracts href attributes from anchor tags in document order. urljoin resolves relative links against the current page, and urldefrag removes fragments so /a#top and /a refer to one page. Only resolved URLs present in the fixture mapping are eligible for the queue. External or unknown links are skipped; that is a fixture boundary, not a claim that those websites do not exist.

Breadth-first search uses deque for an efficient front-of-queue removal. A queued set records URLs as soon as they are scheduled, preventing duplicates and cycles such as home → about → home. A max_pages limit stops traversal after a bounded number of visits even when more pages are queued. The result is a list of visited URLs in stable breadth-first order, which learners can compare with a hand-drawn graph.

The checked task includes a cycle, two paths to one page, a fragment link, an external link, an empty one-page fixture, an invalid start URL, and an invalid limit. Real crawling adds network failures, robots.txt, delays, canonical URLs, content-type checks, and legal or operational restrictions. This offline lesson deliberately isolates the graph algorithm so the browser exercise remains runnable and truthful.

A fixture graph is not a network crawl
HTMLParser extracts links in source order
urljoin and urldefrag normalize local navigation
A queued set and page limit prevent cycles and unbounded traversal

WORKED EXAMPLES

See it step by step

Read the code, predict the output, then compare it with the result.

01. Extract hrefs in document order

PYTHON

from html.parser import HTMLParser
class Links(HTMLParser):
    def __init__(self): super().__init__(); self.found=[]
    def handle_starttag(self, tag, attrs):
        if tag=='a': self.found.extend(v for k,v in attrs if k=='href' and v)
p=Links(); p.feed('<a href="/a">A</a><a href="/b">B</a>'); print(p.found)

EXPECTED OUTPUT

['/a', '/b']

The parser reads anchor attributes without searching HTML with a fragile regex.

02. Resolve a fragment-bearing relative link

PYTHON

from urllib.parse import urljoin, urldefrag
base='https://example.test/'
print(urldefrag(urljoin(base,'/a#part')).url)

EXPECTED OUTPUT

https://example.test/a

Fragment removal maps two sections of one fixture page to one queued URL.

03. Use a queue and seen set for a cycle

PYTHON

from collections import deque
graph={'home':['about'],'about':['home']}
queue=deque(['home']); seen={'home'}; visited=[]
while queue:
    current=queue.popleft(); visited.append(current)
    for nxt in graph[current]:
        if nxt not in seen: seen.add(nxt); queue.append(nxt)
print(visited)

EXPECTED OUTPUT

['home', 'about']

Marking a URL when queued prevents the home/about cycle from revisiting home.

GO DEEPER

A closer look

Follow the reasoning, inspect each result, then try the suggested changes in the console below.

01 / 03

Reproduce a cycle in a fixture graph

A crawler follows a link from home to about and another link from about back to home. Without a seen set, the queue keeps scheduling pages it already visited. A page limit stops this tiny demonstration from looping forever, but the output still wastes visits and may hide a page that would have appeared later.

The graph below is an in-memory model, not a network request. Its purpose is to make traversal order and cycle behavior visible before HTML parsing enters the picture. The corrected algorithm marks a URL as seen when it is queued, so two different source pages cannot schedule it twice.

PYTHON

graph={'home':['about'],'about':['home']}
queue=['home']; visited=[]
while queue and len(visited)<4:
    current=queue.pop(0); visited.append(current)
    queue.extend(graph[current])
print(visited)

EXPECTED OUTPUT

['home', 'about', 'home', 'about']

Follow the reasoning

The graph has only two distinct pages.
Without seen-on-enqueue, the queue repeats them.
A page limit bounds damage but does not fix duplication.

02 / 03

Extract links and resolve fixture URLs

HTMLParser calls handle_starttag for each anchor tag. Collect its href attributes in document order instead of trying to parse HTML with a simple regular expression. urljoin turns a relative path into an absolute URL using the current page as the base. urldefrag removes an in-page fragment before looking up the fixture key.

The resolved URL can be visited only if it exists in the pages dictionary. A link to another host or an absent path is skipped by this offline exercise. It does not imply that the external page is unavailable; it only means the local fixture does not contain it. This boundary keeps the browser demo deterministic.

PYTHON

from html.parser import HTMLParser
from urllib.parse import urljoin, urldefrag
class Links(HTMLParser):
    def __init__(self): super().__init__(); self.found=[]
    def handle_starttag(self,tag,attrs):
        if tag=='a': self.found.extend(v for k,v in attrs if k=='href' and v)
p=Links(); p.feed('<a href="/a#top">A</a><a href="/b">B</a>')
print([urldefrag(urljoin('https://example.test/',href)).url for href in p.found])

EXPECTED OUTPUT

['https://example.test/a', 'https://example.test/b']

Follow the reasoning

Parse anchor attributes in source order.
Resolve each relative path against its page.
Drop fragments before fixture lookup.

03 / 03

Traverse once, breadth first, with a limit

Use deque to remove the next URL from the front of the queue efficiently. Keep a queued set containing every URL already scheduled, starting with the root. After visiting a fixture page, parse its links, normalize them, and queue only fixture URLs that have not appeared before. Stop when the queue empties or max_pages visits have happened.

Validate the start URL and positive integer limit before traversing. The prepared graph has two paths to c and a cycle through home; both should still yield each page once in breadth-first order. The website checker also exercises fragments, unknown links, a one-page fixture, and invalid arguments. This is a graph lesson, not live scraping.

PYTHON

from collections import deque
from html.parser import HTMLParser
from urllib.parse import urljoin, urldefrag
class LinkParser(HTMLParser):
    def __init__(self): super().__init__(); self.links=[]
    def handle_starttag(self,tag,attrs):
        if tag=='a': self.links.extend(value for key,value in attrs if key=='href' and value)
def crawl_fixture(pages,start,max_pages=10):
    if start not in pages or type(max_pages) is not int or max_pages<=0:
        raise ValueError('invalid start or page limit')
    queue=deque([start]); queued={start}; visited=[]
    while queue and len(visited)<max_pages:
        url=queue.popleft(); visited.append(url)
        parser=LinkParser(); parser.feed(pages[url])
        for href in parser.links:
            target=urldefrag(urljoin(url,href)).url
            if target in pages and target not in queued:
                queued.add(target); queue.append(target)
    return visited
root='https://example.test/'
pages={root:'<a href="/a">A</a><a href="/b">B</a>',root+'a':'<a href="/">Home</a><a href="/c#part">C</a>',root+'b':'<a href="/c">C</a>',root+'c':'<a href="/a">A</a>'}
print(crawl_fixture(pages,root,4))

EXPECTED OUTPUT

['https://example.test/', 'https://example.test/a', 'https://example.test/b', 'https://example.test/c']

Follow the reasoning

Mark a URL seen when queued.
Visit only pages present in the fixture mapping.
Stop after the requested number of unique visits.

HANDS-ON

Try it in Python

Edit the example and run it. Python starts in your browser the first time you click Run.

Python console

Ready to run

from html.parser import HTMLParser
from urllib.parse import urljoin

class LinkParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links = []
    def handle_starttag(self, tag, attrs):
        if tag == 'a':
            self.links.extend(value for key, value in attrs if key == 'href' and value)

def crawl_fixture(pages, start, max_pages=10):
    queue, visited = [start], []
    while queue and len(visited) < max_pages:
        url = queue.pop(0)
        visited.append(url)
        parser = LinkParser()
        parser.feed(pages[url])
        queue.extend(urljoin(url, href) for href in parser.links if urljoin(url, href) in pages)
    return visited

root = 'https://example.test/'
pages = {root:'<a href="/a">A</a><a href="/b">B</a>', root+'a':'<a href="/">Home</a><a href="/c#part">C</a>', root+'b':'<a href="/c">C</a>', root+'c':'<a href="/a">A</a>'}
print(crawl_fixture(pages, root, 4))

Need input()? Add one value per line

Input lines

Ctrl / ⌘ + Enter to run

OUTPUT

Your output appears here.

Your turnComplete crawl_fixture(pages, start, max_pages=10). Parse anchor hrefs, resolve relative links with urljoin, remove fragments with urldefrag, and visit only URLs present in pages. Return stable breadth-first visit order; schedule each URL once; stop after max_pages visits. Reject an absent start or a nonpositive/non-integer limit with ValueError. Prepared output must be home, a, b, c once each.

Need a hint?

Use collections.deque and a queued set initialized with start. Add a resolved fixture URL to both when first seen. Before the loop validate start and max_pages. Call urldefrag(urljoin(current, href)).url to remove fragments.

Complete the task and select Check task to verify your code.

DEBUG BETTER

Typical mistakes

Everyone meets these errors. See what causes them and how to fix them.

Never recording scheduled URLs

COMMON MISTAKE

queue.extend(all_links)

FIX

if target in pages and target not in queued:
    queued.add(target)
    queue.append(target)

What happens: A home/about cycle can revisit pages until the limit is exhausted.
Mark a fixture URL when it is scheduled, not after another path finds it.

Claiming the browser fixture is a live crawl

COMMON MISTAKE

html = urllib.request.urlopen(url).read()

FIX

html = pages[url]

What happens: Arbitrary network requests are outside this deterministic browser-console exercise.
Keep the algorithm offline; teach real network crawling separately with operational safeguards.

Browse this lesson’s problem fixes

PROJECT SOURCE

Explore the runnable code

The GitHub project includes setup instructions and source code you can run on your computer. Use this lesson for the guided examples, browser task, and quiz.

View this project on GitHub

VERIFY THE STORY

Sources and further reading