PythonLessonLab
LESSON 57 · DEEPER PYTHON29 min

Understand Unicode Text and UTF-8 Bytes

Keep str and bytes distinct and perform an explicit UTF-8 round trip without changing the text.

Before you start: Variables and Types · Basic String Operations · Functions

Example check: Worked and guided code samples are checked against CPython 3.13. Practice uses Pyodide 0.29.5 (Python 3.13) in your browser. How we check lessons

WATCH

Video lesson: Understand Unicode Text and UTF-8 Bytes

The preview is stored on this site. YouTube loads only when you press Play. Watch on YouTube

Jump to a chapter

Read the video transcript

Select a timestamp to watch that moment on YouTube.

  1. 00:00

    You read a file and see café where you expected café. The text was not randomly damaged. A common cause is decoding bytes with a different encoding from the one that produced them. In Python, str is Unicode text, while bytes is an encoded sequence for storage or transmission. We will encode café as UTF-8, deliberately decode it as Latin-1, inspect the bytes that expose the mismatch, then decode correctly and verify several kinds of input. This is a real program run, not an illustration of imaginary output. The companion lesson includes the browser console, two worked examples, a checked task, common mistakes, and a quiz.

  2. 00:44

    Before running, make two predictions. How many code points are in café? Four: c, a, f, and é. How many bytes will the UTF-8 version contain? Five, because the accented character needs more than one byte in that encoding. These counts answer different questions. They also remind us that a user-visible symbol, a Unicode code point, and a byte are not interchangeable units. We will keep those units named clearly throughout the lesson, especially when the output looks readable but is actually wrong. The first line is café. The second line is the original café, so the mismatch is observable in a two-line run.

  3. 01:27

    UTF-8 encoded the final é as two bytes. Latin-1 treats those two bytes as two separate characters, producing the corrupted pair. Notice that no exception was raised. A decoder can successfully return text and still be the wrong decoder for the data's actual format. That makes this bug easy to miss in logs or a user interface. The mistake is at the bytes-to-text boundary, not in the variable name, the print function, or the font. A tempting workaround is to replace the visible characters afterward with string operations, but that addresses one symptom and can corrupt other inputs. The correct repair is to know which encoding produced the bytes and decode with it.

  4. 02:12

    In our controlled example, we created the bytes ourselves with UTF-8, so that answer is certain. In a real file or protocol, the encoding should come from its specification, metadata, or an explicit agreement with the producer. Let us inspect the two lengths and the byte representation before we change the decoder. The text length is four. The byte length is five. In the bytes representation, ASCII c, a, and f appear directly, while the final accented character is represented by two hexadecimal byte escapes. Those escapes are not literal backslash characters in the original text; they are a readable representation of non-ASCII byte values.

  5. 02:56

    When we decode the same bytes with UTF-8, café returns unchanged. This supports a useful round-trip invariant: for valid text, decoding its UTF-8 encoding as UTF-8 should yield the original string. The invariant depends on matching the encoding at both sides. It does not mean every file in the world is UTF-8; formats can specify other encodings. In modern Python code, pass encoding='utf-8' explicitly when opening a UTF-8 text file so the choice is visible and does not depend on a local default. Also remember that len of a Python string counts Unicode code points, not visual grapheme clusters.

  6. 03:40

    A combining accent can make two code points appear as one perceived letter, and some emoji use multiple code points. We will avoid claiming a general character count from len and focus on code points versus encoded bytes. The function returns three values: code-point length, byte length, and decoded text. Café gives four, five, and the original word. The snake emoji gives one code point, four UTF-8 bytes, and the original emoji. This function is intentionally small because encoding boundaries should be explicit. We use encode with UTF-8, then decode with the same encoding. We do not silently replace undecodable bytes or normalize text, because neither behavior is part of this task's contract.

  7. 04:27

    A real data pipeline might need a policy for malformed input. It should decide whether to fail, quarantine records, or use replacement characters, and it should document that decision. Here we created the bytes ourselves, so a clean round trip is the correct expectation. The browser task asks you to implement these exact return values. The task checks ASCII, an empty string, an accented word, and an emoji. If your result says the byte and text lengths are always equal, print the encoded bytes and test with non-ASCII input. If the restored string differs, look at the decoder first. A successful function preserves data across the representation boundary rather than merely making a corrupted string look plausible on one example.

  8. 05:15

    The ASCII case has equal code-point and byte lengths because those particular characters use one UTF-8 byte each. Empty input stays empty. Café and the snake demonstrate that equality does not generalize to non-ASCII text. The combined a plus snake case confirms we count two code points and five bytes, not two bytes. Every check also requires the decoded text to match exactly. This is a better test than only checking that decode returns some string, because a wrong encoding can return corrupted but valid-looking text. In the browser console, try a word in another writing system and observe its byte length without assuming every letter uses the same number of bytes.

  9. 06:01

    Then complete the task and quiz. The core method is to name the boundary: text to bytes is encode, bytes to text is decode. Keep the encoding agreement on both sides, and never treat a byte length as a universal measure of characters. If you encounter mojibake in a real project, trace where the bytes were produced and where they were decoded. Fix that boundary rather than layering replacements over the damaged display. Clear boundaries prevent subtle corruption from spreading into databases, logs, and exported files.

THE IDEA

Understand the concept

Python str represents Unicode text, while bytes is a sequence of integer byte values. Encoding turns text into bytes for storage or transfer; decoding turns bytes back into text. A character does not always occupy one byte in UTF-8. The word café has four Unicode code points but its UTF-8 encoding has five bytes because é uses two bytes.

If UTF-8 bytes are decoded with a different encoding such as Latin-1, the program may produce plausible but corrupted text rather than an exception. The familiar café is an example of mojibake. A wrong decoder can also raise UnicodeDecodeError, depending on the byte sequence. The remedy is to know the actual encoding of the data and use it consistently at each boundary, not to guess repeatedly until output looks acceptable.

The task utf8_round_trip(text) returns a tuple of the number of Unicode code points, the number of UTF-8 bytes, and the decoded text. It should work for ASCII, accented characters, and emoji. It must use encode('utf-8') and decode('utf-8') explicitly. The returned text must equal the input; no normalization or replacement of invalid data is requested.

When reading or writing files, pass encoding='utf-8' explicitly so the intended format is visible. Network protocols and file formats may specify another encoding, which you must respect. For some user-perceived characters, len(text) differs from what a reader sees because combining marks and emoji sequences can contain multiple code points. Here we compare code points and bytes only.

  • str holds Unicode text; bytes holds encoded values
  • UTF-8 uses variable numbers of bytes
  • Decode with the encoding that produced the bytes
  • len(str) counts code points, not UTF-8 bytes
WORKED EXAMPLES

See it step by step

Read the code, predict the output, then compare it with the result.

01. Compare characters with encoded bytes

PYTHON
text = 'café'
data = text.encode('utf-8')
print(len(text))
print(len(data))
print(data)
print(data.decode('utf-8'))
EXPECTED OUTPUT
4
5
b'caf\xc3\xa9'
café

é becomes two UTF-8 bytes, and matching UTF-8 decoding reconstructs the original text.

02. Observe a mismatched decoder

PYTHON
data = 'café'.encode('utf-8')
print(data.decode('latin-1'))
print(data.decode('utf-8'))
EXPECTED OUTPUT
café
café

Latin-1 maps each byte separately, producing corrupted text; UTF-8 interprets the multibyte sequence correctly.

GO DEEPER

A closer look

Follow the reasoning, inspect each result, then try the suggested changes in the console below.

01 / 03

Observe text and byte length separately

Python str stores Unicode text. Encoding that text as UTF-8 yields bytes for storage or transfer. The four code points in café include an accented é that requires two UTF-8 bytes, so the resulting byte sequence has length five. Neither length is wrong; they answer different questions.

Printing the bytes representation exposes the two non-ASCII byte values as escape sequences. Decoding with the same UTF-8 encoding reconstructs the original string. A program should keep this boundary explicit, especially when reading files or sending network data. Do not equate one user-visible glyph with one byte, because UTF-8 has variable-length encodings.

PYTHON
text = 'café'
data = text.encode('utf-8')
print(len(text))
print(len(data))
print(data)
print(data.decode('utf-8'))
EXPECTED OUTPUT
4
5
b'caf\xc3\xa9'
café
Follow the reasoning
  1. Count Unicode code points in text.
  2. Count encoded byte values in data.
  3. Decode with UTF-8 to recover the same text.
02 / 03

Reproduce corruption from a mismatched decoder

Bytes do not carry their own reliable encoding label. If UTF-8 bytes are decoded as Latin-1, each byte maps separately and the accented character becomes two visible characters. The program may not raise an error, which makes the corruption easy to overlook. This is one form of mojibake.

Fix the boundary by using the encoding specified by the file format or protocol. In our controlled example, we know the producer used UTF-8, so the consumer must decode UTF-8. In an unknown data source, detect or obtain the encoding from metadata rather than trying random decoders and assuming the first readable output is correct.

PYTHON
data = 'café'.encode('utf-8')
print(data.decode('latin-1'))
print(data.decode('utf-8'))
EXPECTED OUTPUT
café
café
Follow the reasoning
  1. The first decoder maps the two UTF-8 bytes independently.
  2. The second decoder interprets them together as é.
  3. A successful decode is not proof that the chosen encoding was correct.
03 / 03

Write a round trip with an explicit contract

The task returns three values: the number of code points, the number of UTF-8 bytes, and the restored text. This makes the distinction testable for ASCII and non-ASCII input. An empty string has zero of each; a snake emoji is one Unicode code point and four UTF-8 bytes.

The function does not normalize text or replace decoding errors. It encodes and immediately decodes with the same known encoding, so the restored string should equal the input. In a real file operation, pass encoding='utf-8' explicitly on open. Some rendered symbols use multiple code points, so this exercise does not claim len(text) equals perceived character count.

PYTHON
def utf8_round_trip(text):
    data = text.encode('utf-8')
    return len(text), len(data), data.decode('utf-8')

print(utf8_round_trip('café'))
print(utf8_round_trip('🐍')[:2])
EXPECTED OUTPUT
(4, 5, 'café')
(1, 4)
Follow the reasoning
  1. Encode text into UTF-8 bytes.
  2. Compare the two lengths before decoding.
  3. Return the exact restored text without replacement.
HANDS-ON

Try it in Python

Edit the example and run it. Python starts in your browser the first time you click Run.

Python console

Ready to run
Need input()? Add one value per line
Ctrl / ⌘ + Enter to run
OUTPUT
Your output appears here.
Your turnImplement utf8_round_trip(text). Encode the string as UTF-8, decode those bytes as UTF-8, and return (len(text), len(encoded_bytes), restored_text). The prepared output is (4, 5, 'café').
Need a hint?

Use data = text.encode('utf-8'); return len(text), len(data), data.decode('utf-8').

Complete the task and select Check task to verify your code.

    KNOWLEDGE CHECK

    Quick quiz

    Three questions. You can change your answers and try again.

    01.What is the type of 'café'.encode('utf-8')?

    02.Why can len(text) differ from len(text.encode('utf-8'))?

    03.What should decode use for bytes produced by encode('utf-8')?

    DEBUG BETTER

    Typical mistakes

    Everyone meets these errors. See what causes them and how to fix them.

    Decoding UTF-8 bytes as Latin-1

    COMMON MISTAKE
    data = 'café'.encode('utf-8')
print(data.decode('latin-1'))
    FIX
    print(data.decode('utf-8'))

    What happens: The output becomes café.
    The decoder must match the data's actual encoding.

    Treating bytes as characters

    COMMON MISTAKE
    print(len('🐍'.encode('utf-8')) == len('🐍'))
    FIX
    text = '🐍'
print(len(text), len(text.encode('utf-8')))

    What happens: The comparison is False because the emoji is four UTF-8 bytes but one code point.
    Name the measurement: code points and encoded bytes are different quantities.

    Browse this lesson’s problem fixes

    VERIFY THE STORY

    Sources and further reading