Read Text Files with Explicit Encoding and Line Rules

Select a timestamp to watch that moment on YouTube.

1. 00:00

   A text file is stored as bytes. Reading it requires an encoding, and line endings can differ across systems: LF, CRLF, and older CR. A program that splits only on '\n' may leave a trailing carriage return in each Windows-style line. The first example prints the result of splitting a CRLF string on LF alone. The remaining carriage-return characters appear as escaped backslash-r sequences. That is a common source of failed string comparisons when data crosses operating systems or is read in binary mode. A visually ordinary text file may hide a carriage return before every line feed.

2. 00:41

   We will make those characters visible, then choose an explicit decoding rule for bytes that are supposed to be UTF-8. Before the run, predict the elements of b'A\r\nB\n'.decode('utf-8').split('\n'). The first element retains a carriage return, and the final separator creates an empty element. Keep that prediction beside the output so invisible characters do not disappear in ordinary printing. Python's text I/O normally translates line endings when newline=None. Explicitly passing encoding='utf-8' makes the byte-to-text rule stable across machines.

3. 01:23

   In a browser exercise with no physical file, bytes.decode('utf-8') and str.splitlines() demonstrate the same two decisions separately. splitlines() recognizes LF, CRLF, and CR boundaries and returns logical lines without their separators. The browser task works on bytes to make the decoding step explicit. For a physical text file, open with a chosen encoding and newline behavior so the contract is visible in the call. The starter splits on LF only and leaves a carriage return attached to A. That extra character can break equality checks or produce odd output after data is moved between systems.

4. 02:05

   The failed result is a list with a hidden carriage return and an unwanted final empty string. Print repr of each element if the screen makes the characters hard to see. This is a logical-line bug, even though the program exits without an exception. Do not silently replace malformed bytes unless the application has an explicit data-recovery rule. Strict UTF-8 decoding raises UnicodeDecodeError and preserves the fact that the file does not match the contract. Once decoded, splitlines() recognizes common line boundaries without leaving '\r' attached to a CRLF line. The second run encodes café into UTF-8 bytes and decodes it back to characters.

5. 02:50

   The byte representation includes two bytes for é, so byte positions do not match character positions. Splitting raw bytes might work for ASCII line-feed bytes, but it leaves character interpretation unresolved. Use data.decode('utf-8') as an explicit first step. This lesson's format contract says input is UTF-8 and rejects other encodings; it does not guess. A real application might have a documented legacy encoding, but that should be a separate, intentional choice rather than an environment-dependent default. The diagnosis prints both the raw split and splitlines result.

6. 03:30

   It also shows café as bytes and as text. The two representations serve different layers and should not be conflated. The UTF-8 bytes for café show a second boundary: bytes must be decoded before text line handling. Compare that output with the raw split. Once the encoding is known, splitlines recognizes CRLF, LF, and CR without keeping their terminators. Implement read_utf8_lines(data: bytes) by decoding UTF-8 strictly and returning a list of logical lines without newline characters. Handle LF, CRLF, and CR; an invalid UTF-8 sequence should raise UnicodeDecodeError.

7. 04:13

   The prepared call must print ['A', 'B']. Decode with data.decode('utf-8'), then call splitlines() on the text. The final run passes ff, which is not a valid standalone UTF-8 byte. Strict decoding raises UnicodeDecodeError. Silently replacing it with a placeholder would make the line look processed even though the source did not meet the promised format. The fix is intentionally short: decode strictly, then split logical lines. For an actual file, open with encoding='utf-8' and an intentional newline policy. Do not infer the encoding from the local machine. Now read the exact b'A\r\nB\n' sample through strict UTF-8 decoding and

8. 04:56

   splitlines. The result should contain only A and B. This function deliberately does not trim ordinary spaces inside a line, because line endings and content are different concerns. The task returns clean lines for mixed line endings and refuses invalid UTF-8. A real file version should use open(path, 'r', encoding='utf-8', newline=None) or pathlib.Path.open with the same parameters. Binary formats should remain in binary mode rather than being forced through a text decoder. The checked function handles empty input, CRLF, LF, and CR while preserving valid

9. 05:38

   Unicode text. In file-based code, errors='replace' can be useful for a deliberate salvage workflow, but then the caller must know that information was lost. Here failure is the honest result for invalid bytes. splitlines recognizes LF, CRLF, and CR boundaries. An explicit encoding makes the data contract portable. The last run covers CR-only input, valid non-ASCII UTF-8, and an invalid byte. An error on malformed input is useful information; a replacement character would hide a broken contract. When debugging a real file, inspect a tiny sample in binary mode first.

10. 06:23

    Print repr of the bytes and the decoded text, then decide whether universal newline handling is appropriate. This avoids changing the data before you understand it. Finally, test a CR-only separator, a valid accented word, and b'\xff'. The last input must raise UnicodeDecodeError. Replacing invalid bytes would conceal a format violation, while guessing another encoding would make results depend on the environment. Check whether an empty input returns an empty list and whether spaces inside an otherwise valid line remain intact. Neither case should be mistaken for a line-ending problem.