Normalize Unicode Text for Search Keys

Select a timestamp to watch that moment on YouTube.

1. 00:00

   A search box says it cannot find a name that appears to be typed exactly as stored. Both versions look like Café on screen, yet Python equality returns False. One version may contain a single precomposed accented character; another may contain an ordinary e followed by a combining accent. Both are valid Unicode text. A second surprise appears with case- insensitive search: lowercasing Straße and STRASSE still gives different strings, while Python's casefold method can treat this pair as equivalent. In this lesson we will reproduce both misses, inspect the difference between

2. 00:40

   lower and casefold, and build a comparison key using NFC normalization. We will preserve the original display text rather than overwriting it with a key. We will also show a non-match that must remain a non-match: unaccented cafe and café. Normalization is a matching policy, not permission to erase every distinction or to decide that two people are the same. The companion website exercise checks composed accents, case variants, surrounding whitespace, and unsupported input types. Run the code along with the video and predict each Boolean before revealing it. The first comparison is False.

3. 01:20

   Looking at the next two lines explains why. Lowercasing Straße leaves its sharp s, giving straße. Lowercasing STRASSE gives strasse. This is not a defect in str.lower; lowercasing and full Unicode caseless matching have different purposes. Python provides str.casefold for more aggressive caseless comparison. It can map the sharp s to ss. Before replacing lower, however, think about what the key will be used for. A search index can deliberately merge these case variants. A unique account identifier or a legal name might need a stricter or locale- specific rule, and a product should not silently rewrite the displayed spelling.

4. 02:03

   The function here will return a derived key, while the original string stays available for display. Lower also does nothing about the two code-point sequences that can render as Café. A successful fix has to handle both case and canonical representation, and the tests must include both. Simply making this one German example pass would leave the accent example broken. The first two words can look identical while containing different code points. Python string equality checks the sequence, so the first answer is False. NFC normalization composes canonically equivalent sequences where possible, and the comparison becomes True.

5. 02:46

   The third line is intentionally False. An accent is meaningful information; NFC does not strip it from café or add it to cafe. That distinction makes canonical normalization a narrower and safer rule than blindly deleting combining marks. It still does not solve every human- language search question. Different scripts, locale rules, punctuation, transliteration, and visually confusable characters require separate policies. For this lesson we will keep the contract small: trim surrounding whitespace, perform caseless matching, and compose canonically equivalent text. We will not collapse internal spaces, remove accents, or claim that a comparison

6. 03:28

   key is a secure identity. Notice also that the original strings were never changed. The normalizer returned new strings for comparison, leaving the input display forms available to the application. The function first rejects values that are not strings. Silently converting None to the text None would create a misleading searchable entry. It strips whitespace only at the edges; two spaces inside a name remain two spaces. Next, casefold performs Unicode caseless mapping. NFC normalizes the resulting sequence so that composed and decomposed forms match under this key.

7. 04:09

   The first printed comparison includes uppercase letters, surrounding spaces, and the two accent representations; the result is True. The second demonstrates the sharp-s mapping; it is also True. The last confirms that the key preserves the accent distinction between cafe and café. The exact order of transformations is a policy choice. Normalizing after case folding makes the final key canonically composed even if the case operation changes code points. For a large search product, you may add indexing, tokenization, or locale-aware ranking, but do not smuggle those behaviors into this compact function.

8. 04:51

   A stable key is useful only when callers know what it equates and what it keeps distinct. On the website, the starter uses lower alone. Replace it with this explicit sequence and then run the practical checker. The first two assertions verify the matching behavior we intentionally chose. The third protects an equally important boundary: café must not collapse into cafe under NFC and case folding. If a future developer adds accent removal to the function, this test would fail and force a discussion about the changed policy. The fourth assertion confirms that strip only trims the ends and leaves internal spacing alone.

9. 05:32

   A different product might normalize all whitespace, but that could merge names or phrases unexpectedly. The final try block makes the type contract visible; None raises TypeError rather than creating a string-shaped key. When the script prints checks passed, we have evidence for both positive and negative cases. There are still limits. A search key is not a complete spelling- correction system, and equal keys do not prove that two records refer to the same person. For account security, validation and identifier policy need separate design. Try adding a decomposed accent after a different letter and predict whether NFC creates an equivalent composed form.

10. 06:15

    Then compare the original string with its returned key and note that the original remains untouched. On the companion page, use Try this example to load each sample into the console, finish the checked task, and answer the quiz about canonical equivalence, accent preservation, and case folding.