Build an Offline Duplicate File Scanner

Select a timestamp to watch that moment on YouTube.

1. 00:00

   A duplicate scanner should identify files with the same bytes, regardless of their names. The browser project models a directory as a mapping from path labels to byte strings. That gives us a deterministic, offline test fixture before we discuss reading actual files. The first example hashes the same byte sequence twice and compares the hexadecimal digests. Equal contents produce equal digests, so a dictionary keyed by digest can collect paths that might match. Hashing is valuable for large files because a fixed-size digest can be stored instead of keeping all contents in a grouping key. A scanner should identify candidate duplicates and report their paths, not

2. 00:42

   silently remove anything. This video stays offline with a dictionary of byte strings, so we can see every input and result. Predict which path names should be paired when a.txt and b.txt contain identical bytes but c.txt does not. A count of distinct contents is insufficient: the user needs the actual path group to inspect before taking any action. A naive implementation stores byte contents in a set. It can say how many unique values exist but loses the paths that need to be reported. Instead, group paths by a digest such as SHA-256, then compare bytes inside each

3. 01:24

   digest group before declaring equality. A digest is an efficient candidate key, not a mathematical proof that two inputs are equal. A digest collision remains possible, so this project treats equal hashes as candidates and verifies the original bytes inside each bucket. With the tiny in-memory fixtures used by the browser task, that second comparison is cheap. The design is faithful to a real scanner without claiming cryptographic impossibility of collisions. The naive starter returns only a number of unique contents. Even if that number is correct, it cannot tell the user which paths are duplicates.

4. 02:06

   We need to carry identifiers through the grouping process. The starter's numeric result may look plausible, yet it discards every path label. The observable failure is missing information, not an exception. Keep the mapping visible so we can compare the desired duplicate group with the scalar that was returned. Return only groups with at least two matching files. Sort path names within each group and sort the groups by their first path so output is stable across dictionary insertion orders. Empty files can be duplicates too; one file alone is never a duplicate group. Treat path labels as names in this browser exercise and do not access the user's

5. 02:47

   actual disk. The second example uses different file names with identical bytes. A set of byte values would know there is one distinct content value, but the user needs the two path labels to inspect the duplicates. Maintain those labels all the way through grouping and output. Do not merge files merely because their names or extensions match, and do not treat two different names as proof of different contents. The scanner's contract is exact byte equality. It does not normalize line endings or encodings; two text files that read similarly but differ in bytes are not duplicates under this definition.

6. 03:29

   A digest makes a compact candidate key, but a digest match alone is not proof. The diagnostic comparison shows that different names can have equal bytes. The algorithm checks actual contents within each candidate bucket. A SHA-256 digest is a fast candidate bucket. Within that bucket compare the original bytes before grouping, because equal digest values alone do not prove equality. Path names stay attached to their byte contents throughout the comparison. Implement find_duplicates(files), where files maps path strings to bytes. Return a sorted list of tuples of paths sharing identical bytes, omitting

7. 04:09

   singleton groups. Use SHA-256 as a candidate key and verify equal bytes within a digest bucket. The prepared example must print [('a.txt', 'b.txt')]. Build digest buckets with hashlib.sha256(data).digest(). Within each bucket, group by the actual bytes and keep groups with two or more sorted paths. The final example sorts paths before creating a tuple. The checked task sorts each duplicate group and the overall list by its first path. That makes output deterministic regardless of the insertion order of the input

8. 04:49

   mapping, which is helpful for review and automated checks. The on-screen fix now handles every digest bucket and every distinct byte sequence inside it. The exact dictionary uses the original bytes as keys, so even a digest collision cannot merge different contents. It sorts paths within each group and sorts the groups before reporting. Real files would be read in chunks with error handling. The corrected scanner builds digest buckets, then exact-byte groups within each bucket. It reports only groups of two or more and sorts their paths. That second grouping keeps distinct contents separate even if a test

9. 05:29

   deliberately forces a digest collision. A real file scanner would walk a chosen root with pathlib, reject symlink surprises according to a documented policy, hash large files in chunks, and handle permission or read errors explicitly. Do not automatically delete duplicates. This project stops at identification and shows a reproducible algorithm suitable for testing before any filesystem actions. No file is deleted or renamed by this project. A real scanner must choose a root directory, decide whether to follow symlinks, read large files in chunks, and handle permission failures.

10. 06:09

    Those operational choices need explicit user control. The in-memory algorithm offers a safe foundation before any disk traversal. Keep the human-readable identifiers alongside candidate digests. Hashes narrow candidate sets; an exact byte comparison confirms a group. The final run deliberately forces every file into one fake digest bucket. It still reports two separate duplicate groups: two empty files and two files containing X. The unrelated singleton is omitted. A real SHA-256 collision is extraordinarily unlikely, but a controlled

11. 06:49

    fake key proves the grouping logic does not assume it is impossible. No file is deleted by this exercise. For real disk scans, separate candidate detection from user-approved cleanup. The forced-collision check puts every item into one fake digest bucket. Two distinct byte values still produce two separate duplicate groups, and a singleton stays out. The scanner reports candidates and never deletes files; cleanup requires an explicit separate decision.