Pandas Drop Duplicates by Column and Keep the Latest Row

Select a timestamp to watch that moment on YouTube.

1. 00:00

   A contact import contains two rows for one email address. The newer update appears first, and the older update appears second. We want to keep the latest contact, but two plausible one-line fixes give the wrong result. Calling `drop_duplicates()` with no arguments compares every column, so both rows remain because their names or timestamps differ. Adding `subset=['email']` identifies the duplicate, but `keep='last'` means last in current row order, not latest in time. If we use it directly on this sample, we keep the old record. This lesson answers the specific search phrase pandas drop duplicates by column

2. 00:40

   while making the choice of surviving row explicit. We will first expose the full-row default, then define email as the identity key, parse and sort update times, and verify ties and missing values. The browser uses a small in-memory DataFrame; local Python needs `python -m pip install pandas`. The default removes exact duplicate rows, and these rows are not exact duplicates because name differs. The subset argument changes the identity rule: only email is considered when deciding whether two rows are duplicates. It does not delete the email column from the result.

3. 01:20

   This is a simple but important distinction. Before removing records, decide whether email is truly the right unique key for your product. Shared email accounts, aliases, case differences, and missing addresses can all change the answer. Our classroom example assumes one contact per exact email string. A real import might normalize case or whitespace, but that is a separate business rule and should not happen silently. We also need to decide which of the two rows survives. Pandas defaults to the first observed row; `keep='last'` chooses the last observed row. Neither option automatically reads the timestamp.

4. 02:01

   Let's make that failure visible with the newer row intentionally placed first. The method has done exactly what `keep='last'` asks: it kept the second input row. But that row is older. The phrase latest is not built into `drop_duplicates`; we must define order ourselves. Parse `updated_at` with `pd.to_datetime(..., utc=True)` rather than assuming a mixed set of date strings will sort chronologically as text. Then sort by that parsed value. Use a stable sort so two equal timestamps keep their original order. In our policy, the later input row wins a tie. If that is not acceptable, add another explicit tie breaker such as a version

5. 02:43

   number. The helper will add a temporary parsed- time column, sort, drop duplicate email values with `keep='last'`, reset the index, and remove that temporary column. This produces a new DataFrame and leaves the source untouched. A new table is safer for a teaching example because it makes the transformation inspectable before replacing an imported dataset. The parsed timestamps put the old row before the new row, then `keep='last'` selects the new one. `ignore_index=True` gives the result a clean sequence of row labels, which helps when later code reads or displays it.

6. 03:24

   The temporary parsed-time column does not leak into the return value. We reject a missing email because otherwise missing identity values may be treated as duplicate candidates and an unrelated contact could disappear. This is our chosen policy, not a claim that pandas itself rejects missing keys. For a production import, inspect rows before dropping them and decide whether to preserve a review log. If one duplicate row has a useful phone number missing from the newer row, deduplication by selection will lose that field. Combining fields is a different reconciliation problem. The function does not solve it automatically.

7. 04:06

   Also note that emails with different capitalization remain different strings here. Normalize only if your application has an explicit rule for that comparison. For the equal timestamps, stable sorting preserves first then second, so keeping the last selects second. That tie rule is predictable but only appropriate if input order has meaning. `keep=False` implements a different policy: it discards every row whose email occurs more than once, leaving only the unique b record. `duplicated` gives a Boolean mask to inspect those rows before removing them. In a real data cleanup, that mask can feed a review report, while the retained result

8. 04:47

   can feed the application. Check that the input frame remains unchanged and test an empty or missing email according to your contract. On the lesson page, fix the starter function, run the reverse-order and tie tests, and explain why `subset` and `keep` solve different parts of the problem. The core lesson is to define identity and survivor order separately. A one-line deduplication call is only correct after both decisions are made.