Pandas to_datetime: Parse Dates and Find Invalid Rows

Select a timestamp to watch that moment on YouTube.

1. 00:00

   Here are three values from an order export. The first is February twenty-ninth in a leap year, the second says February thirtieth, and the third is missing. All three are strings or empty values until we parse them. A string column can look convincing on a screen and still behave incorrectly in time calculations. Today we will use pandas to_datetime to create a real datetime Series, identify every invalid source position, and make our timezone assumption explicit. The source contract says the values use four-digit year, two-digit month, and two- digit day. It also says a valid date represents midnight UTC.

2. 00:41

   Those details are decisions from the data producer, not guesses pandas can safely make for every application. Watch how the first attempt reacts to an impossible date. Imagine a CSV with ten thousand orders: a single visually plausible bad date can distort a monthly total if it is left as text. The parser should therefore expose uncertainty, rather than quietly producing a column that merely looks date-shaped. The strict call uses the stated format but leaves the default error behavior in place. It raises on February thirtieth. That is useful if one bad value should abort the import immediately; the error

3. 01:21

   prevents the program from pretending that the data is clean. But this particular import workflow needs to report all invalid rows at once. A try block around the call can catch the exception, yet it does not hand us a mask for every source position. Notice that the input's order matters. If we drop or reorder rows too early, we might show a user the wrong record when explaining a bad date. We will switch to a mode that marks invalid values, keep the original strings beside the parsed values, and then inspect the mask. Strict parsing is especially useful during development because it exposes an unexpected source format immediately.

4. 02:03

   During operations, a complete error list can be more helpful to the person who repairs the export. Choose that behavior explicitly at the import boundary. With errors set to coerce, pandas returns NaT for an impossible date. NaT is the missing datetime marker, not a substitute date and not midnight. The isna mask is true for the invalid calendar day and for the original missing value. We can use that mask to print the raw values that need review. This lets a file importer choose an explicit policy: reject the whole file, quarantine selected records, or ask the data producer to repair them.

5. 02:44

   The mask alone cannot tell whether a value was blank or malformed; the original column supplies that context. Also notice that an unspecified format invites inference. Since our producer promised year-month- day, the corrected code states that format instead of relying on what looks familiar in this tiny sample. Notice that isna also marks the original None. The parsed mask tells us where the result is unavailable, while the raw values tell us why each row needs investigation. Keep both columns until the import decision has been recorded. The corrected function accepts a sequence, builds a Series to preserve row positions, and calls pandas to_datetime with an explicit format, errors coerce, and UTC

6. 03:27

   enabled. We return the datetime Series itself. Formatting it as text would discard the benefits of datetime-aware filtering and arithmetic. The sample print formats values only for display, replacing NaT with the word INVALID. UTC needs a careful explanation: a naive date has no timezone. In this example we deliberately interpret midnight as UTC. If a producer meant midnight in Bratislava, we would localize in that region first, then convert. In other words, UTC is a declared part of this data contract, not a universal repair

7. 04:07

   setting for date strings. Check the dtype or .dt.tz before performing timezone-sensitive operations. An ordinary display of a timestamp may hide its timezone in a compact table. A trustworthy pipeline tests the semantic type, not just a pretty string representation. The final check includes a leap day, an impossible day, a missing value, and a later valid date. We inspect the timezone, the invalid-value mask, and the original order. A date parser should be tested on edge cases because a few normal weekdays do not prove that the calendar contract is enforced.

8. 04:47

   Keep the raw input for error messages, and resist a silent dropna call before anyone has reviewed the bad records. In a larger pipeline, validate the source column name, record a source row number, and decide whether the entire batch can continue. The key result is a real UTC-aware Series plus a reliable way to locate failures. The related pandas duplicate lesson uses date parsing briefly for ordering; this focused lesson shows what that parsing step must guarantee. One subtle limitation: the format argument describes the intended pattern but does not guarantee zero-padded month and day in every pandas version.

9. 05:29

   If exact textual width matters, validate the raw string separately before parsing; calendar validity and textual shape are different checks.