Process JSON Lines One Record at a Time

Select a timestamp to watch that moment on YouTube.

1. 00:00

   A log export arrives as one JSON object per line. Each event has a name such as view or click, and the next record starts on the next line. If we paste the whole text into json.loads, the decoder reports Extra data. That sounds like a malformed file, but each line can be perfectly valid JSON. The mismatch is between the file convention and the decoder call: json.loads expects one complete top-level JSON value, while this export contains several independent values. Today we will reproduce the failure, then build a line-oriented reader that preserves physical line numbers.

2. 00:41

   We will skip blank lines under a stated policy, parse each nonblank record, validate that it is an object with a nonempty string event, and count accepted event names. The website task uses io.StringIO so the browser console can practice this without uploading private logs. The same loop can process a local text file later. Our checks will include malformed syntax and a syntactically valid JSON value that is not a valid event. That distinction is central to reliable data handling: decoding text and accepting an application record are two different gates. The decoder successfully reads the first object.

3. 01:22

   It then expects only whitespace until the end of this one JSON document. Instead it finds the second opening brace, so it raises JSONDecodeError with the message Extra data. We catch that error here only to keep the demonstration running. The objects are individually valid; the way we grouped them in one call is wrong for the format. One tempting repair is to split the text at every closing brace. That fails as soon as an event contains a nested object or a brace inside a quoted string. Another temptation is to wrap the entire file in square brackets and add commas. That can work as a one-time conversion, but then we are no longer processing the

4. 02:05

   line-oriented export as received. The direct solution is to treat each line as its own JSON document. This lets a large local file be read progressively instead of making one giant string, although our browser exercise uses a short StringIO sample. It also gives us a natural place to report which physical line contains a mistake. Before writing the final function, we need to see how blanks affect those line numbers. enumerate starts at one so its number matches the line number in an editor. We skip the blank second line, but the count of physical lines still advances.

5. 02:45

   If line three were malformed, reporting it as event number two would send a human to the wrong place. The final line shows a second layer of validation. An empty list is perfectly valid JSON, so json.loads accepts it. Yet it is not a dictionary representing an event and does not contain an event name. Likewise, a JSON number, string, or null can be syntactically valid while violating the import contract. We therefore need to validate the decoded Python object after parsing. Our task requires an object with a nonempty string under the key event. A missing key, a number as the event value, and an empty event string all fail.

6. 03:28

   We should not merely write record['event'] and wait for KeyError or TypeError to appear far away from the source. The function can raise ValueError with a useful line number at the point where it detects the bad record. That makes debugging a data export much easier than a generic failure while updating a counter. The function begins with an empty count dictionary. For each physical line, it ignores only blank text and sends every nonblank line to json.loads. A syntax error is translated into ValueError that identifies the line. The original JSONDecodeError is chained with from error, so a developer can still

7. 04:08

   inspect the detailed cause in a traceback. Next comes the record contract. The value must be a dictionary, its event field must be a string, and that string must not be empty. Only after all three conditions pass do we update the counter. The dictionary preserves the first-seen event order in current Python, so the printed result shows view before click. For a production system, we might choose to collect several row errors and continue processing valid records, or cap how much data is held in memory. This introductory importer stops at the first error because a clear failure is easier to reason about than silently dropping data.

8. 04:51

   Notice that a blank line and a malformed line are treated differently. A blank line is ignored by policy; a nonblank line that cannot be decoded is reported as an error. That distinction should be documented whenever an import can discard input. The first assertion checks the full counting behavior, including a repeated view event. The second confirms that a stream of blanks produces an empty result rather than an exception or a made-up event. The next check deliberately places invalid JSON after a blank line. We catch ValueError and require that its message mention line three, because source location is part of the task's useful behavior.

9. 05:35

   Finally, we test records that json.loads would accept but the application must reject: an array, an empty event name, and a numeric event name. This is a stronger test than checking only syntax failure. A parser can be technically correct and still admit unusable data. Try adding an event object with extra keys. This function accepts it because its contract only requires event to be a nonempty string; a stricter schema would need to say otherwise. Then try a record whose event value is whitespace, and decide whether your application should strip or reject it. For this lesson, whitespace is technically a nonempty string, so it passes; changing

10. 06:18

    that policy would require changing both the function and its tests. On the lesson page, the starter function counts only the first line. Replace that shortcut with the line-by- line reader, run the practical checker, and use the quiz to verify that you can explain why valid JSON and valid event records are not synonymous.