Convert JSON to JSON Lines (JSONL) and back

JSON Lines (also called JSONL or NDJSON) is JSON with one record per line. Logs, database exports, BigQuery and machine learning datasets use it because a program can read or append one line at a time without loading the whole file. Converting between the two is a one-liner in each direction, as long as you know where it goes wrong.

Every command below was run with jq 1.7, Python 3.12 and Node 22 on this file, users.json:

[
  { "id": 1, "name": "Ada", "tags": ["admin"] },
  { "id": 2, "name": "Linus", "tags": [] },
  { "id": 3, "name": "Grace", "tags": ["beta", "ops"] }
]

JSON vs JSON Lines

The same three records as JSON Lines:

{"id":1,"name":"Ada","tags":["admin"]}
{"id":2,"name":"Linus","tags":[]}
{"id":3,"name":"Grace","tags":["beta","ops"]}

There are three rules:

1. One complete JSON value per line, usually an object.
2. No [ ] around the file and no commas between lines.
3. Each value written compact, on a single line.

A file that follows them is valid JSON Lines. The file as a whole isn't valid JSON, which is why a JSON parser or a viewer that expects JSON rejects it.

JSON to JSONL with jq

.[] outputs each item of the array, and -c writes each one compact, on its own line:

jq -c '.[]' users.json > users.jsonl

When the list sits inside an object, as in an API response like {"count":3,"users":[...]}, name the field:

jq -c '.users[]' response.json > users.jsonl

Forget -c and jq pretty-prints each record over several lines. The output looks fine, but it isn't JSON Lines.

Without a terminal, paste the array into JSON to JSON Lines and download the .jsonl file.

JSONL to JSON with jq

-s (slurp) reads every line into one array:

jq -s '.' users.jsonl > users.json

Add -c to get the array on one line instead of indented. To check a file without a terminal, paste the lines into JSON Lines to JSON, which gives you the array to copy or download.

In Python

import json

# JSON to JSONL
with open("users.json") as f, open("users.jsonl", "w") as out:
    for rec in json.load(f):
        out.write(json.dumps(rec, ensure_ascii=False) + "\n")

# JSONL to JSON
with open("users.jsonl") as f:
    records = [json.loads(line) for line in f if line.strip()]

ensure_ascii=False keeps accented letters and emoji as they are instead of é escapes. json.dumps puts a space after : and ,, which is still valid JSONL; pass separators=(",", ":") to match jq's output byte for byte.

In Node

import { createReadStream, readFileSync, writeFileSync } from 'node:fs';
import { createInterface } from 'node:readline';

// JSON to JSONL
const rows = JSON.parse(readFileSync('users.json', 'utf8'));
writeFileSync('users.jsonl', rows.map(r => JSON.stringify(r)).join('\n') + '\n');

// JSONL to JSON, one line at a time
const out = [];
for await (const line of createInterface({ input: createReadStream('users.jsonl') }))
  if (line.trim()) out.push(JSON.parse(line));

The file this writes is identical to jq's.

Files too big to load

jq -c '.[]' loads the whole array into memory first. For a JSON file of several gigabytes, use jq's streaming mode, which reads it piece by piece and writes each top-level item as soon as it's complete:

jq -cn --stream 'fromstream(1|truncate_stream(inputs))' big.json > big.jsonl

It's slower than the normal mode, but its memory use stays flat. The other direction needs less care, since you can read JSONL one line at a time (the Python and Node loops above). Just don't collect a huge file into one array unless the next step really needs one. The guide to opening large JSON files covers the other options.

When a line won't parse

One bad line stops jq -s. With this file:

{"id":1}

{"id":2,}
{"id":3}

jq -s '.' fails with parse error: Expected another key-value pair at line 3, column 9. Blank lines and Windows line endings (\r\n) aren't a problem; the trailing comma on line 3 is.

To find every bad line, Python can report each one with its number:

with open("messy.jsonl") as f:
    for n, line in enumerate(f, 1):
        if not line.strip():
            continue
        try:
            json.loads(line)
        except json.JSONDecodeError as e:
            print(n, e.msg)

This prints 3 Expecting property name enclosed in double quotes. To skip bad lines instead and keep the rest:

jq -ncR '[inputs | fromjson? // empty]' messy.jsonl

That gives [{"id":1},{"id":3}]. -R reads each line as text and fromjson? drops the ones that don't parse. Only do this when losing records is acceptable, and count what you dropped.

GigaJSON's JSON Lines to JSON skips blank lines too, and stops at a bad one with its line and column ("Invalid JSON on line 3, column 9"), whether you paste the lines or open the file.

Looking at a JSONL file

A JSONL file is hard to read in an editor: every record is one long line. The JSON Lines Viewer opens .jsonl and .ndjson files as a table with one row per line, plus a tree and a query panel, for files up to 1 GB. Nothing is uploaded. For logs specifically, the guide to API responses and NDJSON logs has jq recipes for filtering and counting. If you need test data, the sample JSON Lines files page makes one in the size you want.