Skip to the content
GigaJSON
All tools
Format and check
View and query
Convert
JSON to code
Encode and decode
Sample files
Guides Open GigaJSON

How to open a large JSON file (100 MB to 1 GB) without crashing

· 8 min read · Updated · By Amir Badi

You download an export, a log dump or an API backup and double-click it. The editor goes grey. A few minutes later you get either the file or a crash report. Below: why that happens, and what works at 100 MB, 500 MB and 1 GB, with commands you can paste.

Start by deciding what you want from the file. Looking at it needs a viewer that doesn't build everything at once. Pulling data out of it needs a streaming parser or jq.

A settings file or a small export doesn't need any of this. For those, how to open a JSON file covers Notepad, TextEdit, browsers, VS Code, Excel and phones.

Open a large JSON file in your browser, step by step

  1. Go to the large JSON viewer. You install nothing, and the file stays on your computer.
  2. Choose Open file and pick the file. A background worker reads it (up to 1 GB), and the page keeps responding.
  3. Wait for the tree. On our test laptop, 200 MB takes about 2 seconds, 500 MB about 9 and 1 GB about 18 (benchmarks).
  4. Open the branch you care about. For a list of records, switch to Table to sort and filter.
  5. Find searches keys and values. Query runs JSONPath and JMESPath; jq works on files up to about 20 MB.
  6. To edit, choose Open in GigaJSON. Files over 512 MB need Preferences → Largest document set to 1 GB first.

If you'd rather stay in a terminal or in Python, keep reading.

GigaJSON tree view with an order selected and its customer details in the InspectorGigaJSON tree view with an order selected and its customer details in the Inspector (dark theme)
Start with one branch of a document, then use Table or Query for records. Screenshots follow the page theme. Click to view full size.

Why editors choke on big JSON

You hit three limits, usually in this order.

The parsed tree is much bigger than the file

JSON text is compact. Parse it, and each object, key and number becomes its own allocation with its own overhead. I measured this on a 15.5 MB file of 200,000 small orders (Intel i5-9300H, Linux):

Tool Peak memory Time
jq '.orders | length' 231 MB 0.6 s
Python json.load 142 MB
Node JSON.parse 104 MB
jq --stream (filter shown below) 3.6 MB 6.0 s
Python ijson.items 14 MB 0.4 s

Plain jq needed about 15 times the file size. At that ratio, a 1 GB file wants well over 10 GB of RAM. The streaming parsers stay flat because they hold one item at a time.

Text editors try to highlight and fold everything

On top of parsing, an editor tokenizes the text for colours, works out folding ranges and runs a language server to validate it. VS Code turns these features off for big files. Its source puts the cut-off at 20 MB or 300,000 lines, and past 50 MB it stops syncing the file to extensions, so you lose JSON validation and the outline. Scrolling a few hundred megabytes of single-line JSON is still slow after all that. Minified JSON, one enormous line, is the worst case for any text editor.

Browsers have a hard string limit

Chrome won't create a JavaScript string longer than 536,870,888 characters (about 512 MB). A web tool that calls file.text() and then JSON.parse stops there, however much RAM you have. You'll see "Invalid string length" or "Cannot create a string longer than 0x1fffffe8 characters". And if the parse runs on the page's main thread, the tab freezes until it finishes.

First: look at the file without opening it

Check what's inside before you choose a tool. These two commands read only the start of the file:

ls -lh export.json          # how big is it really
head -c 2000 export.json    # the first 2 KB: is it one array, an object, NDJSON?

One JSON object per line means you have NDJSON (JSON Lines), and your life gets easier. head, grep, split and wc -l all work on it, and any tool can read it line by line. Viewing and querying NDJSON logs covers that case.

For one big document, note the top-level shape. Most large files look like {"meta": ..., "items": [ ...millions of things... ]}, and the path to the big array (items here) is what you'll stream over.

Option 1: jq, when the machine has the RAM

If the file fits in memory, jq answers a question faster than anything else. Some commands I use all the time:

jq '.orders | length' big.json                 # how many items
jq -c '.orders[:2]' big.json                   # the first two items, compact
jq 'keys' big.json                             # top-level keys
jq -c '.orders[] | select(.status == "refunded") | .id' big.json > refunded.txt

Memory is the problem. Going by the table above, plan for 10 to 20 times the file size: fine for 300 MB on a 16 GB laptop, probably not for 1 GB.

Option 2: jq --stream, slow but constant memory

With --stream, jq emits path/value events and never builds the document. Add fromstream and truncate_stream and you can rebuild one array element at a time:

jq -cn --stream '
  fromstream(2 | truncate_stream(inputs | select(.[0][0] == "orders")))
  | select(.status == "refunded")
  | .id' big.json

The 2 is how many path levels to strip. ["orders", 17, "status"] becomes ["status"], so jq rebuilds each order as a standalone object. For a bare top-level array, drop the select and strip one level: jq -cn --stream 'fromstream(1 | truncate_stream(inputs))' big.json.

On the 15.5 MB test file this took 6.0 s and 3.6 MB of memory. Plain jq took 0.6 s and 231 MB. So you pay ten times the time for a sixtieth of the memory, and for a multi-gigabyte file on a small machine you have no other way.

Option 3: a streaming parser in Python or Node

When the job is to go through every record and compute something, ten lines of code with a streaming parser will do it.

Python with ijson

pip install ijson
import ijson
from decimal import Decimal

refunded = 0
total = Decimal(0)
with open("big.json", "rb") as f:
    for order in ijson.items(f, "orders.item"):
        if order["status"] == "refunded":
            refunded += 1
            total += order["total"]
print(refunded, total)

"orders.item" means each element of the orders array; for a file that is a bare array, use "item". ijson hands back decimal numbers as Decimal, which is why the total starts as one. When the fast C backend (yajl2_c) is installed, ijson uses it. In my test it beat json.load on time and used a tenth of the memory.

Node with stream-json

Node ships without a streaming JSON parser. The stream-json package fills that gap:

npm install stream-json
const fs = require('fs');
const { chain } = require('stream-chain');
const { parser } = require('stream-json');
const { pick } = require('stream-json/filters/Pick');
const { streamArray } = require('stream-json/streamers/StreamArray');

let refunded = 0;
chain([
  fs.createReadStream('big.json'),
  parser(),
  pick({ filter: 'orders' }),
  streamArray(),
])
  .on('data', ({ value }) => { if (value.status === 'refunded') refunded++; })
  .on('end', () => console.log(refunded));

I tested this with stream-json 1.9.1. It runs slower than ijson's C backend, but file size doesn't matter to it. JSON.parse(fs.readFileSync(path, 'utf8')) on a 1 GB file fails at the string limit before parsing even begins.

Option 4: load it into a database

Planning to ask the same file many different questions? Import it once. DuckDB reads JSON and NDJSON with read_json_auto, and SQLite has JSON functions. You wait for the import one time, and after that you get SQL and fast queries. For analysis that goes on for days, I'd pick this. For a one-off look it's too much setup.

Option 5: split it into smaller files

If a colleague needs the data in a tool with a size limit, split it. jq streams it into NDJSON and split cuts that into parts:

jq -cn --stream 'fromstream(2 | truncate_stream(inputs | select(.[0][0] == "orders")))' big.json > orders.ndjson
split -l 100000 orders.ndjson part_

Each of part_aa, part_ab and so on holds 100,000 records, one per line. To turn one back into a JSON array: jq -s . part_aa > part_aa.json.

Option 6: a viewer that parses off the main thread

Sometimes you only want to look: scroll the structure, search for an ID, see what record 1.2 million looks like. A viewer suits that if it gets three things right. It parses in a background thread (a Web Worker, in a browser) so the window stays usable. It keeps the parsed data away from the interface and draws only the rows on screen. And it reads the file in pieces, never as one string.

GigaJSON is a browser tool built on those rules. Open a file in the large JSON viewer or the full app and a worker parses it on your machine; nothing gets uploaded. We benchmarked it on the same i5-9300H laptop in Chrome: the tree appeared after 2.1 s for 200 MB, 9.3 s for 500 MB and 18.3 s for 1 GB (medians of three runs, October 2026), and the page kept responding throughout. The page's own heap stayed between 5 and 7 MB at every size, since it only ever receives the rows on screen. That figure leaves out worker memory, so it isn't the total footprint. The benchmark page has the three raw runs, exact byte counts and a fixture generator; those archived reports don't record the exact source revision.

Once the file is open you can browse the tree, see an array of objects as a table, search with Find, and run JSONPath or JMESPath. The limits you should know about:

  • The largest-document setting starts at 512 MB. Raise it to 1 GB in Preferences.
  • At 1 GB the worker needs a few GB of RAM. On an 8 GB machine with other apps open, close some tabs first.
  • Queries over the whole document are slow at that size: $..sku over the 1 GB file took about a minute.
  • jq in the browser stops at about 20 MB of input (the WebAssembly build aborts past ~25 MB). For big files, use JSONPath or JMESPath in the browser, or jq in a terminal.
  • Native desktop viewers open 1 GB files much faster than a browser tool can. If you open giant files every day, that speed may matter more to you than skipping an install.
  • GigaJSON keeps the file in the browser's storage so you can come back to it. That costs about twice the file size on disk (the document plus its "Original" version).

Which option to pick

You want to... Use
Answer one question, file fits in RAM jq
Process every record of a multi-GB file ijson (Python) or stream-json (Node)
Same, without writing code, on a small machine jq --stream
Ask many questions over days DuckDB or SQLite
Hand the data to someone with a size limit split into NDJSON parts
Browse, search and inspect up to 1 GB a viewer that parses in a worker, such as the large JSON viewer

If you control how the file gets written, write NDJSON instead of one giant array. Every tool on this list handles it better, and so does grep.

Questions

Why does my editor freeze when I open a large JSON file?

Most editors read the whole file, turn it into a tree in memory and then lay it out on screen. The parsed tree needs several times the file's size in RAM, and the parsing often runs on the thread that draws the window, so the editor can't respond until it's done.

What is the largest JSON file a browser can open?

Chrome caps a JavaScript string at about 512 million characters. A tool that reads the file into one string and calls JSON.parse fails around 500 MB. Tools that parse the bytes in pieces get further: GigaJSON opens files up to 1 GB that way.

How do I read a huge JSON file in Python without running out of memory?

Use a streaming parser such as ijson. ijson.items(f, 'orders.item') gives you one element of the orders array at a time, so memory stays flat however big the file gets.

Can jq handle a 1 GB JSON file?

Yes, if the machine has the RAM. Plain jq loads the whole document, which takes many times the file size. With --stream it reads event by event in a few megabytes, at about a tenth of the speed.

Is it safe to open a confidential JSON file in an online viewer?

Only if the tool parses it on your machine. GigaJSON reads the file inside your browser and never uploads it. Open the browser's Network panel while it loads and you can check that yourself.

More from the blog