JSONL

JSON Lines (JSONL) is a text format for storing structured data where each individual record occupies a single line.

JSONL streamlines large-scale data processing by treating every line as a valid JSON object. Unlike standard JSON arrays, this format allows for incremental parsing and streaming: you can append new records without rewriting the file or loading 50GB into memory. It is the industry standard for high-throughput logging, ETL pipelines, and training datasets for LLMs like GPT-4. By using the .jsonl extension and UTF-8 encoding, developers ensure compatibility with Unix tools like grep and awk while maintaining the rich structure of nested data.

https://jsonlines.org/