On-disk graph

On-disk graph technology, or out-of-core processing, enables analysis of massive graphs (terabytes, billions of edges) that exceed a machine's main memory (RAM) capacity.

This technology is mission-critical for scaling graph analytics beyond the limits of in-memory systems. It operates by partitioning the graph data—nodes and edges—into disk-resident blocks, then employing I/O-efficient techniques like bi-sharding (GraphChi) or LSM-tree compaction (LSMGraph) to minimize costly random disk accesses. The system loads only the necessary subgraphs or 'active' blocks into RAM for computation, converting random I/O into sequential I/O for better performance on SSDs. This approach ensures persistence, handles graphs up to the petabyte scale, and supports essential algorithms like PageRank and Breadth-First Search (BFS) on datasets far too large for a single-server memory footprint.

https://graphar.apache.org/