Executive Summary

1 min summary

A fixed single-machine deployment made data representation, rather than adding infrastructure, the critical architectural decision. Moving long-lived grid data into region-based memory mappings kept the existing Java algorithm viable, but shifted the operational burden to paging, access patterns, and resource cleanup. Before funding a rewrite or capacity increase, validate the workload against API limits and the actual deployment constraints.

Deep Tracks: Mapping India Outside the Heap

A telecom client needed to optimize antenna tilt and power settings for maximum coverage across their Indian network. The goal: find the configuration that delivers the strongest signal to the most people. The resolution: 10 meters. The area: all of India.

Do the math in front of yourself. India covers roughly 3.3 million square kilometers. At 10-meter resolution, each square kilometer contains 10,000 grid cells. That’s 33 billion cells. Each cell needs state for the optimization algorithm: signal strength, terrain characteristics, population density, antenna visibility. Even at a conservative 8 bytes per cell, you’re looking at 264 GB of grid data.

The deployment constraint: a single machine in the client’s datacenter. 128 GB of RAM. No cloud. No cluster. One box.

The JVM was the platform. The optimization algorithm was already written in Java, already validated, already trusted by the engineering team. Rewriting it wasn’t on the table. The question was whether the JVM could handle data at this scale without falling apart.

Why the Heap Betrays You

The JVM’s memory model is built for a different kind of workload. Objects are allocated in young generation, promoted to old generation if they survive, and eventually collected by the garbage collector. This works beautifully for the typical case: lots of short-lived objects, a manageable number of long-lived ones.

33 billion grid cells are not the typical case.

Start with object headers. Every Java object carries 12–16 bytes of metadata: class pointer, hash code, GC age, lock state. For a grid cell carrying 8 bytes of actual payload, the header alone costs 2x the useful data. The math gets ugly fast:

33 billion cells × 16 bytes of header = 528 GB in object headers alone.

That’s four times the machine’s total RAM, and you haven’t stored a single data value yet. Game over.

But even if you could somehow compress the representation (packed arrays, primitive arrays, structs-on-heap hacks) you’d still face the GC problem. These 33 billion cells are long-lived. They’re allocated at startup and live for the entire optimization run. Every one of them lands in old generation. And every full GC cycle must scan every live object to determine what’s still reachable.

Here’s what that looks like in practice. A naive allocation attempt:

// Don't do this. Shown for illustration.
int rows = 180_000;  // latitude cells
int cols = 360_000;  // longitude cells
long[][] grid = new long[rows][cols];
// OutOfMemoryError before you finish allocating

And the GC pressure, even at a smaller scale, from jstat -gcutil:

  S0     S1     E      O      M     CCS    YGC    YGCT    FGC   FGCT     GCT
  0.00  94.31  87.22  98.71  97.12  95.88  1842  124.31    47  891.07  1015.38

Old generation at 98.7% capacity. 47 full GCs consuming 891 seconds. The optimization algorithm, which needs responsive, predictable heap for its working set, is spending more time waiting for garbage collection than doing actual work.

G1 and ZGC handle large heaps better than the older collectors, but they’re not magic. When billions of objects live in old generation and never die, the collector must still track them. Card table scanning, remembered sets, concurrent marking — all of it scales with the number of live objects. The collector is doing enormous work to reach the same conclusion every cycle: nothing has changed.

The optimization algorithm needs the heap. The grid data is squatting on it.

The First Solution: MappedByteBuffer

The idea was simple: move the grid data out of the JVM heap entirely. Store it in a file. Map that file into the process’s virtual address space using the operating system’s memory-mapping facility. Let the OS manage paging data in and out of physical RAM. The JVM’s garbage collector would never see it.

Java exposes this through FileChannel.map(), which wraps the mmap system call. When you memory-map a file, the OS creates entries in the process’s page table pointing to the file’s contents on disk. No data is loaded immediately. When your code reads a byte at a given offset, the CPU triggers a page fault, the OS loads that 4KB page from disk into the page cache, and the read completes. Subsequent reads from the same page hit RAM directly.

The critical insight: none of this involves the JVM heap. The mapped memory lives outside the garbage collector’s view. It doesn’t get scanned. It doesn’t cause pauses. It doesn’t count against -Xmx.

The prototype looked straightforward:

RandomAccessFile raf = new RandomAccessFile("grid.dat", "rw");
FileChannel channel = raf.getChannel();

long gridSizeBytes = (long) rows * cols * Long.BYTES;
MappedByteBuffer buffer = channel.map(
    FileChannel.MapMode.READ_WRITE, 0, gridSizeBytes
);
buffer.order(ByteOrder.nativeOrder());

The indexing scheme converted geographic coordinates to byte offsets with O(1) access:

long cellOffset(int row, int col) {
    return ((long) row * cols + col) * Long.BYTES;
}

void writeCell(int row, int col, long value) {
    buffer.putLong((int) cellOffset(row, col), value);
}

long readCell(int row, int col) {
    return buffer.getLong((int) cellOffset(row, col));
}

I tested it on a smaller region, a single Indian state, a few hundred million cells. It worked. GC pauses vanished from the logs. The heap stayed small and responsive. The optimization algorithm ran without interruption. Clean, elegant, solved.

Time to scale up to all of India.

The Wall: “Why Did It Just Crash?”

The full-scale run died within seconds.

java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:895)

MappedByteBuffer extends ByteBuffer. ByteBuffer uses int for positions and limits. An int in Java maxes out at 2,147,483,647 (just under 2 GB).

A single MappedByteBuffer mapping cannot exceed 2 GB.

The grid for all of India was tens of gigabytes. The mapping that worked perfectly for a single state couldn’t cover the whole country. Not even close.

This is the kind of limit that doesn’t surface in tutorials. Every MappedByteBuffer example you’ll find online works with files that fit comfortably under 2 GB. Stack Overflow answers handle megabytes. The API compiles without warning. The code is correct for the small case and fundamentally broken for the large one.

I had the right idea and the wrong assumption about its limits. Back to the drawing board.

Decision checkpoint: A successful small-region prototype did not validate the full-country workload. Test hard API limits and representative access patterns before treating a design as ready for production scale.

This is where system architecture work earns its keep: tracking the workload, library limits, and deployment constraints together, rather than choosing a familiar tool and hoping it scales.

The Redesign: A List of Regions

The solution was to abandon the idea of a single mapping and build a layer on top: split the grid into regions, each small enough for one MappedByteBuffer, managed by a region index that makes the split transparent to the rest of the code.

public class RegionMappedGrid {
    private static final long MAX_REGION_BYTES = 1L << 30; // 1 GB per region
    private final MappedByteBuffer[] regions;
    private final long cellsPerRegion;
    private final int cols;

    public RegionMappedGrid(int rows, int cols, FileChannel channel)
            throws IOException {
        this.cols = cols;
        long totalBytes = (long) rows * cols * Long.BYTES;
        int regionCount = (int) ((totalBytes + MAX_REGION_BYTES - 1)
                                  / MAX_REGION_BYTES);
        this.cellsPerRegion = MAX_REGION_BYTES / Long.BYTES;
        this.regions = new MappedByteBuffer[regionCount];

        for (int i = 0; i < regionCount; i++) {
            long offset = (long) i * MAX_REGION_BYTES;
            long size = Math.min(MAX_REGION_BYTES, totalBytes - offset);
            regions[i] = channel.map(
                FileChannel.MapMode.READ_WRITE, offset, size
            );
            regions[i].order(ByteOrder.nativeOrder());
        }
    }

    public long readCell(int row, int col) {
        long linearIndex = (long) row * cols + col;
        int regionIndex = (int) (linearIndex / cellsPerRegion);
        int offsetInRegion = (int) (
            (linearIndex % cellsPerRegion) * Long.BYTES
        );
        return regions[regionIndex].getLong(offsetInRegion);
    }

    public void writeCell(int row, int col, long value) {
        long linearIndex = (long) row * cols + col;
        int regionIndex = (int) (linearIndex / cellsPerRegion);
        int offsetInRegion = (int) (
            (linearIndex % cellsPerRegion) * Long.BYTES
        );
        regions[regionIndex].putLong(offsetInRegion, value);
    }
}

The callers never see the regions. They call readCell(row, col) and writeCell(row, col, value). The region manager translates coordinates to region index and offset within that region’s buffer. The 2 GB limit becomes invisible.

This architecture brought a bonus: each region could be independently managed. Regions covering areas the algorithm had finished processing could be unmapped to release physical memory back to the OS. Regions about to be needed could be pre-loaded. The grid became elastic in a way a single mapping never could be.

For the India dataset, the grid split into roughly 25–30 regions of 1 GB each. The OS mapped and paged them as needed. The optimization algorithm ran. The GC stayed quiet.

Spatial Locality and Page Faults

Memory-mapped files shift the performance bottleneck from GC pauses to page faults. When the OS needs to load a page that isn’t in RAM, it stalls that thread until the I/O completes. The cost depends entirely on access pattern.

Sequential access is nearly free. The OS detects sequential reads and prefetches upcoming pages before you request them. If your algorithm sweeps across the grid row by row, the vast majority of accesses hit pages already in the cache.

Random access is expensive. Each random read to an unmapped page triggers a disk I/O. If your algorithm jumps between distant grid cells — say, comparing an antenna’s coverage across scattered geographic points — you’ll see thousands of page faults per second. Each one costs microseconds with an SSD, milliseconds with a spinning disk.

The layout of your data in the file determines which access patterns are cheap. Row-major order (the default for our grid) makes row-wise sweeps fast. If the algorithm needs to access square tiles — all cells within a geographic bounding box — a tile-based layout, where adjacent tiles are stored contiguously, can reduce page faults dramatically.

For our use case, row-major worked well enough. The optimization algorithm processed antennas geographically, sweeping through coverage areas that mapped naturally to contiguous rows.

Monitoring page faults is essential. On Linux:

# Per-process page fault counts
grep 'page_faults' /proc/<pid>/stat

# Real-time page fault monitoring
perf stat -e page-faults -p <pid> -- sleep 10

MappedByteBuffer.load() forces all pages into RAM eagerly. For regions you know the algorithm will access heavily, pre-faulting avoids stalls during the computation. But don’t call load() on every region at startup — you’ll saturate I/O and potentially push other regions’ pages out of the cache. Load selectively, based on what the algorithm needs next.

More Pitfalls

Unmapping

MappedByteBuffer has no public unmap() method. The buffer stays mapped until the garbage collector finalizes its Cleaner. If you need to re-map the same file region, or delete the file, or just release memory pressure, you’re stuck waiting for GC to notice.

The workaround is ugly but necessary:

// Java 9+
import sun.misc.Unsafe;

void unmap(MappedByteBuffer buffer) throws Exception {
    Unsafe unsafe = getUnsafe();
    unsafe.invokeCleaner(buffer);
}

private Unsafe getUnsafe() throws Exception {
    Field f = Unsafe.class.getDeclaredField("theUnsafe");
    f.setAccessible(true);
    return (Unsafe) f.get(null);
}

This is internal API. It triggers warnings with --illegal-access. It may break in future JVM versions. Use it anyway. The alternative — letting mapped regions accumulate until GC decides to clean them up — is worse in a memory-constrained environment. Just wrap it, document it, and monitor for breakage on JDK upgrades.

Byte Order

ByteBuffer defaults to big-endian. Your CPU is almost certainly little-endian. Every getInt() and putLong() silently byte-swaps the data. Set ByteOrder.nativeOrder() on every buffer immediately after mapping:

buffer.order(ByteOrder.nativeOrder());

The performance difference is measurable on billions of accesses.

Memory Visibility

MappedByteBuffer.force() flushes dirty pages to disk, guaranteeing durability. But within a single process, you don’t need it for consistency — writes are immediately visible to subsequent reads from the same mapping. Don’t call force() in a hot loop. Save it for checkpointing.

The Swap Trap

Memory-mapped files share physical RAM with everything else. When the OS runs low on free pages, it evicts mapped pages back to disk. If your algorithm then re-accesses those pages, you pay the I/O cost again. Under heavy pressure, this becomes thrashing: constant eviction and re-loading, with throughput collapsing.

Monitor with vmstat:

vmstat 1
# Watch the 'si' (swap in) and 'so' (swap out) columns
# Non-zero values under load indicate memory pressure

On a dedicated machine, tune vm.swappiness low (e.g., 10) to tell the kernel to prefer evicting file-backed pages over anonymous memory. This protects the JVM heap while allowing mapped regions to page out gracefully.

The GC Dividend

Once the grid data moved off-heap, the JVM logs told a different story.

Before, with grid data on-heap (-Xlog:gc*):

[gc] GC(892) Pause Full 114G->113G(120G) 47231.442ms
[gc] GC(893) Pause Full 114G->113G(120G) 48102.118ms

Full GC pauses of 47 seconds. The collector scanning 114 GB of live data, freeing almost nothing, because 33 billion grid cells never die.

After offloading:

[gc] GC(12) Pause Young (G1 Evacuation Pause) 2G->1G(4G) 8.221ms
[gc] GC(13) Pause Young (G1 Evacuation Pause) 2G->1G(4G) 7.893ms

Young generation pauses of 8 milliseconds. No full GCs at all. The heap dropped from 120 GB to 4 GB — just enough for the optimization algorithm’s working set. The collector had nothing heavy to scan.

Heap sizing after offloading becomes straightforward. The -Xmx flag no longer needs to account for grid data. Set it to what the algorithm actually needs, plus headroom for the usual overhead. For this workload, 4–8 GB was generous.

A few JVM flags to consider alongside off-heap patterns:

-XX:MaxDirectMemorySize controls how much memory ByteBuffer.allocateDirect() can use. Memory-mapped buffers don’t count against this limit, but if you’re mixing mapped and direct buffers, set it explicitly.

-XX:+DisableExplicitGC prevents library code from triggering System.gc(). Some NIO cleanup code calls explicit GC to reclaim direct buffers — with off-heap patterns, you want GC to run on its own schedule, not when a library panics about buffer count.

For the collector itself: G1 is the right default for moderate heaps (4–16 GB). If your working set is larger or you need sub-millisecond pauses, ZGC handles multi-GB heaps with consistent low latency.

Alternatives Considered

No technique exists in isolation. Here’s what else we evaluated and why it didn’t fit.

ByteBuffer.allocateDirect(): allocates off-heap memory without a file backing. Same 2 GB limit per buffer. Counts against MaxDirectMemorySize. No persistence, no ability to memory-map from disk. Useful for network I/O buffers, not for datasets this size.

Embedded databases (RocksDB, LMDB): designed for key-value storage with efficient disk access. The problem: serialization overhead. When you’re doing billions of reads and writes per optimization iteration, the cost of encoding and decoding each cell through a database API adds up catastrophically. Our access pattern was direct byte-level reads to known offsets — no query planning, no serialization, no abstraction tax.

Distributed compute (Spark, Hazelcast): the obvious answer for large datasets. And a non-option. The client’s constraint was absolute: this runs on one machine in their datacenter. No network. No cluster. No negotiation. The constraint was driven by data sovereignty requirements that predated our engagement.

Libraries (Chronicle Map, MapDB): Chronicle Map offers memory-mapped hash maps with off-heap storage. MapDB provides tree-based structures backed by memory-mapped files. Both are excellent for their use cases. Ours wasn’t a hash map or a tree — it was a dense, fixed-size grid with coordinate-based indexing. Adding a library’s abstraction layer would have cost performance without adding functionality. Raw MappedByteBuffer was the right tool.

When to Go Off-Heap

Not every large dataset needs this treatment. A decision framework:

Strong candidates for off-heap: Large, long-lived, read-heavy data. Data you structure yourself as flat bytes, eliminating object header overhead. Workloads where predictable GC latency matters alongside large datasets.

Stay on heap: Small datasets that fit comfortably in a few GB. Complex object graphs with rich relationships — the overhead of manual serialization defeats the purpose. Short-lived data that benefits from generational GC. Anything where developer productivity matters more than memory efficiency.

The core question: is the GC spending significant time scanning data that never changes? If yes, that data doesn’t belong on the heap.

Key Takeaways

The JVM’s object model is the enemy at scale. 12–16 bytes of header per object means that at billions of objects, you’re spending more memory on metadata than on data. Off-heap storage eliminates this entirely.

MappedByteBuffer is powerful but has a 2 GB ceiling. The int-based position API limits each mapping to ~2 GB. For larger datasets, you need a region manager that splits data across multiple mappings transparently.

Access pattern determines performance. With memory-mapped files, the bottleneck shifts from GC pauses to page faults. Sequential access is nearly free; random access pays I/O cost per page. Structure your data layout around your algorithm’s access pattern.

Off-heap doesn’t mean unmanaged. You still need to handle unmapping (no public API), byte order (default big-endian costs you on every access), swap pressure (monitor vmstat), and file cleanup.

The GC dividend is dramatic. Moving long-lived bulk data off-heap can reduce heap requirements by an order of magnitude and eliminate full GC pauses entirely. The collector stops scanning data that never changes, and your application’s working set gets the responsive heap it needs.

This is the first post in Deep Tracks — a series on JVM internals, performance, and the kind of problems that don’t show up until you’re past the scale where tutorials help. Next up: GC tuning war stories from production.

Wrestling with JVM performance at scale? Let’s talk about what’s in your heap.

Szymon Kaczorowski

Principal Software Engineer

15 years building systems that can't go down. From Tier-1 banks to fintech scale-ups, I help engineering teams solve the problems other consultants gave up on.