False sharing and native-bridge hot paths

The team spent a week making a counter array lock-free with atomic operations. Under load, it’s slower than the locked version was. The flame graph shows updateCounter wide, IPC 0.42. Meanwhile, a Rust crypto library is idle 92% of the time. The Node service calls it 10,000 times per second — and 40% of CPU is in the N-API stub, not the crypto code.

False sharing: when “lock-free” is slower than locked

False sharing happens when multiple threads write to different fields that happen to share the same cache line. The hardware’s MESI coherency protocol treats a cache line as the atomic unit of ownership. When one CPU writes to any byte in a 64-byte line, it acquires exclusive ownership and invalidates the line in every other CPU’s cache. Every other CPU that subsequently reads or writes any byte in that line must re-fetch it through the coherency fabric — at L3 or DRAM latency (~150–300 cycles), not L1 (~5 cycles).

The result: atomic operations that appear non-contending at the code level contend heavily at the hardware level because their data lives on the same cache line.

Signature in profiles

False sharing does not look like lock contention in a standard CPU profile. There is no visible mutex, no blocked thread. Instead:

• IPC collapses (typically 0.3–0.6 on affected code, compared to 2–4 for compute-bound code).
• Cache-miss rate is extreme (60–80%), even though the data is small and “should” be hot.
• The hot function is innocent-looking — an atomic increment, a simple field write.
• Performance degrades as thread count increases, not improves.

Hardware counters that expose it

The hardware event MEM_LOAD_L3_HIT_RETIRED.XSNP_HITM (Intel) counts loads that were satisfied by a modified copy in another CPU’s cache — a direct false-sharing signal. On Linux, perf stat -e cache-references,cache-misses,instructions paired with scaling the thread count exposes it indirectly.

Fix: cache-line padding

The fix is to ensure each independently-written field occupies its own cache line. On x86, a cache line is 64 bytes; on ARM, 64 or 128 bytes.

// BEFORE: 16 uint64 counters share 2 cache lines (8 per line)
var counters [16]uint64

// AFTER: each counter on its own 64-byte line
type paddedCounter struct {
    value uint64
    _     [56]byte // pad to 64 bytes
}
var counters [16]paddedCounter

In Java, @Contended (from sun.misc.Contended, or jdk.internal.vm.annotation.Contended) inserts padding automatically. In Rust, crossbeam::CachePadded wraps values. In C++, alignas(64) on struct fields. The Disruptor (Java) and DPDK (C) bake explicit cache-line padding into their core data structures as a non-negotiable invariant.

# perf stat -e cache-references,cache-misses,L1-dcache-load-misses,instructions ./service

 1,250,000,000      cache-references
   950,000,000      cache-misses           # 76% miss rate — extreme
 1,200,000,000      L1-dcache-load-misses  # nearly every L1 access misses
 3,000,000,000      instructions
                    IPC = 0.42             # CPU stalled 58% of the time

# Profile shows hot leaf:
#   updateCounter(idx int):
#     atomic.AddUint64(&counters[idx], 1)   # supposed lock-free fast path

# counters[] is a flat array of 16 uint64 values, accessed by 16 worker
# goroutines (each goroutine increments its own index).
# CPU is 16-core. Each uint64 is 8 bytes; cache line is 64 bytes.

Native-bridge hot paths: the FFI overhead trap

Modern runtimes bridge to native code via FFI: Node’s N-API, Java’s JNI, Python’s ctypes / cffi / Cython, Go’s cgo. Each bridge crossing carries fixed overhead:

• N-API (Node → native addon): ~50–200 ns per call.
• JNI (Java → native): ~100–500 ns per call.
• cgo (Go → C): ~200–500 ns per call (includes goroutine stack switch).
• Python ctypes: ~1–5 μs per call.

When the bridged function is expensive (milliseconds), this overhead is irrelevant. When the bridged function is cheap (nanoseconds), the bridge stub can dominate.

Signature in a cross-language flame graph

A standard single-language profiler shows only its own stack. A cross-language profile (eBPF, Datadog continuous profiler, or a manually stitched perf + async-profiler capture) shows both stacks. The false-sharing signature from the profiler’s perspective:

• The native function itself is narrow (small self-time).
• The bridge stub (Cgo_runtime_cgocall, JNIEnv::CallStaticVoidMethod, napi_call_function) is wide.

Real-world example

A Node service called a Rust crypto routine via N-API: 10,000 calls per second, each call computing a 32-byte HMAC. The Rust function itself took ~40 ns. The N-API stub added ~160 ns per call — 4x the work. CPU profile: 40% in the stub, 8% in the actual crypto function.

Fix: batch 64 operations per N-API call. The Rust function receives a slice of 64 inputs and returns a slice of 64 outputs. Per-item overhead drops from 200 ns to 43 ns (160 ns stub / 64 items). CPU profile after: 12% crypto function, stub invisible.

Fix families for native-bridge overhead:

1. Batch per crossing — pass a slice of inputs, receive a slice of outputs. Amortise the fixed overhead over N items.
2. Push the loop into native — instead of calling native N times, call native once with the loop body inside the native function.
3. Raise the boundary — move the FFI boundary to a coarser operation so fewer crossings happen per unit of work.

Edge cases where “wider frame = bigger problem” lies

Three situations where the widest leaf is not the right attack target.

1. Sampled-out short hot paths

A function called 500,000 times per second for 200 ns each runs for 100 ms/s total — 10% of a single CPU second. At a standard 100 Hz sampling rate, the profiler fires ~10 samples per second. Expected samples: 1. Actual samples: 0 or 1, depending on alignment.

The frame is narrow in the flame graph, but it is a top consumer. Diagnosis: instrument with cheap counters (atomic increments + a Prometheus histogram), or raise sample rate temporarily to 1000 Hz during a dedicated profiling window.

2. Spin-wait dominating the CPU profile

A CPU profile shows a function wide because the program spin-waited inside it — busy-looping until a condition holds. The thread is on CPU, consuming cycles, but doing no real work. The fix is not to optimise the spin’s body; it is to convert the spin into a proper wait (futex, condition variable, channel).

Signature: function body is a tight branch back to itself; IPC is low despite being CPU-bound in the profile; context-switch rate is low (the thread never yields).

3. Symbol resolution failures

A wide [unknown] frame is not a function — it is a stack the profiler cannot resolve. Common causes: JIT-compiled code without perf maps (Node needs --perf-basic-prof; JVM needs -XX:+PreserveFramePointer), stripped DWARF debug info, missing kernel symbols.

Before treating [unknown] as a target, fix the symbol resolution. The underlying function may be the real hot path, hidden by a diagnostic gap.