SIMD, SoA vs AoS, and memory bandwidth

A hot ML inference loop is 5x slower than the reference C++ implementation, despite identical algorithm and the same number of multiplications. The profiler shows IPC 0.8 and 25% L3 miss rate. Switching to SIMD intrinsics is suggested — but the fix that actually works is changing how the data is stored, not how it is computed.

SIMD: one instruction, multiple values

Modern CPUs carry wide vector registers: 256-bit (AVX2, standard on x86 since 2013), 512-bit (AVX-512, available on server Intel and recent Ryzen), 128-bit NEON on ARM. One SIMD instruction can add, multiply, or compare 4–8 floats simultaneously.

The critical requirement: the values must be consecutive in memory. A single load instruction fills the vector register from a contiguous block; the CPU then operates on all 4–8 values in parallel.

Array of Structures vs Structure of Arrays

AoS (Array of Structures):

struct Point { float x; float y; float z; };
Point points[N];  // layout: x0 y0 z0 x1 y1 z1 x2 y2 z2 ...

To add all x values with SIMD, you need points[0].x, points[1].x, points[2].x, points[3].x — but they are at offsets 0, 12, 24, 36 bytes. The SIMD load at address 0 gives x0 y0 z0 x1 — mixed types. You must use scatter/gather operations to pull out just the x values, which costs ~10x a contiguous load.

SoA (Structure of Arrays):

float xs[N];  // layout: x0 x1 x2 x3 x4 x5 x6 x7 ...
float ys[N];
float zs[N];

To add all x values with SIMD: load 8 consecutive floats from xs[0], done. One instruction, 8 results. The loop body becomes 8x more productive per instruction.

Game engines (Unity ECS, Bevy), ML inference engines, audio processing, and database column stores all use SoA. V8 (Chrome’s JavaScript engine) uses SoA-like TypedArrays for hot loops. The previous lesson’s ML example: changing from AoS (x,y,z,w) structs to xs[], ys[], zs[], ws[] drops L3 miss rate from 25% to 5% and raises IPC from 0.8 to 3.0.

Auto-vectorisation

Compilers automatically convert simple loops to SIMD when the data layout allows. Conditions for auto-vectorisation success:

• No pointer aliasing (two pointers don’t point to overlapping memory — use restrict in C, Rust handles this via borrow checker).
• Contiguous (not strided or gather/scatter).
• Predictable trip count.
• No data-dependent inner branches.

Check what the compiler emitted: -fopt-info-vec (GCC) or -Rpass=loop-vectorize (Clang) prints when loops are vectorised and why they failed. For hot loops that fail auto-vec, manual SIMD intrinsics or libraries (Intel Highway, simd-everywhere) close the gap.

Memory bandwidth: the other constraint

Cache hit rate is one axis of performance; memory bandwidth is another. A workload that streams through 100 GB of data will hit the RAM bandwidth ceiling (~50–100 GB/s on DDR5) regardless of cache behaviour. Bandwidth-bound code is fixed by:

• Reducing data volume: more compact types (float32 instead of float64 when precision allows), on-the-fly computation instead of materialised intermediate tables.
• Compression at rest, decompression on-the-fly.
• Non-temporal stores (covered in the senior lesson): bypass cache for write-once data.

perf stat -e cache-misses,mem-loads-retired.l3-miss separates cache-miss-bound from bandwidth-bound: if L3 misses are high but total data volume is huge, you are bandwidth-bound; if L3 misses are high but data volume is small (fits in L3 but pattern is random), you are cache-miss-bound.

NUMA: multi-socket memory access

Servers with 2+ CPU sockets are Non-Uniform Memory Access. Each socket has local RAM (~70 ns) and remote RAM (~120–150 ns). A thread that allocates memory on socket 0 but runs on socket 1 pays the remote-access tax on every load. This is a 1.7x latency penalty that perf stat will misreport as “L3 misses” because the access goes through the interconnect.

Mitigations:

• Pin threads to sockets (taskset, hwloc).
• Allocate on the local NUMA node (numactl --membind, jemalloc NUMA-aware mode).
• Distribute work so each thread touches data from its local socket.