Samsung LPDDR5X-PIM: processing in memory aimed squarely at the decode bottleneck

At Hot Chips 2026 on August 26 Samsung showed LPDDR5X-PIM, a mobile-class DRAM with compute logic placed next to the memory cells. The preliminary numbers: 2.28x faster runtime and 3.01x higher token throughput than conventional LPDDR5X, tested with Llama 3.1 8B on an edge AI accelerator. The number underneath those is the one I care about: peak bandwidth goes from 76.8 GB/s to a theoretical 614 GB/s, an 8x jump, because a chunk of the work never leaves the DRAM die. Samsung says accuracy tuning is still ongoing.

Why decode is a bandwidth problem

Generating one token with a dense model means reading every weight once. For an 8B model in 4-bit that is roughly 4 GB per token, and the arithmetic per byte is tiny. On the roofline chart decode sits far to the left, in the memory-bound region, where the only thing that raises tokens per second is bytes per second. Compute units sit idle waiting for DRAM.

That is why the same model runs at wildly different speeds on hardware with similar FLOPS and different memory. It is also why unified-memory Macs punch above their weight: the bandwidth number decides decode, not the GPU core count. And it is why, on a phone or a thin laptop with LPDDR at 76.8 GB/s, an 8B model is a slideshow: my back-of-envelope, 4 GB per token over 76.8 GB/s, caps you near 19 tokens per second before any overhead, and real numbers are lower.

What processing-in-memory changes

PIM attacks the bytes-per-token term directly. If the multiply-accumulate for a weight can happen next to the cells that hold it, the weight never crosses the memory bus; only the much smaller result does. Effective bandwidth, as seen by the model, goes up by however much work you push into the memory die. Samsung's 8x theoretical figure is that ratio; the 3.01x measured throughput is what survived contact with a real accelerator and a real model.

Decode does not need more FLOPS. It needs the weights to stop travelling. Processing-in-memory is the first mainstream memory part built around that fact.

Two things make this announcement different from the datacenter memory race. It is LPDDR, the memory in laptops, phones and edge boxes, not the HBM stacked next to datacenter GPUs; the same week OpenAI's Jalapeño chip made its perf-per-watt claims on HBM4, which is a different market entirely. And the test workload is a real 8B LLM, which is exactly the size class that fits on those devices.

What it means for local inference

Be careful with the timeline; this is a Hot Chips talk, not a product. But the shape of the win is clear enough to plan around.

• Edge decode gets a 3x-class step without a bigger battery, because moving bytes costs energy and PIM moves fewer of them.
• The KV cache is still the other half. PIM helps weight reads; the per-token attention over a long context still streams the cache. The KV cache math does not go away, and at long context it becomes the bottleneck again.
• Software has to cooperate. Somebody has to compile matmuls into PIM operations and fall back gracefully when an op is unsupported. Expect the first real gains to arrive through vendor runtimes, not through llama.cpp overnight.
• Accuracy is an open item. Samsung explicitly says tuning is ongoing, which usually means reduced-precision accumulation inside the memory die. Watch for quality-adjusted numbers, not just tokens per second.
• Apple's answer is bandwidth, not PIM. M-series unified memory hits hundreds of GB/s the conventional way; PIM is aimed at the class of devices below that, and at the Neural Engine style of fixed-function acceleration.

The honest gap

Every number here is Samsung's, from a conference presentation, labelled preliminary by Samsung itself. The 614 GB/s is theoretical; the measured gain is 3.01x on one model on one unnamed edge accelerator with accuracy still being tuned. There is no part number, no ship date, no independent measurement, and no statement about which operations run in-memory versus on the host. I would not put PIM in a roadmap yet. I would put memory bandwidth per watt at the top of the metrics I track for edge deployments, because that is the axis this announcement says the industry is about to compete on.