The brief · about 1 minute
AI memory beyond HBM
AI agents need to keep information available between tasks. The thesis is a broader memory market: HBM for the fastest work, with CPU memory and storage holding less time-sensitive state.
What to watch. Watch whether SOCAMM2 adoption, CPU-side memory and context storage turn into production revenue.
What could break it. The case weakens if agents need little persistent state or new memory tiers remain experiments.
Summary of the original dated research below. This is not a new market update; figures and views retain their original as-of dates.
Read the full analysis · about 10 minutes
Why SOCAMM2, CXL, PIM and context storage could expand — rather than replace — the AI memory thesis
AI memory demand has been closely associated with accelerator capacity and bandwidth. Longer agent workflows broaden the question: systems must also preserve context, tool results and intermediate state between tasks.
The investment question is where that state should live, how frequently it is accessed, and what it costs to retain. HBM remains important, while host memory and storage may capture additional demand. Adoption and supplier revenue—not architectural possibility alone—will determine the payoff.
1. Longer Agent Workflows Increase the Importance of Persistent State
A conventional chatbot often follows a relatively simple sequence:
A simple chatbot may follow a prompt–response loop, but chatbots can also retain conversation history and external memory. The distinction is the duration and complexity of the workload: long-running agents may retain more state, use more tools and revisit more context.
An agent behaves differently.
It may reason, retrieve data, call an API, execute code, observe the result, revise its plan, call another tool and repeat the process several times before completing a task.
During that process, the system must preserve far more than model weights.
It can need:
- KV cache
- conversation state
- retrieved documents
- tool outputs
- code sandboxes
- permission context
- checkpoints
- intermediate reasoning state
- files and external data
That means AI infrastructure increasingly has to optimize not just tokens generated, but what I would call state-hours: how much state must remain available, across how much memory, for how long.
Keeping all of that state inside HBM would be extremely expensive.
The natural response is memory tiering.
2. NVIDIA Vera Is Already Pointing Toward the New Architecture
The architecture around NVIDIA’s Vera CPU offers an important clue.
According to NVIDIA specifications cited in the underlying research, Vera combines 88 Olympus cores with up to 1.2 TB/s of SOCAMM2 LPDDR5X bandwidth and up to 1.8 TB/s of coherent NVLink-C2C bandwidth between CPU and GPU.
NVIDIA is also building a separate context-memory layer through BlueField-4 and CMX.
The investment implication is more important than the specifications themselves.
The CPU is not suddenly replacing the GPU.
Instead, the CPU side of the AI factory is becoming more important because agentic workloads contain more irregular execution: tool calls, Python runtimes, retrieval, orchestration, storage access and sandbox execution.
If the host side cannot feed the accelerator efficiently, expensive GPUs sit idle.
In other words:
GPU utilization increasingly depends on memory outside the GPU.
3. The AI Memory Stack Is Becoming a Hierarchy
The mistake would be to treat HBM, SOCAMM2, CXL, PIM and flash as competing technologies where one eventually eliminates the others.
They solve different problems.
| Memory Tier | Primary Role | Economic Logic |
|---|---|---|
| HBM4 / HBM4E | Active tensors, attention, hot weights | High bandwidth close to the accelerator; access latency depends on the memory hierarchy and workload |
| LPDDR5X-PIM | Selected memory-bound compute | Reduce data movement and energy |
| SOCAMM2 | CPU-side warm KV/state/sandbox | Capacity, bandwidth and power efficiency |
| CXL DRAM | Expandable or pooled warm memory | Improve utilization and capacity elasticity |
| CMX / NVMe SSD | Cold context, checkpoints, long-lived KV | Low-cost capacity and prestaging |
The key insight is simple:
Not every byte deserves HBM economics.
The most performance-sensitive data should stay close to the accelerator.
Less active state can move down the hierarchy.
That makes HBM more specialized rather than obsolete.
4. SOCAMM2 May Be More Important Than the Market Thinks
One of the most interesting changes is the migration of LPDDR economics from mobile devices into the server.
SOCAMM2 attempts to combine the power efficiency of LPDDR with the serviceability and modularity required in data centers.
SK hynix has already announced mass production of a 192GB SOCAMM2 product for the Vera Rubin platform.
Micron has disclosed a 256GB SOCAMM2 product and said that an eight-channel CPU architecture could scale to as much as 2TB of LPDRAM.
Micron also reported a 2.3x improvement in time-to-first-token in a disclosed long-context KV-cache offload benchmark.
That number should not be generalized to every workload; it is a vendor benchmark under specific conditions.
But the mechanism matters.
If lower-priority KV cache can move out of HBM and into a high-bandwidth warm-memory tier, the system can preserve expensive HBM capacity for the data that genuinely requires it.
This changes the question from:
“How many gigabytes of HBM does an agent need?”
to:
“What percentage of an agent’s state must remain hot?”
That is a much more powerful way to think about the memory bill of materials.
5. PIM and CXL Are Not Opposites
At first glance, PIM and CXL appear to represent opposite philosophies.
PIM moves compute closer to memory.
CXL allows memory to move farther from the processor and into expandable or pooled capacity.
Both can still win.
The reason is that AI data has different “temperatures.”
Hot data benefits from being extremely close to compute.
Some repetitive, memory-bound operations may benefit from processing near memory.
Warm state may benefit from SOCAMM2 or host DRAM.
Less latency-sensitive capacity may move into CXL memory pools.
Cold context may reside in flash-based systems such as CMX or enterprise SSDs.
The real moat therefore may not be any single memory device.
It may be the placement policy that decides:
what state moves where, when, and at what cost.
As heterogeneous accelerators proliferate, that orchestration problem could become even more valuable.
6. CXL Is Not Simply a Technology for Buying Less DRAM
A common interpretation of CXL is that better memory utilization should reduce memory demand.
That may be true at the level of a single server.
It does not necessarily follow at the level of the entire AI system.
CXL can recover stranded capacity and allow memory expansion beyond local processor slots.
At the same time, making memory more flexible can increase the amount of economically usable memory available to each AI task.
The likely deployment path is also important.
The easiest stage is host-local Type-3 expansion.
Single-tenant rack pooling is harder.
Multi-tenant pooling adds security, isolation, sanitization, SLA and failure-domain complexity.
So the early CXL investment opportunity may appear first in validation, controllers, protocol testing and infrastructure, before massive pooled-memory deployments become mainstream.
7. Flash Could Become “Context Memory”
The role of NAND may also change.
Historically, investors often thought about AI storage mainly in terms of training datasets.
Agentic AI creates a more direct use case.
Long-running agents accumulate history.
That history does not need to remain permanently inside HBM.
NVIDIA’s CMX concept places network-attached flash between the accelerator memory and conventional shared storage, allowing context and KV data to be prestaged closer to the compute layer.
NVIDIA has claimed material performance and power-efficiency improvements versus traditional storage architectures in its disclosed CMX comparisons.
Again, vendor benchmarks should be treated carefully.
But the architectural message is more important than the exact number:
Flash is moving from passive storage toward an active context-supply layer.
.
8. Why More Efficient Memory Could Still Be Bullish for HBM
This is the most counterintuitive part of the thesis.
If SOCAMM2, CXL and CMX offload data from HBM, it is tempting to conclude that HBM demand must fall.
That conclusion assumes the amount of AI work is fixed.
It probably is not.
If memory tiering lowers the cost of running each agent, the same power envelope can support more agents.
More agents generate more aggregate work.
More aggregate work can require more total accelerator capacity.
And more accelerator capacity can still require more total HBM — even if HBM per agent falls.
This could produce a rebound effect: cheaper memory use per task may encourage more tasks. Aggregate demand grows only if adoption and workload growth outweigh the efficiency gains; that outcome is a scenario, not a law.
Efficiency does not always reduce resource consumption.
Sometimes it makes the resource economical enough to consume much more of it.
The relevant equation is therefore not:
HBM per model ↑ or ↓
It is closer to:
HBM intensity per agent × total number of agents
If the second term grows faster than the first term falls, aggregate HBM demand can continue to rise.
9. The Investment Map Is Broader Than HBM
This framework changes how I would organize the memory trade.
SK hynix — the most direct current AI memory exposure
SK hynix remains the cleanest direct exposure to HBM, but SOCAMM2 extends that position into the warm-memory tier.
Its 192GB SOCAMM2 production for Vera Rubin matters because this is not merely a distant technology option.
The relevant variables now include attach rates, module volumes and economics alongside HBM pricing and share.
Samsung Electronics — the portfolio option
Samsung’s potential advantage is breadth.
HBM, SOCAMM2, LPDDR5X-PIM, enterprise SSD, NAND and foundry capabilities all sit within the same corporate ecosystem.
If AI memory becomes increasingly workload-specific, the ability to optimize across multiple tiers could become more valuable.
The risk is equally clear: portfolio breadth only matters if it converts into customer attachment and revenue.
Micron — density as an economic weapon
Micron’s 256GB SOCAMM2 highlights a different competitive axis: capacity density.
If the number of concurrent agents grows, CPU-side memory becomes a question not only of bandwidth but also of how much persistent state can remain resident per socket.
That makes memory density an increasingly important system-level variable.
Second-order beneficiaries
The thesis can also extend into memory modules, PCB, validation and testing.
In Korea, the underlying research identifies names such as Simmtech and TLB on the module side, while Exicon and Neosem could become relevant as CXL validation moves from proof-of-concept toward deployment.
These are less direct exposures than the memory manufacturers themselves, so qualification, customer concentration and actual revenue conversion matter more than thematic proximity.
10. Five Things the Market May Be Mispricing
The market may still be making five conceptual mistakes.
First: treating SOCAMM2 as merely an RDIMM replacement rather than a new agent warm-memory tier.
Second: treating PIM and CXL as competitors when they may serve different temperature classes of data.
Third: assuming better CXL utilization automatically means lower DRAM demand.
Fourth: assuming more custom ASICs mainly increase hardware-module complexity when the bigger complexity may emerge in memory scheduling and orchestration.
Fifth: assuming better memory efficiency must reduce memory demand.
The broader possibility is the opposite:
agentic AI may widen memory demand from one premium product into an entire hierarchy.
11. What Would Prove This Thesis Wrong?
A strong thesis needs clear failure conditions.
I would become materially less constructive on this framework if several of the following occur:
- Agentic workloads become highly stateless and short-lived, sharply reducing state-hours.
- HBM capacity expands and pricing falls so quickly that tiering becomes economically unnecessary.
- CXL remains stuck in proof-of-concept deployments without meaningful production revenue.
- SOCAMM2 remains largely confined to NVIDIA rather than spreading across broader server ecosystems.
- PIM fails to develop usable software frameworks and remains a demonstration technology.
The strongest near-term signal is simple:
Does Vera/Rubin actually drive meaningful SOCAMM2 attachment and CPU-side memory growth?
That is where the thesis begins moving from architecture to revenue.
Conclusion: HBM Is Not the Whole Memory Trade Anymore
The next AI memory cycle may be misunderstood if investors continue to measure everything through HBM alone.
HBM remains critical.
But agentic AI creates a new economic problem: persistent state must be stored across different latency, bandwidth, capacity and power requirements.
That naturally creates a hierarchy.
The investment framework is a hierarchy of possible roles: HBM for bandwidth-intensive accelerator workloads, CPU memory for host-side capacity, CXL for selected expansion use cases, and storage for less latency-sensitive context. PIM adoption remains workload- and implementation-dependent. These technologies do not all substitute for one another. NVIDIA’s Vera specifications provide a primary-source example of host memory and CPU–GPU connectivity; they do not validate every supplier forecast in this article.
The winner may therefore not be the company that simply produces the fastest memory.
The deeper moat may belong to the companies that help AI systems place the right state in the right memory at the right time.
That is a much broader investment opportunity than the HBM trade alone.
Sources & Method
This article is based on company disclosures and technical material referenced in the underlying research, including SK hynix’s SOCAMM2 announcement, Micron’s SOCAMM2 disclosures, NVIDIA’s Vera CPU and BlueField-4/CMX material, and Samsung Semiconductor’s 2026 AI memory roadmap. Vendor performance claims are presented as vendor claims rather than universally applicable benchmarks.
Disclaimer
This article is for informational and educational purposes only and does not constitute investment advice, a recommendation, or an offer to buy or sell any security. The author may hold positions in securities discussed. Technology roadmaps, vendor benchmarks and market expectations can change materially, and readers should conduct their own research before making investment decisions.
— END OF POST —