Context

KV cache as a content-addressed particle subgraph.

This document specifies how glia's KV cache becomes a first-class citizen of the cybergraph — every KV block addressable by content hash, deduplicated across conversations and across Avatars, persisted in bbg, paged into GPU memory on demand. The result is effectively unbounded context with cryptographic provenance for every cached token range.

Problem

Conventional LLM runtimes (ollama, vllm, llama.cpp) treat the KV cache as:

• Per-conversation, ephemeral. New conversation = new KV cache. Conversation ends = cache freed.
• Linear in context length. A 128k context with a 7B model holds tens of GB of K/V tensors. Memory-bound on consumer hardware.
• Opaque. The cache is a flat tensor with no addressable structure. Two conversations sharing the same system prompt compute and store the same KV twice.
• Non-portable. KV cache cannot move between processes, machines, or Avatars. The expensive prefill is repeated every time the conversation is re-opened on different hardware.

A planet of soma Avatars cannot afford this. The same system prompt, the same RAG retrievals, the same canonical context fragments are paid for over and over. Conversations vanish when an Avatar migrates Bodies. Multi-Avatar coordination has no shared cognitive ground.

Insight

The KV cache is already paged into blocks by PagedAttention (vllm). Block size is typically 16 or 32 tokens. The block table maps logical positions to physical block IDs. This is conceptually OS virtual memory for attention.

The bridge: make the block ID a content hash. Then every block is a particle — addressable, deduplicatable, shareable, persistable. The page table is no longer process-local pointers; it is a chain of cyberlinks in the cybergraph.

conventional:  conversation → flat KV tensor (private, ephemeral)
PagedAttention: conversation → block table → physical blocks (private, ephemeral)
this design:   conversation → cyberlink chain → particles in bbg → blocks in CAS (shared, persistent)

The GPU does not care that block IDs are content hashes. It receives physical memory addresses at dispatch time. Content addressing happens at the management layer — block lookup, dedup, fetch. The GPU never sees a hash.

PagedAttention recap

Brief — the upstream design is documented in the vllm paper (Kwon et al. 2023).

KV cache is split into fixed-size blocks, typically block_size = 16 tokens. Each layer holds two block pools: K blocks and V blocks. Each block stores [block_size, num_kv_heads, head_dim] of one dtype.

A logical conversation is a list of block IDs per layer. Decode reads from this list to compute attention. Append writes a new K and V block when the previous one fills.

The block table is a tensor of shape [num_layers, seq_len / block_size] holding physical block IDs. Position-to-block math is integer division; intra-block offset is the modulo.

This already gives:

• Internal fragmentation only at the tail of a sequence.
• Copy-on-write block sharing for prefix caching (vllm v0.4+).

What's missing: the block IDs are local-process integers, the blocks are private RAM allocations, the sharing is restricted to one runtime instance.

Architecture

Five layers, each independent of the layer above:

1. block layer       fixed-size [block_size, num_kv_heads, head_dim × 2] tensor (K and V)
2. hash layer        content hash over (model_id, layer_idx, token_window, K_bytes, V_bytes)
3. particle layer    block hash registered as particle in bbg
4. cyberlink layer   page table = chain of next_block cyberlinks
5. residency layer   GPU HBM / unified memory / local CAS / remote bbg fetch

A KV block is generated by attention at the end of prefill or every block_size decode steps. The runtime hashes the block, looks up the hash in the local block index, and either reuses an existing physical block (cache hit) or registers a new one (cache miss).

Block identity

A block must hash to the same value if and only if reproducing it would yield identical K and V tensors. The hash is:

block_hash = H(model_id, layer_idx, position_anchor, token_window, prev_block_hash, K_bytes, V_bytes)

Each component matters:

K_bytes and V_bytes are required to avoid hash collisions from numerical nondeterminism (different backends producing slightly different K/V from the same tokens). The hash is the artifact, not a derivation of the inputs alone.

Position anchoring is the subtle part. Naive PagedAttention stores K post-RoPE (rotated by absolute position). Two conversations with the same prefix produce identical K blocks only if they share the same starting position. For prefix caching to work, blocks must be position-relative — RoPE is applied at attention dispatch, not at block write. vllm's prefix caching uses this trick. We adopt the same.

With position-relative storage:

block_hash = H(model_id, layer_idx, token_window, prev_block_hash, K_unrotated_bytes, V_bytes)

Two Avatars running the same model on the same prompt prefix produce identical block hashes for the prefix region. Dedup works.

Page table as cyberlinks

A conversation's KV cache is a chain of blocks. Per layer, the chain is:

block_0 ──next_block──> block_1 ──next_block──> block_2 ──> ...

Each edge is a typed cyberlink with type = next_block, carrying:

cyberlink<next_block>:
  from:           block particle (prev block hash)
  to:             block particle (next block hash)
  layer:          attention layer index
  position_start: absolute token position of the next block's first token
  model:          model_id

The conversation's KV cache state is the set of leaf blocks across all layers. To resume a conversation, walk the chain forward from a known root; to fork at a point, share the prefix chain and branch a new leaf.

Multiple conversations sharing a prefix share the head of the chain. Cross-Avatar sharing is automatic: if Avatar B asks for next_block(block_X, layer_7) and the chain exists in bbg from any prior computation, Avatar B fetches the block instead of recomputing it.

Residency hierarchy

Blocks live in a four-tier memory hierarchy. The runtime manages residency; the user never sees it.

Promotion is on-demand. Demotion is by LRU within each tier, with the constraint that any block referenced by a currently-running attention dispatch is pinned in L0.

For a 7B Llama-class model with block_size = 16, num_kv_heads = 8, head_dim = 128, F16:

bytes_per_block = block_size × num_kv_heads × head_dim × 2 (K and V) × 2 (F16)
               = 16 × 8 × 128 × 2 × 2
               = 65,536 bytes  (64 KiB per block, per layer)

128k tokens, 32 layers:
  blocks_per_layer = 128000 / 16 = 8,000
  total_blocks      = 8,000 × 32 = 256,000
  total_bytes       = 256,000 × 64 KiB = 16 GiB

That fits in a 64 GB M-series unified memory but not in HBM of a consumer GPU. The hierarchy is essential.

Block deduplication

The dedup unit is the block hash. When dedup pays off, in descending order of impact:

1. System prompts. Every Neuron using the same system prompt template hits the same prefix blocks. The first computation is the only computation across the entire network.
2. RAG context. Common retrieved documents (canonical Wikipedia paragraphs, common code snippets, frequent policy docs) all produce reusable blocks.
3. Boilerplate. Tool descriptions, JSON schemas, format examples that prefix nearly every agent prompt.
4. Conversation forks. Branching from a conversation point shares the entire prefix.
5. Cross-Avatar resumption. Avatar B opens a conversation that Avatar A held; the prefill is replaced by a chain walk.

In practice, agent ensembles using shared infrastructure spend the majority of their tokens on shared prefixes. Dedup makes the marginal cost of an additional agent close to zero for the shared part.

GPU does not care

The hash lives at the management layer. The dispatch path is unchanged:

attention dispatch:
  for each layer:
    block_table = page_table_to_physical_addrs(conversation, layer)
    K_blocks    = gather(L0_pool, block_table)
    V_blocks    = gather(L0_pool, block_table)
    out         = paged_attention_kernel(q, K_blocks, V_blocks, block_table)

block_table is a tensor of physical pool offsets; the kernel reads it just like vllm does today. The content-addressed metadata never enters the kernel.

What changes is the path that populates the L0 pool:

on conversation step:
  for each block_id needed:
    if block_id in L0_pool:        no-op
    elif block_id in L1_pool:      DMA L1 → L0, evict LRU L0
    elif block_id in L2_pool:      load from disk → L1 → L0
    else:                          fetch from bbg → L2 → L1 → L0

A prefetcher walks the cyberlink chain ahead of the decode head and warms L1 with the next N blocks. For deterministic decode, prefetch is just sequential chain walk. For speculative decoding or beam search, prefetch follows multiple chain heads.

Network fetch

Block fetch from a remote Avatar is a bbg read:

fetch(block_hash):
  particle = bbg.get_particle(block_hash)
  if particle present:
    cas_data = bbg.get_content(particle)        # 64 KiB transfer
    verify_hash(cas_data, block_hash)            # cheap
    return cas_data
  else:
    raise NotFound

Bbg's bounded-locality guarantee makes the query constant-cost. The 64 KiB block transfer is the dominant time at typical link speeds. On a 100 Mbps link, ~5 ms per block — comparable to a deep tier-2 model load step. On LAN, sub-millisecond.

Avatars near each other on the network cache the common shared prefixes for the same agent flock. Cold start for a new Avatar joining the flock is a chain walk that pulls only blocks not already present.

Privacy

KV blocks can leak the tokens that produced them — the K/V values are a function of those tokens under the model weights. Publishing every block to bbg without consideration would leak prompts.

The bbg privacy model applies:

• Private blocks: stored locally only. The block hash is never published. Used for private conversations, secret keys, PII.
• Public blocks: hash and content published to bbg. Available to any Avatar that knows or can compute the hash. Used for shared system prompts, public docs, canonical RAG sources.
• Aggregate blocks: hash published with content under access control (zero-knowledge proof of access right). Used for org-shared but non-public material.

The decision is per-conversation, configured by the Soul's privacy policy. The default is private; publishing is explicit. Shared prefixes that everyone uses anyway (open-source model defaults, popular system prompts) are pre-published as part of the model's .model artifact distribution.

When a conversation contains a mix of private and public spans, only the public spans get published as blocks. The private spans become local-only blocks chained off public anchors.

Eviction

Each tier has its own eviction policy:

• L0 (HBM): pinned-while-dispatching, LRU among unpinned. Eviction target: physical block in L1.
• L1 (unified memory): LRU with a hot bias for blocks referenced in the last N decode steps. Eviction target: L2 if dirty (not yet persisted), drop if clean and reproducible.
• L2 (local CAS): size-capped, LRU. Eviction target: drop. Re-fetchable from bbg if the block was published.
• L3 (bbg): permanent for published blocks; private blocks never reach L3.

A block is "reproducible" if its inputs (model, prior block, tokens) are all known and the runtime can regenerate it. Pure decode blocks from public prompts are always reproducible. Prefill blocks from private prompts are not.

What this enables

1. Effectively unbounded context. Conversations longer than the model's positional limit become walks over chains where only the relevant blocks are loaded into L0. The attention window is what the kernel computes over; the conversation can extend much further with retrieval at the block level.

2. Cross-Avatar continuity. The same conversation can move between Bodies as the Avatar migrates. The Soul carries the chain root; the new Body warms the cache from bbg.

3. Multi-Avatar shared cognition. Multiple Avatars working on a shared task share the prefix blocks of their working context. Marginal cost per additional Avatar drops to the unique-suffix size.

4. Free prefix caching. vllm-style prefix caching becomes a side effect of content-addressed storage instead of a separate feature.

5. Persistent agent memory. Episodic memory and working memory blur — the KV blocks themselves are the substrate of episodic memory, queryable by content hash and walkable as cyberlinks.

6. Provable inference traces. Every block has a hash; chains are reproducible; the runtime can prove "this output was produced from this KV history." Fits glia's existing model-as-particle posture.

Block-level RAG

The conventional RAG stack retrieves token text by similarity, prepends to context, paying the prefill cost every time.

With content-addressed KV blocks, RAG can operate one level lower. The cybergraph holds particle embeddings (from glia's tier-0 embedding model). When a query needs context, the runtime:

1. Searches embeddings for relevant particles.
2. For each particle, walks to its KV block subgraph (if computed).
3. Splices those blocks into the current conversation's chain.
4. Skips prefill — the blocks already exist.

This trades retrieval-quality (block-level granularity is coarser than token-level) for inference cost (no prefill). For most agent tasks the tradeoff is favorable; for tasks needing precise quoting, fall back to token-level retrieval.

Block size

The choice of block_size is a tradeoff:

Defaults: block_size = 16 for chat-style workloads (frequent forks, short shared prefixes); block_size = 64 for batch document processing (long shared prefixes, infrequent forks).

Block size is per-model in the .model manifest. Mixing block sizes across runtimes for the same model means no dedup — coordination required.

Open questions

1. Quantized KV cache. K and V often stored in F16 today, but Q8/Q4 KV cache cuts memory 2-4×. Quantization is lossy; the same tokens through different quantizers produce different blocks. How does the block hash incorporate the quantizer ID? Probably: extend model_id to (model_id, kv_quant_scheme).

2. Block content distribution. 64 KiB per block × millions of blocks puts pressure on bbg's content layer. IPFS is the obvious backing CAS; the open question is the indexing path from bbg particle to IPFS CID.

3. Sliding window models. Models with sliding window attention (Mistral, recent Gemma) evict K/V older than the window. Blocks past the window cannot be re-loaded into attention dispatch but remain valuable as episodic memory. How to model the lifecycle distinction in the chain?

4. Speculative decoding. The draft and target models produce different block sequences; the draft's KV is throwaway. Should draft blocks be hashed and stored, or kept ephemeral? Probably ephemeral — the verification step throws most of them away.

5. Continuous batching. vllm's continuous batching shares one runtime across many conversations. With content-addressed blocks, the gather kernel is unchanged but the residency manager needs awareness of multi-conversation block reuse patterns.

6. Cross-model transfer. Two distillations of the same teacher might produce KV blocks of similar geometry. Could a transformation map blocks between models, enabling transfer? Long-term research direction.

7. KV from .model itself. Some published blocks (a default system prompt's KV) could ship inside the .model artifact. The runtime preloads them on model load. Frees the network from re-fetching the most common blocks every time.

8. Block-level proofs. Every block produced through nox carries a STARK proof of its computation from the prior block and the new tokens. A conversation's chain becomes a provable history.

Relationship to soma

soma's soma-spec Memory section names four memory types: working, episodic, semantic, procedural. Content-addressed KV blocks unify the first two:

• Working memory IS the L0/L1 cache of currently-active blocks.
• Episodic memory IS the L2/L3 chain of past blocks, walkable by conversation root.
• Semantic memory remains the cybergraph of particles and typed cyberlinks (independent of KV).
• Procedural memory remains the Skill particles in agentskills.io format.

When a Soul checkpoints its state for migration, it commits its current conversation chain roots. The new Body warms its cache from those roots and the Avatar continues with continuity of working and episodic memory.

What changes in glia

To realize this design, glia's runtime needs:

1. Block-level KV storage replacing the contiguous KV tensor. PagedAttention kernel for honeycrisp, wgpu, and cpu backends.
2. Block hasher: deterministic across backends (CPU reference is arbiter, as per execution.md).
3. Block index: maps block_hash → (residency_tier, physical_location).
4. Block fetcher: bbg client for L3 fetch; local CAS interface for L2.
5. Prefetcher: walks the chain ahead of decode, warms L1.
6. Cyberlink emitter: publishes next_block cyberlinks when a new block is appended and the conversation is configured public.
7. .model manifest extension: declare block_size, kv_quant_scheme, position_anchor_mode.

Backwards compatibility: a runtime with no bbg connection runs purely on L0/L1/L2 with private blocks. Dedup still works locally. The cybergraph integration is additive.

See also

• soma-spec — Memory layer, where this fits in the cognitive architecture
• bbg — the authenticated state layer this builds on
• tru — model compilation; .model artifact carrying block_size
• mir — consumer of inference outputs
• vLLM PagedAttention paper: Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al. 2023