Quantization formats

Canonical reference: the authoritative encoding set is §weights — five fixed encodings (u32, u16, q8, q4, ternary). Layouts below are the runtime decoder contract; encoders live in import::quant::canonical.

Canonical encodings (.model on-disk)

The only encodings allowed in a canonical .model tensor index:

Bytes-on-disk identities (verified by round-trip tests in import/tests/canonical_quant_roundtrip.rs):

• u32: i32 little-endian; dequant f = i32 / 65536
• u16: i16 little-endian; dequant f = i16 / 256
• q8 block: [i16 scale][i8 × 32]; dequant f[i] = i8[i] * scale_f / 127 where scale_f = i16 / 256
• q4 block: [i16 scale][packed nibbles × 16], split layout — byte i holds value[i] in the low nibble and value[i + 16] in the high nibble; dequant (nibble - 8) * scale_f / 8
• ternary byte: 4 values packed 2 bits each; codes 00 / 01 / 10 = 0.0 / +1.0 / −1.0; 11 reserved (decoded as 0)

Source-side dtypes (import-only)

The import crate also reads source files (HF safetensors, GGUF) at import time. Source dtypes never appear in a canonical .model; they are dequantized to f32 on the way through, then re-encoded into the canonical set above.

The source decoders live in run/backend/cpu/quant/{q4_0,q4_k,q5_k,q6_k,q8_0}.rs and import/types.rs::dequantize_to_f32. Both ports of llama.cpp's reference k-quant kernels.

Endianness

All multi-byte integer and float fields are little-endian. Same as GGUF, x86, ARM, WGSL.

F32

IEEE 754 single-precision.

F16

IEEE 754 half-precision: 1 sign bit, 5 exponent, 10 mantissa. Range ±65504. Gradual underflow via denormals.

BF16

"Brain floating point 16": 1 sign bit, 8 exponent (same as F32), 7 mantissa. Same dynamic range as F32, lower precision.

Import rule: BF16 weights convert to F16 at import time (import.md). Runtime does not carry BF16.

Q8_0 — 8-bit blocks of 32

Symmetric quantization, per-block scale.

Block layout (34 bytes):

struct block_q8_0 {
    f16 d;            // 2 bytes — scale
    i8 qs[32];        // 32 bytes — quantized values [-128, 127]
}

Dequant:

x[i] = d * qs[i]

Quant (F32 → Q8_0):

amax = max(abs(x[i]))
d = amax / 127.0
qs[i] = round(x[i] / d)   # clip to [-128, 127]

Q4_0 — 4-bit blocks of 32

Symmetric, two nibbles per byte.

Block layout (18 bytes):

struct block_q4_0 {
    f16 d;            // 2 bytes — scale
    u8 qs[16];        // 16 bytes — 32 nibbles
}

Nibble encoding: bottom nibble stores x[j] for j in 0..16, top nibble for j in 16..32.

qs[j] = nibble + 8         # nibble in [-8, 7]
x[j] = d * (nibble_of(qs[j >= 16], j % 16) - 8)

Dequant:

for j in 0..16:
  x[j]      = d * ((qs[j] & 0x0F) - 8)
  x[j + 16] = d * ((qs[j] >> 4)   - 8)

Note: llama.cpp's Q4_0 interleaves lo/hi differently in some older versions. Canonical layout above (lo → first half, hi → second half) matches current GGUF Q4_0.

K-quants (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K)

Superblock of 256 values, divided into 8 or 16 sub-blocks with 6-bit scales and optional 6-bit mins. Higher precision than legacy Q4_0 at similar size.

Q4_K — 144 bytes per 256 values

struct block_q4_K {
    f16 d;              // 2 bytes — superblock scale
    f16 dmin;           // 2 bytes — superblock min offset
    u8 scales[12];      // 12 bytes — packed 6-bit: 8 scales + 8 mins
    u8 qs[128];         // 128 bytes — 256 nibbles
}

Sub-block structure: 8 sub-blocks of 32 values each. Each sub-block j has a 6-bit scale s[j] and a 6-bit min m[j].

Scale/min extraction from scales[12] for sub-block j in 0..8:

if j < 4:
    s[j] = scales[j] & 0x3F                        # low 6 bits
    m[j] = scales[j + 4] & 0x3F
else:
    s[j] = (scales[j + 4] & 0x0F) | ((scales[j - 4] >> 6) << 4)
    m[j] = (scales[j + 4] >> 4)   | ((scales[j]     >> 6) << 4)

Nibbles: qs[128] holds 256 nibbles. For sub-block j (0..8), values are at positions [j*32, j*32+32). Two nibbles per byte:

qs_byte_off = (j / 2) * 32          # byte range for a pair of sub-blocks
is_upper = j % 2                    # low nibble (0) or high nibble (1)

for l in 0..32:
    byte = qs[qs_byte_off + l]
    nibble = (byte >> (is_upper * 4)) & 0x0F
    x[j*32 + l] = d * s[j] * nibble - dmin * m[j]

Critical: dequant subtracts dmin * m[j], not adds. This is the asymmetric min offset. Common bug: forgetting the min term entirely (matches Q4_0 behavior but produces wrong Q4_K values).

Q5_K — 176 bytes per 256 values

Same superblock structure as Q4_K, plus 1 extra high bit per nibble packed in 32 extra bytes:

struct block_q5_K {
    f16 d;
    f16 dmin;
    u8 scales[12];
    u8 qh[32];          // high bits (1 bit per value)
    u8 qs[128];         // low 4 bits
}

5-bit quant value: q = (qh_bit << 4) | qs_nibble, where qh_bit is the bit for this value pulled from qh[32] (32 bytes = 256 bits).

Q6_K — 210 bytes per 256 values

8 sub-blocks of 32, 6-bit values:

struct block_q6_K {
    u8 ql[128];         // low 4 bits of each value
    u8 qh[64];          // high 2 bits of each value
    i8 scales[16];      // 16 sub-block scales (i8)
    f16 d;              // superblock scale
}

Note: sub-blocks of 16 values here, not 32. Scale array is length 16.

Dequant:

for j in 0..16:
    s = scales[j]
    for l in 0..16:
        idx = j * 16 + l
        q = ((ql[bit position] & 0x0F) | ((qh[bit position] & 0x03) << 4)) - 32
        x[idx] = d * s * q

Full bit-packing details follow GGUF's dequantize_row_q6_K.

Q3_K — 110 bytes per 256 values

3-bit with 6-bit scales:

struct block_q3_K {
    u8 hmask[32];       // high bits (1 per value)
    u8 qs[64];          // low 2 bits
    u8 scales[12];      // packed 6-bit scales
    f16 d;
}

Q2_K — 84 bytes per 256 values

2-bit with 4-bit scales + 4-bit mins:

struct block_q2_K {
    u8 scales[16];      // 4-bit scale + 4-bit min per sub-block of 16
    u8 qs[64];          // 256 values × 2 bits
    f16 d;
    f16 dmin;
}

Ternary (BitNet 1.58-bit)

Four ternary values per byte, 2 bits each. Used by BitNet 1.58.

Encoding:

00 → -1
01 →  0
10 → +1
11 →  0  (unused; defensive default)

Dequant:

for each byte:
    for i in 0..4:
        bits = (byte >> (i * 2)) & 0x03
        v = {-1, 0, +1, 0}[bits]
        x[4*byte_idx + i] = scale * v

Scale storage: ternary tensors carry a per-tensor f32 scale separately from the quantized bytes. Convention: the tensor's scale is stored as a sibling tensor with the same name + .scale suffix:

["model.layers.0.self_attn.q_proj.weight"]
shape = [4096, 1024]
encoding = "ternary"
offset = X
size = 4096 * 1024 / 4             # 1M bytes

["model.layers.0.self_attn.q_proj.weight.scale"]
shape = [1]
encoding = "u32"                    # f32 as raw bytes
offset = X + 4096*1024/4
size = 4

At load time, the runtime pairs each ternary tensor with its .scale sibling. Missing scale = import error.

Some models use per-row scales instead of per-tensor: shape = [N] with one f32 per output row. Detected from scale tensor shape.

Format dispatch

A matmul with quantized weights runs a format-specific kernel. The backend (wgpu, honeycrisp) has one kernel per supported format. Missing format → error at dispatch ("backend does not implement Q3_K matmul").

Conversion between formats (e.g. Q4_0 → Q4_K for storage normalization) happens only at import time (import.md), not at runtime.

Storage across multiple blocks

A weight tensor W: [N, K] with block-quantized dtype stores its data as a flat byte array of contiguous blocks. Layout is row-major by N (outer), then blocks along K (inner).

For Q4_K with K=1024:

blocks_per_row = K / 256 = 4
bytes_per_row = blocks_per_row × 144 = 576
total_bytes = N × bytes_per_row

Byte offset of block at row r, block index b (within row):

offset = (r * blocks_per_row + b) * block_bytes

No headers, no padding between blocks, no inter-tensor alignment within the quantized bytes of a single tensor. Tensors themselves align to 64 bytes at the tensor-data level (format.md).

K dimension alignment

All K-quants use 256-value superblocks. Q4_0 and Q8_0 use 32-value blocks.

Requirement: for a weight tensor of shape [N, K] stored in format F with block size B(F), K must be a multiple of B(F):

K % B(F) == 0

Import checks this. Models with non-aligned K cannot use that format — import must either:

• Pad K with zeros to the next multiple and store metadata (not preferred — wastes memory)
• Fall back to a finer-grained format (Q8_0 if K % 32 == 0 but K % 256 != 0)
• Reject the tensor and fall back to F16 for that tensor

Spec: reject and fall back. Tolerant mixing across tensors is supported (scope.md).

Partial blocks are NEVER stored. Every block is exactly B(F) values.

Tolerance

Dequant tolerance is the error bound of the format itself:

These are per-weight, not per-output. Output error of a matmul depends on accumulation length (√K scaling). Expect 10-100× the per-weight bound on final activations.