Performance Measurements and Tuning

① Model Specs (Parsed from the GGUF Header, Not Estimated)

general.architecture     = qwen35          (Qwen3.5 hybrid attention)
total parameters         = 27.32B          (dense model, not MoE)
block_count              = 65              (blk.64 is the nextn/MTP layer, ignored by llama.cpp)
attention.head_count     = 24
attention.head_count_kv  = 4               (KV has only 4 heads)
key_length/value_length  = 256             (256 dims per head)
full_attention_interval  = 4               (only 1 of every 4 layers is full attention)
ssm.*                                      (the remaining ~49 layers are linear-attention GatedDeltaNet)
file size                = 19,653,896,608 bytes (18.30 GiB, nvfp4 era)

Key inference: only 16 layers participate in the KV cache (layers where (i+1)%4==0); the other 49 are SSM linear attention — fixed-size state that does not grow with tokens. This is where a same-size traditional Transformer overflows VRAM, and it is why this model can run a 260K context. Prefill is also extremely fast: 64K-token prompt in 33.5 seconds (1,905 t/s), prompt cache hit rate 99.7%.

② Generation Speed Drops with Context Length (measured, not a fault)

On 2026-09-21 the eval time was read directly from llama_server.log. Every generated token attends over all cached KV — longer context, slower; it is arithmetic.

Three levers if it feels slow (by cost-effectiveness): ① LLM_THINK=0 to turn off the thinking chain, ~2× faster (or use a better --reasoning-budget, see ⑤); ② shorten the context (cost: compression happens more often); ③ reduce tool noise so the same length carries more useful content. Self-check: findstr /C:"eval time" llama_server.log.

③ The 27 t/s Mystery = the UE Editor Competing for GPU

Speed flips between two tiers of 26~28 and 71 t/s (12h log: 1,780 fast / 74 slow), and it jumps even mid-task.

Countermeasures (while the model is generating): minimize UE / Ctrl+R to disable viewport real-time rendering / console t.MaxFPS 30 / check "Use Less CPU in Background" in editor preferences.

④ ZCode's Token Estimation Cannot Be Trusted (three measurements, unstable bias)

The third one directly 400'd the server: request (82324 tokens) exceeds the available context size (81920 tokens). Therefore: llama-server's --context-shift is off by default, overshoot gives a hard error, not a silent truncate; ZCode's contextWindow is not protection, only a "when to start compressing" trigger point — compression cannot stop it from overshooting on its own. The server ctx must be a large margin above the client cap as an absolute buffer (pairing formula on the integration page).

⑤ Curing Overthinking: --reasoning-budget 1000

Background: one UE task ran 2h36m; item-by-item decomposition found 85% was model generation, and the model often burns the entire output budget on thinking, with a single word in the body.

Probe measurement (the same UE reasoning problem, max_tokens=4000):

Root cause: the template injects a "please verify key assumptions, consider alternatives" system instruction on the xhigh tier — that sentence is literally teaching the model to overthink, and the high that ZCode sends every turn is forced to xhigh, so every turn gets this instruction.

Accuracy comparison (speed alone is not enough): three tests — sequence reasoning / instruction following / tool_calls — all matched the as-is answers at RB=1000 — what was cut was indeed useless content (the model itself spotted the numeric contradiction in the prompt within the body, so reasoning quality held).

⑥ Four Optimizations in Full: KV4 Adopted, MTP Rolled Back

Executed on 2026-10-04 by "build the baseline first, optimize item by item, self-check every step". Stages: T0 = before optimization; T_KV4 / T_MTP = intermediate states.

Why MTP was rolled back — user's measured feedback: "feels slower: VRAM indeed dropped, but GPU utilization is only in the twenties". Same-time comparison at ~100K context:

Root cause (upstream known defect llama.cpp #28049): speculative decoding's boundary states are bound to the exact last position of the cached prefix. Real agent traffic is "partial prefix hit" (this machine's log f_sim=0.997 / f_keep=0.993) — 0.3% off and cache reuse fails, every step recomputed. A synthetic fresh prefill or exact hit cannot expose this problem, so the user's "feels slower" is right and the machine data confirms it. During troubleshooting, four confounders were also ruled out one by one (all measured): the context cliff, VRAM overflow, tool schema, and UE using the GPU.

⑥ Verification of 1.25× Context Extrapolation (2026-10-04 evening)

yarn extrapolation is not just "turn up -c" — it must be verified that it holds up after crossing the training line. Verdict: all three needles green; the speed cost is only paid on super-long contexts.

Three-needle embedding method (source 36-10): three different passwords embedded in the same prompt, at 10% / 50% / 90% depth (testing full-range retrieval, not just the tail), plus a math reasoning problem:

Speed: decode ≈ 27.8 t/s at 300K depth, prefill ≈ 1,000 t/s — ~100-130K daily session speed unchanged (the length-correlated decode decay is gradual; the cost is only paid on long contexts).

A/B decision: Option A (conservative -c 262144 + window ≈206,000, not adopted) vs Option B (yarn extrapolation -c 327680 + window 256,000, adopted). Fallback path = Option A (drop the 3 new params, -c back to 262144, window 206000); A/B are two independent knobs — falling back to A does not require giving up the KV4 and b10754 gains.

Limitations (recorded honestly): each depth ran once, 3 needles, reasoning verified on only 1 problem — "can retrieve, can reason" holds, but no controlled comparison with 100K-depth inference of equivalent quality. For critical decisions in super-long sessions, important conclusions are still recommended to be reread and checked.

⑦ Methodology Lessons (Biggest Gain This Round)