Qwen3.8-27B · GGUF · KV q8_0

Qwen3.8-27B GGUF on llama.cpp with KV q8_0, 56.1 t/s warm, stable run with high spec acceptance.

56.1

WARM DECODE (T/S)

52.4

COLD DECODE (T/S)

16.65

WARM TTFT (S)

87.17

COLD TTFT (S)

23.4

PEAK VRAM (GB)

CONCURRENT REQS

llm-servingllamacppggufb70ASRock B70▼ −0.7 vs prev verdict: keep

Based on: https://huggingface.co/Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF

Date: 2026-10-17 · Endpoint: your-LAN-IP:9932 (llamacpp) Status: benchmarked (client stats); server-log pass pending

Config

Field	Value
Model	`Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF` → `/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-Intel-Arc-Tuned-MTP-Q4_K.gguf` (IQ3_S ≈3.5 bpw, MTP draft head Q4_K embedded)
Inference platform	llama.cpp `ghcr.io/ggml-org/llama.cpp:full-intel`, SYCL/Level-Zero backend, single B70 (`SYCL_DEVICE_FILTER=level_zero:0`)
GPU env	`GGML_SYCL_ENABLE_GRAPH=1`, `GGML_SYCL_F16=1`, `GGML_SYCL_ENABLE_COPY=1`, `ZES_ENABLE_SYSMAN=1`, `SYCL_USE_XMX=1`, `ZE_AFFINITY_MASK=0`; `SYCL_CACHE_PERSISTENT=0` (recompiles every start)
Port / host	your-LAN-IP:9932
Context	`--ctx-size 163840`
Layers	`--n-gpu-layers all`, `--fit-target 256` (256 MiB margin/device for auto-fit)
Flash attn	on
KV cache	`--cache-type-k q8_0 --cache-type-v q8_0` (only diff vs run #4), `--kv-unified --kv-offload`
Speculative decoding	`--spec-type ngram-mod,draft-mtp`: ngram-mod n-match 24, n-min 1, n-max 3; draft-mtp n-max 3, p-min 0.5
Prefill batching	`--batch-size 16384 --ubatch-size 2048`
CPU	`--threads 8 --threads-batch 12` (7900X = 12C); `OMP_NUM_THREADS=12`
Prompt caching	`--cache-prompt --cache-reuse 2048 --cache-ram 2048` (disk prompt cache, 2 GiB RAM budget, reuse chunks ≥2048 tokens via KV shifting)
Concurrency	`--parallel 1`, `--n-predict -1`
Sampling defaults	temp 0.7, top-p 0.9, top-k 40, repeat-penalty 1.05 (last-n 256), presence 0.0, dry-penalty last-n 512, client overrides to greedy-equivalent for comparability
Reasoning / tools	`--reasoning on`; tools: read_file, file_glob_search, grep_search, write_file, edit_file, get_info; `--jinja` + `/models/qwen-sharp.jinja` (from the model repo's `chat_templates/`) template; alias `Qwen3.8-27B`
Monitoring	`--metrics --slots --warmup --perf`

Parameter diff vs run #4 (single variable, isolation run C)

Flag	Run #4	This run
`--cache-type-k/v`	q4_0	q8_0

Everything else identical to run #4: ctx 163840, ubatch 2048, batch 16384, --kv-offload, --flash-attn on, spec ngram-mod+draft-mtp, APC via --cache-prompt, greedy-equivalent client override.

Stats

Metric	Cold (req #1)	Warm (APC hit)
Prompt tokens	64,501	64,501
TTFT (s)	client 87.17 / server prompt eval 63.24 s	client 16.65 / server prompt eval 0.36 s (4 tokens re-prefilled)
Prefill (tok/s)	1,020.01 (server; #4 q4_0 was 1,013.59 → KV dtype makes no difference)	cache-served (cached_tokens 64,497/64,501)
Decode (tok/s, generated)	client 52.4 (1,237 tok / 23.63 s); server 26.04 (38.4 ms/tok)	client 56.1 (1,156 tok / 20.59 s); server 31.40 (31.9 ms/tok)
Completion tokens	1,237	1,156
E2E (s)	client 110.80 / server total 110.71	client 37.24 / server total 37.14
VRAM (GB)	23.1 at start (user)	23.4 peak (user)
Spec decode	draft accept 0.743 (749/1,008), mean len 2.81	draft accept 0.832 (766/921), mean len 3.19, best of all llama.cpp runs

Baselines: run #4 (q4_0): prefill 1,014, decode 30.1 cold / 44.9 warm (client), server 22.81/25.87, VRAM peak 20.9. Run #5 attempt 3 (f16 + ubatch 4k): prefill 1,036, decode 60.3 / 56.8 (client), VRAM peak 29.8.

Findings

Prefill unchanged by KV dtype: 1,020.01 vs 1,013.59 tok/s (+0.6 %, noise). q8_0 costs nothing on prefill.
Server-side decode up vs #4: 26.04 vs 22.81 cold (+14 %), 31.40 vs 25.87 warm (+21 %), but draft acceptance improved in the same direction (cold 74.3 %/2.81 vs 69.3 %/2.61; warm 83.2 %/3.19 vs 75.9 %/2.87), so part of the gain is better spec acceptance, not provably raw kernel speed. Isolating that needs a no-spec A/B (open).
Server total ≈ client E2E this time (110.71 vs 110.80; 37.14 vs 37.24), the SSE first-token delay was small on this run (~24 s cold, ~16 s warm vs prompt-eval completion), confirming it's variable delivery behavior, not a fixed cost.
VRAM peak 23.4 GB (+0.3 over start), matches the linear model (weights ≈17.5 + q8_0 pool ≈5.9 at 163k ctx).
Output trajectory matches attempt 3's (f16), not #4's (q4_0); summary coherent and on-corpus.

Recommendations

Speed variant

Adopt q8_0, zero prefill cost (1,020 vs 1,014), server decode +14 %/+21 % vs q4_0 (partly via better spec acceptance), quality ≈ f16, and the only KV dtype that fits ctx 262k on one B70 (~27 GB).
ctx-size → 262144 with q8_0 KV, the user's real use case (131k–262k), now feasible on a single B70.
No-spec A/B (drop --spec-type), splits raw decode speed from spec-acceptance gains; also settles whether ubatch matters at all. The ubatch-only isolation run from #5 can be skipped if this covers it.
SYCL_CACHE_PERSISTENT=1, pending since run #4.
SSE first-token delay, curl -N check, pending since run #4.

Quality variant

Verify bullet accuracy vs corpus for this q8_0 output (trajectory already matches f16's, no red flags visible).
If quality holds: q8_0 + ctx 262k is the recommended config for the 131k–262k target.
Draft KV stays f16 (default).

Appendix: docker-compose (verbatim, as provided 2026-10-21)

services:
  llamacpp:
    container_name: b70
    image: ghcr.io/ggml-org/llama.cpp:full-intel
    restart: unless-stopped
    ports:
      - "9932:9932"
    volumes:
      - ./models:/models
      - ./cache/llamacpp:/root/.cache/llamacpp:Z
      - ./cache/sycl:/root/.cache/sycl:Z
      - ./cache/huggingface:/root/.cache/huggingface:Z
    devices:
      - /dev/dri
    group_add:
      - "992"
    environment:
      GGML_SYCL_ENABLE_GRAPH: 1
      UR_LOADER_USE_LEVEL_ZERO_V2: 0
      ZE_AFFINITY_MASK: 0
      SYCL_DEVICE_FILTER: level_zero:0
      ONEAPI_DEVICE_SELECTOR: level_zero:0
      SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS: "1"
      GGML_SYCL_ENABLE_COPY: "1"
      GGML_SYCL_F16: "1"
      ZES_ENABLE_SYSMAN: "1"
      OMP_NUM_THREADS: "12"
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface
      LLAMA_CPP_CACHE: /root/.cache/llamacpp
      SYCL_CACHE_PERSISTENT: "0"
      SYCL_CACHE_PATH: /root/.cache/sycl
      SYCL_USE_XMX: 1
    command:
      - --server
      - --model
      - /models/Qwen3.8-27B-GSQ-RCO-IQ3_S-Intel-Arc-Tuned-MTP-Q4_K.gguf
      - --spec-type 
      - "ngram-mod,draft-mtp"
      - --spec-ngram-mod-n-match 
      - "24"
      - --spec-ngram-mod-n-min
      - "1"
      - --spec-ngram-mod-n-max
      - "3"
      - --spec-draft-n-max
      - "3"
      - --spec-draft-p-min
      - "0.5" 
      - --jinja 
      - --chat-template-file 
      - /models/qwen-sharp.jinja
      - --alias 
      - "Qwen3.8-27B"
      - --host
      - 0.0.0.0
      - --port
      - "9932"
      - --n-gpu-layers
      - "all"
      - --flash-attn
      - "on"
      - --fit-target
      - "256"
      - --ctx-size
      - "163840"
      - --n-predict 
      - "-1"
      - --temp 
      - "0.7"
      - --top-p 
      - "0.9"
      - --top-k 
      - "40"
      - --repeat-penalty
      - "1.05"
      - --repeat-last-n
      - "256"
      - --presence-penalty
      - "0.0"
      - --dry-penalty-last-n
      - "512"
      - --threads
      - "8"
      - --threads-batch
      - "12"
      - --batch-size
      - "16384"
      - --ubatch-size
      - "2048"
      - --cors-origins
      - "*"
      - --cache-type-k
      - "q8_0"
      - --cache-type-v
      - "q8_0"
      - --cache-prompt
      - --cache-reuse
      - "2048"
      - --cache-ram
      - "2048"
      - --parallel
      - "1"
      - --kv-unified
      - --kv-offload
      - --reasoning
      - "on"
      - --tools
      - "read_file,file_glob_search,grep_search,write_file,edit_file,get_info"
      - --metrics
      - --slots
      - --warmup
      - --perf

2026-10-17 · llama.cpp SYCL/XPU · 1x B70 · ctx 163840