Qwen3.8-27B · GGUF · ubatch 4096 + KV q8_0

Multi-variable run vs run #4, three changes at once, so attribution is partial:

Prefill gains can't be split between ubatch and offload; decode shifts could come from either offload or q8_0; the quality delta is attributable to q8_0 alone. Isolation follow-ups listed in Recommendations.

Status: Attempt 3 SURVIVED, --no-kv-offload identified as the crash culprit. Benchmark data below.

Crash history

--no-kv-offload is the only variable present in both crashes and absent from the survivor → culprit. ubatch 4096 and f16 KV are safe. Exception text for the record:

W ggml_sycl_flash_attn_ext_onednn: oneDNN SDPA failed (Graph nodes cannot depend on events from outside the graph.); falling back to TILE kernel
Exception caught at file:/app/ggml/src/ggml-sycl/ggml-sycl.cpp, line:3240

Pitfall: don't use --no-kv-offload with this SYCL/flash-attn setup, long prefills crash-loop the container.

Config

Stats (attempt 3: ubatch 4096 + kv-offload + f16 KV)

Baseline to beat (run #4: ubatch 2048 + kv-offload + q4_0 KV): prefill 1,013.59 tok/s, warm decode 44.9 tok/s, cold decode 30.1, VRAM peak 20.9 GB.

Findings

• ubatch 4096 did not improve prefill: 1,036.07 vs 1,013.59 tok/s (+2 %, noise). Consistent with the EXL3 run #3 A/B, batch budget doesn't move single-request long-context prefill on this hardware. llama.cpp's ~1k tok/s prefill is a SYCL/flash-attn limitation for this hybrid model, not a chunk-size problem (6× gap vs vLLM/exl3xpu's ~6,439 stands).
• Warm cache reuse perfect again (4 tokens / 350 ms, f_sim_best = 1.000); client TTFT 16.0 s. SSE first-token delay again but smaller than run #4's 23.45 s. The delivery artifact is real and variable (~16–31 s across runs), so llama.cpp client TTFT numbers are unreliable until the streaming path is understood.
• Client decode (60.3 / 56.8) exceeds server window rates (~30 / ~33–49), server eval time includes draft-model forwards and the SSE delivery distorts spans; per journal convention client remains primary, server windows recorded as context.
• Draft acceptance slightly up vs run #4: 74.4 %/78.5 % (mean len 2.81/3.02) vs 69.3 %/75.9 %.
• Output quality: both summaries coherent and on-corpus (mutiny in the forecastle, news of Moby Dick), no red flags visible at f16 KV; trajectory still diverged from #4 due to KV numerics.
• VRAM: 29.2 GB start → 29.8 peak (+0.6). The 29.2 start vs attempt 2's 19.8 (no-offload, same f16 KV) is unexplained, no KV-split line in the pasted startup log; open question. Practical consequence: with f16 KV at 163k ctx there's only ~2.8 GB headroom, so ctx→262k will not fit on one B70 (needs ~+5 GiB); it would fit at q4_0.

Recommendations

Quality variant

1. f16 KV is now active (attempt 3), compare summary output vs runs #4/#2/#3 (same corpus, greedy; note trajectory diverged due to KV numerics).
2. q8_0 was never actually tested, both q8_0 attempts crashed (culprit was no-kv-offload, not q8_0). If a middle ground between q4_0 and f16 is wanted, test q8_0 with kv-offload ON.
3. Draft KV stays f16 (default).

Speed variant

1. Resolved: --no-kv-offload crashes (SYCL graph exception), keep --kv-offload.
2. Isolate the decode gain cleanly: rerun run #4 config with only ubatch 4096 (keep q4_0 KV), splits the +26 % warm decode gain between ubatch and f16 KV.
3. ctx-size → 262144, confirmed NOT to fit with f16 KV (only ~2.8 GB headroom at 163k ctx, needs ~+5 GiB); fits at q4_0 (~+2 GiB). Pick the KV dtype first.
4. SYCL_CACHE_PERSISTENT=1, still pending from run #4.
5. SSE first-token delay, verify with curl -N before trusting llama.cpp TTFT numbers (open since run #4).

Appendix: docker-compose (verbatim, as provided 2026-10-21, attempt 3, the running config)

Attempts 1–2 (both crashed) used --no-kv-offload; attempt 1 additionally had --cache-type-k/v q8_0 active.

services:
  llamacpp:
    container_name: b70
    image: ghcr.io/ggml-org/llama.cpp:full-intel
    restart: unless-stopped
    ports:
      - "9932:9932"
    volumes:
      - ./models:/models
      - ./cache/llamacpp:/root/.cache/llamacpp:Z
      - ./cache/sycl:/root/.cache/sycl:Z
      - ./cache/huggingface:/root/.cache/huggingface:Z
    devices:
      - /dev/dri
    group_add:
      - "992"
    environment:
      GGML_SYCL_ENABLE_GRAPH: 1
      UR_LOADER_USE_LEVEL_ZERO_V2: 0
      ZE_AFFINITY_MASK: 0
      SYCL_DEVICE_FILTER: level_zero:0
      ONEAPI_DEVICE_SELECTOR: level_zero:0
      SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS: "1"
      GGML_SYCL_ENABLE_COPY: "1"
      GGML_SYCL_F16: "1"
      ZES_ENABLE_SYSMAN: "1"
      OMP_NUM_THREADS: "12"
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface
      LLAMA_CPP_CACHE: /root/.cache/llamacpp
      SYCL_CACHE_PERSISTENT: "0"
      SYCL_CACHE_PATH: /root/.cache/sycl
      SYCL_USE_XMX: 1
    command:
      - --server
      - --model
      - /models/Qwen3.8-27B-GSQ-RCO-IQ3_S-Intel-Arc-Tuned-MTP-Q4_K.gguf
      - --spec-type 
      - "ngram-mod,draft-mtp"
      - --spec-ngram-mod-n-match 
      - "24"
      - --spec-ngram-mod-n-min
      - "1"
      - --spec-ngram-mod-n-max
      - "3"
      - --spec-draft-n-max
      - "3"
      - --spec-draft-p-min
      - "0.5" 
      - --jinja 
      - --chat-template-file 
      - /models/qwen-sharp.jinja
      - --alias 
      - "Qwen3.8-27B"
      - --host
      - 0.0.0.0
      - --port
      - "9932"
      - --n-gpu-layers
      - "all"
      - --flash-attn
      - "on"
      - --fit-target
      - "256"
      - --ctx-size
      - "163840"
      - --n-predict 
      - "-1"
      - --temp 
      - "0.7"
      - --top-p 
      - "0.9"
      - --top-k 
      - "40"
      - --repeat-penalty
      - "1.05"
      - --repeat-last-n
      - "256"
      - --presence-penalty
      - "0.0"
      - --dry-penalty-last-n
      - "512"
      - --threads
      - "8"
      - --threads-batch
      - "12"
      - --batch-size
      - "16384"
      - --ubatch-size
      - "4096"
      - --cors-origins
      - "*"
      - --cache-prompt
      - --cache-reuse
      - "2048"
      - --cache-ram
      - "2048"
      - --parallel
      - "1"
      - --kv-unified
      - --kv-offload
      - --reasoning
      - "on"
      - --tools
      - "read_file,file_glob_search,grep_search,write_file,edit_file,get_info"
      - --metrics
      - --slots
      - --warmup
      - --perf