Qwen3.6-35B-A3B · GPTQ-Int4 · vLLM XPU

Config

Stats (measured 2026-10-17 ~00:15 UTC, client-side streaming + /metrics cross-check)

Server /metrics cross-check: TTFT sum/count = 17.46/2, E2E = 27.17/2, prompt tokens = 128,692/2, all consistent with client timings.

Server log lines (as logged)

• Cold window: Avg prompt throughput: 6434.7 tok/s, Avg generation throughput: 51.2 tok/s, both are 10s-window averages; 51.2 = 512 tokens ÷ 10s window (decode only occupied part of it), not the real decode rate. Client-side 105.8 tok/s stands.
• Warm window: Prefix cache hit rate: 47.3%, cumulative since start: (0% cold + ~95% warm) / 2. Consistent with warm TTFT 1.50 s.
• JIT caveat: cold run triggered one-time Triton JIT compiles (precopy_mamba_align_fused_kernel, postprocess_mamba_fused_kernel, eagle_*, rejection_greedy_sample_kernel) → latency spikes inside the measured cold request. True steady-state cold prefill is faster than 15.99 s suggests.
• Raw-decode estimate (derived): 512 tokens in ~167 steps (matches 170 chunks); each step ≈ 1 target forward + 4 draft forwards. With 3.07 acceptance length, raw non-spec decode is roughly 40–60 tok/s. MTP is worth ~2× here. Confirm with a no-spec run.

Findings

• Decode: 105.8 tok/s with MTP speculation; raw non-spec estimated 40–60 tok/s (from 3.07 acceptance length, ~167 steps for 512 tokens). MTP ≈ 2× gain.
• Prefill: server reports ~6,434 tok/s; cold TTFT 15.99 s includes one-time Triton JIT spikes (mamba/eagle kernels). Steady-state cold will be faster.
• Warm (APC ~95% hit): TTFT 1.50 s, repeated long prompts are nearly free after the first.
• VRAM budget at startup: weights 19.9 GiB + peak activation 2.14 + cudagraphs 3.08 + KV 4.23 ≈ 29.4 GB; peak 31.4/32 GB during test.
• Concurrency ceiling: 322,451 KV tokens total → only ~2 full-context (163,840) requests fit. Second concurrent user would OOM.
• Startup time: weights 28.4 s, torch.compile 57.6 s, engine init 81.9 s → serving in ~2.5 min.

Recommendations

Quality variant (improve output fidelity)

1. --dtype bfloat16, logs show Casting torch.bfloat16 to torch.float16; the model is natively bf16 and we're downcasting it. Free fix, small fidelity gain.
2. KV cache fp8 → bf16, vLLM itself warns fp8 KV "may cause accuracy drop"; matters most at 131k+ context. Cost: KV token capacity halves (322k → ~161k ≈ exactly one full-context request); needs the VRAM trade-off accepted.
3. Weights GPTQ-Int4 → BF16 with TP=2 across both B70s, biggest quality jump; ~35 GiB of weights won't fit one 32 GB card but fits the pair. Also fixes the concurrency ceiling (see speed).

Speed variant (improve tok/s / TTFT)

1. Raise --max-num-batched-tokens (16384 → 32768+), 64k prefill currently chunks across ≥4 iterations.
2. Verify MTP is net-positive first (run without --speculative-config), then tune draft depth: per-position acceptance decays 0.78 → 0.29, so num_speculative_tokens=2 may cost less per step for similar effective speed.
3. TP=2 across both B70s, parallelizes prefill and decode (shared with quality #3); second GPU is currently idle.
4. APC stays ON, warm long prompts are nearly free (TTFT 1.5 s vs 16 s cold).

Measurement hygiene

• Compare cold runs across configs (server restart clears APC); always record cold AND warm.
• Cold numbers can carry one-time Triton JIT spikes, rerun to get steady state when precision matters.

Follow-up

• [ ] Re-test this exact setup without --speculative-config → raw decode baseline, same endpoint/test.
• Then proceed to next model/config (new run file).

Appendix: docker-compose (verbatim, as provided 2026-10-20)

services:
  vllm:
    image: vllm/vllm-openai-xpu@sha256:2c427ef477da092eb6f2cdbbbd24950b5fa171565b916db69d4c7bb10e68ca97
    container_name: b70
    privileged: true
    restart: unless-stopped
    ports:
      - "9932:9932"
    devices:
      - /dev/dri
    group_add:
      - "992"
    cap_add:
      - SYS_PTRACE
    volumes:
      - ./models:/models:Z
      - ./cache:/root/.cache/huggingface:Z
      - ./patches/patch_mtp_nightly.py:/patch_mtp.py:ro
      - ./patches/patch_mtp_boundary.py:/patch_boundary.py:ro
      - ./patches/patch_mtp_ptr_wrap.py:/patch_mtp_ptr_wrap.py:ro
    environment:
      HF_HOME: /root/.cache/huggingface
      HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
      VLLM_TARGET_DEVICE: xpu
      ZE_FLAT_DEVICE_HIERARCHY: COMPOSITE
      ZE_AFFINITY_MASK: "0"
      B70_MTP_BF16_DRAFT: "1"
      VLLM_XPU_ENABLE_XPU_GRAPH: "1"
      PYTORCH_ALLOC_CONF: "expandable_segments:True"
      TORCHINDUCTOR_COMPILE_THREADS: "1"
    entrypoint:
      - bash
      - -lc
    command:
      - |
        set -e

        echo "========================================"
        echo " B70 vLLM"
        echo "========================================"

        echo "=== GPU CHECK ==="
        python -c 'import torch; print("XPU count:", torch.xpu.device_count()); [print(i, torch.xpu.get_device_name(i)) for i in range(torch.xpu.device_count())]'

        echo "=== MTP PATCH ==="
        python /patch_mtp.py

        echo "=== patch_boundary PATCH ==="
        python /patch_boundary.py

        echo "=== patch_mtp_ptr_wrap PATCH ==="
        python /patch_mtp_ptr_wrap.py

        echo "=== STARTING vLLM ==="

        exec vllm serve "llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GPTQ-Int4" \
          --quantization gptq \
          --dtype float16 \
          --max-model-len 163840 \
          --gpu-memory-utilization 0.9 \
          --kv-cache-dtype fp8 \
          --port 9932 \
          --max-num-seqs 64 \
          --max-num-batched-tokens 16384 \
          --enable-auto-tool-choice \
          --tool-call-parser qwen3_coder \
          --served-model-name Qwen3.6-35B-A3B \
          --language-model-only \
          --reasoning-parser qwen3 \
          --speculative-config '{"method":"mtp","num_speculative_tokens":4}'

The three MTP patch scripts are from the Intel Arc Pro B70 Inference Cookbook by SergiioB.