malaikat

Personal AMD Strix Halo coding inference launcher: ROCm llama.cpp + MoE + MTP.

No chat UI. Profiles, config file or CLI → OpenAI-compatible API.

The default profile is glm47-flash (GLM-4.7-Flash 30B-A3B, 192k) — it is what bare malaikat serve starts. For the fastest coding model use coding: the Qwen3.8-35B-A3B Distill plus DFlash n=15, 40/40 at 172 tok/s mixed (unique 165). coding-qwen3 (Qwen3-Coder-30B-A3B + DFlash n=15) is 164 tok/s mixed (37/40) and 185 unique. coding-tiel (Tiel-Coder-35B-A3B) is 40/40 at 177 tok/s. gemma-moe (Gemma 4 26B-A4B, DFlash n=15) is 40/40 at 152 tok/s. Dense gemma-31b 31B QAT at MTP n=12 is 45 tok/s. gpt-oss adds EAGLE-3 on MXFP4 20B (~76 tok/s).

runtime.GOOS selects the platform stack:

	Windows	Linux
Runtime	lemonade-sdk windows-rocm-gfx1151 zip (PATH/DLLs)	llama.cpp built against system ROCm (gfx1151)
Extra env	HIP_VISIBLE_DEVICES=0, high-priority process	HIP_VISIBLE_DEVICES=0, ROCR_VISIBLE_DEVICES=0, ROCBLAS_USE_HIPBLASLT=1, GPU_MAX_HW_QUEUES=16, HSA_ENABLE_SDMA=0
Context	-c 256k|512k|1m (YaRN auto for 512k/1m)	same, and --fit off so the window is not shrunk
Models	%LOCALAPPDATA%\malaikat\models	~/.local/share/malaikat/models
last.yaml	%AppData%\malaikat\last.yaml	~/.config/malaikat/last.yaml
Profiles	%AppData%\malaikat\profiles	~/.config/malaikat/profiles

Profiles

A profile fixes the model, context size and system prompt for one job, and carries the Hugging Face coordinates so its weights can be fetched. Thirty-four ship inside the binary:

malaikat profiles                  # list what is available and what is on disk
malaikat profiles -json            # same, machine readable (used by the eval runner)
malaikat serve                     # serve the default profile (glm47-flash)
malaikat pull  -profile coding     # download that profile's weights
malaikat serve -profile coding     # serve a named one

WEIGHTS in the listing is ok when the model file is present and pull when it is not.

Generating profiles (kind: chat, the default):

Profile	Model	Ctx	For
glm47-flash	GLM-4.7-Flash (30B-A3B, MLA) UD-Q4_K_XL	192k	default profile; GGUF is deepseek2-arch, ngram only
coding	Qwen3.8-35B-A3B Distill + DFlash n=15	256k	fastest coding agent, 172 tok/s
coding-qwen38	Qwen3.8-35B-A3B Distill + DFlash n=15	256k	same weights as coding
agent	Qwen3.8-35B-A3B Distill + DFlash n=15	256k	tool/agent prompt, same weights as coding
coding-tiel	Tiel-Coder-35B-A3B + DFlash n=15	128k	code-specialist MoE, 177 tok/s
coding-qwen3	Qwen3-Coder-30B-A3B + DFlash n=15	128k	coder MoE, 185 tok/s unique
chat	Qwen3.8-27B + DFlash2 n=7	64k	dense chat, ~35 tok/s
muse	Muse Glimmer 30B + DFlash2 n=15	64k	dense 30B, 40/40 at 35 tok/s
reasoning	Qwen3.8-27B + DFlash2 n=7	128k	thinking on, same draft as chat
server	Qwen3.8-35B-A3B Distill + DFlash n=15	512k	4 concurrent clients, 128k each
vision	Qwen3.8-27B + DFlash2 + mmproj	64k	images (same weights as chat)
vision-moe	Qwen3.8-35B-A3B Distill + mmproj	64k	images at MoE decode speed
vision-gemma-moe	Gemma 4 26B-A4B + DFlash n=15 + mmproj	64k	images at 155 tok/s
dense-27b	Qwen3.6-27B + DFlash n=15	128k	dense 27B, 38/40 at 65 tok/s
compact	Qwen3.8-27B IQ3_S + DFlash2 n=7	128k	27B in ~12 GB, ~36 tok/s
gemma-31b	Gemma 4 31B QAT + MTP n=12 + ngram	64k	dense 31B, 40/40 at 45 tok/s
gpt-oss	gpt-oss-20b MXFP4 + EAGLE-3	32k	MXFP4 MoE, ngram follow-ups, ~76 tok/s
gemma-e4b	Gemma 4 E4B + MTP	128k	4B dense, ~135 tok/s
qwen-4b	Qwen3.5-4B + DFlash n=15 p-min 0.6	128k	smallest Qwen, 37/40 at 198 tok/s
qwen38-9b	Qwen3.8-9B Distill + DFlash n=15 p-min 0.6	128k	mid-size Distill, 38/40 at 218 tok/s
gemma-e2b	Gemma 4 E2B QAT + MTP n=6	128k	smallest Gemma, ~236 tok/s
gemma-moe	Gemma 4 26B-A4B (MoE) + DFlash n=15	128k	3.8B active, 40/40 at 152 tok/s
fast-moe	LFM2.5-8B-A1B (hybrid MoE)	128k	1.5B active, ~140 tok/s
tiny	LFM2.5-2.6B (hybrid) + DSpark	32k	instant replies, ~2.4 GB
bonsai	Ternary Bonsai 2 27B (PQ2_0)	128k	27B in 7 GB; needs a forked runtime
laguna	Laguna S 2.1 (118B-A8B MoE) IQ3_M	256k	118B agentic coder, ~56 GiB solo; no DFlash (needs the poolside fork)
nemotron-3-nano	Nemotron 3 Nano 30B-A3B (Mamba2 hybrid) UD-Q4_K_XL	256k	fastest here at 111 tok/s, 37/40
coding-next	Qwen3-Coder-Next (80B-A3B hybrid) IQ2_XXS	256k	40/40; DFlash hangs on qwen3next, so ngram only
flash-next-iq4	Qwen3.8-Flash-Next (125B-A6B, qwen4exp) UD-IQ4_XS + MTP n=5	256k	177B-class Qwen4-preview arch, 40/40; needs the fork runtime + a raised GTT (see Qwen3.8-Flash-Next)
flash-next-q4	Qwen3.8-Flash-Next UD-Q4_K_XL + MTP n=5	256k	one quant up (92.3% vs 89.6% top-1 recovery); 104 GiB of weights, at this box's ceiling

Embedding and reranking profiles (kind: embed / kind: rerank) serve /v1/embeddings and /v1/rerank instead of generating text, so they are evaluated differently (see below). They need ubatch at least as large as the longest chunk, because llama-server rejects any input larger than the physical batch — 4096 here rather than the 1024 used for chat.

Profile	Model	Dim	For
embed	Qwen3-Embedding-0.6B	1024	retrieval, RAG, code search
embed-gemma	EmbeddingGemma 300M	768	smallest embedder, 2k window
rerank	Qwen3-Reranker-0.6B	—	reorder candidates before the prompt

kind only changes which llama-server flags are emitted: embed adds --embeddings (plus --pooling if set), rerank adds --reranking, classify adds --embeddings --pooling none for per-token hidden states. Everything else — model, context, runtime, system prompt plumbing — is shared.

Classification sidecar (kind: classify): Laya is a System-1 decision model. The GGUF is only the ModernBERT encoder. pull also fetches laya-head.safetensors and tokenizer.json. serve runs the head in this binary and listens for POST /predict (the same body as the Python laya package). llama-server stays on an internal port and supplies per-token hidden states. Clients do not call /embedding. Serve it next to a chat profile on port 8081:

malaikat pull  -profile laya
malaikat serve -secondary laya         # default chat :8080, Laya :8081
malaikat serve -profile laya            # Laya only, :8081

Profile	Model	Ctx	For
laya	Laya ModernBERT Q8_0 + Go head	2k	System-1 decisions, POST /predict on a port chosen at serve time

curl -s http://127.0.0.1:8081/predict -H 'Content-Type: application/json' -d '{
  "state": {"subject": "Duplicate charge on invoice 4411",
            "body": "We were billed twice for March. Please refund the duplicate."},
  "questions": {
    "department": {
      "type": "choice",
      "instructions": "Which department should handle this?",
      "criteria": {
        "billing": "invoices, payments, refunds",
        "technical": "bugs and outages",
        "sales": "pricing"
      }
    }
  }
}'

Forked runtimes

A model can need a llama.cpp build that mainline does not have. Bonsai 2's ternary weights use PQ2_0/PTQ1_0 (ggml type ids 142/143), which exist only in Prism's fork, so the profile declares both the runtime and the model:

runtime_url: https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10709-9a9394a/llama-prism-b10709-9a9394a-bin-ubuntu-rocm-7.2-x64.tar.gz
llama_tag: llama-prism-b10709-rocm72

malaikat pull -profile bonsai then fetches the weights and unpacks that runtime into <data>/runtimes/<llama_tag>/, beside the mainline runtime rather than replacing it. llama_dir: overrides it with an explicit path. Nothing else changes: serve still probes the runtime's --spec-type list and its --help flags before launching.

A fork with no release tarball is built from source instead. Mainline llama.cpp knows the qwen4exp architecture but cannot load Qwen3.8-Flash-Next's MTP head — src/models/qwen4exp.cpp has no load_mtp support — so flash-next-* needs Unsloth's qwen4exp/mtp branch. scripts/build-qwen4exp-mtp.sh clones, builds against system ROCm (gfx1151) and installs runtimes/qwen4exp-mtp-<sha>/, which the profile names with llama_tag::

This runs a third-party prebuilt binary. It is the model publisher's own release, linked from the model card, but it is not built by malaikat and the mainline runtime is untouched. Archives are extracted with path-escape and symlink-escape checks.

Profiles are plain YAML: any config field works, plus name, description, tags, system_prompt/system_prompt_file, hf_repo/hf_file/hf_draft_repo/hf_draft_file/hf_mmproj/hf_extra. A profile can also name secondary: (or you pass -secondary NAME) so serve starts that profile as a sidecar on its own port.

Edit the shipped ones with:

malaikat profiles -write      # extract to ~/.config/malaikat/profiles (then edit)

A profile in the user directory shadows the shipped one of the same name. -profile and -config are mutually exclusive; -m, -c and the other flags still override the profile.

System prompts are injected

llama-server has no server-side system prompt (-sys is not a server option), so when a profile carries one, serve starts llama-server on an internal port and puts an injecting proxy on the requested port:

system:  injected by proxy (73 chars)
upstream: http://127.0.0.1:44125
listen:  http://127.0.0.1:8080

The prompt is added to /v1/chat/completions (and /v1/messages) requests that arrive without a system message. A client that sends its own system message wins — the profile never overrides an explicit prompt. Everything else (/health, /v1/models, streaming, /completion) passes straight through. Profiles without a system prompt exec llama-server directly, with no proxy in the path.

Several models at once

Nothing serialises llama-server instances, so one box can serve several models at the same time — each on its own port and alias, which is what lets N agents each drive a different model. Two ways:

• One process, one sidecar: malaikat serve -secondary chat (the default profile plus chat), or malaikat serve -profile coding -secondary chat. -secondary is not classify-specific — the sidecar is any profile, started on its own port (the profile's port: when free, otherwise an ephemeral one). One primary + one sidecar.
• N processes: run malaikat serve once per model with -port and -no-save. There is no registry or lock, so N is bounded only by GTT/unified memory.

Pass -no-save when running more than one: every serve writes last.yaml, so the last one started wins.

Measured here (60.7 GiB GTT): three 30B-class models at once — tiny (2.6B), glm47-flash (30B-A3B) and nemotron-3-nano (30B-A3B) — answered three simultaneous requests in 2.6 s wall at 46 GiB GTT. They share the iGPU, so aggregate throughput does not scale with model count and per-request latency rises as bandwidth splits. Budget by weights: two 30B-class models plus KV fit together, three fit if contexts stay modest, and laguna (118B-A8B, IQ3_M) needs ~52 GiB on its own, so it runs solo.

Setup

Linux (from source)

Requirements: Go 1.22+, ROCm HIP SDK (Arch: sudo pacman -S --needed base-devel cmake git rocm-hip-sdk rocblas hipblas hipblaslt).

go build -o malaikat .
./malaikat setup

setup clones llama.cpp and compiles natively (-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151). Flash-attn uses HIP WMMA builtins. --bundle falls back to the lemonade Ubuntu zip.

./malaikat setup --force        # git pull master + rebuild
./malaikat setup --ref b1311    # pin a llama.cpp tag
./malaikat setup --bundle       # lemonade Ubuntu ROCm zip instead

Large GGUFs come from unified (GTT) memory; if allocation fails, check dmesg | grep -i amdgpu. The GPU-visible window defaults to half of RAM (~60 GiB on a 128 GB box). Qwen3.8-Flash-Next is the first shipped model that does not fit there, so its profiles need it raised — and two knobs are needed: amdgpu.gttsize (the GTT aperture) and ttm.pages_limit (the allocator that actually hands out the pages). Setting only the former leaves the kernel logging Configuring gttsize via module parameter is deprecated, please use ttm.pages_limit and still capping allocations at the old TTM size, so the model OOMs at ~61 GiB. On Omarchy/limine:

printf 'KERNEL_CMDLINE[default]+=" amdgpu.gttsize=102400 ttm.pages_limit=26214400"\n' \
  | sudo tee /etc/limine-entry-tool.d/amdgpu-gtt.conf
sudo limine-update
sudo reboot

Both are 100 GiB (102400 MiB of GTT; ttm.pages_limit counts 4 KiB pages, so 100 GiB = 26214400). See Qwen3.8-Flash-Next.

Windows

go build -o malaikat.exe .
.\malaikat.exe setup

setup pulls the lemonade-sdk/llamacpp-rocm Windows gfx1151 zip into %LocalAppData%\malaikat\runtime. All-in-one release builds skip the download and unpack the embedded zip. --source is available if you have a HIP SDK, but is not the default.

Download (all-in-one, no toolchain)

Grab malaikat-* from Releases. It embeds the lemonade ROCm gfx1151 runtime for that OS (not the model):

.\malaikat-0.2.0-windows-amd64.exe serve -profile coding

chmod +x malaikat-0.2.0-linux-amd64
./malaikat-0.2.0-linux-amd64 pull  -profile coding
./malaikat-0.2.0-linux-amd64 serve -profile coding

Weights come from Hugging Face:

malaikat pull -profile coding     # this profile's model, into the models dir
malaikat pull -profile chat       # resumable; re-run to continue an interrupted pull

pull also fetches a profile's companion files (draft/MTP head, mmproj, hf_extra). If the profile names secondary:, that sidecar is pulled too. A profile can also need a forked runtime with no release tarball for pull to fetch: flash-next-* needs scripts/build-qwen4exp-mtp.sh run once before serve, otherwise the tag is advisory and serve falls back to mainline (which cannot load the MTP head). If a scripts/download_*.py script is easier (it uses huggingface_hub), these still work:

python3 scripts/download_qwen38_35b.py  # Qwen3.8-35B-A3B Distill + DFlash head (the default `coding`)
python3 scripts/download_qwen36.py      # Qwen3.6-35B-A3B MTP (draft source for the above)
python3 scripts/download_qwen38_27b.py  # Qwen3.8-27B

Serve

./malaikat serve                           # the default profile (glm47-flash)
./malaikat serve -profile coding           # a named profile (see Profiles)
./malaikat serve -profile coding -secondary laya  # chat plus Laya on a port chosen at serve time
./malaikat serve -profile chat -c 128k      # flags still override the profile

# A plain config file works too:
./malaikat serve -config ./coding.example.yaml -c 256k

# Bare serve reloads the last successful settings, falling back to the default
# profile the first time:
./malaikat serve

-c	Tokens	YaRN
256k	262144	off — native window for the Qwen3.5/3.6 family
512k	524288	2× from 262144
1m	1048576	4× from 262144

512k and 1m pass --rope-scaling yarn --rope-scale N --yarn-orig-ctx 262144 and raise <arch>.context_length (rope_arch, e.g. qwen35moe) so llama.cpp does not cap the slot at the model's training window.

Model paths in YAML may be relative to the platform models dir, or use ~ / $ENV.

On Linux, serve passes --fit off unless you override it after --. Windows lemonade is left as originally shipped.

API: http://127.0.0.1:8080/v1

Shared defaults: -ngl 999, -b/-ub 2048/1024, -fa on, --cache-type-k/v q8_0, --jinja, --spec-type draft-mtp,ngram-map-k,ngram-mod, HIP_VISIBLE_DEVICES=0. Linux also sets GPU_MAX_HW_QUEUES=16 and HSA_ENABLE_SDMA=0 (measured ~1-3% on the 35B-A3B MoEs). The batch sizes were picked by measurement on this APU:

batch/ubatch	16k prefill	short decode
2048/1024	899 tok/s	69.1 tok/s
4096/2048	890 tok/s	68.7 tok/s
512/512	775 tok/s	68.7 tok/s
2048/512	758 tok/s	67.0 tok/s

Speculative depth is per-model — see Tuning. -no-mtp disables speculative decode; -spec-draft-n-max N sets the MTP depth.

A sidecar whose profile pins a port (the classify profile pins 8081) keeps that port only when it is free: with something else holding it, serve takes a free port instead and the banner prints the one it took, rather than dying with "bind: address already in use" and reporting it as a model-server failure.

Agent clients

The endpoint is OpenAI-compatible, so a coding agent that speaks that wire drives a served profile with no adapter. http://127.0.0.1:8080/v1 is the base URL and the profile's alias is the model id (malaikat profiles prints it).

malaikat serve                        # terminal A (the default profile)
opencode                              # terminal B
cmd                                   # terminal C

llama-server ignores the request's model field in single-model mode — it answers any id and echoes its own --alias — so the client-side ids are labels. The field that does matter is the declared context window: both agents decide when to compact from it, so a 128k entry pointed at a 64k profile overflows. Keep it equal to the served profile's ctx_size.

These seven are declared in both clients; each id is that profile's alias:

Model id	Profile	Ctx
qwen3.8-35b-a3b-distill	coding (and server)	256k
qwen3.8-35b-a3b-agent	agent	256k
qwen3-coder-30b-a3b	coding-qwen3	128k
tiel-coder-35b-a3b	coding-tiel	128k
qwen3.6-27b	dense-27b	128k
gemma-4-26b-a4b	gemma-moe	128k
glm-4.7-flash	glm47-flash	192k

Serving server instead of coding puts four slots behind the same id, so 128k per client rather than one 256k window — worth it only when both agents run at once.

OpenCode — ~/.config/opencode/opencode.json:

"provider": {
  "malaikat": {
    "npm": "@ai-sdk/openai-compatible",
    "name": "malaikat (local Strix Halo)",
    "options": { "baseURL": "http://127.0.0.1:8080/v1", "apiKey": "local" },
    "models": {
      "qwen3.8-35b-a3b-distill": {
        "name": "Qwen3.8-35B-A3B Distill - coding",
        "limit": { "context": 262144, "output": 32768 },
        "tool_call": true,
        "reasoning": true,
        "interleaved": { "field": "reasoning_content" }
      }
    }
  }
}

tool_call is what lets the agent call its tools, and interleaved is where llama-server puts the <think> block. apiKey is a placeholder: llama-server only checks it under --api-key.

opencode run -m malaikat/qwen3.8-35b-a3b-distill "read go.mod"

Command Code — ~/.commandcode/providers.json, keyless:

"provider": {
  "malaikat": {
    "name": "malaikat (local Strix Halo)",
    "baseURL": "http://127.0.0.1:8080/v1",
    "apiKey": false,
    "models": {
      "qwen3.8-35b-a3b-distill": {
        "name": "Qwen3.8-35B-A3B Distill (coding)",
        "contextWindow": 262144,
        "maxOutput": 32768
      }
    }
  }
}

cmd -p --model malaikat/qwen3.8-35b-a3b-distill "read go.mod"

Both were checked end-to-end against a served coding profile: each agent called its read tool and answered (opencode run printed the module path, cmd -p --output-format json showed tool_running read_file then stopReason: end_turn). A bare /v1/chat/completions call carrying tools returned finish_reason: tool_calls with the right arguments, so --jinja is doing the tool work rather than the agent prompt-wrangling around it.

Long-horizon agent loops

A local model driving a tool-calling agent can fall into a loop: it re-issues the same tool call and re-states the same plan until the harness stops it (stopped: no progress after repeated tool calls). It is not specific to the profiles here — laguna, coding-next and the Ornith-1.5 profiles did loop, so it is the models and the task rather than the profile wiring. On a short task every one of them uses tools correctly (tool call on turn 1, right answer on turn 2).

The Ornith-1.5 profiles are no longer shipped. They looped on long-horizon agent tasks often enough to be unusable in practice, and every one of them had a same-generation alternative that measured better (see Tuning). The loop is stochastic rather than deterministic — the same model and profile sometimes worked a task through and sometimes stalled — but the rate on repeat runs was high enough that the profiles were dropped rather than tuned. The history of what was tried is kept below so the finding is not re-run from scratch.

glm47-flash does not loop in the same way. It is the daily-driver profile on this box and completes long agent tasks; the earlier laguna/coding-next finding above does not apply to it.

What did not change the loop, on this box (src-a894dae, 60.7 GiB GTT):

• Profile settings. -reasoning off, -spec-type none, --no-reasoning-preserve and cache_prompt: false each left it identical. DRY repetition-breaking (--dry-multiplier 0.8) helped once and failed to reproduce.
• Quantisation. coding-next loops the same on UD-IQ3_S (29.7 GB) as on UD-IQ2_XXS (23.3 GB), so the shipped profile keeps the smaller file.
• The output budget. Raising the client's maxTokens from its 4096 default to 32768 changed nothing.

The loop is stochastic: the same model and profile sometimes work a task through and sometimes stall, so expect it to be a property of the model and the task rather than something a profile setting can remove.

What a long prompt costs

An agent client does not feel the decode rate the profile table quotes. Every step resends the whole conversation, so it pays prefill first, and prefill is where a long context hurts. Measured on this box with coding-qwen3 (scripts/agent-prefill.sh, one fresh server per size so the prompt cache cannot help):

prompt tokens	prefill	rate
2,394	1.7 s	1,446 tok/s
9,594	8.0 s	1,205 tok/s
23,994	29.9 s	803 tok/s
33,494	57.4 s	583 tok/s

So a 25k-token step costs about half a minute before the model says anything, and a 33k one almost a minute - which is why an agent run on a 256k profile feels far slower than the profile table suggests. The table's numbers are decode rates for a short reply (malaikat bench -profile coding-qwen3: 160 tok/s decode, 1,200-1,400 tok/s prefill at 3.5k), and they stay true; they are just not what a long conversation spends its time on.

Two things that do help, both already on by default: prompt caching, which reuses an unchanged prefix, and keeping the context small - halving the prompt roughly halves the wait. The profiles differ here in a way that matters more to a long conversation than their decode rates do:

profile	decode (short reply)	prompt reused on a follow-up
coding-qwen3	~160 tok/s	99.5% (19 tokens reprocessed)
coding	~106 tok/s	70-85% (1,043 tokens reprocessed)

Measured with malaikat bench -profile X -n 128 -reuse 8192. A follow-up turn on coding reprocesses about a thousand tokens where coding-qwen3 reprocesses nineteen, so the profile that is faster on a one-shot answer is not the faster one across a conversation.

One thing that does not, measured rather than assumed: --cache-reuse. With a 7k prompt whose early half changed and whose recent half was identical, the server reprocessed all 7,210 tokens and spent 6.7 s with the flag and 6.7 s without it. It is not enabled in any profile.

One thing that is not obvious, and costs a long run its speed: the prompt cache itself. Measured on flash-next-iq4 with sixteen requests in one session, the cost per evaluated prefill token rises from 1.2 ms to 16 ms after roughly ten requests, decode falls to 0-1 tok/s and the server's RSS grows a gigabyte - in two different shapes (a growing prompt, and small prompts with the slot pinned at 930 tokens). The same session with cache_prompt: false never degrades: prefill 285 to 333 tok/s, decode 32 to 29, RSS flat. So the cache's per-request saved states, and the similarity search over them (selected slot by LCP similarity in the log), are what accumulate.

No flag bounds it in the runtime here; it is a per-request field, so a client turns it off. It is a trade either way: without the cache every step repays its whole context (18.7k tokens cost 56s cold), where a healthy cache makes the same step near-free. A client that wants the knob can set cache_prompt per request - mcode does, via MCODE_PROMPT_CACHE=0.

The GPU stability question is separate and still open. Three amdgpu page faults in llama-server were seen in agent runs on coding-next (the harness restarts the server and carries on). Three attempts to reproduce them all failed: sequential 33k-token requests, the same with spec decoding off, and immediate restarts where a new server loads while the old one is still holding its 29 GiB of GTT. So the cause is not prompt size, not the ngram spec-decoding path, and not a restart race - it is still unknown, and worth recording as unknown rather than as a fix.

A collapse is silent, and now it is reported

A loop is loud: the harness sees the same call repeated and stops it. The other failure is quiet — output stops arriving mid-stream while the server stays up, the process is alive and the GPU goes idle. A client sees only a request that never returns, which is why a run can sit for half an hour without saying anything useful.

serve now watches for it and says so, both for a stream that went quiet and as a one-line summary of each generation:

request 2: first byte 71ms, 450ms of output in 64 writes
request 3: no output for 47s after 12 writes - the model stopped making progress mid-stream

Flush is passed straight through, so streaming is unaffected, and health probes are not reported as generations.

Tuning

Everything below was measured on Linux, system ROCm HIP gfx1151, ctx 262144, -fa on, KV q8_0, -b/-ub 2048/1024.

Draft depth follows the model's MTP accept rate. Drafting deeper than a model can verify costs throughput — a weak MTP head can be slower than no speculation at all.

The clearest measurement of this was on Ornith-1.5-35B-A3B, a profile that no longer ships. Its accept rate collapsed as depth grew, so each step down was faster than the last:

Ornith-1.5-35B-A3B (Q4_K_M)	draft accept	short tg	16k tg	32k tg
-no-mtp	—	56.4	54.4	—
draft-mtp n=3	38%	52.4	42.9	39.6
draft-mtp n=2	57%	62.8	—	—
draft-mtp n=1	90%	71.1	59.7	56.4
draft-mtp,ngram-map-k n=1	90%	71.6	61.2	55.8

DFlash drafts from an external head rather than the model's own MTP layer, which sidesteps the problem entirely: the shipped 35B-A3B profiles run draft-dflash n=15 plus ngram. The Qwen3.8 distill (coding) measured unique 165 tok/s versus 97 with MTP, mixed 40/40 at 172 tok/s.

ngram-map-k adds verified self-speculation on top of the draft head: it replays spans that already appeared in the context. Free on fresh code, large on rewrites and repetitive output, and it does not change the emitted tokens (byte-identical greedy output vs -no-mtp):

Ornith n=1 + ngram-map-k	tg	vs n=1
write a function from scratch	71.6	71.0
rewrite code that is already in the prompt	71.6	71.0
repeat a fixed line 30 times	98.7	70.1 (+41%)

ngram-mod is the follow-up-turn lever. It hashes recent tokens into a shared pool and drafts from matches. On coding-qwen38 (MTP n=3, thinking off, 128-token decode, n=2 after warmup):

Spec chain	First-style merge prompt	Same prompt again	Repeat a line 30 times	Gauge
draft-mtp,ngram-map-k	91 tok/s	91 tok/s	132 tok/s	24/24
... + ngram-map-k4v	92 tok/s	—	126 tok/s	—
... + ngram-mod	93 tok/s first request	148 tok/s	369 tok/s	24/24
KV f16 instead of q8_0	86 tok/s	—	134 tok/s	—

ngram-map-k4v and short-prompt KV f16 did not beat the baseline here, so they stay off the default chain. ngram-mod does not change cold unique prompts much. It pays when the next turn reuses ngrams — the coding-agent case. The expanded eval still scored 24/24 with it on.

A runtime that does not list ngram-mod in --help has it dropped, same as ngram-map-k.

Follow-up turns re-evaluate 4 + ubatch tokens, not the whole prompt — this is what makes multi-turn chat and tool loops feel snappy. Measured with malaikat bench:

./malaikat bench -url http://127.0.0.1:8080 -reuse 4096
# turn 1: prompt=4037 ttft=3891ms pp=1037.5 tok/s
# turn 2: prompt=4052 cached=3009 (74.3%) reprocessed=1043 ttft=1177ms

At 32k context the same probe reuses 95.9% (24410/25451) with ubatch 1024. Lowering ubatch halves the reprocess (ubatch/2) but costs prefill throughput (-16% at ubatch 512); raise it if cold prefill matters more than follow-ups. -ub N or ubatch: in the config.

Thinking is on by default and costs time-to-answer. These models emit a <think> block first, so a short chat question can spend its whole token budget thinking and return no answer at all:

reasoning	chat wall	answer chars (256 max)
auto / on	4.2s	0
off	1.5s	420

Set reasoning: off in the config or malaikat serve -reasoning off for snappy chat; keep auto for coding, where the thinking block pays off. Clients can also request it per call with chat_template_kwargs.

cache-reuse and keep are not exposed: llama.cpp disables KV shifting (Server: cache_reuse is not supported by this context, it will be disabled) for this model family, and --keep only applies to the context shift that is disabled with it.

serve probes llama-server --help and drops any spec type the installed runtime does not know (an unknown type is a startup error in llama.cpp), so a newer config still runs against an older bundled Windows runtime:

warning: runtime does not support spec type ngram-map-k; using draft-mtp

-t is tuned by default (NumCPU-4) and further changes measured neutral or worse (-tb 32: -2% prefill, -t 16: -5%).

KV cache type. cache_type_k/v: f16 is lossless and measured faster than the default q8_0 once the context is deep (16k: 66.8 vs 59.9 tg, pp 891 vs 914), because there is no dequant work per attention step. It doubles KV memory (20.5 GiB vs 10.25 GiB at 256k), so it only fits at -c 256k on 64 GiB unified memory — do not use it with 512k/1m. Mixed types (k q8_0 + v f16) hit a slow path: 16k pp 271 vs 914, avoid.

Concurrency (-np, parallel:). The box is bandwidth-bound, so serving several clients at once buys aggregate throughput. The server profile, four requests of 256 tokens each:

	Aggregate	Wall	Per client
sequential (-np 1)	68 tok/s	6.8s	~71 tok/s
concurrent (-np 4)	98 tok/s	5.4s	~33 tok/s

+44% throughput for the same work, at the cost of per-request latency. Concurrency does not help when -np 1 — four simultaneous requests are simply serialised at the same speed.

Context is split across slots (n_slots = 4, n_ctx_slot = 65536 for -c 256k -np 4), so the server profile asks for -c 512k to give each of its four clients 128k. KV memory is unchanged by -np; the per-client window is not.

Capability gauge

scripts/eval-profile.sh serves each profile and runs an objective battery instead of eyeballing output:

scripts/eval-profile.sh -neutral coding chat compact     # one JSON report per profile
scripts/eval-profile.sh -neutral -thinking on reasoning   # compare with thinking forced on

Category	Check
code	generated Python is executed against unit tests (smoke trio plus LRU, interval merge, search-insert)
tool	function calls must carry the right name and arguments
json	output must parse and match the requested keys/values (plus nested sku/qty)
math	word problems with exact numeric answers (plus LCM, work-rate, mod, rabbits)
instruct	hard constraints (one word, exactly 3 lines, capitals)
recall	a code planted at ~6k and ~24k tokens must come back
speed	prefill tok/s, decode tok/s, TTFT

-neutral sends one fixed system message so models are compared on the same prompt; it also suppresses each profile's own prompt (the proxy leaves requests that carry a system message alone). -thinking off|on pins enable_thinking per request, so a profile's reasoning flag does not skew the comparison. Reports land in build/eval/<profile>.json.

The runner reads each profile's kind (malaikat profiles -json) and picks the battery: a chat gauge is meaningless for a server that does not generate text. Embedding and reranking profiles go through scripts/eval-embed.py, which checks the property that actually matters — whether the vectors rank things correctly:

Kind	Check
embed	vector dimension; steady-state reproducibility; retrieval top-1 over five labelled query/passage sets; throughput batched vs one-at-a-time; a long chunk is accepted
rerank	top-1 ordering on the same five sets, and score separation between the relevant passage and distractors
classify	POST /predict on the invoice ticket: choice, score, and noul answers

Two measurements there are worth knowing about:

• Reproducibility depends on shape, not luck. llama-server captures a graph per input shape, and the first call for a shape returns a vector ~0.9996 cosine from the replayed one. After two calls of the same shape, vectors are bit-identical (cosine 1.000000). So warm the shape before caching vectors, and never expect the same text embedded at a different batch size to hash the same.
• ubatch caps chunk size. llama-server rejects any input larger than n_ubatch (input (1383 tokens) is too large to process. increase the physical batch size). The chat profiles use 1024; the embedding and reranking profiles ship 4096 so real chunks fit.

The recall checks send cache_prompt: false and use a different filler text per task. With one shared repeated filler, the 24k prompt reuses the 6k prompt's cached prefix, and in practice the server then answered an earlier prompt entirely (a 24k recall returning 55 from the maths task, or the previous code task's function). Asking the same needle as the first request to a fresh server retrieved it correctly every time, so those misses were an artefact of the harness, not of the models. Prefill numbers come from this cache-free prefill.

Measured

All profiles, served as shipped (-neutral system message, thinking pinned off, short code prompt for decode, 24k-token prompt for prefill), on this machine:

Profile	Score	Decode	Prefill @24k	Failing checks
gemma-e2b	37/40	232 tok/s	2174 tok/s	one_word, mod_exp, rabbits
tiny	23/24	154 tok/s	2565 tok/s	lru_cache
fast-moe	22/24	139 tok/s	3481 tok/s	nested, three_lines
gemma-e4b	27/30	135 tok/s	1263 tok/s	word_count, work_rate, rabbits
qwen-4b	37/40	198 tok/s	1105 tok/s	DFlash n=15 --spec-draft-p-min 0.6. Unique 132. Misses: word_count, median_sorted, mod_exp
qwen38-9b	38/40	218 tok/s	—	DFlash n=15 --spec-draft-p-min 0.6. Unique first 141. Misses: median_sorted, catalan8
gemma-moe	40/40	152 tok/s	1148 tok/s	unique merge 157 tok/s with DFlash n=15
vision-gemma-moe	42/42	155 tok/s	1152 tok/s	unique 151 tok/s; vision 2/2 with DFlash
coding	40/40	172 tok/s	986 tok/s	unique merge 165 tok/s with DFlash n=15
coding-qwen3	37/40	164 tok/s	948 tok/s	unique merge 185 tok/s with DFlash n=15; one_word, lcm, mod_exp
coding-tiel	40/40	177 tok/s	1144 tok/s	unique merge 122 tok/s with Qwen3.5 DFlash n=15
gpt-oss	38/40	76 tok/s	1233 tok/s	unique_paths, min_window at n=3
vision-moe	26/26	68 tok/s	865 tok/s	— (includes vision 2/2)
dense-27b	38/40	65 tok/s	227 tok/s	mod_exp, catalan8
gemma-31b	40/40	45 tok/s	201 tok/s	MTP n=12 + ngram. Same 40/40 as MTP-only. Follow-up 114 vs 41
chat	38/40	35 tok/s	237 tok/s	median_sorted, mod_exp
muse	40/40	35 tok/s	—	unique merge 28 tok/s with DFlash2 n=15; 24k prefill not re-run
reasoning	38/40	35 tok/s	237 tok/s	same weights as chat, thinking pinned off
vision	40/42	21 tok/s	230 tok/s	median_sorted, mod_exp (includes vision 2/2)
bonsai	16/18	22 tok/s	239 tok/s	math:distance, recall_6k¹
compact	39/40	36 tok/s	332 tok/s	mod_exp; unique merge 42 tok/s with DFlash2 n=7
nemotron-3-nano	37/40	123 tok/s	1449 tok/s	hybrid Mamba-MoE, f16 KV; lcm, rabbits, widgets
glm47-flash	37/40	46 tok/s	300 tok/s	deepseek2-arch GGUF, ngram only; word_count, mod_exp, rabbits
laguna	40/40	22 tok/s	443 tok/s	118B-A8B IQ3_M, 256k, ngram only (DFlash needs the poolside fork)
coding-next	40/40	36 tok/s	660 tok/s	f16 KV; DFlash disabled (it hangs on qwen3next)
flash-next-iq4	40/40	43.6 tok/s	358 tok/s	qwen4exp 177B-class (6B active), MTP n=5; 74.7 GiB GTT
flash-next-q4	40/40	44.1 tok/s	371 tok/s	same weights one quant up; 91.2 GiB GTT

Scores of 38 used that 36-check plus minimum window substring and longest valid parentheses. decode_ways and median-of-two-sorted did not split E2B (37/40, same three misses). Scores of 36 used that 33-check plus Levenshtein distance, unique grid paths, and C_8. Scores of 33 used the later 2026-09-20 battery plus word-break, trapping rain, and C(10,4). Scores of 30 used that set without those three. Scores of 28 used that set without the last two code tasks. Scores of 24 or 26 used the earlier expanded set. Scores of 16 or 18 used the first smoke battery and were not re-run.

¹ Sequence-dependent, and not a capability verdict: bonsai retrieves its needle correctly when it is the first request to a fresh server, and fails it part-way through a long battery with the answer to an earlier task. This was reproduced with prompt caching off and a distinct filler per task, so it is a server-side artefact of the run rather than something the model cannot do.

² Recorded on the now-unshipped Ornith profile: on llama.cpp 6d05498 it returned the numeric tail of each needle (4471, 9032) and dropped the prefix (ZX-, QK-). Re-run on a894dae with the shipped ngram-mod chain: 24/24, both needles complete.

³ Prefill after rebuilding llama.cpp to a894dae (HIP RDNA3.5 MoE tiles). The Qwen3.8 distill profile on the August 6d05498 runtime was 869 tok/s at 24k, and 986 tok/s on the same rebuild.

Embedding and reranking profiles are scored by scripts/eval-embed.py instead:

Profile	Score	Numbers
laya	3/3	/predict billing 0.97, urgency 1.62, churn 0.19
embed	10/10	dim 1024, retrieval 5/5, 127 texts/s (batch 32), 1562-token chunk OK
embed-gemma	10/10	dim 768, retrieval 5/5, 267 texts/s, 1383-token chunk OK
rerank	7/7	top-1 5/5, relevant scored 0.98-0.997 against ~0.000 for distractors

The battery is pass/fail by category. It is a smoke-level gauge, not a benchmark suite. The 35B MoEs still pass the executed code and tool checks. Read decode tok/s for ranking among the models that pass. Vision is only run for profiles with an mmproj, and is skipped rather than failed for the rest.

Rows for profiles that no longer ship (coding, agent and server on Ornith weights, fast, small-qwen, qwen-27b, coding-qwen, gemma 12B) were removed when those profiles were dropped. Their numbers live in the git history.

What the sweep found

Budget first: this APU exposes ~60 GiB of GTT, so a 35B-A3B MoE at Q4 (~22 GB) can hold the full 256k KV cache and a dense 27B at Q4 fits at 128k. GTT is carved out of the same unified memory, so the whole model plus KV has to fit in the ~121.5 GiB the box reports: a 30B-A3B at Q4 is comfortable and laguna (118B-A8B IQ3_M, ~52 GiB) still fits. Qwen3.8-Flash-Next is the first shipped model that does not at the default GTT — it is a 125B MoE (6B active) plus a 51B n-gram embedding and a 4B MTP head, 177B parameters on the new qwen4exp architecture. Its UD-IQ4_XS build is 87.3 GiB and UD-Q4_K_XL 103.7 GiB, so the GTT has to be raised (amdgpu.gttsize) before every layer fits; see Qwen3.8-Flash-Next.

The single biggest capability/price split is MoE vs dense, not model size:

Family	Weights/token	Decode	Prefill (24k)
35B-A3B MoE (Qwen3.8 distill, Tiel-Coder)	~3B active	164-177 tok/s	~948-1144 tok/s
Dense 27B Qwen3.8 + DFlash2 n=7	all 27B	35 tok/s	~237 tok/s
Dense 27B Qwen3.6 + DFlash n=15	all 27B	65 tok/s	~227 tok/s
Dense 31B + separate MTP head (Gemma 4 QAT, n=12)	all 31B	45 tok/s	201 tok/s

Notes from testing each:

• Dense 27B without a block draft stays memory-bandwidth-bound. MTP-only Qwen3.8-27B reads all ~17.6 GB of weights per token: 11 tok/s with -no-mtp, 17 at depth 1, 24 at depth 3. Prefill is ~3.5x slower than the MoE, so filling a long context is the real cost.
• DFlash more than doubles Qwen3.6-27B decode on this APU. dense-27b now ships draft-dflash n=15 plus ngram (williamliao Q8_0, 1.8 GB). Unique merge was 69 tok/s versus 28 with MTP n=3. Mixed eval 38/40 at 65 tok/s. Follow-up 110 tok/s. n=16 was 68 tok/s unique (same mean length 13.9 as n=15). MTP is no longer the spec on this profile. Qwen3.8-27B (chat, reasoning, vision) uses DFlash2 n=7 plus ngram (incoai Q8_0). Unique merge was 39 tok/s versus 28 with MTP. Mixed chat is 38/40 at 35 tok/s. Follow-up 73 tok/s. n=15 did not raise unique past n=7 (mean length 7.6). The 35B-A3B MoEs use DFlash n=15 plus ngram from a small external head (0.42-0.51 GB), which beats each model's own MTP head by a wide margin: the coding distill (Qwen3.8-35B-A3B Distill) is unique 165 tok/s versus 97 with MTP, mixed 40/40 at 172 tok/s, follow-up 245 tok/s. IQ3 compact uses Qwen3.8-27B DFlash2 n=7: unique 42 tok/s versus 22 with MTP, mixed 39/40 at 36 tok/s (mod_exp). coding-tiel uses the Qwen3.5-35B-A3B DFlash head (Anbeeld Q8_0, 0.42 GB): unique 122 tok/s versus 67 with MTP, mixed 40/40 at 177 tok/s. coding-qwen3 uses Anbeeld Qwen3-Coder-30B-A3B DFlash Q8_0 (0.51 GB) at n=15: unique 185 tok/s versus 58 with ngram only (86% accept, mean length 14), mixed 37/40 at 164 tok/s with the same three misses (one_word, lcm, mod_exp). muse (Muse Glimmer 30B, Unsloth UD-Q4_K_XL + Anbeeld DFlash2 Q8_0 n=15) scored 40/40 at 35 tok/s mixed (unique 28, follow-up 33). Same speed class as chat (38/40 at 35 tok/s) with a full 40-check. DFlash2 only added ~4 tok/s over no draft (mean length 5). dense-27b remains the faster dense pick (38/40 at 65). gemma-moe uses Anbeeld Gemma 4 26B-A4B DFlash Q8_0 (0.47 GB) at n=15 plus ngram: unique 157 tok/s versus 100 with MTP n=6, mixed 40/40 at 152 tok/s, follow-up 494 tok/s. Gemma 4 31B DFlash was a wash (50 vs 48 unique), so gemma-31b stays on MTP. Qwen3.5-4B DFlash n=15 unique was 170 tok/s versus 109 with no draft, but mixed eval dropped unique_paths (36/40 vs 37/40) until --spec-draft-p-min 0.6. That setting keeps mixed 37/40 (same three misses as none) at unique 132 tok/s and mixed decode 198 tok/s, so qwen-4b ships DFlash plus ngram with p-min 0.6. p-min 0.8 was 37/40 but unique only 111. That p-min 0.6 chain is what qwen38-9b ships on the 9B Distill weights. vision-gemma-moe uses the same 26B-A4B DFlash draft plus mmproj: unique 151 tok/s versus 106 MTP, mixed 42/42 at 155 tok/s, vision 2/2. DFlash plus mmproj on vision-moe (Qwen3.8 distill) failed image decode (failed to process speculative batch), so that vision profile stays MTP.
• Gemma 4 ships its MTP head as a separate gemma4-assistant GGUF used via -md. On this APU that head wants a much larger draft depth than Qwen: on gemma-31b, n=12 is 106 tok/s cold unique and 112 tok/s on the 33-check battery (33/33), while n=3 was 75 tok/s and n=16 collapsed to 69. The 12B Gemma 4 profile that shared this head was removed as superseded by gemma-moe; the 31B profile below stays because it is a different point in the family.
• Gemma 4 31B QAT is the dense 30B-class pick with a Gemma MTP head. gemma-31b (Unsloth UD-Q4_K_XL + MTP Q8_0, n=12, plus ngram-map-k,ngram-mod) scored 40/40 at 45 tok/s mixed and 201 tok/s at 24k. MTP-only was also 40/40 at 45 tok/s; ngram kept that score and raised a same-prompt follow-up to 114 tok/s from 41. n=6 was 45 tok/s cold. n=16 collapsed to 31 tok/s. Slower than DFlash dense-27b (65 tok/s) but the higher 40/40 score and a much larger context than the MoE pick justify keeping both. ngram on gemma-e2b dropped mixed to 34/40, so that profile stays MTP-only.
• Quantisation degrades the MTP head, not just the weights. The IQ3_S build of Qwen3.8-27B accepts 58% at MTP depth 3 against 76% for the Q4 build, so MTP shipped at depth 1. DFlash2 n=7 plus ngram now replaces that: unique 42 tok/s versus 22, mixed 39/40 at 36 tok/s.
• Draft depth has to follow the measured accept rate, per model: the 9B dense holds 86% (depth 3, +12% over depth 1) and Qwen3.8-27B holds 76% (depth 3, +19% over depth 1), while Ornith-1.5-35B dropped to 38% at depth 3 and had to ship depth 1.
• Small dense models are not automatically faster. The 9B Ornith without DFlash was 47 tok/s — slower than the 35B-A3B MoE at 67 — because the MoE activates only ~3B parameters per token while the 9B pays for all of them. A dense model still wins on prefill-per-byte for tiny prompts and on memory.
• An MTP head is worth ~60% at the same size. Two 9B dense models, same context, same quantization: Ornith-1.5-9B with MTP at depth 3 managed 52 tok/s where Qwen3.5-9B without a draft managed 32. When choosing between comparable models, prefer one with a usable draft head — it is a bigger lever than quantisation.
• A hybrid attention model is dramatically faster to prefill. LFM2.5-2.6B (tiny) with DSpark still does 2565 tok/s at 24k against ~980-1004 for the 35B-A3B MoEs, because most of its layers carry no KV cache. Without DSpark it was ~3790 tok/s; the sidecar is the right trade for decode. fast-moe (LFM2.5-8B-A1B) prefills at 3232 tok/s.
• Gemma 4 E4B with MTP sits between tiny and gemma-moe. gemma-e4b at MTP n=4 scored 27/30 at 135 tok/s cold (n=3 was 125; n=6 was 126). 24k prefill 1263 tok/s. It passed cycle-detect, anagrams, k-th largest, and rotated search, and missed word_count, work_rate, and rabbits. Its vision projector failed sees_red, so there is no vision-e4b profile.
• Gemma 4 E2B is the fastest shipped chat profile. gemma-e2b (Unsloth QAT UD-Q4_K_XL + MTP Q8_0, n=6) scored 30/33 at 236 tok/s on the mixed battery and 272 tok/s cold unique. 24k prefill 2174 tok/s. n=4 was 231 cold. n=8 was 235. Misses: one_word (Pineapple vs pineapple), mod_exp, rabbits. Faster than tiny (154 tok/s) and gemma-e4b (135).
• Qwen3.8-9B Distill is the mid-size Qwen pick. qwen38-9b (Empero Q4_K_M + the Qwen3.5-9B DFlash Q8 at p-min 0.6) scored 38/40 at 218 tok/s mixed, unique first 141. Misses: median_sorted, catalan8. It replaced small-qwen (Qwen3.5-9B), which scored 34/40 at 134 tok/s on the same DFlash chain. The 4B Distill of the same teacher scored 33/40 at 195 tok/s, so it is not shipped.
• Qwen3.5-4B is the small Qwen pick. qwen-4b (Unsloth UD-Q4_K_XL + Anbeeld DFlash Q8_0 n=15, --spec-draft-p-min 0.6, ngram) scored 37/40 at 198 tok/s mixed and 132 tok/s unique. 24k prefill 1105 tok/s. Misses: word_count, median_sorted, mod_exp. DFlash with default p-min 0 dropped unique_paths (36/40). p-min 0.6 recovered the 37/40 none score at nearly 2× none decode (109 tok/s). Same mixed score as gemma-e2b, slower (232 tok/s). qwen38-9b beats it on score at similar speed.
• Qwen3.5-27B is superseded by dense-27b. On DFlash p-min 0.6 the 3.5 build scored 39/40 at 52 tok/s (unique 45), while dense-27b (Qwen3.6-27B + DFlash) scores 38/40 at 65 tok/s. The one-check score loss is worth the extra 25% decode and the newer generation, so the 3.5 profile was dropped.
• Empero Ridge 27B (3.7 bpw) is not a win on this APU. Same DFlash2 n=7 draft as chat/compact: unique first 34 tok/s, mixed 37/40 at 36 tok/s. Misses: longest_valid_parens, mod_exp, catalan8. compact is 39/40 at 36 and chat is 38/40 at 35. The profile is not shipped.
• GLM-4.7-Flash (30B-A3B) is the default profile (glm47-flash). It is mid-pack on the gauge (37/40, word_count, mod_exp, rabbits) and slow for a 30B-A3B (46 tok/s decode, 300 tok/s at 24k), but it is what bare malaikat serve starts and it is the profile that gets used every day. The GGUF is deepseek2-arch, so it runs ngram only — no MTP or DFlash head is available for it, which is why config.Default() now uses rope_arch: deepseek2, native_ctx: 202752 and spec_type: ngram-map-k,ngram-mod. It does not show the long-horizon agent loop that laguna and coding-next do.
• Qwen3-Next 80B-A3B is not a win on this APU. Hybrid MoE (512 experts, 10 used), no MTP head. 36/36 at 45 tok/s decode and 736 tok/s at 24k. Slower than coding (the Qwen3.8 distill) with the same gauge score. The GGUF can stay on disk. The profile is not shipped.
• coding-next (Qwen3-Coder-Next IQ2_XXS) is slower than coding at the same score. Unique DFlash n=15 was 54 tok/s versus 54 with no draft (accept 35-50%). Mixed 40/40 either way, but prefill at 24k is 618 tok/s versus 986 for coding. The 23 GB IQ2 file fits. It ships because it is a different architecture (qwen3next) rather than a faster option, and DFlash is disabled on it because it hangs. But spec decoding is not off on it: the shipped profile sets spec_type: ngram-map-k,ngram-mod, and serve's own default is -spec-type draft-mtp,ngram-map-k,ngram-mod, so any harness or bench that starts a server without an explicit -spec-type gets one of them. Observed 2026-10-16 in an mcode session on this profile: the server generated at 30 tok/s, then produced nothing for 2m03s (server log: task 482 | n_gen = 100 ... W srv stop: cancel task), with draft acceptance = 0.21429 reported on every response; the client's stall watch then restarted the server and retried until its budget ran out, ending the session mid-task. This is now measured, and the profile ships with speculation off. Controlled A/B, one fresh server per arm, 40 requests of 512 generated tokens each at a ~24k-token prompt:
  • as shipped (ngram): the server stalls mid-generation at request 19 — twice, the same request — after a normal timing line, with silence until the client's 150s timeout. The last thing it logged before the silence was common_ngram_map_begin: refresh map: idx_last_draft=40761, new begin=40341. The second bare request (the token budget is the thing that scales with the map) explains why long sessions die rather than slow down, and why it reproduces around a 40k context.
  • --spec-type none: no stall in either run. One arm ended at request 30 only because the probe injected a mid-prompt rewrite, which forces a full re-prefill of its own; the other hit the probe's own 150s timeout on a cold 38.7k-token first request while its log showed steady 35 tok/s progress — slow, not stuck.
  • Decode is identical either way: 17.2s vs 16.9s per 512 tokens (~30 tok/s), which matches the note above that DFlash n=15 measured 54 tok/s against 54 with no draft. A stall costs a whole session; speculation costs nothing here. coding-next therefore sets spec_type: none, which drops both --spec-type and the draft -md from its argv. To turn speculation back on for an experiment, pass it after --: malaikat serve -profile coding-next -- --spec-type ngram-map-k,ngram-mod. The flag needs two dashes; the single-dash form is rejected with invalid argument, and --spec-draft-n-max 0 does not disable the ngram path.
• EAGLE-3 works on this APU. gpt-oss (ggml-org MXFP4 20B + eagle3 Q8, n=3) scored 38/40 at 76 tok/s and 1233 tok/s at 24k. Draft accept sat near mean length 3. n=6 reached 38/38 and the removed 12B Gemma profile (73). Slower than gemma-31b (45 tok/s, 40/40) and coding (172 tok/s, 40/40). Chaining ngram-map-k,ngram-mod after EAGLE-3 left unique decode flat (~99 tok/s) and raised a follow-up turn to 160 tok/s (repeat 407). The profile ships that chain. On Gemma 4 31B the same method lost: eagle3 Q8 (2.4 GB draft, n=4) was 38/38 at 32 tok/s versus MTP n=12 at 52 tok/s. That probe is not shipped.
• Gemma 4 26B-A4B MoE now ships DFlash n=15. Unique merge 157 tok/s versus 100 with MTP n=6. Mixed 40/40 at 152 tok/s (was 36/36 at 107 on an earlier battery). Follow-up 494 tok/s with ngram. It replaced the dense 12B Gemma 4 profile, which sat at 109 tok/s on the same 40/40 score, and also carries vision-gemma-moe.
• Qwen3.8 distilled into the 35B-A3B MoE leads the 35B-A3B class and is the default. coding ships DFlash n=15 (0.42 GB Qwen3.6-35B-A3B draft). Mixed 40/40 at 172 tok/s. Unique merge 165 tok/s versus 97 with MTP. MTP n=3 was 83-90 tok/s on earlier batteries. It took over the coding name from Ornith, and replaced coding-qwen (Qwen3.6-35B-A3B) which was 40/40 at 84 tok/s on the same draft. coding-tiel on Qwen3.5 DFlash is 40/40 at 177 tok/s mixed (unique 122).
• ngram-mod speeds follow-up turns, not cold unique prompts. Chained after MTP + ngram-map-k on coding-qwen38, the first merge-function request stayed ~93 tok/s. Asking the same prompt again (4 tokens reprocessed) went to 148 tok/s, and a 30-line repeat hit 369 tok/s. The 24-check gauge stayed 24/24. ngram-map-k4v and short-prompt KV f16 did not win, so they are not in the default chain. On coding-qwen3 DFlash n=15 plus ngram is now shipped: unique 185 vs 58 tok/s ngram-only, mixed 37/40 either way (one_word, lcm, mod_exp).
• MoE vision is about 7× the dense 27B vision profile. vision-gemma-moe (same Gemma 4 26B-A4B weights plus mmproj-F16.gguf and DFlash n=15) scored 42/42 including both colour checks, 155 tok/s decode. vision-moe (Qwen3.8 distill, MTP) is 68 tok/s. vision (Qwen3.8-27B plus DFlash2) is 21 tok/s (vision 2/2).
• llama.cpp master a894dae (2026-09-20) beats the August 6d05498 HIP runtime on this APU. The RDNA3.5 MoE tile heuristic (ggml-org/llama.cpp#28935) is the likely cause. Same coding-qwen38 weights and spec, thinking off: short decode 96.8 vs 91.0 tok/s; 24k prefill 1004 vs 869 tok/s (+16%); gauge still 24/24. Rebuild with ./malaikat setup --force.
• Vulkan RADV is not the default on this APU. Same llama.cpp a894dae, same Qwen3.8 35B-A3B weights, unique short decode: Vulkan 111 tok/s vs HIP 73. After an 18k prompt, decode flipped: HIP 71 tok/s vs Vulkan 41. Prefill at 18k was 990 vs 977 tok/s. HIP stays the shipped backend because coding windows are long. Vulkan headers are not a system package here; a side build used user-space Khronos headers.
• Laguna S 2.1 is the quality leader but needs a fork for its drafter. The 118B-A8B MoE passed 40/40 at IQ3_M (51.5 GB) with a full 256k window (56 GiB GTT) — the largest model that fits here, and the only score tied with the best shipped profiles at 22 tok/s. Poolside's DFlash drafter does not load on mainline (done_getting_tensors: wrong number of tensors; expected 76, got 69), and every third-party GGUF re-upload of it has the same 69-tensor layout. Poolside's own laguna branch is dated 2026-08-04, seven weeks behind a894dae, so building it would give back the ~16% RDNA3.5 MoE prefill win to gain ~35% decode. There is no mainline path until upstream's DFlash arch covers Laguna.
• DFlash hangs on qwen3next. With draft-dflash the Coder-Next server intermittently livelocks a slot — the process spins at ~190% CPU forever, no error, reproducible within 5-18 requests (seen on merge_intervals and median_sorted, so it is not prompt-specific) and independent of the ngram chain. Without it the profile is stable and scores 40/40 at 36 tok/s. DFlash buys little on this arch anyway (README: 54 vs 54 tok/s unique), so coding-next ships ngram-only.
• GLM-4.7-Flash loads as deepseek2, not glm4_moe_lite. Its GGUF declares general.architecture = deepseek2 (MLA), so the arch gap in the runtime is moot. No GGUF drafter exists for it (the HF DFlash/Eagle3 drafters are safetensors, and llama.cpp's converter needs a torch install), so it ships ngram-only: 37/40 at 46 tok/s, 192k window.
• Nemotron 3 Nano 30B-A3B is the fastest of the four. The hybrid Mamba2-MoE (nemotron_h_moe) does 111 tok/s decode and 1232 tok/s at 24k — a 24k prefill roughly 3× the other three — because most layers carry no KV cache. 37/40, all 12 misses in the maths category (lcm, rabbits, widgets). It has no MTP head, so ngram-only.
• The f16 KV win is nearly free on the hybrid profiles. Same weights, same prompt, decoded at ~14k depth: nemotron-3-nano 52.3 → 57.0 tok/s and coding-next 32.4 → 36.3 tok/s, and glm47-flash at 192k 28.9 → 32.5 tok/s. Prefill is unchanged in every case (1037 vs 1033, 523 vs 529, 300 vs 296), and f16 is lossless where q8_0 is not. It doubles KV, which is ~1 GB on Nemotron (6 attention layers) and ~2 GiB on Coder-Next, so both ship f16; glm47-flash (MLA, 53 KB/token → +10.4 GB) and laguna (56 GiB of weights already near the 60.7 GiB GTT ceiling) stay on q8_0.

Qwen3.8-Flash-Next

The first release under Qwen's new qwen4exp design (the preview of Qwen4). A 125B MoE with 6B active, hybrid Gated DeltaNet + Qwen Sparse Attention across 48 layers — 12 full-attention, the rest linear — plus a 20M-entry n-gram / per-layer embedding (51B params) and a 4B MTP head. Because only 12 layers carry a KV cache, the 256k window is ~6.75 GiB even at f16.

Two things set it apart from every other profile here:

• It needs the fork. Mainline llama.cpp knows the qwen4exp arch but not its MTP head, so build Unsloth's qwen4exp/mtp branch with scripts/build-qwen4exp-mtp.sh.
• It does not fit at the default GTT. GTT is a slice of the same unified memory, so the weights plus KV must land inside the ~121.5 GiB the box reports. The window defaults to half of RAM (60.7 GiB); UD-IQ4_XS is 87.3 GiB and UD-Q4_K_XL 103.7 GiB, so both want it raised, and it takes ttm.pages_limit as well as amdgpu.gttsize — see Setup.

The n-gram table is marked TENSOR_READ_LAZY (mmap'd, rows read on demand) — the whole point of the design — but llama.cpp's --lazy-mode auto resolves to off on this iGPU: llama-model.cpp falls back to OFF when a device reports no mmap support, and the loader only logs tensor per_layer_token_embd.weight (size = 27465 MiB) lazy read enabled when it is forced on. Left at auto, the whole 26.8 GiB table lands in the model buffer on the GPU. With --lazy-mode on (which both profiles set) the split is:

buffer	size	holds
ROCm0	59.8 GiB	the 49 offloaded layers
CPU_Mapped	26.8 GiB	per_layer_token_embd, lazy and off GTT

so GTT only has to hold ~60 GiB of layers plus the 2.6 GiB MTP head and the KV cache: flash-next-iq4 runs in 74.7 GiB, flash-next-q4 in 91.2 GiB, both at the full 256k window.

Both quants score 40/40 (100%) on the full battery with thinking pinned off — every code task executed, both needles recalled, all 12 maths. They also measure the same speed, which is the interesting part:

Profile	Score	Decode	Prefill @24k	MTP accept
flash-next-iq4	40/40	43.6 tok/s	358 tok/s	53-93% (mean len 3.0-4.9)
flash-next-q4	40/40	44.1 tok/s	371 tok/s	—

Q4_K_XL carries 16 GiB more weights for the same decode, because the extra bytes are mostly in the lazy n-gram table (off GTT) while decode is bound by the 6B of active experts. Prefill is the weaker number — ~360 tok/s against ~990 for the 35B-A3B MoEs — which is what 6B active plus a 16-head n-gram gather per token costs.

Ternary (Bonsai 2)

Bonsai 2 27B puts a 27B hybrid-attention model into 7.2 GB of ternary weights. It runs here, but only through Prism's runtime, and the memory saving does not buy proportional speed:

Build	Size	Decode	Prefill @24k	Gauge
PQ2_0 (2-bit slots)	7.21 GB	21.8 tok/s	277 tok/s	16/18
PTQ1_0 (dense trits)	5.95 GB	17.4 tok/s	252 tok/s	—

• It needs the fork, and there is no mainline path. PQ2_0/PTQ1_0 are ggml types 142/143, added only in PrismML-Eng/llama.cpp; mainline master still ends at 42. The older Ternary-Bonsai-27B-Q2_0.gguf looks mainline-compatible (standard type id 42) but fails with tensor ... has offset X, expected Y — the block layout differs, so it is a fork file too. It no longer loads with the current fork either; use the -2 release.
• The custom kernels are the bottleneck on ROCm, not memory. At 21.8 tok/s the model moves ~157 GB/s, where the dense Q4 27B reaches ~230 GB/s — an ideal kernel would give ~32 tok/s. The fork's ternary matmuls are tuned for CUDA and Metal; they compile for gfx1151 through hipify but do not saturate the bus.
• PTQ1_0 is smaller but slower to decode (17.4 vs 21.8 tok/s) — unpacking trits costs more than the bytes it saves. PQ2_0 is the better default; switch with malaikat serve -profile bonsai -m Ternary-Bonsai-2-27B/Ternary-Bonsai-2-27B-PTQ1_0.gguf.
• Self-speculation does not help. There is no MTP head, and --spec-type ngram-map-k produced 45 drafts with 0 accepted (21.1 vs 21.8 tok/s), so the profile runs with spec_type: none.
• Quality is close but not equal. 16/18, the same band as the IQ3_S build: it missed the train-distance word problem (answered 60, the speed) and the 6k needle, while passing every executed code task and both vision checks. Fewer than half the weights, and about the same answer quality as a 12 GB IQ3 quant.

A separate draft model is a net loss here

The obvious next profile type is a big target paired with a small draft (-md + --spec-type draft-simple). Measured on Qwen3-Coder-30B-A3B (a MoE with no MTP head) paired with same-tokenizer Qwen3 drafts:

Configuration	Decode	Drafts accepted
no speculation	64.5 tok/s	—
ngram-map-k only	64.3 tok/s	0 generated
draft Qwen3-0.6B, depth 1	43.4 tok/s	33/76 (43%)
draft Qwen3-1.7B, depth 3	30.9 tok/s	65/132 (49%)
draft Qwen3-0.6B, depth 3	30.8 tok/s	48/183 (26%)

Every draft configuration is slower than doing nothing, at every depth. The cause is structural rather than a tuning problem: the box is memory-bandwidth-bound, so the draft has to read its own weights on every step, and the target pays a verification pass for every rejected draft. A dense 0.6B draft at Q8 is ~0.64 GB per step against ~1.7 GB for the target's 3B active parameters at Q4 — the draft costs roughly a third of a token, and acceptance below 50% never earns that back. A MoE target is the worst case for drafting, because its whole advantage is how little it reads per token.

Two further reasons not to ship a generic small draft: a foreign draft shifted greedy output on one of the two test prompts (sha changed), so it is not output-preserving either. Speculation is worth using here as a model's own MTP head, verified ngram-map-k, ngram-mod on follow-up turns, and Liquid DSpark on LFM2.5-2.6B only (a 664 MB F16 sidecar that shares embeddings with the target). On tiny, thinking off, 128-token decode, DSpark went from 99 tok/s to 172 tok/s (~1.7×). Chaining ngram-mod raised a same-prompt follow-up to 210 tok/s. Accept was 104/198 (~53%). DSpark is exact: the target verifies every draft token. The same sidecar on LFM2.5-8B-A1B (fast-moe) was 147 vs 142 tok/s on a unique prompt and slower on a repeated line (123 vs 137), so that profile ships ngram-map-k,ngram-mod with no draft file.

Bench vs Ollama

# terminal A
./malaikat serve -config ./coding.yaml -c 256k

# terminal B
./malaikat bench -url http://127.0.0.1:8080 -ollama qwen3.6:35b-a3b -n 128

Optional kernel microbench: bench -kernel -m path/to/model.gguf. The -reuse N probe (on by default, N = 4096) measures follow-up-turn prompt-cache reuse; -reuse 0 skips it.

Measured on Linux (system ROCm HIP gfx1151, ctx 262144, FA on, KV q8, 2048/1024):

Model / path	tok/s
Qwen3.8-35B-A3B Distill, DFlash n=15 + ngram-map-k,ngram-mod, llama.cpp a894dae	~165 tg unique, 172 mixed, 986 pp @ 24k
LFM2.5-2.6B + DSpark F16 + ngram-mod	~172 tg cold, ~210 tg follow-up, 2565 pp @ 24k
Gemma 4 26B-A4B UD-Q4_K_XL + DFlash n=15	~157 tg unique, 152 mixed, 1148 pp @ 24k
Gemma 4 E4B UD-Q4_K_XL + MTP n=4	~135 tg cold, 1263 pp @ 24k
LFM2.5-8B-A1B + ngram-map-k,ngram-mod	~142 tg cold, ~255–730 tg on repeated prompts

Windows (Qwen3.6, lemonade ROCm, 128 completion tokens): tuned MTP n=3 + FA + KV q8 ~73 tok/s vs Ollama ~51.

Re-sweep: scripts/sweep-speed.sh (Linux) or scripts/sweep-speed.ps1 (Windows).

Config

JSON or YAML. See coding.example.yaml; a profile is the same file plus name, description, tags, system_prompt and the hf_* download coordinates. CLI overrides the file. ctx_size accepts 256k, 512k, 1m, or a raw token count.

Full field list (all optional except model):

Field	Meaning
model	GGUF path, relative to the models dir or absolute
mmproj	multimodal projector for vision models (--mmproj)
spec_draft_model	separate draft/MTP head file (-md)
ctx_size, native_ctx, rope_arch	window, training window, YaRN KV prefix
batch, ubatch, n_gpu_layers, threads	throughput knobs
flash_attn, cache_type_k, cache_type_v	attention and KV cache
spec_type, spec_draft_n_max	speculative decoding chain and depth
reasoning	on/off/auto thinking for chat
system_prompt, system_prompt_file	prompt injected for clients that send none
hf_repo, hf_file, hf_draft_repo, hf_draft_file, hf_mmproj, hf_extra	what malaikat pull downloads (hf_draft_repo defaults to hf_repo; hf_extra is companion files that are not passed to llama-server)
head, tokenizer	classify decision head and tokenizer.json (Go /predict; siblings of the GGUF if omitted)
secondary	profile name to start as a sidecar (-secondary NAME, -no-secondary)
env, extra_args, host, port, alias, parallel	platform and server wiring

Release build

scripts/release.sh              # → dist/malaikat-0.2.0-linux-amd64
scripts/release.sh --publish    # tag v0.2.0 + GitHub release (needs gh auth)

.\scripts\release.ps1           # → dist\malaikat-0.2.0-windows-amd64.exe