Qwen3.8 Speed Recipes, Gemma 4 Heatmaps, and GhostSwitch LoRA Routing Fit 16GB Macs

Today's findings

1. #1 Qwen3.8-27B Speed Recipes: Native llama.cpp and Smarter GPU Splitstechnique

   Two local benchmarks show Qwen3.8-27B jumping from about 69 to 100 tok/s short-context and 26 to 131 tok/s on code by changing the serving stack and GPU placement.

   LOCAL INFERENCE · QWEN3.8-27B

   Rebuild native, split smarter — the speedup is free before any new hardware

   5×

   faster code generation: 26 → 131 tok/s

   Short-context: 69 → 100 tok/s (+45%)

   2

   fixes: native llama.cpp build + smarter GPU layer splits

   131

   tok/s peak on code after tuning

   $0

   hardware spend

   Benchmark your app wrapper against a native build, then rebalance layer splits so the fast GPU holds the hot layers.

   Why it matters: Most local teams lose throughput to default app wrappers and naive tensor splitting; these are free performance gains before buying hardware.

   How to apply: Rebuild llama.cpp natively with the right CUDA flags, benchmark LM Studio versus native, then test layer and offload splits so the fastest GPU handles hot layers and the slower PCIe card avoids becoming the bottleneck.

   local-llmllama.cppinferenceperformance

   Read more: Qwen3.8-27B Q6 on one RTX 5090: 69 → 100 tok/s short, 35 → 64 tok/s at 120K context, just by leaving LM Studio for a native llama.cpp build (full benchmarks + recipe) · Qwen3.8 27B went from 26 tk/s to 131 tk/s on code on my box. Same model, same two GPUs, I just stopped doing the obvious thing

2. #2 Gemma 4 26B-A4B Runs on a 16GB Mac via Expert Heatmapstechnique

   Kraits, a stock llama.cpp plugin, keeps only the hottest experts per layer on GPU so Google's 14.4GB Gemma 4 26B-A4B file runs offline on a 16GB M4 Mac mini.

   Gemma 4 26B-A4B · Kraits expert heatmaps

   14.4GB model runs offline in 16GB unified memory

   HOT

   GPU (unified memory) Hottest experts per layer, heatmap-pinned

   COLD

   System RAM Remaining experts, loaded on demand

   No weight cuts — swap the heatmap per task domain instead of maintaining separate model cuts.

   Why it matters: It shows how to run a larger MoE model on consumer unified memory without cutting weights, by exploiting per-task expert routing.

   How to apply: Use the stock QAT Q4_0 GGUF, generate a workload-specific expert heatmap, pin the top experts per layer to GPU, and swap heatmaps for different task domains instead of maintaining separate model cuts.

   local-llmggufmoemac

   Read more: Gemma 4 26B-A4B on a 16 GB Mac: 33/40 on TypeScript tasks, 31 tok/s, offline, using Google's stock file · Gemma 4 26B-A4B on a 16 GB Mac: 33/40 on TypeScript tasks, 31 tok/s, offline, using Google's stock file · Gemma 4 26B-A4B on a 16 GB Mac: 33/40 on TypeScript tasks, 31 tok/s, offline, using Google's stock file

3. #3 GhostSwitch Routes Domain LoRAs Around One Small Base Modeltechnique

   A 16GB-Mac experiment keeps Qwen3.5 9B resident and attaches coding, finance, or legal LoRA adapters per request, falling back to the base model when no specialist fits.

   GHOSTSWITCH · LORA ROUTING

   One resident base, three domain specialists, one fallback

   Resident on a 16GB Mac — a router attaches one specialist per request; misses run on base and get logged.

   Why it matters: LoRA routing lets a small local model cover multiple domains without loading several full models or paying for a larger context window.

   How to apply: Start with a router prompt or small classifier that picks a domain, keep adapters hot-swappable, and log fallback cases so you can add or retrain specialists where the base model underperforms.

   local-llmloraroutingagents

   Read more: I built a LoRA router for Qwen3.5 9B as an experiment: it picks a domain expert for every message

4. #4 NeuronDump Inspects and Steers GGUF Models Layer by Layertool

   NeuronDump opens Ollama/llama.cpp GGUFs, traces a token through layers and heads, lets you zero, scale, or steer parts live, and exports a slimmer GGUF plus Modelfile.

   GGUF interpretability

   Reach inside a local model — without retraining

   1

   Open GGUF

   Ollama / llama.cpp

   2

   Trace token

   through layers & heads

   3

   Edit live

   zero · scale · steer

   steer heads & experts live

   4

   Export

   slimmer GGUF + Modelfile

   Verify each edit: compare baseline vs edited logits on a fixed prompt.

   Why it matters: Local model debugging is usually blind; this gives engineers a practical way to see which heads or experts drive behavior and to prune or steer without retraining.

   How to apply: Point it at a suspect GGUF, compare baseline versus edited logits on a fixed prompt, then use the export path to package a pruned or steered model for repeatable tests.

   ggufinterpretabilityollamatooling

   Read more: LLM inspector NeuronDump · LLM Inspector NeuronDump

5. #5 Giraffe Regression-Tests LLM Serving Changes Against a Baselinetool

   Giraffe is an open-source CLI and local UI that probes OpenAI-compatible endpoints for latency, streaming pauses, answer correctness, JSON and tool-call behavior, and short-versus-long prompt interference.

   Tool · LLM serving regression tests

   Five silent regressions a smoke prompt misses

   Answer quality Correctness drifts silently

   JSON & tools Structured calls break

   Latency Tail latency worsens

   Streaming Pauses mid-stream

   Prompt length Short vs long interference

   Output quality Performance Prompt sensitivity

   Giraffe saves a baseline, re-probes an OpenAI-compatible endpoint after any serving change, and gates deploys on drift.

   Why it matters: Changing batching, inference servers, or models often leaves an endpoint answering but silently degrading structured output or tail latency.

   How to apply: Save a baseline before upgrades, run Giraffe after each serving change, and gate deploys on both correctness and streaming/latency regressions rather than a single smoke prompt.

   llmopstestingservingtooling

   Read more: I built Giraffe to check what breaks when you change your LLM serving setup · I built Giraffe to check what breaks when you change your LLM serving setup

6. #6 TokenGate Paper Claims Up to 11.3x Faster Long-Context Prefillpaper

   Stanford and UC Berkeley researchers propose token-gated attention that skips irrelevant tokens during prefill, reporting large speedups on long-context workloads.

   Why it matters: Prefill cost is the main wall for long-document RAG and agent transcripts; a token-gating approach could make large contexts cheaper without changing the model's generation path.

   How to apply: Read the paper for the gating criterion and kernel assumptions, then prototype it on a fixed long-context eval before assuming the 11.3x number transfers to your model or hardware.

   paperlong-contextattentioninference

   Read more: Researchers from Stanford and UC Berkeley Introduce TokenGate: Token-Gated Attention Makes Long-Context Prefill Up to 11.3x Faster

7. #7 DSpark Speculative Decoding Lifts Qwen3-8B from 59 to 133 tok/s on an RX 7900 GREtechnique

   A local test shows llama.cpp's DSpark speculative decoding more than doubling Qwen3-8B throughput on a consumer AMD GPU, despite the paper's datacenter focus.

   Why it matters: Speculative decoding can be a drop-in local speedup when the draft model and acceptance rate fit the workload, and this confirms it is not only for high-end NVIDIA servers.

   How to apply: Enable DSpark in a recent llama.cpp build, pair a small draft model with your target model, and measure acceptance rate and memory overhead on your own prompts before enabling it broadly.

   local-llmspeculative-decodingllama.cppamd

   Read more: DSpark on an RX 7900 GRE: Qwen3-8B goes from 59 to 133 tok/s in llama.cpp

8. #8 SQLite BM25 and a Rust Sentinel Beat a Vector DB for Multi-Agent Loopstechnique

   A rebuilt open-source multi-agent pipeline replaced its vector DB with SQLite BM25 and added a 3.6ns Rust sentinel to catch bad state before the LLM reasons over it.

   Multi-agent reliability

   Simpler beats the vector DB

   vs

   Vector DB stack

   SQLite BM25 + sentinel

   Keyword-heavy retrieval

   Embedding service

   BM25 in SQLite

   Raw metrics

   LLM interprets

   Hard checks in code

   Stale or bad state

   Reaches the LLM

   3.6ns sentinel rejects

   Fuzzy semantic recall

   Embeddings win

   Lexical only

   Infrastructure

   Dedicated DB to run

   One SQLite file

   Vector DB stack wins the row SQLite BM25 + sentinel wins the row

   Failures traced to raw metrics and stale state — not retrieval quality.

   Why it matters: Agent failures often come from raw metrics and stale state, not retrieval quality; simpler lexical retrieval plus deterministic guards can be more reliable and cheaper.

   How to apply: Audit where the LLM interprets raw numbers, move hard checks into code, try SQLite FTS/BM25 for keyword-heavy retrieval, and keep a fast sentinel that rejects invalid state before it reaches the model.

   agentsragsqlitearchitecture

   Read more: Why we killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: hard lessons from running multi-agent loops · Why we killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: hard lessons from running multi-agent loops

9. #9 PaperPilot Is a Zero-Cloud Rust PDF Engine with WASM and MCPtool

   PaperPilot packages PDF parsing as a pure-Rust, local-first engine with WASM and MCP interfaces, aimed at document workflows that cannot send files to a cloud service.

   PDF PARSING

   Same extraction job, opposite trust model

   Cloud-hosted parser

   • PDFs leave your infrastructure
   • Live dependency on a hosted API
   • Non-starter for sensitive document sets

   vs

   PaperPilot · local

   • Pure Rust — runs fully on-device
   • WASM: embeds straight in the browser
   • MCP: callable from Claude / agent tooling

   Zero-cloud is the feature — docs stay local, agents still call it in via MCP.

   Worth a bake-off on your worst PDFs before swapping parsers.

   Why it matters: RAG and agent pipelines need reliable local document ingestion; an MCP-ready PDF engine can slot into Claude or agent tooling without a hosted parser.

   How to apply: Test PaperPilot on your worst PDFs, compare extracted tables and reading order against your current parser, and expose it as an MCP tool for local agents if the output holds up.

   mcpragpdfrust

   Read more: PaperPilot – Zero-Cloud, Pure-Rust PDF Engine with WASM and MCP

10. #10 Treat Agent Skill Evals Like Software: Holdouts, Repeats, and Separate Diffstip

    Three eval practices from skillEd: keep repair examples out of the judging set, report pass distributions instead of best runs, and review skill changes separately from expected-answer changes.

    Why it matters: Agent behavior is variable; a single green run or a test set edited alongside the prompt can hide regressions and make releases look safer than they are.

    How to apply: Maintain a dev suite and an untouched holdout, run each case multiple times and record failures and timeouts, and require separate review for instruction edits versus expected-output edits.

    evaluationagentstestingprompts

    Read more: Keep the skill diff and the expected-answer diff separate during review · A single successful run is a weak release signal for a variable agent · The examples used to repair a prompt should not be the only examples used to judge it

11. #11 Agent Refactors Can Bury Auth Logic; Review Security Paths Separatelytip

    A ClaudeAI post describes an automated refactor that renamed users to principals and left a new impersonation path three ifs deep inside authorize().

    AGENT REFACTOR RISK

    Buried three ifs deep in authorize()

    authorize()

    one mechanical rename: users → principals

    if #1

    first gate down

    if #2

    second gate down

    if #3

    impersonation path, hidden here

    After agent refactors, diff auth paths explicitly — keep security changes out of cleanup PRs.

    Why it matters: Coding agents are good at mechanical cleanup but can miss security-critical control flow; auth changes need human review and dedicated tests.

    How to apply: After agent refactors, diff authorization paths explicitly, require tests for impersonation and privilege escalation, and keep security-sensitive changes out of broad cleanup pull requests.

    claudesecuritycode-reviewagents

    Read more: The agent “cleaned up” our access check and left impersonation inside it

12. #12 Turn 'Deeper, Not Bigger' Into a Reusable Claude Skilltip

    A PromptEngineering post turns Felix Rieseberg's depth prompt into a skill.md that asks what a top professional would add before expanding scope.

    Why it matters: It gives Claude a concrete way to improve task quality without ballooning the request into a larger project, which is useful for planning and review workflows.

    How to apply: Add the prompt as a project skill, require it to name missing professional moves and non-goals before rewriting, and test it on tasks where scope creep has been a problem.

    claudepromptsskillsplanning

    Read more: Felix Rieseberg's "deeper, not bigger" prompt, turned into a skill.md that knows when NOT to make your task more ambitious