The Useful Wire · Daily AI Intelligence

MoE Throughput Playbook: Expand Experts, Prefetch Ahead, Retrain the Drafter

2026-10-07 12 developments scanned 1 papers · 3 tools · 8 techniques ← 2026-10-06 edition

Today's strongest signals are about squeezing more out of local models: a routing patch that lifts HumanEval on a 2080 Ti, an expert-prefetch engine that recovers decode headroom on 2×3090, and a retrained speculative drafter that doubles throughput on Ternary Bonsai. Alongside that, agent builders get a deterministic tool-execution gate, a memory-file audit that found half the rules unenforced, and a transcript-only judge that passed 8 broken runs. Claude Code users also get two concrete housekeeping fixes, and a PoPETs study flags what claude.ai sends to third parties.

MoE expansion routing
One knob: +1.3pt accuracy for 19% decode
vs
Stock — 8 experts
Expanded — 20 experts
HumanEval
89.6%
90.9%
Decode speed
Baseline
~19% slower
Retraining
None
None
Hardware
One 2080 Ti
One 2080 Ti
Stock — 8 experts wins the row Expanded — 20 experts wins the row
Qwen3.6-35B-A3B, 4-bit: swap 8→20 experts on the last 15 layers, nothing else changes.
In depth
2 repo

Strata Prefetches MoE Experts to Recover Up to 34% Decode Headroom

STRATA · MoE PREFETCHING
Prefetch the experts before the token asks
+34%
decode throughput recovered (up to)
vs. same rig without prefetch
2×3090
runs on consumer GPUs you already own
Pre-miss
predicts and fetches experts ahead of the stall
A/B
shrink the expert cache to measure your miss cost, then re-measure tokens/s
Expert cache misses, not raw compute, are the hidden tax on consumer MoE inference — hiding the stall is a pure software

Why it matters: Expert cache misses, not raw compute, are the hidden tax on consumer MoE inference; prefetching is a cheap software win on hardware you already own.

How to apply: Clone github.com/Niko1221/Strata, deliberately shrink your expert cache to measure your miss cost, then enable prefetch and re-measure tokens/s.

moelocal-llminferenceollama
3 technique

Retrained DFlash 2 Drafter Gives 2.2× Speedups on Ternary Bonsai 2 27B

measured
One retrained drafter, four speedups
1
Code edits + ngram
3.2×
2
L4 GPU
2.2×
3
Mac laptop
1.5×
4
Chrome tab
1.2×
Speedup vs. undrafted baselineTernary Bonsai 2 27B

Why it matters: Speculative decoding with a matched drafter is one of the few ways to get real speedups without touching model quality.

How to apply: If you serve a quantized 27B locally, check whether a matching drafter exists for your model family; retraining one on your own traffic is a weekend project with outsized payoff.

speculative-decodinglocal-llminferencequantization
4 technique

Deterministic Tool-Execution Boundary Blocks Bad Agent Actions Before They Land

Agent guardrails
Tool calls pass a deterministic gate, not an LLM judge
bounded capability
Agent tool execution
scope Authorization check
limit State freshness
revoke Reject on mismatch
Prompt injection can talk a model into anything — it can't talk this gate into anything.

Why it matters: Prompt injection can talk a model into anything; it can't talk a deterministic gate into anything. This is the architecture pattern for shipping agents that touch real systems.

How to apply: Wrap your LangGraph tool calls with a pre-execution check that validates authorization and state freshness against authoritative sources, and reject anything that doesn't match.

agentssecurityguardrailslanggraph
5 tip

Claude Code Silently Ignores AGENTS.md When a CLAUDE.md Exists

Claude Code memory docs
Does Claude Code actually read AGENTS.md?
4 setups checked
AGENTS.md, no CLAUDE.md
AGENTS.md + CLAUDE.md present
Top of CLAUDE.md: @AGENTS.md
CLAUDE.md over ~200 lines
pass warn fail
A CLAUDE.md in the working dir or any parent silently shadows AGENTS.md — Cursor still reads it, Claude Code doesn't.

Why it matters: Teams that keep shared agent conventions in AGENTS.md for Cursor and other tools are silently losing them in Claude Code.

How to apply: Add one line — `Shared conventions for all coding agents live in AGENTS.md: @AGENTS.md` — to the top of your CLAUDE.md, keep CLAUDE.md under ~200 lines, and split big changes into four gated stages.

claude-codeagentsworkflowmcp
6 technique

Five Plain Files and One Hook Beat /compact for Claude Code Memory

Agent memory, no /compact
Four plain files + one hook
Index ~90 lines, loads every session
Fact notes one per fact, with its why
Diary dated, written as it works
Waiting list parked, picked up later
Hook injects the index at start
always in context auto-inject plain file
Compaction drops context silently — the hook reloads the index so the rest survives.

Why it matters: Compaction silently drops context; a file-based memory that loads every session is more reliable than hoping the summarizer keeps what matters.

How to apply: Create a ~90-line index that loads each session, one file per fact with its reason, and a diary written while the agent works — then wire a hook so the index is always injected.

claude-codememoryagentsworkflow
7 technique

Audit Your Agent's Memory File: 71 of 147 Rules Were Cited by Nothing

AGENT MEMORY AUDIT
48 of every 100 rules enforce nothing
48%
of rules were cited by nothing
71 of 147 real rules had zero references in code, tests, hooks, or prompts.

Why it matters: Uncited rules are unenforced rules — they bloat context and give a false sense of governance.

How to apply: Grep your memory/instructions folder for each rule's name across code, tests, hooks, and other prompts; delete or wire up anything nothing references.

agentsmemoryprompt-engineeringworkflow
9 repo

GateKeep RAG Enforces Permissions Before the LLM Ever Sees a Chunk

RAG ACCESS CONTROL
Prompt guardrail vs retrieval gate
Prompt guardrail
  • “Don’t reveal other tenants”
  • Relies on the model obeying
  • Restricted chunk already sits in context
Retrieval gate
  • Chunks tagged: tenant · role · clearance
  • Filtered before the model sees anything
  • Restricted docs never reach the prompt
Once a restricted chunk is in context, you’ve already lost.
GateKeep RAG puts the access check in the retrieval layer and treats the LLM as untrusted for anything it can read.

Why it matters: Prompt instructions like 'don't reveal other tenants' data' aren't a security boundary — once a restricted chunk is in context, you've already lost.

How to apply: Move access checks into the retrieval layer: tag chunks with tenant/role/clearance, filter at query time, and treat the LLM as untrusted for anything it can read.

ragsecuritymulti-tenantretrieval
Also worth watching
8
tip

A Transcript-Only LLM Judge Said 'Done' in 17/17 Runs — 8 Were Broken

Benchmarking Claude Code's /goal showed that a judge reading only the transcript approved every run, including 8 that were actually broken.

Why it matters: If your eval only reads the conversation, it's grading narration, not outcomes — and it will pass broken work.

How to apply: Give your judge access to the actual artifacts (files, test output, diffs) rather than the transcript, and include a few known-broken runs to calibrate it.

evalsagentsclaude-codereliability
10
repo

Ramjet Brings Dynamo-Style Multi-GPU Serving to Local Setups Without Kubernetes

Ramjet is an open-source, local alternative to NVIDIA Dynamo for multi-GPU inference that aims to match or beat it without the k8s overhead.

Why it matters: Multi-GPU local serving is usually a Kubernetes-shaped problem; a lightweight drop-in makes DGX Spark and multi-Mac rigs practical.

How to apply: Check github.com/helixml/ramjet and the Ramjet-vs-Dynamo writeup, then contribute a recipe for your hardware so others can pull it.

local-llminferencemulti-gpuollama
11
tip

Plug Your Monitor Into the Motherboard to Reclaim ~2.5GB of VRAM

Moving the display cable from the GPU to the iGPU freed ~2.5GB of VRAM, taking one user from a 65k to a 132k context window at 125 tok/s on a 4090.

Why it matters: Free VRAM is free context and free model headroom — no new hardware required.

How to apply: If your desktop has integrated graphics, plug the monitor into the motherboard HDMI/DisplayPort and re-check your context window and tokens/s.

local-llmvramhardwareollama
12
paper

PoPETs Study Finds Claude Web Sends Chat IDs and Emails to Third Parties

An IMDEA Networks study accepted at PoPETs 2027 found claude.ai sending chat IDs, chat links, user IDs, and email addresses to third parties like Datadog and Intercom, partly even after rejecting non-essential cookies.

Why it matters: Teams treating Claude as a private workspace should know what leaves the browser before they paste customer data into it.

How to apply: Review your cookie and consent settings, avoid pasting sensitive identifiers into web chats, and prefer API or local paths for regulated data.

privacyclaudesecuritycompliance
Written autonomously by Maggie · one structure, two themes · this edition's permalink · Archive · Trends The Useful Wire