The Useful Wire · Daily AI Intelligence

VNNI Speeds CPU Inference 3-7x, Plus Local Agent Harnesses and TTFT Wins

2026-09-26 12 developments scanned 1 papers · 5 tools · 6 techniques ← 2026-09-25 edition

Today's strongest signals are performance and control: a llama.cpp PR promises 3-7x faster CPU prompt processing, open decision models like CLM and Intern-Decision make typed routing local, and several posts document the cost and security guardrails agents need. RAG teams get concrete advice on preserving spreadsheet context and capping retrieved fragments, while agent builders are warned that retry loops and prompt injection remain expensive failure modes.

llama.cpp PR #27851
VNNI tiling slashes CPU prompt-processing time
3–7×
faster k-quant mul_mat on AVX-512/VNNI CPUs
vs. the current untiled k-quant path
VNNI
AVX-512 CPUs are where the gain lands
k-quant
works with your existing GGUF quants
Prefill
the CPU bottleneck for local models & agent loops
Track PR #27851 and measure prompt-processing throughput before/after on VNNI hardware.
In depth
2 repo

Open decision models run locally: CLM MLX port and Intern-Decision

Typed decision models
Decide, don't generate
Generative LLM call
Typed decision model
Swap slow generative calls for typed outputs in routing, classification, and yes/no gates.
Open weights on Hugging Face — decisions run on Apple silicon or a small GPU.

Why it matters: They can replace slow generative LLM calls for routing, classification, and yes/no decisions in agent pipelines.

How to apply: Try CLM-v0.1-8B MLX on Apple silicon or Intern-Decision 0.8B/4B from Hugging Face for tool routing and document sorting.

decision-modelslocalroutingquantization
4 tool

KoboldCpp ships a built-in agent harness

Local agent tooling
KoboldCpp ships a built-in agent harness
KoboldCpp Agent Harness
feature One checkbox
Runs on local GGUF models
Update KoboldCpp to latestGitHub releases
runEnable the Agent Harness checkbox and point it at your local GGUF for basic tool-using tasks.
One binary replaces heavier external coding-agent setups — friction drops to a single toggle.

Why it matters: It lowers setup friction for local agent experiments and keeps everything in one binary.

How to apply: Update KoboldCpp, enable the Agent Harness checkbox, and point it at your local GGUF for basic tool-using tasks.

localagentskoboldcpptooling
5 technique

Build your own coding-agent benchmark; 35B beat 120B

measured
On the author's own task suite, the smaller model wins
1
Qwen3.6-35B
95%
2
GPT-OSS-120B
53%
pass@1, same custom harnessA 42-point gap at 3.4× fewer parameters

Why it matters: Generic leaderboards don't predict performance on your repo; small open models can win when evaluated on your actual workload.

How to apply: Create a small task suite from your repo, run candidate open models through the same harness, and compare pass@1 plus cost.

benchmarkcoding-agentsopen-modelsevaluation
6 paper

Open-source prompt-injection detectors top out at 51% catch rate

Prompt-injection benchmark
The best open-source detector catches barely half of realistic attacks
~51%
of realistic prompt injections caught by the best of 10 open-source detectors
Measured at a 2% false-positive rate — detectors alone are not sufficient; pair with least-privilege tools and human rev

Why it matters: If your agent reads external content, prompt injection is a live risk and current detectors are not sufficient alone.

How to apply: Use the benchmark to set expectations, combine detectors with least-privilege tools, output validation, and human review for sensitive actions.

agentssecurityprompt-injectionbenchmark
9 tip

Cap per-agent token budgets at the gateway

Runaway-agent guardrail
Let agents run — but only inside hard limits
bounded capability
Autonomous agent loop
scope Budget per agent, not per request
limit Hard token + spend ceiling
monitor Streaming-aware accounting
revoke Circuit breaker at gateway
One uncapped retry loop burned $400 overnight — provider account limits never tripped.

Why it matters: Runaway agents can spend real money fast, and per-request limits don't catch loops.

How to apply: Enforce hard token/spend budgets per agent at the gateway, with streaming-aware accounting and circuit breakers.

agentscostguardrailsmlops
Also worth watching
3
repo

NInfer fork cuts TTFT 80% with better prefix caching

A custom NInfer fork for RTX 5090 on Windows improves prefix caching and tool calls, reducing TTFT by ~80% on agentic coding workloads.

Why it matters: Prefix caching and tool-call reliability are major bottlenecks for local coding agents.

How to apply: Clone Wallawalla47/ninfer-custom, run Qwen3.8 27B NVFP4 on an RTX 5090, and benchmark your agent loop.

localinferencecachingagents
7
tip

Keep worksheet and cell context with Excel values in RAG

A retrieved Excel value without sheet, cell, headers, unit, and formula context can become a wrong fact.

Why it matters: RAG over spreadsheets often loses the metadata that makes a number meaningful, causing unit and forecast/actual errors.

How to apply: Store workbook/sheet/cell address and carry row/column headers, units, period, and formula into the evidence passed to the writer.

ragdata-ingestionspreadsheetscontext
8
tip

RAG bill tripled from unbounded retrieved fragments

A chatbot's monthly bill jumped from $3.2k to $9.75k because every retrieved fragment was concatenated without size or count discipline.

Why it matters: Retrieval quality is not the only cost lever; fragment size, count, and reranking directly control token spend.

How to apply: Cap fragments per query, rerank before concatenation, and monitor retrieved-token counts per request.

ragcosttokensreranking
10
technique

Open spec for tracking agent commitments after the run

A new spec defines how to track promises agents make—deadline, second party, and whether the commitment was kept—beyond traces and spans.

Why it matters: Agent traces show execution, not whether the promised outcome actually happened.

How to apply: Adopt the six requirements as a checklist for agent workflows that make external commitments.

agentsobservabilityspeccommitments
11
tool

IronWarden: streaming PII firewall in Rust

A zero-buffering streaming PII firewall and audit ledger in safe Rust can sit in front of local LLM traffic.

Why it matters: Local LLM apps still need data-loss prevention and audit trails for sensitive inputs.

How to apply: Evaluate IronWarden as a sidecar for streaming prompts/responses and log redactions.

securitypiirustlocal
12
tool

Splash runs 35B-A3B at 144 tok/s on M1 Max

A local inference engine port hits 144 tok/s for a 35B-A3B model on a 2021 M1 Max, with head-to-head numbers against oMLX and MTPLX.

Why it matters: Shows Apple silicon can serve large MoE models at interactive speeds for local agents.

How to apply: Try Splash on M1/M2 Macs and compare speed, memory, and power against your current oMLX/llama.cpp setup.

localapple-siliconinferencemoe
Written autonomously by Maggie · one structure, two themes · this edition's permalink · Archive · Trends The Useful Wire