Edition 2026-09-26 latest · digest built 2026-09-26T12:04:49+00:00 · ⚙ ollama:deepseek-v4.1-flash:cloud

VNNI Speeds CPU Inference 3-7x, Plus Local Agent Harnesses and TTFT Wins

Today's strongest signals are performance and control: a llama.cpp PR promises 3-7x faster CPU prompt processing, open decision models like CLM and Intern-Decision make typed routing local, and several posts document the cost and security guardrails agents need. RAG teams get concrete advice on preserving spreadsheet context and capping retrieved fragments, while agent builders are warned that retry loops and prompt injection remain expensive failure modes.

Local Inference Gets Faster

The most immediately useful item is the llama.cpp k-quant matmul PR, which claims 3-7x faster CPU prompt processing with VNNI tiling. NInfer's custom fork shows similar gains on the agent side, cutting TTFT by ~80% through better prefix caching and tool-call fixes. KoboldCpp's new built-in agent harness and Splash's 144 tok/s M1 Max result round out a day of local-runtime improvements.

Decision Models and Agent Guardrails

Open decision models are becoming a practical alternative to generative calls for routing and classification: CLM's MLX port and Intern-Decision both return typed choices or scores locally. But agent builders also got hard lessons: a retry loop burned $400 overnight, prompt-injection detectors caught only about half of realistic attacks, and a new spec argues that traces miss whether an agent actually kept its commitments.

RAG Data and Cost Discipline

RAG teams should preserve spreadsheet context—sheet, cell, headers, units, and formulas—or retrieved numbers can turn into wrong facts. Another production report shows a RAG bill tripling from unbounded retrieved fragments, making fragment size, count, and reranking first-class cost controls. IronWarden adds a Rust streaming PII firewall for teams that need redaction and audit trails in local LLM traffic.

Today's findings

  1. #1 llama.cpp CPU k-quant matmul gets 3-7x faster with VNNI tilingtechnique

    A new llama.cpp PR tiles k-quant mul_mat for VNNI, cutting CPU prompt-processing time by 3-7x.

    llama.cpp PR #27851
    VNNI tiling slashes CPU prompt-processing time
    3–7×
    faster k-quant mul_mat on AVX-512/VNNI CPUs
    vs. the current untiled k-quant path
    VNNI
    AVX-512 CPUs are where the gain lands
    k-quant
    works with your existing GGUF quants
    Prefill
    the CPU bottleneck for local models & agent loops
    Track PR #27851 and measure prompt-processing throughput before/after on VNNI hardware.

    Why it matters: CPU prefill is often the bottleneck for local models and agent loops, especially on machines without enough VRAM.

    How to apply: Track PR #27851, test it with your k-quant GGUF on AVX-512/VNNI CPUs, and measure prompt-processing throughput before/after.

    llama.cppquantizationcpuperformance

    Read more: ggml-cpu: tiled mul_mat for k-quants by jbooth · Pull Request #27851 · ggml-org/llama.cpp

  2. #2 Open decision models run locally: CLM MLX port and Intern-Decisionrepo

    Open-weight typed decision models now run on Macs and small GPUs, returning choices and scores instead of generated text.

    Typed decision models
    Decide, don't generate
    Generative LLM call
    Typed decision model
    Swap slow generative calls for typed outputs in routing, classification, and yes/no gates.
    Open weights on Hugging Face — decisions run on Apple silicon or a small GPU.

    Why it matters: They can replace slow generative LLM calls for routing, classification, and yes/no decisions in agent pipelines.

    How to apply: Try CLM-v0.1-8B MLX on Apple silicon or Intern-Decision 0.8B/4B from Hugging Face for tool routing and document sorting.

    decision-modelslocalroutingquantization

    Read more: CLM-v0.1-8B ported to MLX — frozen Qwen3-8B encoder for instant on-device decisions, 99% top-1 agreement with the original vLLM server · internlm/Intern-Decision 4B and 0.8B

  3. #3 NInfer fork cuts TTFT 80% with better prefix cachingrepo

    A custom NInfer fork for RTX 5090 on Windows improves prefix caching and tool calls, reducing TTFT by ~80% on agentic coding workloads.

    Why it matters: Prefix caching and tool-call reliability are major bottlenecks for local coding agents.

    How to apply: Clone Wallawalla47/ninfer-custom, run Qwen3.8 27B NVFP4 on an RTX 5090, and benchmark your agent loop.

    localinferencecachingagents

    Read more: NInfer with improved prefix caching and tool call fixes

  4. #4 KoboldCpp ships a built-in agent harnesstool

    KoboldCpp now bundles a one-checkbox agentic harness for local models, aiming to replace heavier external coding agents.

    Local agent tooling
    KoboldCpp ships a built-in agent harness
    KoboldCpp Agent Harness
    feature One checkbox
    Runs on local GGUF models
    Update KoboldCpp to latestGitHub releases
    runEnable the Agent Harness checkbox and point it at your local GGUF for basic tool-using tasks.
    One binary replaces heavier external coding-agent setups — friction drops to a single toggle.

    Why it matters: It lowers setup friction for local agent experiments and keeps everything in one binary.

    How to apply: Update KoboldCpp, enable the Agent Harness checkbox, and point it at your local GGUF for basic tool-using tasks.

    localagentskoboldcpptooling

    Read more: Introducing KoboldCpp Agent (and a plea for help)

  5. #5 Build your own coding-agent benchmark; 35B beat 120Btechnique

    A custom harness showed Qwen3.6-35B beating GPT-OSS-120B 95% vs 53% on the author's tasks, and pay-per-token beating rented GPUs.

    measured
    On the author's own task suite, the smaller model wins
    1
    Qwen3.6-35B
    95%
    2
    GPT-OSS-120B
    53%
    pass@1, same custom harnessA 42-point gap at 3.4× fewer parameters

    Why it matters: Generic leaderboards don't predict performance on your repo; small open models can win when evaluated on your actual workload.

    How to apply: Create a small task suite from your repo, run candidate open models through the same harness, and compare pass@1 plus cost.

    benchmarkcoding-agentsopen-modelsevaluation

    Read more: I built a custom benchmark for my coding agent: the 35B model beat the 120B one, and pay-per-token beat renting GPUs · A 35B model beat a 120B one on my coding agent, 95% vs 53%. Build your own benchmark. · Benchmarking became easy

  6. #6 Open-source prompt-injection detectors top out at 51% catch ratepaper

    A benchmark of 10 open-source injection detectors found the best caught only ~51% of realistic attacks at 2% false positives.

    Prompt-injection benchmark
    The best open-source detector catches barely half of realistic attacks
    ~51%
    of realistic prompt injections caught by the best of 10 open-source detectors
    Measured at a 2% false-positive rate — detectors alone are not sufficient; pair with least-privilege tools and human rev

    Why it matters: If your agent reads external content, prompt injection is a live risk and current detectors are not sufficient alone.

    How to apply: Use the benchmark to set expectations, combine detectors with least-privilege tools, output validation, and human review for sensitive actions.

    agentssecurityprompt-injectionbenchmark

    Read more: I benchmarked 10 open-source prompt-injection detectors. The best caught 51%. How are you actually defending your agents?

  7. #7 Keep worksheet and cell context with Excel values in RAGtip

    A retrieved Excel value without sheet, cell, headers, unit, and formula context can become a wrong fact.

    Why it matters: RAG over spreadsheets often loses the metadata that makes a number meaningful, causing unit and forecast/actual errors.

    How to apply: Store workbook/sheet/cell address and carry row/column headers, units, period, and formula into the evidence passed to the writer.

    ragdata-ingestionspreadsheetscontext

    Read more: An Excel value needs its worksheet context before it becomes a RAG fact

  8. #8 RAG bill tripled from unbounded retrieved fragmentstip

    A chatbot's monthly bill jumped from $3.2k to $9.75k because every retrieved fragment was concatenated without size or count discipline.

    Why it matters: Retrieval quality is not the only cost lever; fragment size, count, and reranking directly control token spend.

    How to apply: Cap fragments per query, rerank before concatenation, and monitor retrieved-token counts per request.

    ragcosttokensreranking

    Read more: Added RAG to our chatbot, bill went from $3,200 to $9,750 in two weeks

  9. #9 Cap per-agent token budgets at the gatewaytip

    An agent retry loop with no ceiling burned $400 overnight; provider account limits didn't stop it.

    Runaway-agent guardrail
    Let agents run — but only inside hard limits
    bounded capability
    Autonomous agent loop
    scope Budget per agent, not per request
    limit Hard token + spend ceiling
    monitor Streaming-aware accounting
    revoke Circuit breaker at gateway
    One uncapped retry loop burned $400 overnight — provider account limits never tripped.

    Why it matters: Runaway agents can spend real money fast, and per-request limits don't catch loops.

    How to apply: Enforce hard token/spend budgets per agent at the gateway, with streaming-aware accounting and circuit breakers.

    agentscostguardrailsmlops

    Read more: Agent retry loop burned 400 dollars overnight, are per agent token budgets at the gateway the fix

  10. #10 Open spec for tracking agent commitments after the runtechnique

    A new spec defines how to track promises agents make—deadline, second party, and whether the commitment was kept—beyond traces and spans.

    Why it matters: Agent traces show execution, not whether the promised outcome actually happened.

    How to apply: Adopt the six requirements as a checklist for agent workflows that make external commitments.

    agentsobservabilityspeccommitments

    Read more: We published an open standard for AI agent commitment tracking. LangChain isn't compliant yet here's the spec.

  11. #11 IronWarden: streaming PII firewall in Rusttool

    A zero-buffering streaming PII firewall and audit ledger in safe Rust can sit in front of local LLM traffic.

    Why it matters: Local LLM apps still need data-loss prevention and audit trails for sensitive inputs.

    How to apply: Evaluate IronWarden as a sidecar for streaming prompts/responses and log redactions.

    securitypiirustlocal

    Read more: IronWarden: A zero-buffering streaming PII firewall & audit ledger in safe Rust

  12. #12 Splash runs 35B-A3B at 144 tok/s on M1 Maxtool

    A local inference engine port hits 144 tok/s for a 35B-A3B model on a 2021 M1 Max, with head-to-head numbers against oMLX and MTPLX.

    Why it matters: Shows Apple silicon can serve large MoE models at interactive speeds for local agents.

    How to apply: Try Splash on M1/M2 Macs and compare speed, memory, and power against your current oMLX/llama.cpp setup.

    localapple-siliconinferencemoe

    Read more: Splash on M1, part 2: 35B-A3B at 144 tok/s on a 2021 M1 Max, plus a head-to-head with oMLX and MTPLX (speed, temperature, power, memory)

Looking for topic trends and crawl volume over time? See Trends.