Edition 2026-09-06 latest · digest built 2026-09-06T12:05:01+00:00 · ⚙ fallback: ollama:deepseek-v4-flash:cloud

Local-first agent tooling, KV-cache tricks, and fast uncertainty estimation

Today's digest is heavy on practical local and Claude-centric tooling: a sliding-window attention implementation to shrink KV caches, a fast lexical entropy method for uncertainty, new agent permission and routing tools, and performance tips for Apple Silicon and Claude Code. These are concrete, open-source pieces you can adopt this week.

Local inference and memory

The day's most actionable work is about making local LLMs cheaper and faster. The sliding-window attention repo lets you bound KV cache to attention sinks plus a recent window, enabling longer contexts on modest GPUs. On Apple Silicon, Ollama's new MLX support delivers ~80 tok/s for Qwen 3.6 35B on an M1 Ultra, while a careful benchmark shows speculative decoding can actually slow things down on M2. If you have multiple machines, NVIDIA's Personal AI Router can pool Ollama and LM Studio instances across your network.

Agent tooling and safety

Agent orchestration is maturing. Pigeon introduces signed passes to constrain sub-agent actions, and Routed provides a sub-20ms local router for skill selection, saving tokens and latency. For Claude Code users, a new CLI helps manage skill packs without conflicts, and the Project Athena pattern shows how to externalize memory and verification to avoid token burn. These tools directly address the pain of building reliable multi-agent systems.

Uncertainty and evaluation

On the evaluation side, the Spanda paper shows that a normalized lexical self-consistency ratio can match DeBERTa-based semantic entropy for hallucination detection at a fraction of the compute. This is a practical win for real-time monitoring of agent outputs. The open-source benchmark lets you test it on your own models.

Today's findings

  1. #1 Sliding Window Attention for Hugging Face LLMsrepo

    Implement sliding window attention on pretrained HF LLMs to bound KV cache and enable longer contexts on limited VRAM.

    KV-CACHE BOUND
    Sliding window attention: keep only the recent past
    Full attention
    • KV cache grows with context
    • All tokens attended
    Sliding window
    • Only sinks + recent window
    • Cache size bounded
    Drop-in inference layer for HF causal LLMs

    Why it matters: Long-context inference is memory-bound; SWA keeps only attention sinks and a recent window, cutting cache size dramatically.

    How to apply: Use the open-source SWA repo (github.com/oraby8/SWA) as a drop-in inference layer for Hugging Face causal LLMs; test with your own models.

    kv-cacheinferencememorylocal-llm

    Read more: I implemented Sliding Window Attention for Hugging Face LLM inference — looking for feedback · Applying Sliding Window Attention to pretrained LLMs at inference time [P] · I implemented Sliding Window Attention for Hugging Face LLM inference — looking for feedback · I implemented Sliding Window Attention for Hugging Face LLM inference — looking for feedback

  2. #2 Spanda: Fast Lexical Entropy for Uncertaintypaper

    A normalized lexical self-consistency ratio (R_sc) matches DeBERTa-based semantic entropy for hallucination detection at a fraction of the cost.

    Uncertainty detection
    Spanda matches semantic entropy at a fraction of the cost
    vs
    Semantic Entropy (DeBERTa)
    Spanda (R_sc)
    Detection quality
    baseline
    matches
    CPU time
    100+ sec
    fraction
    Semantic Entropy (DeBERTa) wins the row Spanda (R_sc) wins the row
    R_sc matches DeBERTa-based semantic entropy for hallucination detection at a fraction of the cost.

    Why it matters: Semantic entropy is expensive (100+ sec CPU clustering); Spanda gives a fast, open-source alternative for real-time uncertainty.

    How to apply: Run the Spanda benchmark (github.com/Adarshent/Spnda) on your models to see if R_sc works for your use case; integrate as a lightweight uncertainty signal.

    uncertaintyhallucinationevaluationopen-source

    Read more: Evaluating lexical vs neural semantic entropy across 1.5B to 120B models · [R] Spanda: Fast Lexical Entropy Matches DeBERTa Semantic Uncertainty—Until Frontier Models Break It · [R] Spanda: Fast Lexical Entropy Matches DeBERTa Semantic Uncertainty—Until Frontier Models Break It

  3. #3 Pigeon: Signed Pass for Sub-Agent Permissionstool

    Pigeon is a signed pass system that constrains what a sub-agent may do, adding a security layer to agent orchestration.

    Agent security
    Signed passes bound sub-agent actions
    bounded capability
    Sub-agent actions
    scope Scoped to task
    limit Limited blast radius
    revoke Revocable at will
    Pigeon issues signed, scoped passes so sub-agents act only within defined limits.

    Why it matters: As agents get more autonomy, blast-radius control is critical; signed passes let you enforce permissions without hardcoding.

    How to apply: Adopt Pigeon in your agent harness to issue scoped, signed capabilities to sub-agents; see github.com/pigeonlabsHQ/pigeon.

    agentssecuritypermissions

    Read more: Pigeon, a signed Pass for what a sub-agent may do

  4. #4 Routed: Local-First Hybrid Router for Agent Skillstool

    Routed is a local-first, sub-20ms router that selects the right agent skill without burning tokens or adding latency.

    Local-first agent router
    Route skills in under 20ms — no tokens, no LLM call
    sub-20ms
    router latency on CPU
    0 tokens burned
    0
    tokens spent
    CPU
    runs on-device
    Routed classifies prompts locally before the LLM call.

    Why it matters: Stuffing all skill definitions into the system prompt degrades instruction-following; an LLM router costs tokens and time. Routed runs on CPU.

    How to apply: Install Routed (github.com/bshea-1/Routed) and point it at your skill/rule files; it classifies prompts locally before hitting the LLM.

    agentsroutingskillslocal

    Read more: I built a local-first hybrid router for AI Agent Skills (sub-20ms, zero tokens, runs on CPU)

  5. #5 NVIDIA Personal AI Routertool

    NVIDIA's Personal AI Router routes Ollama and LM Studio inference across all machines on your home network.

    Local LLM
    NVIDIA Personal AI Router
    RouterOllamaLM Studio
    Routes each request to the best available device across your home network.

    Why it matters: If you have multiple GPUs or Macs, you can pool them and route requests to the best available device.

    How to apply: Set up the router on your network and configure it to discover Ollama/LM Studio instances; use it to balance load and maximize throughput.

    local-llmroutingollamalm-studio

    Read more: NVIDIA Personal AI Router - routes Ollama and LM Studio inference across every machine on your home network

  6. #6 Ollama MLX Support for Qwen 3.6tip

    Ollama 0.33.3 adds MLX support, giving ~80 tok/s for Qwen 3.6 35B on an M1 Ultra Mac Studio.

    Why it matters: Apple Silicon users can now get near-native performance from Ollama without switching to MLX directly.

    How to apply: Update Ollama to 0.33.3+ and pull the Qwen 3.6 35B model; test on your Mac to see the speedup.

    ollamamlxapple-siliconperformance

    Read more: new Ollama for qwen3.6:35b on Mac Studio m1 ultra 64G got ~80 tok/s · new Ollama for qwen3.6:35b on Mac Studio m1 ultra 64G got ~80 tok/s

  7. #7 Speculative Decoding on Apple Silicon: Negative Resulttip

    Speculative decoding on an M2 Mac with Ollama was 5-10% slower than no draft, contradicting server-side benchmarks.

    Why it matters: Draft models add overhead on memory-bandwidth-bound Apple Silicon; don't assume speedups from papers.

    How to apply: Benchmark speculative decoding on your own hardware before enabling it; consider disabling it on Macs.

    speculative-decodingapple-siliconperformanceollama

    Read more: I tested speculative decoding on my M2 Mac. It's 5-10% slower, not 2–3x faster.

  8. #8 Claude Code Skill Pack CLItool

    A CLI to discover and manage Claude Code skill packs, avoiding conflicts between overlapping packs.

    Why it matters: Skill packs like Superpowers, Agent Skills, and Matt Pocock's can clash; a CLI helps you pick and manage them.

    How to apply: Use the CLI to list, install, and resolve conflicts between skill packs; see the Reddit post for the tool.

    claude-codeskillsclitooling

    Read more: I got tired of guessing which Claude Code skill pack to install, so I built a CLI for it

  9. #9 Project Athena: Harness Architecture for Claude Codetechnique

    Project Athena is a harness pattern that moves memory, sub-agents, and verification out of the LLM context into durable state.

    Why it matters: Keeping planning and QA in the model's context window causes token burn and instability; externalizing them makes agents more reliable.

    How to apply: Study the Athena approach (from the Reddit post) and refactor your harness to use external memory and verification steps.

    agentsharnessmemoryverification

    Read more: Stop overengineering agent harnesses from scratch. How Project Athena manages memory, sub-agents, and verification in Claude Code

  10. #10 Prompt Cache TTL Optimizationtip

    For slow workflows, a 1-hour prompt cache TTL can be cheaper than the default 5 minutes despite higher write cost.

    Why it matters: If you often take 15-45 minutes between turns, the 5-minute cache expires and you pay full input again; 1-hour TTL avoids that.

    How to apply: In Claude Code or API, set the cache TTL to 1 hour if your sessions have long gaps; measure your cache hit rate to confirm.

    claudecachingcostapi

    Read more: Prompt cache TTL: when does 1 hour beat the default 5 minutes?

Looking for topic trends and crawl volume over time? See Trends.