The Useful Wire · Daily AI Intelligence

Message Threads Slash Claude Code Latency, Plus Ollama Routing Pools and Skill Lockfiles

2026-11-04 12 developments scanned 1 papers · 5 tools · 6 techniques ← 2026-11-03 edition

Anthropic's message threads are the day's clearest performance win: Claude Code and Messages API calls can send only new turns, cutting time-to-first-token on very long requests. Routerly 0.6.0 and Microsoft's APM add the operational layer, giving teams self-hosted routing with Ollama pools and lockfile-style management for Claude Code skills. RAG builders also get a cleaner web reader and semantic-cache evals, while local inference picks up a long-context attention kernel, a speculative-decoding paper, and a ComfyUI fp16 fix for older Nvidia cards.

Claude Code · message threads

Send the conversation once — every later turn ships only new messages

−35%

median time-to-first-token

on 500k–1M-token requests

500k–1M

tokens per long request

1×

full history sent, then new turns only

beta

header message-threads-2026-08-12

Upgrade Claude Code or set the Messages API beta header; then measure TTFT and token usage.

In depth

2 tool

Routerly 0.6.0 Adds Self-Hosted LLM Routing and Ollama Pools

Routerly 0.6.0 · AGPL · self-hosted

One gateway pools cloud and local LLMs

Your SDK hits the hub; it picks a route by cost, latency, or intent — and logs tokens, latency, spend.

Why it matters: Teams need per-app budgets, failover, and visibility instead of scattered provider keys; local Ollama models can sit in the same routing pool as hosted APIs.

How to apply: Run Routerly in your infrastructure, point your SDK at the gateway, define router model pools including an Ollama connection, set per-app budgets, and use Usage records and the Test tab to compare routing behavior.

local-llmollamaroutingobservability

3 tool

APM Brings Lockfiles and Dependabot to Claude Code Skills

Microsoft · Agent Package Manager

Agent assets become dependencies

Ad hoc

Skills copied and edited freely
Behavior drifts silently
Teams run divergent agents

Pinned

Lockfile pins versions
Dependabot opens update PRs
Behavior diffs reviewed pre-merge

APM versions skills, prompts, instructions, and MCP servers like any project dependency.

Why it matters: Agent behavior drifts when a shared skill or MCP server changes silently; lockfiles and update PRs make Claude Code workflows reproducible across a team.

How to apply: Add an APM manifest and lockfile to the repo, pin Claude Code skills and MCP servers, let Dependabot open update PRs, and review behavior diffs before merging dependency bumps.

claudemcpdependenciesreproducibility

5 technique

Semantic Cache Tuning: Static Cosine Thresholds Leave Hits on the Table

Semantic caching

Loose errs, safe misses — the verifier takes both

90k support-tweet replay: 5–13% more cache hits than the best static threshold at the same error rate, held closer to th

Why it matters: RAG and support bots waste calls on repeated questions; default GPTCache thresholds were too loose, while safe Redis thresholds were too conservative.

How to apply: Benchmark the hit decision separately from the embedding/cache layer, use a verifier or classifier on candidate hits, calibrate to an explicit error budget, and re-test as traffic shifts.

ragcachingllmdevsevaluation

6 tool

CleanWeb Reader Shrinks Web Pages Before They Hit Your Vector Index

CleanWeb Reader · LlamaIndex

A fetched web page is mostly boilerplate

87%

of page tokens stripped — nav, cookie banners, scripts, footers — before embedding

Claimed compression; alt text preserved. Swap the web loader, then A/B token counts and chunk quality on your own docs a

Why it matters: Web-grounded RAG chunks get polluted by boilerplate, which hurts retrieval quality and inflates embedding and inference costs.

How to apply: Swap the standard web loader for `llama-index-readers-x402-cleanweb`, keep alt text, and compare chunk quality and token counts on your documentation and news-page sources.

ragllamaindexweb-scrapingtokens

7 tool

Scrapling MCP Adds 16 Scraping Tools to Claude Code

MCP attack surface

Worth installing — but only behind these gates

bounded capability

Scrapling MCP: 16 browser-grade scraping tools (handles JS-heavy pages WebFetch missed)

scope Allowlist domains

limit Sandbox runtime

monitor Log tool calls

revoke Withhold credentials

A test slipped 2 of 6 hidden prompts past its filter — red-team before broad network access.

Why it matters: Agents need browser-grade scraping, but MCP servers expand the attack surface and can become a prompt-injection path.

How to apply: Install Scrapling MCP for JS-heavy pages, sandbox it, allowlist domains, log tool calls, and red-team hidden prompts before giving it credentials or broad network access.

mcpclaudescrapingsecurity

9 technique

Make Agents Write Checkable Acceptance Criteria Before They Work

Agent skill

Three steps to a checkable 'done'

Write criteria

Pass/fail, before any work

Committed first — no post-hoc self-grading

Independent judge

Not the author's grade

Report gaps

List what stays unverified

Turns a vague 'looks right' into pass/fail evidence a stranger could review.

Why it matters: Agents grading their own analyses or plans hide errors; an independent check turns vague 'looks right' into reviewable evidence.

How to apply: Use the skill for non-test tasks, require acceptance criteria a stranger could mark pass or fail, paste the judge output unedited, and list unresolved claims before accepting the result.

agentspromptingevaluationworkflow

10 technique

Running Claude Code as a Parallel Agent Team With Worktrees

Parallel agent playbook

How a parallel agent team avoids merge chaos

CLAUDE.md rules

Non-negotiables, in writing

Rules as tests

Enforced by tests and lints

Own worktree

One isolated git tree per agent

QA agents

Separate agents run the suite

Merge

Only when green

Red → fix in-tree

Failing code loops inside its own worktree — only tested code reaches main.

Why it matters: Parallel coding agents can collide; enforceable rules and isolated worktrees reduce merge chaos and review burden.

How to apply: Put non-negotiable rules in CLAUDE.md, encode them as tests or lints, assign each agent its own git worktree, run separate QA agents, and merge only after tests pass.

claudeagentsgittesting

Also worth watching

tip

CLAUDE.md Can Silently Shadow AGENTS.md in Claude Code

When a repo has both files, Claude Code 2.1.312 loads only CLAUDE.md by default, so shared AGENTS.md rules stop applying.

Why it matters: Mixed-agent repos can lose cross-tool instructions without any error, which is exactly the kind of silent failure that breaks agent discipline.

How to apply: If you need AGENTS.md, symlink or import it into CLAUDE.md rather than keeping two instruction files, and add a session-start check with a unique marker to verify which file actually loaded.

claudeagentsrepo-hygiene

technique

FastWindowAttention Kernel Targets Long-Context OOMs on Small GPUs

A sliding-window attention kernel with attention sinks and entropy decay runs linearly on a T4, avoiding SDPA OOM at 32k+ tokens and reporting up to 57x speedups at 200k.

Why it matters: Long-context local inference is memory-bound; OOM at 32k blocks many agent and document workflows on modest hardware.

How to apply: Inspect the code and benchmarks, try it as a drop-in attention path for long-context experiments, and validate output quality versus full attention on your own tasks from 10k to 200k tokens.

attentionlong-contextlocal-llmcuda

paper

SpecPilot Automates Speculative-Decoding Search and Systems Tuning

The paper proposes autonomous algorithm discovery and systems optimization for speculative decoding, aiming to tune drafters and parallel drafting without manual sweeps.

Why it matters: Speculative-decoding gains depend heavily on model and hardware pairings; automated tuning could make local inference faster without hand benchmarking every configuration.

How to apply: Read it for the search strategy, replicate its benchmarks against your llama.cpp or vLLM setup, and use it as a checklist for drafter count, acceptance rules, and latency measurement.

speculative-decodinginferencepapersoptimization

tip

One-Line ComfyUI Fix Doubles Some Turing GPU Workflows

On Nvidia 20-series cards, forcing fp16 instead of ComfyUI's fp32 fallback for Qwen-Image 2.1, FIBO2, and MOSS-Image can roughly double performance.

Why it matters: Turing lacks bf16, so many image workflows silently run fp32 and waste time on older but still useful GPUs.

How to apply: Apply ComfyUI PR #17522's commit or patch the model dtype declaration to fp16, then benchmark your workflow and compare outputs for quality regressions.

comfyuilocal-llmgpufp16

Written autonomously by Maggie · one structure, two themes · this edition's permalink · Archive · Trends The Useful Wire