Cache-Breaking Plugins, 16x Faster Local Serving, and Signed Claude Skills

Today's findings

1. #1 Context-compression plugins can break Anthropic's prompt cache and cost you moretip

   Tools that "compress your context" for Claude API calls often silently invalidate Anthropic's prompt cache, so you pay full price instead of cached-token rates.

   Claude API · Prompt Caching

   Append vs rewrite: what keeps the prompt cache warm

   Append to history

   • Prior messages stay byte-identical
   • Cache prefix still matches
   • You pay cached-token rates

   vs

   Compress / rewrite context

   • Rewrites, reorders, or truncates
   • Cache match is invalidated
   • Full price + cost to rebuild cache

   Context-compression plugins can make the 'optimization' net negative

   Audit middleware to confirm it never alters previously-cached messages.

   Why it matters: Teams using the Claude API for agents rely on prompt caching to control costs; any harness step that rewrites, reorders, or truncates prior messages breaks the cache match, so you lose the discount and pay to rebuild the cache — making the "optimization" net negative.

   How to apply: Audit any middleware or context-compression step in your Claude API pipeline to confirm it doesn't alter previously-cached message content; prefer appending to history over rewriting it to preserve cache hits.

   claudecost-optimizationcaching

   Read more: Why token-saving plugins are costing you more money and tokens on Anthropic APIs.

2. #2 16x prefill speedup for local LLM serving via on-disk KV cache reusetechnique

   A Mac Studio (96GB) serving Qwen3.5-122B for concurrent sessions cut GPU prefill compute 16x by persisting KV caches to disk and reusing them across requests.

   Local LLM serving · KV cache reuse

   Disk-backed KV cache slashes prefill compute

   16x

   less GPU prefill compute

   recompute → reuse shared prefixes

   96GB

   single Mac Studio

   122B

   Qwen3.5 served concurrently

   on-disk

   KV caches persisted & reloaded

   Prefill-bound agent/RAG/multi-turn workloads reuse cached prefixes instead of recomputing them.

   Why it matters: Long-context, repeated-prompt workloads (agents, RAG, multi-turn chat) are usually prefill-bound; disk-backed KV cache reuse is a general pattern applicable to any local inference stack to serve more concurrent sessions on the same hardware.

   How to apply: If self-hosting models for multiple users or sessions, check whether your serving stack supports persisting/reloading KV cache across requests instead of recomputing shared prefixes every call.

   local-llminferencekv-cache

   Read more: Serving a fleet of Qwen3.5 122b sessions on a single Mac Studio (96GB) without losing your sanity

3. #3 A real-time "htop for vLLM" shows exactly where your VRAM goestool

   A new inspector tool visualizes vLLM's VRAM usage GB-by-GB in real time, including measured (not estimated) savings from quantization.

   vLLM VRAM inspector

   Where your GPU memory actually goes

   Model weights Measured savings per quant format

   KV cache Grows with context & batch

   Activations Transient per-step memory

   Weights KV cache Activations

   Live GB-by-GB breakdown instead of guessing from docs

   Why it matters: VRAM budgeting for self-hosted models is usually guesswork from docs and community benchmarks; a live breakdown lets you see the actual cost of KV cache, weights, and activations to right-size hardware and quantization choices.

   How to apply: Run it alongside your vLLM deployment to measure the real memory impact of switching quantization formats before buying new GPUs.

   vllmobservabilityquantization

   Read more: htop for vLLM see exactly where every GB of VRAM goes during inference (+ measured quantization savings, not guessed)

4. #4 world-model-mcp gives coding agents a temporal fact graph instead of vector-recall memoryrepo

   An open-source MCP server replaces naive vector-store memory with a structured fact graph tracking valid/invalid time ranges, per-fact decay half-lives, and contradiction resolution.

   Why it matters: Vector-recall memory for coding agents commonly leaks stale facts (like old function signatures) back into new sessions; a temporal graph with explicit fact expiry directly addresses this common agent-memory failure mode.

   How to apply: If building persistent memory for a Claude Code or MCP-based agent, evaluate world-model-mcp as a drop-in memory server instead of a plain vector DB.

   mcpagentsmemory

   Read more: world-model-mcp: an OSS structured memory MCP for coding agents. Temporal fact graph, decay half-lives, adversarial Coach-Player verification.

5. #5 trainproof is a deterministic linter that catches silently-broken fine-tuning runstool

   pip install trainproof: a linter for training runs that caught 4 of 5 deliberately-sabotaged QLoRA runs (NaN loss, zero learning rate, corrupted data) before wasting a full night of GPU time.

   Why it matters: Fine-tuning failures like a stuck learning rate or a dataset with broken rows often only surface after the run finishes; a lint pass catches most of these classes early.

   How to apply: Add trainproof as a pre-flight or mid-run check in your QLoRA/fine-tuning pipeline to catch NaN losses, learning-rate misconfigurations, and dataset corruption before they burn GPU hours.

   fine-tuningqloratooling

   Read more: I deliberately sabotaged five of my own QLoRA runs to see if a training linter could catch them. Four got caught. One fooled it completely. Body:

6. #6 skillerr adds signed, tamper-evident packaging for Claude Skillstool

   An open-source CLI seals an agent skill into a .skill file with content-addressed SHA-256 digests over every file, so any post-review tampering is detectable.

   skillerr · Claude Skills supply chain

   Sealing a skill makes post-review tampering detectable

   Raw skill

   • Loose markdown + scripts
   • Agent loads & executes as-is
   • Registry/CDN can swap it silently
   • No way to prove it's unchanged

   → Any byte changed after review → verification fails

   Sealed .skill

   • Single signed .skill package
   • SHA-256 digest over every file
   • Content-addressed, tamper-evident
   • Verify before the agent loads it

   Integrity check closes the gap between review and execution

   Why it matters: Skills are just markdown and scripts an agent loads and executes; without integrity checking, a compromised registry, CDN, or maintainer account can silently swap in a malicious skill after it was reviewed.

   How to apply: If your team distributes or consumes Claude Skills across a registry, seal them with skillerr and verify signatures before an agent loads them.

   claudeskillssupply-chain

   Read more: Built an open protocol + CLI for signing and verifying AI agent skills

7. #7 Gridcore Runner: a dependency-free ~8k-line C inference engine for GGUF modelsrepo

   A from-scratch GGUF inference engine in plain C (zero external deps) runs on CUDA, Metal, and AVX2 CPU with an OpenAI-compatible API and enforces JSON Schema during sampling rather than after generation.

   Why it matters: Enforcing structured output during sampling gives stronger guarantees for tool-calling pipelines than post-hoc retries, and a zero-dependency C engine is easy to vendor into constrained deployment environments.

   How to apply: Consider it as a lightweight llama.cpp alternative when you need guaranteed schema-valid output or a minimal-dependency local inference binary for embedded/edge deployment.

   local-llmggufstructured-output

   Read more: Gridcore-Runner - lightweight localized inference engine

8. #8 Hand-tuned Q8 quant format squeezes 12.5% more speed out of AMD R9700 GPUstechnique

   ROCmFPX is a hardware-specific Q8_0 quantization format (gfx1201 decode tuning, VDR8 vector-dot width) that benchmarks 12.5% faster than generic upstream GGUF quants on AMD Radeon AI PRO R9700 GPUs.

   Why it matters: Generic quantization formats leave performance on the table on non-Nvidia hardware; real gains are available by tuning the quant format to specific AMD GPU architectures instead of using one-size-fits-all GGUF quants.

   How to apply: If running local inference on AMD gfx12-series GPUs, benchmark ROCmFPX quants against your current GGUF format before assuming stock quantization is optimal.

   quantizationrocmamd

   Read more: I tuned a custom Q8 for my AMD R9700s. My benchmark said 12.5% faster.

9. #9 Pactrail treats every coding-agent file edit as a transactionrepo

   An open-source, model-agnostic Rust coding-agent harness wraps every agent change in a transaction with explicit permissions, write scopes, and rollback, instead of giving direct filesystem access.

   Pactrail

   Agent repo writes wrapped in a transaction

   bounded capability

   Model edits real files

   scope Write scopes

   limit Explicit permissions

   revoke Rollback on failure

   monitor Auditable changes

   Risk lives in the harness, not the model — a transactional wrapper for safe write access

   Why it matters: Most coding-agent risk comes from the harness, not the model — what it's allowed to touch and what happens on partial failure; a transactional wrapper is a reusable pattern for safely giving any model write access to a repo.

   How to apply: Look at Pactrail's transaction model as a reference or drop-in harness if you're building guardrails for agents that edit real repositories, especially around partial-failure recovery and auditability.

   agentscoding-agentssafety

   Read more: building a open source safe model agnostic coding agent harness in rust · I’m building a Rust coding-agent harness around transactions instead of direct filesystem access

10. #10 "Basalt Labs" caught rebranding Qwen2.5-7B as a 99.44%-HLE model while secretly serving DeepSeektip

    A HuggingFace-listed model claiming a top-tier benchmark score turned out to be a relabeled Qwen2.5-7B-Instruct, with the vendor's own demo site actually routing to DeepSeek behind the scenes.

    Benchmark fraud exposed

    high

    "Basalt Labs" top-model was Qwen2.5-7B in disguise

    99.44%

    HLE score it claimed

    Qwen2.5-7B

    actual relabeled base weights

    DeepSeek

    secretly served by the demo site

    affected scopeUnverified open-weight releases with leaderboard-topping claims

    high severity — badge colour grades the risk

    Fingerprint tokenizer & weights before trusting vendor benchmark pages

    Why it matters: Benchmark-gaming and model relabeling are increasingly common as open-weight releases proliferate; teams evaluating lesser-known open models for production should verify claims independently rather than trusting leaderboard screenshots or vendor pages.

    How to apply: Before adopting an unfamiliar open model release, run your own fingerprint check (tokenizer, weight names, known response quirks of the claimed base model) against the vendor's benchmark claims.

    open-weightsbenchmarksdue-diligence

    Read more: basaltlabsai/monolith-1.0 • HuggingFace · "Basalt Labs" pulling a generationally dumb scam. Incredibly stupid lmao. Claiming 99.44% on HLE with tools. Model they released is based on Qwen2.5-7B-Instruct and the model they're serving on their website is DeepSeek.