Cheaper Inference, Longer Context, and Agents That Actually Remember

Today's findings

1. #1 AsymSpec: full-context drafter + compressed verifier cuts inference compute 70-80%technique

   A full-context "drafter" model paired with a compressed verifier gets ~90% of full-context accuracy at 0.2-0.3x the compute.

   AsymSpec · serving efficiency

   Keep most of the accuracy, pay a quarter of the compute

   70–80%

   inference compute cut

   0.2–0.3× baseline compute

   ~90%

   of full-context accuracy retained

   2-model

   full-context drafter + compressed verifier

   Self-host

   prototype on vLLM or llama.cpp

   Cheap full-context draft pass, then verify with the compressed model.

   Why it matters: This is a concrete architecture pattern (not just a benchmark claim) for slashing serving cost on self-hosted LLMs without a large accuracy hit — directly relevant to anyone running local or on-prem inference at any scale.

   How to apply: Use the writeup as a blueprint: run a cheap full-context draft pass, then verify with a compressed model, similar in spirit to speculative decoding. Worth prototyping on top of vLLM or a llama.cpp-based serving stack before investing in bigger hardware.

   inferencespeculative-decodinglocal-llm

   Read more: AsymSpec: full-context drafter + compressed verifier gets ~90% of full-context accuracy at 0.2–0.3× compute

2. #2 Full ik_llama.cpp quant ladder for DeepSeek-V4-Flash with paired KLD testingtool

   A complete GGUF quant ladder (65GB-149GB) for DeepSeek-V4-Flash-0731 is up on Hugging Face with measured KLD accuracy loss per quant, not just size/perplexity guesses.

   Quantization

   DeepSeek-V4-Flash quants: guesswork vs measured accuracy loss

   Size + perplexity guess

   • Download, test, repeat
   • Perplexity only a proxy
   • No quality budget to cite

   vs

   Paired KLD per quant

   • 65GB-149GB GGUF ladder on HF
   • Measured KLD loss per rung
   • Pick smallest safe quant for VRAM

   Needs a recent ik_llama.cpp build - mainline won't load IQK/IQT tensors

   Published KLD deltas turn quant choice into a defensible decision.

   Why it matters: Lets teams pick the smallest quant that doesn't measurably hurt output quality for their VRAM budget, instead of trial-and-error across multiple downloads.

   How to apply: Pull the matching quant from the HF repo for your hardware; note it requires a recent ik_llama.cpp build (mainline llama.cpp won't load the IQK/IQT tensor types). Use the published KLD deltas to justify your quant choice to the team.

   quantizationgguflocal-llm

   Read more: I made ik_llama.cpp quants for DeepSeek V4 Flash 0731, with paired KLD testing against AtomicChat

3. #3 Adaptive KV-cache streaming pushes 262K context onto a 16GB GPUtechnique

   A llama.cpp modification streams KV-cache to run Qwen3.8 27B at 262K context on a single 16GB VRAM card.

   Why it matters: Long-context local inference is usually a hard VRAM wall; this trades some latency for a context window that would otherwise require far more memory, useful for RAG-heavy or long-document workloads on modest hardware.

   How to apply: If you're VRAM-constrained but need long context locally, review this KV-cache streaming approach as an alternative or complement to standard --cache-type-k/v quantization flags.

   context-managementgguflocal-llm

   Read more: Breaking VRAM Barrier: Qwen 3.8 27B at 262K Context with Adaptive KV-Cache Streaming on a 16GB VRAM GPU

4. #4 Two MIT-licensed tools cut context waste in Claude Code sessionstool

   A pair of free, open-source plugins trim stale/dead-end context from long Claude Code sessions without invoking another LLM call.

   Claude Code · context hygiene

   Pruning stale tool output before it degrades later turns

   Untrimmed session

   • Stale tool output
   • Dead-end branches
   • Full file dumps
   • Shrinking headroom

   → Mechanical trim — no extra LLM call

   After trimming

   • Live context only
   • Dead ends pruned
   • Room to reason
   • No restart needed

   Two free, MIT-licensed plugins compact context automatically as sessions grow.

   Why it matters: Long agent sessions accumulate tool output long after it stops being useful, quietly eating context budget and degrading later responses — a mechanical fix beats hoping the model self-summarizes well.

   How to apply: Install both plugins from the linked repos and point them at Claude Code sessions to automatically compact/trim context as sessions grow, rather than manually restarting threads.

   claude-codecontext-managementagents

   Read more: Two MIT tools for reducing context waste in coding agents without another LLM in the loop · I built two free open-source plugins around long Claude Code sessions

5. #5 autoharness: a self-learning skill layer for Claude Codetool

   autoharness is an MIT-licensed Claude Code plugin that captures solved problems from real sessions and turns them into reusable skills automatically.

   Claude Code · autoharness

   Solve it once, and the fix comes back on its own

   MIT-licensed plugin that learns from sessions you already run — no curation step.

   Why it matters: Multiple posts today independently complain about the same thing: Claude Code re-solves the same edge case from scratch days later because nothing from prior sessions sticks. This removes the manual burden of writing/maintaining skill files by hand.

   How to apply: Install as a Claude Code plugin; it learns from sessions you're already running, so no separate curation step is needed before skills start accumulating.

   claude-codeskillsagents

   Read more: Claude Code forgets every skill it ever learned the second the session ends

6. #6 A Claude Code skill that vets GitHub repos before writing new codetip

   A simple skill makes Claude search and vet existing GitHub repos before it starts implementing something like rate limiting from scratch.

   Why it matters: Claude Code defaults to reimplementing common functionality instead of checking whether a solid existing library already solves it, wasting time and adding unreviewed code.

   How to apply: Add the skill to your Claude Code setup so feature requests trigger a repo search/vetting pass before implementation begins.

   claude-codeskillstooling

   Read more: Built a skill to search and vet any available GitHub repo before building anything

7. #7 Fine-tuning a 4B model on ~3k browser trajectories: 22% → 62% accuracytechnique

   Fine-tuning Qwen3.5-4B on roughly 3,000 browser-automation trajectories nearly tripled benchmark accuracy on procedural browser tasks.

   Why it matters: Shows a small local model can get competitive at a narrow, repeatable agent task with modest targeted fine-tuning data — a much cheaper alternative to routing every browser-automation step through a large frontier model.

   How to apply: For repeatable, procedural browser or tool-use workflows, collect a few thousand trajectory examples from your own agent runs and fine-tune a small open model rather than defaulting to a large general-purpose one.

   fine-tuningagentslocal-llm

   Read more: I fine-tuned Qwen3.5-4B on ~3k browser trajectories and improved benchmark results from 22% to 62%

8. #8 Tencent releases Hy4-preview open weights (770B MoE, 49B active, 1M context)repo

   Tencent open-sourced Hy4-preview, a 770B-parameter MoE model with 49B active parameters and a 1M-token context window.

   architecture

   Hy4-preview: 770B parameters, only ~49B awake per token

   router

   top-4 gate

   770B total weights (grid ∝ share, not expert count)≈6% active per token

   weighted

   merge

   770B

   total params

   49B

   active per token

   1M

   context tokens

   Open

   weights on Hugging Face

   Why it matters: A new large open-weight MoE with a very large context window is a meaningful option for teams doing self-hosted long-context work, provided hardware supports it.

   How to apply: Pull the weights from Hugging Face (tencent/Hy4-preview) and benchmark against your current open-weight model on your own tasks before committing serving infrastructure to it.

   open-weightsmoelocal-llm

   Read more: Tencent/Hy4-preview 770B-A49B weight dropped · Introducing to tencent’s Hy4 preview Open weights: 770B MoE, 49B active, 1M context

9. #9 TontaubeV1: a fully local, open-weight TTS model with voice cloningrepo

   TontaubeV1 is a new 2.9B-parameter open-weight TTS model for expressive, long-form, low-latency local speech generation in English and German.

   Open-weight TTS

   TontaubeV1: a 2.9B text-to-speech model that runs entirely on your own machine

   2.9B params Open weights

   Fully local No cloud API

   Low latency Built for real-time

   Voice cloning Zero-shot from ~1 min audio

   English + German Two languages

   Long-form Expressive speech

   4-stage codec Autoregressive; learn it first

   What it is Runs local Capability Know before you wire it in

   Reference audio of up to one minute is enough for zero-shot cloning.

   Why it matters: Gives teams a genuinely local, open TTS option with zero-shot voice cloning, avoiding cloud API dependency for speech features.

   How to apply: Try it for local text-to-speech or voice-cloning prototypes using up to one minute of reference audio; its four-stage autoregressive codec design is worth understanding before wiring it into a product.

   ttslocal-llmopen-weights

   Read more: TontaubeV1 - Open TTS model release for local long-form generation

10. #10 AQuA: changing an agent's tool schema means you're no longer evaluating the same systempaper

    An arXiv preprint argues that altering tool descriptions, retry policy, or metric feedback invalidates prior agent evals even when model weights stay fixed.

    AQuA · arXiv preprint

    Four layers define the agent under test — not just the weights

    Model weights

    Usually the only versioned part

    Tool schemas

    Names, args, descriptions

    Retry policy

    Attempts, backoff, fallbacks

    Metric feedback

    What the agent sees on failure

    Change any layer and prior eval scores no longer compare.

    Why it matters: Teams that tweak tool schemas or prompts without re-running their eval suite may be silently comparing incompatible agent versions release over release.

    How to apply: Version tool schemas and retry policies alongside model versions, and re-run your eval suite whenever they change — not only when you swap the underlying model.

    evalsagents

    Read more: If an agent can change its tool schema, are you still evaluating the same system?

11. #11 MCP-driven agent pipeline turns a track into a music video via ComfyUItechnique

    An agent (Hermes) used an MCP server for ComfyUI to drive MiniMax prompts end-to-end, turning a music track into a short video with no manual workflow wiring.

    Why it matters: A concrete demonstration of MCP giving an agent real, tool-level control over a local generation pipeline — the pattern generalizes to any tool-heavy local workflow beyond media generation.

    How to apply: Look at the ComfyUI-MCP server setup as a reference if you want an agent to drive a local multi-step pipeline directly instead of manually chaining workflow nodes each time.

    mcpagentscomfyui

    Read more: I used Hermes Agent + ComfyUI MCP + MiniMax prompts to turn a music track into a short music video