Edition 2026-10-20 latest · digest built 2026-10-20T12:07:12+00:00 · ⚙ ollama:deepseek-v4.1-flash:cloud
DeepSeek Open-Sources DeepInfer Inference Engine, Plus Token Waste Audits and RAG Citation Checks
DeepSeek's open-source DeepInfer engine headlines a day of practical infrastructure and agent-cost work. Teams also get new ways to audit wasted tokens, verify RAG citations, and structure Claude Code context for large repos. Local deployment options expanded with adaptive MoE streaming and measured vLLM cancellation behavior.
Inference and Local Deployment
DeepSeek's DeepInfer release gives teams a new open-source serving stack to benchmark and study, while Adaptive Streaming pushes large MoE models onto smaller devices. A measured vLLM cancellation test is a useful reminder that client behavior can burn tokens even when the user has walked away, and an inference-time fusion experiment shows how to combine small specialist models without merging weights.
Agent Cost and Reliability
The agent tooling on display is mostly about making runs cheaper and easier to trust: optscale-kompressor trims repeated tool output, ChewMe-Audit finds wasted tokens in traces, QA Cockpit adds visual testing, and StepFork makes failures replayable. Agentic GraphRAG extends that reliability work into memory by linking decisions back to source knowledge.
Claude Code and RAG Workflows
For Claude Code users, the practical advice is to keep root CLAUDE.md files small and push domain rules into subdirectories, while delegate-to-local offers a split where Claude plans and reviews and a local model writes code. RAG teams get a concrete citation-verification pattern that checks verbatim matches, entailment, and chunk self-containment before an answer ships.
Today's findings
-
#1 DeepSeek Open-Sources DeepInfer Inference Enginerepo
DeepSeek released DeepInfer, an open-source high-performance inference framework for its models.
Open-source releaseDeepInfer: DeepSeek's Serving Engine, Now in the OpenDeepInferrunClone the repo, benchmark it against your current vLLM or llama.cpp setup on representative prompts, then mine its schedA production-grade engine you can inspect and reuse — inference no longer a black box.Why it matters: Teams self-hosting or studying large-model serving can inspect and reuse a production-grade engine instead of treating inference as a black box.
How to apply: Clone the repo, benchmark it against your current vLLM or llama.cpp setup on representative prompts, and mine its scheduling and caching code for your own serving stack.
inferenceopen-sourcedeepseeklocal-llm
Read more: DeepSeek just open-sourced its inference engine · Deepseek released DeepInfer - Open Source high-performance, large-scale inference framework for DeepSeek models! · New from DeepSeek - DeepInfer - · DeepSeek open sources the inference engine for their models
-
#2 Optscale-Kompressor Shrinks Agent Prompt Bloattool
A MIT-licensed single Rust binary compresses repeated tool output and compiler noise before it reaches the model.
Prompt compressionAgent prompt bloat, trimmed before it reaches the modelRaw prompt- Same apt-get logs re-sent each turn
- Compiler warnings repeated
- Full verbose tool output
Compressed- Repeated output collapsed
- Duplicated noise compressed
- New information kept
One MIT-licensed Rust binary sits between the agent and its providerWhy it matters: Coding agents often re-send the same apt-get logs and warnings every turn; trimming that can cut a large share of prompt cost.
How to apply: Put the compressor between your agent and provider, start with traces that show repeated command output, and measure token savings before rolling it out broadly.
agentscostcompressionrust
Read more: We open-sourced optscale-kompressor (MIT, single Rust binary)
-
#3 ChewMe-Audit Finds Wasted Agent Tokenstool
An MIT CLI audits Langfuse, LangSmith, JSONL, and Claude Code traces for token waste and suggests copy-paste fixes.
Why it matters: Most agent traces carry measurable waste, so teams can cut spend and latency without changing models.
How to apply: Install with pip, run it on a recent trace export, and fix the highest-priced repeated patterns first, especially self-verification loops.
agentstokensobservabilitycli
-
#4 Verify RAG Citations Before They Shiptechnique
A three-check layer validates that citations are verbatim, actually support the claim, and come from self-contained chunks.
RAG citation guardThree checks before a citation ships1Verbatimstring match in source2Entailmentclaim actually supportedcatches real sources that don't back the claim3Self-containedchunk stands aloneAny failed check drops the citation before the answer is returned.Why it matters: RAG answers can cite real sources that do not support the generated claim, which erodes trust in document QA.
How to apply: Add string-match verification, an NLI or LLM entailment check, and chunk self-containment scoring before returning any cited answer.
ragcitationsevaluationverification
Read more: RAG citations are often fake and I got tired of manually checking them
-
#5 Keep Root CLAUDE.md Small, Push Rules Downtip
A large monorepo user cut the root CLAUDE.md to about 80 lines and moved domain rules into subdirectory files.
Why it matters: Bloated root instructions consume tokens every turn and get ignored; scoped files load only when the agent touches that area.
How to apply: Keep build and test commands plus directory boundaries at the root, then add per-folder CLAUDE.md files for schema quirks and local conventions.
claude-codecontextmonorepoprompting
-
#6 Delegate-to-Local Splits Planning From Codingtool
A Claude Code skill lets Claude plan and review while a local OpenCode model writes the code.
Why it matters: It can preserve Claude's high-level judgment while shifting token-heavy code generation to a local model.
How to apply: Point Claude Code at the delegate-to-local skill, configure your local OpenCode model, and test on a small repo before using it on critical branches.
claudelocal-llmcoding-agentsworkflow
Read more: delegate-to-local: Claude plans and reviews. Your local model writes the code.
-
#7 Closing the vLLM Connection Stops Token Burntip
A measured test found zero tokens accepted after a real connection close, but +44 tokens when the client merely stops reading.
vLLM · streamingTwo ways a client walks away from a streamClose the streamJust stop readingClose or cancel streams explicitly — and audit gateway abandonment before blaming throughput.Measured on self-hosted vLLM: stop-reading ≠ stop-generating.Why it matters: Self-hosted vLLM users can waste GPU time and tokens when clients abandon requests without closing the socket.
How to apply: Make clients explicitly close or cancel streams, and audit abandoned-request behavior in your gateway before blaming model throughput.
vllmtokenscostlocal-llm
-
#8 QA Cockpit Lets Agents See User Screenstool
An MIT-licensed npm package gives coding agents a shared view of every user's screen for web-app testing.
Why it matters: Visual QA is a common gap for coding agents; a shared screen view can make UI bugs reproducible and debuggable.
How to apply: Install the npm package in your test environment, wire it into your agent's tool loop, and start with a small set of visual regression cases.
testingagentsopen-sourceqa
Read more: I built QA Cockpit so you and your coding agent can see every user’s screen at once
-
#9 StepFork Replays and Debugs Agent Runstool
StepFork records agent executions, replays failures offline, forks runs, and generates regression tests.
Why it matters: Agent failures are hard to reproduce; replayable traces turn one-off bugs into testable cases.
How to apply: Record a failing production or staging run, minimize the trace, then fork it to test fixes and keep the generated regression test.
agentsdebuggingreplayopen-source
-
#10 Agentic GraphRAG Links Memory to Decisionsrepo
An open-source Neo4j project connects document knowledge, task activity, and prior decisions for agent retrieval.
Why it matters: Agents often save conclusions but lose the source passages and assumptions behind them; graph-linked memory keeps that context retrievable.
How to apply: Model documents, tasks, and decisions as graph nodes, then retrieve prior decisions with their evidence when starting a new agent session.
raggraphragagentsneo4j
Read more: Context Layer + Agent Memory in a 3D connected graph · Connecting agent memory to knowledge, and decisions with DeepAgents and Neo4j
-
#11 Adaptive Streaming Runs Big MoE Models Locallytool
A local OpenAI-compatible server streams large MoE models across CPU, GPU, and NPU on phones, desktops, and servers.
Local MoE servingOne model, streamed across three device tiersHOTGPU / NPU Active expertsWARMCPU Streamed expertsCOLDRAM / flash Idle expertsExpert weights stream up the tiers on demand — big sparse models run locally on 12–16 GB devices.Why it matters: It lowers the hardware floor for running big sparse models locally, including on 12 to 16 GB devices.
How to apply: Try it with a supported MoE model on your target device, benchmark prefill and decode against llama.cpp, and check memory pressure before adopting.
local-llmmoestreaminginference
-
#12 Fuse Small Local Models at Inference Timetechnique
An experiment combines coder, math, and instruct models with entropy-triggered auxiliary calls instead of merging weights.
Why it matters: It offers a way to get specialist help on hard coding cases while keeping each model quantized and separate.
How to apply: Use the coder model as primary, measure token entropy, and call auxiliary models only above a threshold; combine normalized log-probabilities rather than raw logits.
local-llmensemblingcodingquantization
Read more: I experimented with inference-time fusion of 3 small LLMs instead of merging their weights · I experimented with inference-time fusion of 3 small LLMs instead of merging their weights