MoE Cache Beats RAM, Hybrid RAG Hits 93%, Hidden Reasoning Halves Tokens

Today's findings

1. #1 moe-cache Accelerates MoE Models Bigger Than RAM on Stock llama.cpptool

   A caching layer lets MoE models exceed available RAM while running 2–12× faster on stock llama.cpp with identical outputs.

   moe-cache · llama.cpp

   Bigger than RAM, faster than stock

   HOT

   RAM Busy experts in active rotation, served at full speed

   COLD

   Disk All remaining experts — the model now exceeds RAM

   Only hot experts hit fast memory: 2–12× faster on stock llama.cpp, identical outputs.

   Why it matters: Local teams can run larger MoE models on modest hardware without switching runtimes or accepting quality loss.

   How to apply: Try moe-cache with Qwen3.8-Flash-Next or Qwen3.6-35B on a machine where the model exceeds RAM; benchmark tokens/sec against your current llama.cpp setup.

   local-llmllama-cppmoeperformance

   Read more: moe-cache: MoE models bigger than your RAM, 2-12x faster on stock llama.cpp with identical tokens (Qwen3.8-Flash-Next 68 GB @ 24 GB RAM: 5.2 → 12.5 tok/s, Qwen3.6-35B @ 12 GB RAM: 6.9 → 22.6 tok/s)

2. #2 optscale-kompressor Shrinks Coding-Agent Tool Output Deterministicallytool

   A Rust single-binary compressor reduces coding-agent tool output while staying cache-safe and deterministic.

   AGENT TOOLING

   Squeeze tool output before it reaches the model

   1

   Coding agent

   runs tools

   2

   Raw output

   token sink

   3

   kompressor

   deterministic rewrite

   drop-in between harness and tool results

   4

   Context window

   cache preserved

   Single Rust binary; deterministic, cache-safe rewrites — same behavior, fewer tokens.

   Why it matters: Tool output is a major token sink in agent loops; compressing it cuts cost and context pressure without changing model behavior.

   How to apply: Insert the compressor between your agent harness and tool results, then verify cache hits and output fidelity on a representative coding session.

   agentstoken-efficiencyrustcaching

   Read more: optscale-kompressor: a deterministic, cache-safe compressor for coding-agent tool output, in Rust (MIT, single binary)

3. #3 Measured RAG Study: Hybrid Search Puts 93% of Evidence in 16K Tokenstechnique

   On million-token corpora, hybrid BM25 plus embeddings found most answer evidence in a small token budget, so retrieval still beats stuffing the context window.

   Why it matters: It gives engineers a concrete reason to keep investing in retrieval instead of assuming long-context models make search obsolete.

   How to apply: Benchmark your own corpus with a fixed retrieval budget; compare hybrid search plus reranking against full-context prompting on cost, latency, and answer quality.

   ragretrievallong-contextbenchmark

   Read more: If the Context Window Is a Million Tokens, Why Bother With Search?

4. #4 Hybrid Reasoning Agents Cut Tokens by Thinking Internallypaper

   A new paper proposes hidden internal reasoning with text emitted only for tool calls and final answers, reporting ~10% higher scores at half the tokens.

   arXiv 2610.18861

   Reasoning leaves the transcript

   ReAct: think in text

   • Thoughts → text
   • Tool calls → text
   • Answer → text

   → +~10% scores at ½ the tokens

   Hybrid: think internally

   • Thoughts → hidden
   • Tool calls → text
   • Answer → text

   Only tool calls and final answers get emitted; internal reasoning stays off the token bill.

   Why it matters: Agent loops are often token-bound; moving reasoning out of visible text can reduce cost and latency without sacrificing accuracy.

   How to apply: Read arXiv 2610.18861 and prototype the pattern in a trusted agent workflow where you can compare token usage and task success against a standard ReAct loop.

   agentsreasoningefficiencypaper

   Read more: New paper from Stanford, Together AI, UIUC, and Meta introduces Hybrid Reasoning Agents. These AI models think mostly in a hidden internal way (faster and cheaper), and only use normal text when they need tools or give the final answer.

5. #5 Fix LLM Comparison Bias by Running the Prompt Twice with Swapped Ordertip

   Models often favor whichever option appears first, so swap the order and run the comparison again to expose and cancel position bias.

   Why it matters: Any AI-assisted choice between drafts, candidates, or strategies can be silently skewed by prompt order.

   How to apply: For A/B comparisons, run both orderings and only trust a winner that wins in both; log disagreements for human review.

   promptingevaluationbias

   Read more: Ask AI to compare two options and it leans towards whichever one you listed first. It's called position bias, and the fix is running it twice

6. #6 Use Git Notes as a Shared Timeline for Parallel Claude Code Sessionstechnique

   Instead of having Claude Code sessions message each other, each session writes a git note and reads others' notes at startup.

   Coordination pattern

   Three parallel sessions, one shared notes ref

   Each session appends a short note after work and reads the ref at startup — coordination without cross-chatter, outside

   Why it matters: It avoids token-costly inter-agent chatter and duplicate status messages while keeping coordination outside commit history.

   How to apply: Create a dedicated notes ref, have each session append a short note after work, and read the notes ref before starting a new task.

   claude-codeagentsgitcoordination

   Read more: Stop making your Claude Code sessions message each other: use git notes as a shared timeline

7. #7 optimAIzr Audits Where Claude Code Tokens Gotool

   An open-source local tool breaks down AI coding usage turn by turn, tracks costs and rate limits, and simulates optimizations.

   Open-source token audit

   See where Claude Code tokens actually go

   optimAIzr

   tool Open source

   Local, turn-by-turn audit of AI coding sessions

   runRun it against your Claude Code or OpenCode sessions

   Find the top wasteful turns, simulate optimizations, then apply what works.

   Why it matters: Token spend is now a real engineering budget; visibility into waste helps teams cut costs without guessing.

   How to apply: Run optimAIzr against your Claude Code or OpenCode sessions, identify the top wasteful turns, and test its recommended changes before rolling them out.

   claude-codetokenscostopen-source

   Read more: I got tired of guessing where my AI coding tokens were going

8. #8 WatchDucks Adds a Live Sidebar for Delegated Claude Code Agentstool

   A free open-source Claude Code mod shows what subagents and delegated coding agents are doing in one live view.

   Why it matters: Multi-agent workflows become hard to debug when activity is scattered across plugins; observability is a prerequisite for trust.

   How to apply: Install WatchDucks in your Claude Code setup, run a delegated task, and use the sidebar to spot stuck or overlapping agent work.

   claude-codeagentsobservabilityopen-source

   Read more: I built a free Claude Code mod to see what all my delegated agents are doing

9. #9 lgtmaybe Brings Open-Source PR Review to Local Modelsrepo

   An open-source code reviewer splits large diffs and reviews hunks so a 27B local model can handle PR reviews.

   CODE REVIEW

   Hosted bots vs. a local 27B reviewer

   Hosted review bot

   • PR code goes to a third party
   • Recurring service costs
   • Vendor-managed, no local setup

   vs

   lgtmaybe · local 27B

   • Code never leaves your machines
   • Large diffs split into hunks
   • Runs a 27B open model locally
   • Open-source, self-hosted

   Hunk-by-hunk review is what makes a 27B local model practical for real PRs.

   lgtmaybe: open-source PR review for local models.

   Why it matters: It makes local-model code review practical for teams that want privacy or lower cost than hosted review bots.

   How to apply: Point lgtmaybe at a repository, configure a local 27B model, and compare its review comments against your current review process on a few real PRs.

   code-reviewlocal-llmagentsopen-source

   Read more: Running PR reviews on a local model: what i learned building an open-source code reviewer

10. #10 llama.cpp PR Reduces AVX2 Tiled Accumulators for More CPU Tokens/secrepo

    A llama.cpp pull request optimizes AVX2 tiled accumulator reduction, targeting better CPU inference throughput.

    Why it matters: CPU-only and GPU-poor setups benefit directly from lower-level inference speedups without new hardware.

    How to apply: Track PR #30296, build the branch on an AVX2-capable CPU, and benchmark tokens/sec against your current llama.cpp build.

    llama-cppcpuperformancelocal-llm

    Read more: cpu : reduce AVX2 tiled accumulators together by trivalik · Pull Request #30296 · ggml-org/llama.cpp

11. #11 Qdrant Tests 1-bit TurboQuant for Visual PDF Embeddingstechnique

    Qdrant reports 1-bit TurboQuant shrinks compressed PDF-page vectors from 856 KB to 40 KB while retaining 96.5% of float32 retrieval score, rising to 99.9% with rescoring.

    Qdrant · TurboQuant

    TurboQuant packs visual PDF-page vectors to 1 bit

    21×

    less storage per page vector

    856 KB → 40 KB

    96.5%

    of float32 retrieval score kept as-is

    99.9%

    recovered with rescoring

    1 bit

    per dimension, down from 32-bit floats

    Original float32 vectors stay stored for rescoring

    Why it matters: Document search over image-heavy PDFs can cut vector storage dramatically, but the original vectors still need to be stored for rescoring.

    How to apply: If you use visual document embeddings, test TurboQuant compression plus rescoring against originals on your own retrieval benchmark before changing storage budgets.

    ragembeddingsquantizationvector-search

    Read more: Qdrant tests Perplexity’s PDF embeddings: 21× smaller compressed vectors, with originals still stored

12. #12 MemScope Patches Agent Memory Code from Failure Tracespaper

    A new paper describes an agent that reads its own failure traces and patches one line of memory code instead of rewriting the whole memory system.

    AGENT SELF-REPAIR

    MemScope debugs its own memory

    1

    Run

    agent on a long-running task

    2

    Fail

    the error leaves a trace

    3

    Read trace

    agent inspects its own failure

    4

    Patch

    one line of memory code

    rerun, repeat

    The failure-trace-to-patch loop: one-line fixes, no retraining or full rewrite.

    Why it matters: Targeted memory repairs can make long-running agents more reliable without expensive retraining or full rewrites.

    How to apply: Read arXiv 2610.21170 and try the failure-trace-to-patch loop on a sandboxed agent memory module with human review before deployment.

    agentsmemorypaperdebugging

    Read more: MemScope: an agent reads its failure traces and patches one line of its memory code instead of rewriting it (arXiv 2610.21170, my 80 s explainer)