EAGLE3 Lands in llama.cpp, InfiniteKV Breaks the Context Wall, Kimi K2.7-Code Goes Open

Today's findings

1. #1 EAGLE3 Speculative Decoding Merged into llama.cpptool

   EAGLE3 is now in mainline llama.cpp — it guides the draft model with main-model hidden states for higher acceptance rates than vanilla MTP.

   Speculative Decoding

   EAGLE3: Guided Drafting Beats Blind MTP

   EAGLE3

   • Uses main-model hidden states
   • Higher acceptance rate
   • Requires compatible draft checkpoint

   vs

   Vanilla MTP

   • Uses only previous tokens
   • Lower acceptance rate
   • Works with any draft model

   EAGLE3 consistently outperforms vanilla MTP.

   EAGLE3 merged into llama.cpp main — drop-in throughput boost.

   Why it matters: Drop-in throughput multiplier for any llama.cpp deployment; guided drafting consistently outperforms blind MTP because the draft model sees the main model's hidden state rather than guessing from scratch, yielding more accepted tokens per pass.

   How to apply: Update llama.cpp to latest main, add an EAGLE3-compatible draft checkpoint for your base model, and launch with --spec-type eagle3. Check the merged PR for supported model families and compatible draft weights on HuggingFace.

   llama.cppspeculative-decodinginferencelocal-llm

   Read more: EAGLE3 has landed in llama.cpp

2. #2 MTP Draft Assistant Selection is the Decisive Variable — Wrong Model Gives Zero Speeduptip

   Benchmarks show the right MTP assistant doubles throughput; a mismatched one at the same size and quant gives no improvement at all.

   MTP Draft Assistant

   Right Assistant Doubles Throughput; Wrong One Gives Zero

   vs

   Wrong Assistant

   Right Assistant

   Throughput

   28 t/s

   63 t/s

   Speedup

   0x

   2.25x

   Assistant Match

   Mismatched

   Matched

   Wrong Assistant wins the row Right Assistant wins the row

   On Apple Silicon, MTPLX V1 automates MLX MTP setup and pushed Qwen 3.6 27B from 28 to 63 t/s with the right assistant.

   Why it matters: Many users enable MTP and see no gain, concluding the feature is broken — the culprit is almost always assistant mismatch. On Apple Silicon, MTPLX V1 automates MLX MTP setup and pushed Qwen 3.6 27B from 28 to 63 t/s with the right assistant.

   How to apply: Test at least two or three candidate draft assistants for your base model before giving up on MTP. On Mac use MTPLX V1 (Swift GUI) to benchmark interactively. On llama.cpp compare --spec-draft outputs with different assistant GGUFs and measure with llama-bench.

   llama.cppspeculative-decodingmlxlocal-llm

   Read more: Not All MTP Assistants Are Created Equal · llama.cpp + MTP · MTPLX V1: The Swift App For Running & Creating MLX MTP Models (2x TPS Qwen 3.6 27B)

3. #3 Kimi K2.7-Code — Open-Weight Agentic Coding Model from Moonshot AIrepo

   Moonshot's new open-weight model improves long-horizon agentic coding and token efficiency over K2.6, available now on HuggingFace.

   New Open-Weight Model

   Kimi K2.7-Code: Agentic Coding Model

   Kimi K2.7-Code

   model Open-Weight

   Moonshot AI

   HuggingFace

   runmoonshotai/Kimi-K2.7-Code

   Optimized for long-horizon agentic coding with improved token efficiency.

   Why it matters: A freely downloadable coding-agent model explicitly optimized for multi-step software engineering workflows — not a general chat model that can code, but a model designed for agent loops like Aider and OpenCode with end-to-end task completion in mind.

   How to apply: Pull moonshotai/Kimi-K2.7-Code from HuggingFace and wire it into your coding agent harness. Compare against Qwen 3.6 on your actual agentic task suite — the token efficiency gains matter most on sessions with large context accumulation.

   open-weightscoding-agentsmodels

   Read more: moonshotai/Kimi-K2.7-Code · Hugging Face · Kimi K2.7-Code: open-source coding model with better token efficiency

4. #4 InfiniteKV — Open-Source Two-Tier KV Cache Extends Effective Context Past the Trained Windowrepo

   InfiniteKV compresses old KV entries to 104-byte disk records instead of dropping them, letting Mistral-7B answer coherently from token 76k on a 32k trained window.

   InfiniteKV

   Extends Context Past 32k Window

   HOT

   GPU 256 recent tokens (exact)

   COLD

   Disk Old tokens compressed to 104B each

   Mistral-7B answers coherently from token 76k on a 32k trained window.

   Why it matters: When context overflows, serving stacks silently drop the oldest tokens and the model literally cannot see that content anymore. InfiniteKV eliminates this without retraining or larger GPUs — relevant for any long RAG session or multi-turn agent loop.

   How to apply: Integrate InfiniteKV into your inference stack: keep the 256 most recent tokens in GPU exact, compress all older tokens to 104-byte records in RAM or disk, retrieve on demand. A Colab demo is included to verify correctness on your use case before integration.

   kv-cachelong-contextlocal-llmmemory

   Read more: Open sourcing InfiniteKV: a KV cache that files old tokens as 104-byte searchable records in RAM or on disk instead of deleting them. Mistral-7B answered from token 76,747, 2.3x past its trained window. Colab demo

5. #5 HyperTool — Collapse Multi-Step MCP Tool Calls Into One Code Block to Slash Agent Contextpaper

   HyperTool lets the model emit a single code block that calls multiple tools with local intermediate values, replacing long step-wise traces that inflate agent context windows.

   Paper

   HyperTool: Collapse Multi-Step Tool Calls Into One Code Block

   Step-wise traces

   • Atomic call-return pairs
   • Long reasoning trace

   → Reduces token count, slashes context bloat

   HyperTool block

   • Composite code block
   • Local intermediate values

   Replace long step-wise traces with a single executable code block that calls multiple tools with local intermediate valu

   Why it matters: Step-wise tool traces are the dominant cause of context bloat in long agent sessions; collapsing them into composite operations both reduces token count and removes low-level dataflow decisions from the model's visible reasoning trace.

   How to apply: When building MCP tool servers, expose composite operations as executable code rather than atomic call-return pairs. Model the unit of tool use as a script, not a single function call — the paper is directly applicable to custom MCP server design and Anthropic's dynamic workflows pattern.

   mcpagentstool-callingcontext

   Read more: HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents

6. #6 Audit Prompt Sorts Your AI Workflows Into Frontier-Necessary vs Local-Ready vs Never-Leave-Machinetip

   A structured paste-and-run audit prompt classifies every use case in your workflow into cloud-required, local-Ollama-ready, or data that should never hit a third-party retention window.

   WORKFLOW AUDIT

   Sort every AI use case into one of three tiers

   Paste the audit prompt into your context model to classify each workflow.

   Why it matters: Most workflows default to frontier APIs out of habit; many tasks run fine on a local 27B model, and some containing PII, secrets, or proprietary logic should never touch cloud retention windows. The audit makes trade-offs explicit and suggests concrete models per use case.

   How to apply: Paste the audit prompt from the linked thread into whatever model knows your workflow context. For each local-ready task, swap immediately to the suggested Ollama model. For never-leave-machine tasks, set up local inference before continuing to use those workflows.

   ollamalocal-llmworkflowprivacy

   Read more: I audited which parts of my AI workflow actually need a frontier model. Most didn't. (audit prompt included)

7. #7 Sandbox Only Agent-Written Code, Not the Agent — Open-Source Patterntechnique

   Run the trusted agent on the host with full credentials, but route all model-generated shell commands and code through a lightweight sandbox so the threat surface is contained without breaking developer UX.

   Why it matters: Sandboxing the agent breaks host auth and forces API key piping into env vars — and plain containers still share the host kernel. The real threat is model-written code, not the agent binary itself. This pattern lets coding agents auth as you while isolating untrusted output.

   How to apply: Intercept the agent's shell and code execution calls; redirect them to a subprocess inside a container, gVisor, or landlock namespace while the agent process runs on the host with normal credentials. The open-source project in the linked post is actively seeking design feedback.

   agentssecuritysandboxing

   Read more: Instead of sandboxing the AI agent, sandbox only the code it runs.

8. #8 Greplica — Graph-Based Coding Agent Memory That Propagates Staleness Automaticallyrepo

   Stores repo knowledge as a graph of components, flows, and memory items so that when code changes, stale facts cascade to linked nodes instead of silently persisting.

   Why it matters: Flat CLAUDE.md or scratch-file agent memory rots silently — a function description written today is subtly wrong after refactoring and the agent retrieves it as truth. Graph structure makes rot visible and purgeable rather than a silent corruption source.

   How to apply: Replace flat agent context files with Greplica's component-flow-item graph. As you code, annotate changed components; Greplica flags linked memory items for review rather than serving stale facts. The project is in active development and seeking design critique.

   agentsmemorycoding-agents

   Read more: Greplica - Memory system for coding agents. Specifically solving for stale memory

9. #9 RA-RFT Trains a Retriever to Rank by Reasoning Benefit, Not Semantic Similaritypaper

   Retrieval-Augmented Reinforcement Fine-Tuning co-trains the retriever with the LLM so it surfaces documents that help solve the problem rather than documents that merely look similar.

   Why it matters: Standard RAG retrievers using cosine similarity frequently surface semantically close but reasoning-useless documents; RA-RFT's retriever is trained end-to-end with the reasoning reward signal, aligning retrieval quality with downstream task performance rather than embedding proximity.

   How to apply: Apply RA-RFT as a post-training step on your RAG retriever using gold-relevance distillation to label training pairs by expected reasoning benefit, then fine-tune the retriever with that signal. The approach is model-agnostic and layers on top of any existing retriever without changing the LLM.

   ragfine-tuningreasoning

   Read more: Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning · Learning to Reason by Analogy via Retrieval-Augmented Reinforcement Fine-Tuning

10. #10 Portable Ollama Runs Off a Single exFAT USB Drive on Mac and Windowstool

    An open-source Electron app bundles Ollama binaries and model blobs on a USB stick with no host installation, automatic port conflict resolution, and macOS quarantine handling built in.

    Why it matters: Useful for teams sharing machines, working in restricted environments where software installation requires approval, or carrying a specific model configuration between workstations without reinstalling everything each time.

    How to apply: Clone github.com/isthatseyi/portable-ai, copy your GGUF model blobs to app_data/models/ on an exFAT-formatted drive, and run the app on any Mac or Windows machine. The app probes ports 11434-11440 to avoid collisions with existing Ollama instances and kills the server cleanly on quit.

    ollamalocal-llmtooling

    Read more: I built a free portable Ollama app that runs from a single exFAT USB drive on both Mac and Windows

11. #11 Claude Fable 5 Is Relentlessly Proactive — Scope It or It Will Fix Things You Did Not Ask Abouttip

    Simon Willison finds Fable 5 autonomously deploys every available trick to reach its goal, diagnosing and fixing adjacent issues without being asked — great for complex tasks, requires explicit scope constraints for narrow ones.

    Why it matters: If you are building with claude-fable-5 as your coding agent backbone, expect substantially more autonomous action than prior Claude versions; this changes how you write system prompts and how narrowly you need to scope tasks to avoid unintended changes.

    How to apply: Lean into Fable 5's proactiveness for ambiguous engineering problems where you want it to explore. For narrow, well-defined tasks, add explicit boundary instructions in the system prompt — for example, only change files in a specific directory or do not modify unrelated code.

    claudeanthropicagentsprompt-engineering

    Read more: Claude Fable is relentlessly proactive · Claude Fable is relentlessly proactive

12. #12 Test --threads in llama.cpp — Wrong Setting Can Cost 80% Performancetip

    The common advice to limit threads to P-core count on hybrid CPUs is wrong in many configurations; benchmarking the full thread range on your actual model can recover up to 80% throughput.

    llama.cpp Performance Tip

    Wrong Thread Setting Can Cost 80% Throughput

    80%

    throughput recovery

    Common advice

    Limit threads to P-core count on hybrid CPUs

    Reality

    Benchmark full thread range on your actual model

    Run llama-bench with --threads sweeping from P-core count to logical core count on your hardware and model.

    Why it matters: A fixed thread count derived from old advice on a different CPU may be severely suboptimal; the gap is large enough to matter even on GPU-offloaded runs that still use the CPU for prefill or non-offloaded layers.

    How to apply: Run llama-bench with --threads sweeping from P-core count up to logical core count on your actual model and hardware. Take five minutes to measure rather than assuming the community default applies to your specific CPU and model size combination.

    llama.cppperformancelocal-llm

    Read more: PSA: Test your "threads" argument in llama.cpp (+80% performance in my case)