EAGLE Lands in llama.cpp, KV Quant Breaks 256K on One GPU, and Local Agents Level Up

Today's findings

1. #1 EAGLE Speculative Decoding Merged Into llama.cpp Mainlinetool

   llama.cpp b9606 ships EAGLE natively — enable multi-token speculative decoding on any supported model with no extra setup.

   llama.cpp b9606

   EAGLE speculative decoding merges into llama.cpp mainline

   1.5–2×

   chat throughput vs standard decoding

   no extra setup needed

   1 pass

   target model verifies drafted tokens

   4–8

   typical spec-draft-n-max

   auto

   inherited by Ollama & downstream stacks

   Small draft model proposes tokens; target model verifies them in one forward pass

   Why it matters: EAGLE consistently yields 1.5–2× throughput on chat workloads by having a small draft model propose tokens that the target model verifies in a single forward pass. Mainline inclusion means Ollama and other llama.cpp-based stacks inherit it automatically on next update.

   How to apply: Update llama.cpp to b9606 or later. Download the matching EAGLE draft model for your base model from HuggingFace. Pass --model-draft draft.gguf --spec-type eagle to llama-cli or llama-server. Tune --spec-draft-n-max (typically 4–8) based on your GPU memory headroom and target batch size.

   llama.cppspeculative-decodinglocal-llminference

   Read more: EAGLE support merged into llama.cpp

2. #2 KV Quantization Enables True 256K Context on a Single RTX 3090technique

   Qwen3.6-27B Q4_K_M now runs a full 256K context with only 72 MB of KV cache resident, 88–100% needle recall, and 38.6 tok/s on one RTX 3090.

   Why it matters: KV cache is normally the VRAM bottleneck for long contexts. Q8_0 KV quantization has matured to near-zero accuracy regression while cutting KV memory by roughly 10×, making very long contexts practical on consumer hardware that was previously limited to 26K or less.

   How to apply: Add -ctk q8_0 -ctv q8_0 flags to llama-cli or llama-server. Pair with -c 262144 for 256K context. For extra throughput combine with MTP via --model-draft mtp-model.gguf --spec-type draft-mtp. Validate with a needle-in-haystack test at your target depth before shipping to production.

   quantizationllama.cppkv-cachelocal-llm

   Read more: This is amazing. Token speed doubled + kv cache now need low vram - qwen 27b · I'm still surprised on how good the kv quantization has become

3. #3 Fused Triton INT8 GEMM Unlocks True Tensor Core Speedups on Consumer Ampere GPUstechnique

   A published Triton kernel shows how to stop the quantize-then-dequantize-to-BF16 anti-pattern and actually fire INT8 tensor cores on consumer Ampere GPUs.

   Kernel technique

   Skipping the BF16 detour to fire INT8 tensor cores

   Dequant-then-GEMM

   • Quantize to INT8
   • Dequant to BF16
   • BF16 GEMM
   • Tensor cores idle

   → Promised INT8 speedup finally lands

   Fused INT8 kernel

   • INT8 × INT8
   • Accumulate INT32
   • Dequant in epilogue
   • Tensor cores engaged

   Fused Triton kernel keeps consumer Ampere tensor cores busy

   Why it matters: Most deployed W8A8 code quantizes weights and activations only to immediately dequantize them back to BF16 for the matmul, so the INT8 tensor cores never engage and the promised speedup evaporates. The fused kernel — INT8 × INT8 → INT32 with per-token × per-channel dequant and bias folded into the epilogue, autotuned per GEMM shape — closes the gap entirely.

   How to apply: Replace the dequant + BF16-GEMM pattern in your model's linear layers with the fused INT8 Triton kernel from the paper. The implementation targets Ampere consumer GPUs (3090, 4090) but the principle applies to any accelerator with native INT8 tensor cores. Profile with ncu before and after to confirm tensor core utilization rises to expected levels. The technique is general — the paper uses a diffusion transformer as a testbed but the kernel is model-agnostic.

   quantizationinferencetritonlocal-llm

   Read more: Realizing Native INT8 Compute for Diffusion Transformers on Consumer GPUs: A Fused INT8 GEMM Kernel for Ideogram 4.0

4. #4 lgtmaybe: Open-Source PR Reviewer With Parallel Lanes and Ollama Supporttool

   Drop-in AI code reviewer that fans out five review categories in parallel, runs a reflection pass to kill false positives, and redacts secrets — works fully local via Ollama.

   TOOL · CODE REVIEW

   Reflection Pass Loops Back to Kill False Positives

   1

   Diff

   PR changes in

   2

   5 Lanes

   parallel categories

   3

   Reflect

   self-critique

   4

   Redact

   strip secrets

   5

   Comment

   post review

   re-check flagged issues

   Runs fully local via Ollama — no diffs leave your machine

   Why it matters: Single-pass LLM review catches less and generates more noise. Parallel category specialization combined with a self-critique reflection step cuts false positives and improves coverage. The secret-redaction pass makes it safe to use on proprietary code without sending raw diffs to the cloud.

   How to apply: Clone the lgtmaybe repo and point it at your Ollama endpoint in the config. Wire it into CI as a GitHub Action or a pre-push hook. Run the reflection step at a lower temperature than the review step for more conservative filtering. Start with the default five categories before customizing review prompts for your codebase's specific patterns.

   agentslocal-llmollamacode-review

   Read more: Building lgtmaybe: a PR reviewer for any model

5. #5 Context Rot: Long Agent Sessions Degrade From Noise, Not Just Lengthtip

   In multi-turn coding agents, accumulated stale tool traces and abandoned plans actively poison reasoning quality — aggressive context compaction beats bigger windows.

   Why it matters: Larger context windows are often presented as the solution to long-running agents, but the community is discovering that quality degrades from noise accumulation independently of raw token count. Models reason through old junk, not around it, producing hallucinated continuations of dead code paths.

   How to apply: Implement a compaction step every N agent turns: summarize the session into a structured state artifact capturing the current goal, confirmed facts, and tool outcomes, then discard the raw turns. Treat failed tool calls as ephemeral and strip them before they enter long-term context. Measure task success rate rather than just context fill percentage to catch rot early.

   agentscontextlocal-llm

   Read more: Do long agent sessions get “context rot” for you too?

6. #6 Hybrid Frontier-Plus-Local Agent: Plan With the Cloud, Execute Locallytechnique

   Use a frontier model only for task decomposition and route nearly all execution tokens to a local model to get frontier-quality plans at a fraction of the API cost.

   Hybrid agent architecture

   Frontier plans, local executes

   Frontier (cloud)

   • Task decomposition
   • Superior high-level plans
   • Final synthesis only
   • Few, expensive tokens

   vs

   Local 27B–31B

   • Executes each subtask
   • Bulk of all tokens
   • Runs on consumer HW
   • Fraction of API cost

   Split the roles: frontier-quality plans, local-priced execution

   Route nearly all execution tokens to the local model; call the frontier model only to plan and synthesize.

   Why it matters: Local 27B–31B models excel at executing well-specified subtasks but struggle with high-level decomposition. Frontier models are expensive for bulk token generation but produce superior plans. Splitting these roles dramatically cuts API spend while maintaining output quality — a pattern validated at scale on consumer hardware.

   How to apply: Write a planning prompt that extracts a structured, self-contained step list from the frontier model. Pass each step independently to the local model with only the context needed to execute it. Call the frontier model again only for final synthesis or when the local model signals low-confidence uncertainty. Track frontier-vs-local token counts per session to find the optimal boundary for your task distribution.

   agentslocal-llmarchitecture

   Read more: An agent that plans with a frontier model but runs most of tokens locally (built it for my own dual-3090 rig)

7. #7 QAT GGUFs for Gemma 4 12B and 31B Now Availablerepo

   Quantization-aware training GGUFs for Gemma 4 12B and 31B are published with KLD near full-precision baseline — noticeably better quality than standard post-training Q4 builds.

   Why it matters: QAT bakes quantization error compensation into the weights during a fine-tune pass, so the model adapts rather than having errors blindly applied post-hoc. The result is Q4_0 GGUFs that the community reports matching full-cache harness accuracy — raising the quality floor for anyone running Gemma 4 locally.

   How to apply: Pull the QAT GGUFs from idkwhattoputherenow/gemma-4-12B-it-qat-q4_0-maxerr and the 31B counterpart on HuggingFace. Load them in Ollama or llama.cpp as you would any GGUF. If you previously used standard Q4_K_M Gemma 4 builds, run a side-by-side on instruction-following and coding benchmarks — most users report a lift especially on nuanced instruction compliance.

   quantizationfine-tuninggguflocal-llm

   Read more: moar QAT stuff and hairy ticks

8. #8 Local VLM Benchmark June 2026: Gemma 4 12B and InternVL3.5 Lead for On-Prem Visiontip

   A fresh June 2026 local VLM shootout covering Gemma 4, InternVL3.5 8B, Qwen3-VL, and GLM-4.6V gives concrete rankings for teams replacing cloud vision APIs with local MCP servers.

   Local VLM shootout · June 2026

   Which open-weight vision model to run on-prem

   Gemma 4 12B Default pick, general vision QA

   InternVL3.5 8B When VRAM is tight

   Gemma 4 26B-A4B Higher accuracy, needs headroom

   Recommended default Low-VRAM option High-accuracy option

   Replacing cloud vision APIs with local MCP servers; test on your own images first.

   Why it matters: Most published VLM benchmarks are months out of date and miss the latest open-weight releases. This benchmark is specifically motivated by replacing cloud-hosted MCP image-analysis endpoints with local alternatives — a direct cost and privacy win — and tests the exact models available today.

   How to apply: For general vision QA on consumer hardware Gemma 4 12B is the default recommendation. Use InternVL3.5 8B when VRAM is tight. Use Gemma 4 26B-A4B for higher accuracy tasks where you have headroom. Test on a representative sample of your own image distribution before committing; benchmark rankings can shift significantly depending on domain.

   local-llmvlmmcpollama

   Read more: Which is the best local VLM? Benchmark results June 2026

9. #9 PonyExl3: EXL3 Quantization Format Ported to Apple Silicon via Metalrepo

   A new open-source project brings ExLlamaV3 quantization to M-series Macs through a Metal compute backend, giving Apple Silicon users access to EXL3 quality-per-bit ratios for the first time.

   Why it matters: EXL3 achieves better perplexity-per-bit than standard GGUF quants at the same file size. Previously this format was NVIDIA-only. PonyExl3 opens the quality gap to Apple Silicon users who make up a large share of the local LLM community.

   How to apply: Clone PonyExl3 from github.com/beamivalice/PonyExl3 and build with the Metal backend flag. Load EXL3 models converted from HuggingFace checkpoints. Run a side-by-side perplexity comparison against the equivalent Q4_K_M GGUF on your target model to quantify the quality improvement before switching your default.

   quantizationlocal-llmapple-silicongguf

   Read more: I ported EXL3 to run well on Apple Silicon - PonyExl3

10. #10 Aionforge Memory: Rust Agent Memory Layer With GraphDB and Hybrid Retrievalrepo

    Open-source Rust library for structured long-term agent memory — episodes, facts, skills, bad patterns — with BM25 + vector + graph-traversal retrieval and trust/recency scoring.

    Why it matters: Most agent memory implementations are ad-hoc string stores. Aionforge Memory provides a typed, queryable graph-backed store that surfaces relevant context across very long task horizons and tracks entity relationships, making it practical for autonomous agents running multi-day workflows.

    How to apply: Add aionforge-memory as a Rust dependency or run it as a sidecar service against your agent host. Define your memory schema using the built-in types: episodes for turn history, facts for extracted knowledge, skills for reusable procedures, bad-patterns to avoid repeated mistakes. At each agent turn, write the current episode then retrieve top-K memories by blended recency-importance-trust score before building the system prompt.

    agentsmemorylocal-llm

    Read more: Aionforge Memory - Long Term Agent Memory

11. #11 Parallel-Synthesis: Agent Workers Share KV Caches Instead of Serializing to Textpaper

    A plug-and-play framework lets a synthesizer agent consume raw KV caches from parallel worker agents directly, eliminating the expensive re-prefill of concatenated text outputs.

    Why it matters: Standard multi-agent pipelines concatenate worker text outputs and re-prefill the entire combined context at the synthesizer — wasting compute proportional to worker count. Parallel-Synthesis eliminates that re-prefill by passing KV cache pointers instead of text, which could become a standard efficiency pattern as agentic frameworks mature.

    How to apply: The paper is currently research-grade but the concept is implementable today on inference servers that expose KV cache management APIs such as vLLM and SGLang. When workers run on the same inference host, export their attention cache state after generation and pass cache handles to the synthesizer instead of decoded text strings. Watch for native framework support landing in vLLM or SGLang over the next few months.

    agentsinferencekv-cache

    Read more: Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows · Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows

12. #12 DiffusionGemma 26B Is Neither Parallel Nor Sequential — Actual Commit Order Measuredpaper

    Instrumented sampling of DiffusionGemma 26B finds a partial left-to-right commit bias, not the fully parallel decoding its marketing implies.

    DiffusionGemma 26B · measured commit order

    Neither fully parallel nor sequential — it drifts toward block-autoregressive

    Instrumented sampling shows partial left-to-right commit, not the parallel decoding marketing implies.

    Why it matters: Teams evaluating diffusion LMs against autoregressive models for latency-critical workloads should know the parallel-decoding narrative is overstated. The model exhibits weak per-token order bias that strengthens to strong chunk-level bias — behavior closer to block-autoregressive than fully parallel, which changes latency modeling assumptions significantly.

    How to apply: If benchmarking DiffusionGemma for a latency-sensitive application, measure actual time-to-first-meaningful-chunk rather than assuming full parallelism in your throughput model. The finding also suggests that attention-mask interventions on specific canvas positions during late denoising steps may be a low-cost steering mechanism — worth exploring for controlled generation or constrained output tasks.

    local-llminferencediffusion-llm

    Read more: Neither Parallel Nor Sequential: How DiffusionGemma Actually Commits Tokens