Local Inference Levels Up: EAGLE3 for Qwen, Cohere-MoE in llama.cpp, and a Context Compression Trick

Today's findings

1. #1 Fix Qwen 3.6 Looping with Community-Validated Sampling Parameterstip

   A specific parameter set — temp=1.0, top-k=25, min-p=0.2, repeat-penalty=1.05, presence-penalty=0.1 — reliably eliminates Qwen 3.6's tool-repetition and thinking-loop failures in llama.cpp.

   Qwen 3.6 Looping Fix

   Sampling recipe that stops tool-repetition and thinking loops

   temp = 1.0

   top-k = 25

   min-p = 0.2

   repeat-penalty = 1.05

   presence-penalty = 0.1

   swa-full = on

   ctx-size = ≤125000 avoids SWA edge cases

   Community-validated llama.cpp config for Qwen 3.6

   Why it matters: Qwen 3.6 27B is the dominant local coding and agent model right now, but looping is its most-complained-about failure mode in production. Agents get stuck calling the same tool repeatedly or reasoning in circles, burning context budget and producing no useful output.

   How to apply: In your llama.cpp server config or CLI, set: temp=1.0 top-k=25 min-p=0.2 repeat-penalty=1.05 presence-penalty=0.1. Also enable swa-full=on and set chat-template-kwargs preserve_thinking=true if using the thinking variant. Keep ctx-size at 125000 or lower to avoid SWA edge cases.

   local-llmsamplingqwenagents

   Read more: Qwen 3.6 looping/repetition problem (Tesla p40s and Halo strix)

2. #2 EAGLE3 Speculative Decoding Coming to Qwen in llama.cpprepo

   A WIP pull request (#24593) ports EAGLE3 speculative decoding to Qwen models in llama.cpp, targeting 3-4x throughput gains with no quality loss.

   llama.cpp PR #24593 (WIP)

   EAGLE3 draft-verify decoding could triple Qwen throughput

   3-4x

   projected speedup

   ~20 → 60-80 tok/s

   60-80 tok/s

   Qwen3.6 27B, 1 GPU

   0%

   quality loss

   Same hardware, draft head trained on model's own hidden states

   Why it matters: EAGLE3 generates several candidate tokens using a lightweight draft head trained on the main model's hidden states, then verifies them in a single batched forward pass. For Qwen 3.6 27B this could push typical single-GPU throughput from ~20 tok/s to 60-80 tok/s without any hardware changes.

   How to apply: Watch llama.cpp PR #24593 for merge status. Once merged, supply a compatible EAGLE3 draft model via --model-draft and --spec-type eagle3. The draft model weights will be published alongside the PR. In the meantime you can follow the WIP branch if you want to test early.

   speculative-decodingllama-cppqweninference

   Read more: WIP EAGLE3 for Qwens

3. #3 50 tok/s on a GTX 1080 Ti with Gemma 4 12B QAT and MTP Drafttip

   Unsloth's Gemma 4 12B QAT-Q4_K_XL plus its bundled MTP draft file delivers 50 tok/s on a nine-year-old GTX 1080 Ti using a single llama-server command.

   Why it matters: QAT preserves quality better than post-training quant at the same bit-depth, and the bundled MTP draft head enables native speculative decoding without maintaining a separate model repo. This config proves older consumer GPUs remain competitive for real workloads.

   How to apply: Run: llama-server --hf-repo unsloth/gemma-4-12B-it-qat-GGUF --hf-file gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -ngl 99 -c 16384 --cache-type-k q8_0 --cache-type-v q8_0 -b 512 --spec-draft-hf unsloth/gemma-4-12B-it-qat-GGUF --model-draft MTP/gemma-4-12B-it-Q8_0-MTP.gguf --spec-type draft-mtp --spec-draft-n-max 2

   quantizationspeculative-decodinglocal-llmllama-cpp

   Read more: Yay got Gemma 12B QAT working on old 1080ti (maybe with speculative decoding?)

4. #4 Cohere2-MoE Merged into llama.cpp — North Mini Code Available Locally Nowrepo

   llama.cpp build b9626 adds Cohere2-MoE architecture support, making Cohere Labs' North Mini Code 1.0 coding models runnable locally as GGUFs with no special flags.

   Local LLM Release

   North Mini Code lands in llama.cpp — runs as a plain GGUF

   North Mini Code 1.0

   model MoE Coding

   Cohere Labs · Cohere2-MoE architecturebuild b9626

   llama.cpp (b9626+)unsloth/North-Mini-Code-1.0-GGUF

   runllama-server, no arch flags needed

   Sparse-expert coding model, now autodetected by llama.cpp

   Why it matters: North Mini Code is a compact MoE coding model that competes with Qwen 3.6 27B on code tasks while fitting into less VRAM due to its sparse-expert design. Having it in llama.cpp means it slots into existing llama-server toolchains immediately.

   How to apply: Pull llama.cpp at build b9626 or later (tag: b9626 on the ggml-org/llama.cpp releases page). Download the GGUF from unsloth/North-Mini-Code-1.0-GGUF on Hugging Face and load with llama-server as any other model. No architecture flags needed — the format autodetects.

   llama-cppmoecodinglocal-llm

   Read more: #24260 merged Llama.cpp Arch Cohere-Moe Support Added · Add arch support for cohere2-MoE by michaelw9999 · Pull Request #24260 · ggml-org/llama.cpp

5. #5 Snapcompact: Encode Long Contexts as Images to Save Tokenstechnique

   Snapcompact renders long text contexts as screenshot images and feeds them to a VLM, replacing many text tokens with a single image block and cutting effective context cost.

   Context Compression

   Long context becomes one image, not thousands of tokens

   Text Tokens

   • Doc text
   • Code files
   • Chat history

   → Many tokens → 1 image

   Image Block

   • 1 screenshot

   VLM reads rendered text as accurately as tokens

   Why it matters: In long-context agentic workflows — RAG, codebase analysis, document review — the dominant cost driver is input token count. If a VLM reads a screenshot as accurately as tokenized text, you shrink the per-call budget without losing information and without changing your model.

   How to apply: The technique works with any locally-running VLM that accepts image inputs (Qwen2-VL, LLaVA variants via llama.cpp). Render your context document or code file as a PNG and pass it as an image attachment instead of pasted text. See the blog post linked in item 10169 for implementation details and token-count comparisons.

   context-compressionlocal-llmagentsvision

   Read more: Snapcompact: Saving Tokens With Images

6. #6 Self-Improving Open-Source Coding Agent Stack as Local Claude Code Alternativetool

   An open-source coding agent pipeline that logs successful execution patterns and automatically tightens future prompts offers a local, self-improving alternative to Claude Code that gets smarter with every run.

   Local Coding Agent

   A coding agent that gets smarter with every run

   Local llama-server loop, MIT licensed, no API needed

   Why it matters: Unlike API-based coding assistants, this stack runs entirely on local models, keeps code private, and accumulates institutional knowledge via a feedback loop. Each successful tool-call sequence is stored and reused as a few-shot example for future runs.

   How to apply: See the GitHub link in item 10180 (MIT licensed). Connect it to a local llama-server instance (Qwen 3.6 27B or similar) and let it run iterative coding loops. The system logs successful patterns into a retrieval store that is automatically prepended to future prompts, improving reliability over time.

   agentscodinglocal-llmopen-source

   Read more: I Replaced Claude Code and Codex With an Open Source Stack That Gets Smarter Every Run, & Built Itself Along the Way

7. #7 Q4_0 Scale Deduplication Could Trim 318 MB from Qwen 27B Losslesslytechnique

   Qwen 3.6 27B's Q4_0 GGUF contains thousands of duplicate scale values; replacing them with a compact index table saves 318 MB with zero quality loss, at the cost of custom inference code.

   Why it matters: This is an unexploited lossless compression opportunity baked into the GGUF format. Even though it requires engine changes, it shows that quantization formats still have headroom to reduce memory bandwidth demands without touching weight precision — important as models keep growing.

   How to apply: Not yet productized. If you maintain a llama.cpp fork or gguf-tools: scan the quant file for duplicate f32 scale values, build a compact lookup table, store indices in place of raw scales, and dereference the index during dequantization. The author confirmed this pattern holds for both Qwen 3.5 2B and Qwen 3.6 27B.

   quantizationgguflocal-llmresearch

   Read more: Storing an index to a scale instead of the scale itself with Q4_0 quant reduces scale size by ~31% (small gain but interesting)

8. #8 RTX 5080 + RTX 3090 Reaches 80+ tok/s on Qwen 27B Q8 via Tensor Splittingtip

   Pairing a RTX 5080 16GB with a RTX 3090 24GB gives 40 GB pooled VRAM, enough to run Qwen 3.6 27B Q8 fully on-GPU at 80+ tok/s via llama.cpp tensor splitting.

   Why it matters: Single consumer GPUs cap at 24 GB, forcing either lower quantization or slow CPU offload. This mixed-card approach stays fully on-GPU at Q8 — the highest quality GGUF tier — without professional-grade hardware, at a cost accessible to individual developers.

   How to apply: Install both GPUs and configure llama.cpp with --tensor-split set proportional to VRAM ratio (approximately 0.4:0.6 for 16 GB:24 GB). Use the same PCIe bus generation where possible for bandwidth. The author's blog post at imil.net has exact parameters and throughput measurements across different tensor split ratios.

   local-llmhardwareinferenceqwen

   Read more: RTX 5080 + RTX 3090 Setup: 80+ Tok/s on Qwen 3.6 27B Q8 · RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

9. #9 Don't Trust Large Context Windows for Fact Retrievaltip

   LLMs reliably lose information placed in the middle of long contexts well before hitting their advertised window limit, making RAG and chunking safer defaults than raw context stuffing.

   Long-Context Recall

   Recall dips in the middle of long contexts

   Degradation hits well before the advertised token limit — on local and cloud models alike

   Why it matters: Teams switching to 128K+ context models often stop investing in retrieval pipelines, assuming the raw window is sufficient. Recall accuracy degrades non-linearly as context grows, with middle sections most vulnerable — this applies to local models and cloud APIs alike.

   How to apply: For tasks where recall matters (document Q&A, codebase navigation), use RAG or sliding-window chunking even if your model supports a large context. Reserve long-context mode for single-pass tasks where the model processes the full document once (summarization, rewriting) rather than retrieves specific facts from it.

   local-llmragcontext-windowstip

   Read more: Don't trust large context windows

10. #10 Mid-2026 Enabling Techniques Behind Consumer-Viable Open Weightstip

    Sparse MoE, latent KV compression, multi-token prediction, and 4-bit QAT have collectively made frontier-quality open-weight models runnable on consumer hardware in mid-2026 — not by needing more RAM but by needing less.

    Local LLM Architecture

    Four techniques make frontier weights fit on consumer GPUs

    Sparse MoE Fewer active params per token

    Latent KV Compression Shrinks memory for long context

    Multi-Token Prediction Speeds inference via draft tokens

    4-bit QAT Trains for quantization, not after

    Compute Memory Speed Quality

    Less RAM, not more — the shift enabling local frontier-quality models

    Why it matters: Understanding which architectural innovations enable consumer-viable local inference helps you evaluate new model releases: MoE reduces active compute per token, latent KV compression shrinks memory pressure during long contexts, MTP boosts throughput, and QAT retains more quality than PTQ at the same bit-depth.

    How to apply: When evaluating new open-weight releases, check for: MoE (compare active parameters vs total), SWA or sliding-window attention (critical for long context with limited VRAM), MTP draft support (enables speculative decoding speedups), and QAT vs PTQ quantization. Models hitting all four are the ones worth prioritizing for local deployment.

    local-llmmoequantizationarchitecture

    Read more: Local models in mid-2026

11. #11 Fable 5 US-Only and Anthropic Crackdown: Harden Your Open-Weight Fallbackstip

    Amazon CEO conversations with US officials reportedly triggered a crackdown on Anthropic model availability, and Claude Fable 5 launched US-only — a concrete signal that API-only AI stacks carry real geopolitical risk.

    Geopolitical Risk

    API-only stacks have a single point of failure

    Closed API-Only

    • One provider, one policy decision
    • Fable 5 launched US-only
    • Anthropic access reportedly curtailed

    vs

    Open-Weight Fallback

    • Qwen 3.6 27B — code
    • GLM Air 4.5 — fast completions
    • DeepSeek V4 Flash — reasoning

    Test fallbacks now, not under pressure

    Audit critical workflows for single-provider dependency

    Why it matters: Teams relying exclusively on closed API providers are one regulatory action away from a service disruption. The Fable 5 restriction is not hypothetical: non-US teams lost access to a flagship model on launch day. Open-weight local deployments provide a hedge that closed APIs structurally cannot.

    How to apply: Audit which critical workflows depend on a single closed-source provider. For each, identify and test an open-weight fallback now: Qwen 3.6 27B for code, GLM Air 4.5 for fast completions, DeepSeek V4 Flash for reasoning-heavy tasks. Run your test suite against them before you need them under pressure.

    open-sourceanthropiclocal-llmpolicy

    Read more: The takedown of Fable 5 may lead to the ban of local inference · Amazon CEO's talks with U.S. officials triggered crackdown on Anthropic models

12. #12 AgentPerfBench: Evaluate Local Models on Real Agent Taskstool

    AgentPerfBench is a HuggingFace dataset benchmarking LLM agents on planning, tool use, and multi-step execution — letting you compare local models on agentic behavior rather than static knowledge.

    Why it matters: Standard benchmarks (MMLU, HumanEval) measure single-turn knowledge or code generation, not the planning and tool-calling behaviors that matter in production agent systems. AgentPerfBench is already appearing in model comparison articles and fills that evaluation gap.

    How to apply: Load agent-perf-bench/AgentPerfBench from Hugging Face and run your local model (via llama.cpp server with OpenAI-compatible API) through the evaluation harness. Use results to decide between candidate models for coding-agent or tool-calling use cases before committing to a deployment.

    agentsbenchmarksevaluationlocal-llm

    Read more: AgentPerfBench