Daily AI Digest: Adaptive Speculation, Tiny Tool-Callers, and Qwen's Next MoE

Today's findings

1. #1 Llama.cpp adaptive speculation speeds up Qwen3.8 by up to 50%technique

   A llama.cpp fork that auto-tunes speculative token counts delivers up to 50% faster generation on Qwen3.8 and other models.

   llama.cpp fork

   Adaptive speculation vs fixed token counts

   Fixed speculation

   • One token count for all content

   → up to 50% faster

   Adaptive speculation

   • Auto-tuned per token

   Speculative decoding settings are content-dependent; adaptive tuning removes the burden.

   Why it matters: Speculative decoding settings are content-dependent; a single fixed value leaves performance on the table. Adaptive speculation removes that tuning burden.

   How to apply: Use the fork (link in thread) and set min/max speculation bounds; it works with MTP and DFlash, especially on dense models like Qwen3.8.

   llama.cppspeculative-decodinginferenceqwen

   Read more: New: Llama.cpp adaptive speculation for faster inference · New: Llama.cpp adaptive speculation for faster inference

2. #2 Turing Engine runs 70B-120B models on a single 24GB GPUtool

   Open-source engine uses subspace activation pruning and KV compression to serve 70B+ models on consumer GPUs at 3,064 tok/s.

   Open-source inference engine

   70B–120B models on a single 24GB GPU

   3,064

   tok/s on consumer GPU

   No heavy quantization

   70B–120B

   parameter models served

   24GB

   VRAM on one consumer GPU

   Subspace

   activation pruning + KV compression

   Turing Engine runs frontier-scale open models without multi-GPU setups or quality-degrading quantization.

   Why it matters: Makes frontier-sized open models accessible without multiple GPUs or heavy quantization that degrades quality.

   How to apply: Check the repo (Intutic) and try it with LLaMA-3.1-70B or Qwen-2.5-72B; supports Unsloth checkpoints.

   inferencequantizationlocal-llmopen-source

   Read more: [Release] Turing Engine: Serve LLaMA-3.1-70B, Qwen-2.5-72B & DeepSeek on a Single 24GB GPU (3,064 tok/s, 75% KV Compression, Unsloth Checkpoint Support) · [Release] Turing Engine: Serve LLaMA-3.1-70B, Qwen-2.5-72B & DeepSeek on a Single 24GB GPU (3,064 tok/s, 75% KV Compression, Unsloth Checkpoint Support)

3. #3 thimble: a 48M-param tool-calling model with a 48MB C enginetool

   A tiny model that only does tool calling, with grammar-constrained decoding, beats much larger models on tool selection benchmarks.

   Why it matters: For agent pipelines, a dedicated tool-calling model can be faster and cheaper than a general LLM, and it never invents tools.

   How to apply: Download weights from HF (flashvenom/thimble) and use the single-file C engine for zero-config tool calls.

   agentstool-callingsmall-modelc

   Read more: released thimble: a 48M param tool-calling-only model, now with a 48MB single-file C engine that beats the torch stack · what worked and what didn't when training a 48M param tool-calling model from scratch, with the measurement behind each call

4. #4 Quantization-Aware Healing: 4-bit models that beat full precisionpaper

   A new technique compresses models to 4-bit while recovering (and sometimes exceeding) full-precision accuracy.

   Why it matters: Quantization usually costs accuracy; this method flips that, making local deployment more attractive.

   How to apply: Read the HF blog and consider applying the healing step to your own quantized models.

   quantizationpapercompression

   Read more: Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original · Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

5. #5 Qwen 3.8-Flash-Next (125B a6B) drops tomorrow with Unsloth day-0 supportrepo

   Qwen's new MoE model is releasing with immediate fine-tuning support from Unsloth, promising strong performance per active parameter.

   Why it matters: A 125B MoE with 6B active could be a sweet spot for local deployment on high-end hardware.

   How to apply: Watch the ModelScope page; prepare disk space and test with llama.cpp once quants appear.

   qwenmodel-releasemoeunsloth

   Read more: Qwen 3.8-Flash-Next releasing tomorrow (125B a6B) · Qwen3.8 flash next · Qwen3.8-Flash-Next tomorrow · Qwen3.8-Flash-Next announced · Qwen 3.8 Flash Next day 0 support from unsloth

6. #6 Tencent's WeMM-Embedding: multimodal embeddings from Qwen3.5repo

   Open-source 9B/4B/2B embedding models that accept text, images, video, and documents, returning 4096-dim vectors.

   Open-source release

   WeMM-Embedding: 4096-dim multimodal embeddings

   WeMM-Embedding

   model open-source

   from Qwen3.52025

   Hugging Face

   run9B/4B/2B · text/image/video/doc

   Unified retrieval across modalities without separate encoders.

   Why it matters: Enables unified retrieval across modalities for RAG and agent memory without separate encoders.

   How to apply: Use the HF models for multimodal search; they're L2-normalized and ready for vector DBs.

   embeddingsmultimodalragopen-source

   Read more: tencent/WeMM-Embedding 9B/4B/2B

7. #7 Qwen3.8-27B UD-Q2_K_XL: 9.8GB quant that behaves like 4-bittechnique

   A new dynamic quant at Q2_K_XL fits in 9.8GB and retains most of the 4-bit quality, making it usable on 12GB cards.

   Why it matters: Expands local deployment to lower VRAM GPUs without the usual quality cliff.

   How to apply: Download the UD-Q2_K_XL GGUF and test with llama.cpp; watch for the caveat about context length.

   quantizationqwengguflocal-llm

   Read more: Qwen3.8-27B UD-Q2_K_XL is usable at 9.8 GB — the smallest file that still behaves like the 4-bit one, but here's the catch.

8. #8 Ornith-1.5 on Strix Halo iGPU nearly matches Qwen3.8-27B on dual 3090stechnique

   A 35B MoE with 3B active runs on a Strix Halo iGPU and lands only 4 problems behind Qwen3.8-27B on LiveCodeBench v6.

   Why it matters: Shows that edge hardware can handle serious coding tasks, reducing need for expensive GPUs.

   How to apply: If you have a Strix Halo or similar, try Ornith-1.5 for local coding agents; it uses 8/256 experts per token.

   edgemoecodinglocal-llm

   Read more: Ornith-1.5-35B-A3B on a Strix Halo iGPU lands 4 problems behind Qwen3.8-27B on 2x 3090s. Full LiveCodeBench v6 numbers. · Ornith-1.5-35B-A3B on a Strix Halo iGPU lands 4 problems behind Qwen3.8-27B on 2x 3090s. Full LiveCodeBench v6 numbers.

9. #9 Designing agent memory with forgetting curvestechnique

   A solo dev shares how they built a forgetting curve for a single-user agent, showing which memories decay and when.

   Why it matters: Agent memory isn't just storage; it needs decay to stay relevant and avoid context bloat.

   How to apply: Read the blog post and implement a similar decay mechanism in your agent's memory layer.

   agentsmemorydesign

   Read more: I built a forgetting curve for an agent with one user · I built a forgetting curve for an agent with one user

10. #10 Row-Bot v4.6: agent with execution budgets and loop protectiontool

    An open-source agent harness that adds concurrency limits, loop protection, and local embedding fallback.

    Why it matters: Production agents need guardrails; Row-Bot provides them out of the box.

    How to apply: Check the repo and integrate it for your agent workflows, especially for desktop control.

    agentsguardrailsopen-source

    Read more: Significant improvements to Row-Bot recently. Looking for feedback. · Significant improvements to Row-Bot recently. Looking for feedback. · Significant improvements to Row-Bot recently. Looking for feedback.

11. #11 CLAUDE.md should document exceptions, not the whole projecttip

    Instead of a second README, put only the non-obvious constraints and decisions in CLAUDE.md to get better Claude Code behavior.

    Claude Code tip

    CLAUDE.md: exceptions, not a second README

    Full project docs

    • Repeats what code already says
    • Long, hard to maintain
    • Dilutes the important stuff

    vs

    Exceptions only

    • Only non-obvious constraints
    • Weird decisions and 'don't change this'
    • Short, high-signal

    Claude reads code; give it the context it can't see.

    CLAUDE.md should be a list of exceptions, not a manual.

    Why it matters: Claude can read code; it needs the context that isn't in the code, like 'don't change this' or 'this is intentional'.

    How to apply: Rewrite your CLAUDE.md to include only exceptions, weird constraints, and decisions that aren't obvious from the code.

    claude-codepromptingdocumentation

    Read more: I think the best use of CLAUDE.md is documenting exceptions, not everything

12. #12 Prompt-Evaluation-Engineer skill turns prompts into testable contractstool

    A single-file Claude Code skill that runs deterministic checks and semantic rubrics to evaluate prompt quality.

    Why it matters: Prompts are code; they need regression testing. This skill makes evaluation reproducible.

    How to apply: Add the skill to your Claude Code setup and use it to validate prompts before shipping.

    promptingevaluationclaude-code

    Read more: Prompt-Evaluation-Engineer skill.md (Claude Code)