Local LLM Speedups, MCP Security, and RAG Efficiency: Today's AI Digest

Today's findings

1. #1 Qwen3.8-27B hits 2k prefill and 132 decode tok/s on a single RTX 3090repo

   A custom kernel and int8 quantization push Qwen3.8-27B to 2,000 prefill and 132 decode tokens/sec on a 24GB RTX 3090.

   Local inference speedup

   27B model hits 132 decode tok/s on a single RTX 3090

   132

   decode tok/s

   2,000 prefill tok/s

   2,000

   prefill tok/s

   0.99997

   similarity to fp32

   int8

   quantization

   Custom kernel + int8 quantization on consumer hardware.

   Why it matters: This is a massive speedup for local inference on consumer hardware, making 27B models viable for interactive use without a datacenter GPU.

   How to apply: Clone the repo (github.com/syv-ai/qwen38-27b-rtx3090) and follow the setup; the custom kernel matches fp32 quality at int8 with 0.99997 similarity.

   local llminferencequantizationoptimization

   Read more: I pushed Qwen3.8-27B to 2.000 prefill per second and 132 decode per second on A RTX 3090.

2. #2 llama.cpp Metal PR speeds up IQ3_XXS MoE decode on Apple Silicontechnique

   A new llama.cpp Metal optimization boosts decode from ~65.6 to ~73.9 tok/s for 35B-A3B MoE models on Apple Silicon.

   llama.cpp Metal PR

   Decode speedup for 35B-A3B MoE on Apple Silicon

   Before PR

   • 65.6 tok/s

   → +8.3 tok/s (+12.6%)

   After PR

   • 73.9 tok/s

   IQ3_XXS quantized MoE decode, Apple Silicon

   Why it matters: Apple Silicon users get a free decode speedup for quantized MoE models, improving local LLM responsiveness.

   How to apply: Test the PR (github.com/ggml-org/llama.cpp/pull/28086) and report before/after numbers; expect a follow-up for prefill.

   llama.cppapple siliconmoeoptimization

   Read more: Llama cpp metal moe optimization

3. #3 ExLlama v3 adds CPU offload, GLM-5.3-Flash, Qwen3.8-Flash, and SC quantstool

   ExLlama v3's latest release brings CPU offload for MoE experts and support for new models and SC quantizations.

   Inference engine update

   ExLlama v3 adds CPU offload, new models, SC quants

   ExLlama v3

   tool CPU offload

   ExLlamaLatest release

   GitHub

   runRun large MoE models on limited VRAM

   Expands local inference flexibility with CPU offload for MoE experts.

   Why it matters: ExLlama is a high-performance inference engine; these updates expand what you can run locally and improve memory flexibility.

   How to apply: Update to the latest ExLlama v3 release and try CPU offload for large MoE models that don't fit in VRAM.

   exllamainferencecpu offloadquants

   Read more: ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

4. #4 42 MCP-related CVEs in 7 days: auth, path traversal, and injectiontip

   A week of MCP CVEs shows 8 with missing auth, 5 path traversal, and 3 prompt injection—many with CVSS 9+.

   MCP Security Alert

   critical

   42 MCP-related CVEs in 7 days

   42

   CVEs in 7 days

   8

   missing auth

   5

   path traversal

   3

   prompt injection

   9+

   CVSS score on many

   affected scopeMCP servers

   critical severity — badge colour grades the risk

   Audit your MCP servers: require auth, bind to localhost, pin dependencies, sanitize paths.

   Why it matters: MCP servers are proliferating, and many are insecure by default; this is a wake-up call for anyone deploying MCP.

   How to apply: Audit your MCP servers: require auth, bind to localhost, pin dependencies, and sanitize paths. Check the NVD list for your stack.

   mcpsecuritycvebest practices

   Read more: 42 CVEs mentioning MCP published in 7 days. 3 of them mention injection

5. #5 json-to-yaml-lite: shrink JSON payloads before hitting your LLMtool

   Two zero-dependency Python tools convert JSON to YAML or compact formats, cutting token usage and latency in RAG/agents.

   Why it matters: JSON is token-inefficient; converting to YAML can reduce prompt size by 30-50%, saving cost and context.

   How to apply: pip install json-to-yaml-lite and wrap your API responses/database rows before injecting into prompts.

   promptcontextjsonrag

   Read more: Stop feeding raw JSON to your LLMs (I built two zero-dependency tools to shrink your prompt payloads)

6. #6 LangChain added ~15% CPU overhead in our RAG hot path—so we made it optionaltip

   Profiling showed LangChain's abstraction layer consumed ~15% of CPU in a RAG query path; removing it from the hot path improved performance.

   Performance tip

   LangChain's abstraction costs ~15% CPU in the RAG hot path

   vs

   With LangChain

   Thin client

   CPU overhead

   ~15% extra

   baseline

   Provider abstraction

   Full

   Minimal

   With LangChain wins the row Thin client wins the row

   Profile your pipeline — if LangChain is only for provider abstraction, a thin client cuts the overhead.

   Why it matters: Framework overhead is real; for high-throughput RAG, a thin provider-agnostic layer may be better than a full framework.

   How to apply: Profile your own pipeline; if LangChain is only used for provider abstraction, replace it with a lightweight client or direct calls.

   langchainragperformanceprofiling

   Read more: Measured LangChain's overhead in our RAG pipeline, ended up moving it off the hot path

7. #7 67M-param LaTeX OCR model runs on CPU—open weights and training coderepo

   A tiny 67M-param model converts images to LaTeX on a laptop CPU, with a new style-aware dataset, all MIT-licensed.

   Why it matters: This makes LaTeX OCR accessible for local, private document processing without GPU requirements.

   How to apply: Grab the weights and code from the repo; integrate into your document pipeline for math/formula extraction.

   ocrlatexlocalopen source

   Read more: I trained a 67M-param LaTeX OCR model that runs on a laptop CPU — and built a new style-aware dataset to train it. Weights, data, and training code all open (MIT). · I trained a 67M-param LaTeX OCR model that runs on a laptop CPU — and built a new style-aware dataset to train it. Weights, data, and training code all open (MIT).

8. #8 Krea2 Turbo Distill 4-step LoRA reaches 8-step teacher parityrepo

   New Krea2 LoRA checkpoint (chk42K) delivers 4-step generation with texture/detail matching the 8-step teacher, trained at 1440×1440.

   Why it matters: Faster image generation with fewer steps means lower latency and cost for production image pipelines.

   How to apply: Download the LoRA from the release and use it in ComfyUI/Stable Diffusion with 4-step samplers; adjust prompts for prompt-aware training.

   image genlorakrea2distillation

   Read more: Krea2 Turbo Distill 4 step LoRA - new checkpoint (chk42K) released (texture and detail now at 8-step teacher parity, prompt-aware training added, NF4 fully retired for full-int8 training, 1440×1440 now a trained resolution)

9. #9 MiniMax H3 fused turbo int8 checkpoint merges text, image, and video in one modelrepo

   A single all-in-one MiniMax H3 checkpoint supports text-to-video, image-to-video, and 4-step turbo generation in int8.

   MODEL RELEASE

   One int8 checkpoint, three video modes

   Single ComfyUI checkpoint replaces multiple models, cutting VRAM with int8.

   Why it matters: This simplifies video generation workflows and reduces VRAM requirements with int8 quantization.

   How to apply: Use the MATLOWAI/minimax-h3-fused-turbo-int8-convrot checkpoint in ComfyUI; no need to switch between models.

   video genminimaxint8comfyui

   Read more: MATLOWAI/minimax-h3-fused-turbo-int8-convrot · Hugging Face

10. #10 AURA Harness: a thin, local-first agent harness that conforms to your looptool

    AURA Harness wraps your existing agent loop (Ollama, LangGraph, etc.) with recording and optional gating, without forcing a rewrite.

    Local-first agent harness

    A thin membrane over your existing loop

    AURA Harness

    recording + optional gating

    Your agent loop

    Ollama, LangGraph, etc.

    No rewrite — AURA conforms to what you already run.

    Why it matters: Most harnesses force you to rebuild your runtime; AURA is a membrane that adds observability and control to what you already have.

    How to apply: Install AURA and point it at your existing agent loop; it works offline with session IDs and has an Ollama integration.

    agentsharnesslocalollama

    Read more: Built an offline harness that conforms to your agent loop, not the other way around

11. #11 Tokenly: an open-source macOS dock that shows Claude Code usage limitstool

    Tokenly puts a ring on your screen edge showing Claude, Codex, and Gemini usage, with warnings at 80/90/95%.

    Why it matters: Avoid hitting Claude Code session limits mid-work; visibility into usage prevents surprise interruptions.

    How to apply: Download the open-source app and configure your provider API keys; it reads the usage API and displays live limits.

    claudelimitsmacosopen source

    Read more: I got tired of hitting my Claude limits blind, so I built a dock with Claude Code that shows them on the edge of my screen. Free and open source

12. #12 Popperian falsification prompt: force your LLM to attack your plantechnique

    A prompt framework that makes the AI act as a harsh auditor, trying to falsify your premise with realistic stress tests.

    Prompting technique

    Force your LLM to attack your plan

    Default LLM

    • Agrees with your premise
    • Seeks confirming evidence
    • Misses edge cases

    vs

    Popperian prompt

    • Acts as harsh auditor
    • Tries to falsify your plan
    • Runs realistic stress tests

    Surfaces flaws before you invest time/money.

    Use the prompt template when evaluating business models, architectures, or decisions.

    Why it matters: LLMs default to confirmation bias; this technique surfaces flaws before you invest time/money.

    How to apply: Use the provided prompt template when evaluating business models, architectures, or decisions; ask for the top 5 ways your plan fails.

    promptingreasoningfalsificationdecision making

    Read more: I built a Popperian Falsification Prompt to stop LLM confirmation bias (Testing your plans against plausible black swans)