Local Inference Tuning Wins, an Open-Source Agent Harness, and MiniMax H3's ComfyUI Tooling Matures

Today's findings

1. #1 Manual MoE-Offload Tuning Nearly Doubles Prompt-Processing Speed on a Single 3090technique

   Hand-tuning which MoE expert layers get offloaded to CPU (plus batch size) took prompt processing from 564 to 1330 tok/s on a 24GB 3090 running Qwen3.6-35B-A3B, with decode speed unchanged.

   llama.cpp · MoE offload tuning

   Hand-tuned expert offload takes prompt processing from 564 to 1330 tok/s

   Auto-fit heuristic

   • 564 tok/s prompt processing
   • All expert layers on GPU
   • -b 512 · -ub 128
   • VRAM headroom left unused

   → 2.36x faster prompt processing

   Manual tuning

   • 1330 tok/s prompt processing
   • 8 expert layers spilled to CPU
   • -b 1024 · -ub 512
   • Decode speed unchanged

   Qwen3.6-35B-A3B on one 24GB RTX 3090 — same hardware, benchmark PP and TG separately.

   Why it matters: llama.cpp-style auto-fit heuristics leave real throughput on the table for MoE models; manually spilling eight expert layers to CPU freed enough VRAM to double batch size (-b 512→1024, -ub 128→512) for a 2.36x prompt-processing win at no hardware cost.

   How to apply: If you self-host MoE GGUF models on a single consumer GPU, don't trust auto-fit — manually offload a handful of expert layers to CPU, then raise -b/-ub until VRAM is full again, and benchmark PP and TG separately since they scale differently.

   local-llmllama.cppquantizationperformance

   Read more: Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090) · Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)

2. #2 Run GPT-OSS-120B on a 16GB MacBook Air by Streaming Experts Straight From SSDtechnique

   A dependency-free Swift/Metal runtime keeps only the ~2GB dense trunk of GPT-OSS-120B resident in RAM and streams the active experts per token from disk, running the 120B model on a 16GB laptop at 1.4 tok/s.

   GPT-OSS-120B on Apple Silicon

   Dense trunk stays in RAM; experts stream off SSD per token

   HOT

   Unified RAM ~2GB dense trunk, always resident

   WARM

   Stream buffer Only the experts this token activates

   COLD

   SSD Full 120B checkpoint, read on demand

   1.4 tok/s on a 16GB MacBook Air — no quantization crush, dependency-free Swift/Metal runtime.

   Why it matters: Sparse MoE models only touch a few percent of weights per token, so disk footprint doesn't have to define your memory budget — a reusable insight for running large open-weight models on modest hardware without crushing quality via quantization.

   How to apply: For memory-constrained deployments, identify which layers of your MoE model are actually touched per forward pass and stream inactive experts from NVMe instead of loading the full checkpoint; the trunk-resident/experts-on-demand pattern is worth copying outside Swift/Metal too.

   local-llmmoequantizationapple-silicon

   Read more: GPT-OSS-120B on a 16GB MacBook Air, by streaming experts off SSD (Swift/Metal, no deps)

3. #3 50% Token-Speed Gain on DeepSeek-V4-Flash at 128K Context, Single 3090technique

   A detailed GGUF/llama.cpp tuning pass pushed DeepSeek-V4-Flash-0731 from ~10 to ~15 tok/s at full 128K context on one RTX 3090.

   Local inference · GGUF tuning

   DeepSeek-V4-Flash went ~10 → ~15 tok/s at full context — same GPU, different flags

   15

   tok/s at 128K context

   +50% vs ~10 tok/s baseline

   128K

   full context, no truncation

   1× RTX 3090

   no new hardware

   KV + offload

   cache placement and CPU/GPU split

   Tuning pass on DeepSeek-V4-Flash-0731 GGUF under llama.cpp; approach transfers to other large MoE GGUFs.

   Why it matters: Long-context local inference is usually VRAM- and bandwidth-bound; this writeup documents specific flag and offload choices that got a 50% speedup with no new hardware, transferable to other large MoE GGUFs.

   How to apply: Replicate the tuning pass on your own DeepSeek/Qwen GGUF setup — check context-length-dependent KV cache placement and CPU/GPU split settings from the linked thread before assuming you're stuck at out-of-the-box speed.

   local-llmggufperformancedeepseek

   Read more: Final optimization: from ~10 tok/s to ~15 tok/s on DeepSeek-V4-Flash-0731 at 128K ctx - 1 RTX 3090 · Optimisation finale : de ~10 tok/s à ~15 tok/s sur DeepSeek-V4-Flash-0731 à 128K ctx - 1 RTX 3090

4. #4 Prime Intellect Open-Sources Prime Agent, a Sub-Agents-as-Function-Calls Harnessrepo

   Prime Agent runs sub-agents as ordinary function calls inside a persistent IPython kernel, and paired with an Opus 5 backend it reportedly scored 95% on ARC-AGI-3.

   Agent orchestration

   Two ways to wire sub-agents together

   Message passing

   • Orchestration bolted on top
   • State hidden between agents
   • Intermediate results serialized
   • Framework-specific protocol

   vs

   Function calls

   • Sub-agent = ordinary call
   • Persistent IPython kernel
   • Shared, inspectable state
   • Open-source harness

   Prime Agent on an Opus 5 backend: 95% reported on ARC-AGI-3

   Prime Intellect's Prime Agent runs sub-agents inside one stateful kernel.

   Why it matters: Most multi-agent frameworks bolt orchestration on top via message-passing; running sub-agents as callable functions in a shared, stateful kernel keeps intermediate state directly inspectable instead of hidden between agents.

   How to apply: If you're building an in-house agent orchestrator, evaluate the 'sub-agent = function call in a persistent kernel' pattern as an alternative to message-passing frameworks — it's open-source, so read the harness code directly rather than reverse-engineering the idea.

   agentsopen-sourceclaudebenchmarks

   Read more: Prime Intellect Releases Prime Agent: An Open-Source RLM Harness Where Sub-Agents Are Function Calls Inside Persistent IPython Kernel · Prime Agent scores 95% on ARC-AGI-3 (with Opus 5 backend)

5. #5 A Local-First Hybrid Retrieval Stack for Agent Memory: BM25 + pgvector + RRFrepo

   An open build of an agent memory service fuses Postgres full-text search (BM25) and pgvector embeddings via Reciprocal Rank Fusion, running entirely on local Ollama embeddings.

   Why it matters: It's a concrete, replicable reference architecture for giving agents persistent memory without a dedicated vector DB or cloud embedding API — useful when you want RAG/memory without adding new infra.

   How to apply: For an internal agent memory or RAG layer, mirror the recipe: Postgres tsvector + pgvector + RRF fusion, with local embedding/rerank models (e.g. qwen3-embedding) served via Ollama, instead of standing up a separate vector database.

   agentsragollamamemory

   Read more: Built a local-first hybrid retrieval memory layer for agents (BM25 + pgvector + RRF + local Qwen3 embeddings/rerank)

6. #6 A Fail-Closed Policy Engine Sitting in Front of LangGraph Agent Actionstechnique

   A standalone policy engine evaluates every agent action before execution and blocks by default if it can't confirm safety, logging the exact rule that triggered a block.

   Why it matters: Most agent guardrails are prompt-level and fail open; a deterministic policy layer that defaults to 'deny' unless a rule explicitly clears the action is a stronger pattern for anything touching money, infra, or user data.

   How to apply: Wrap agent tool calls in a pre-execution policy check that defaults to deny rather than allow, and log the triggered rule on every block — the pattern applies even if you're not on LangGraph.

   agentssecurityguardrails

   Read more: Built a fail-closed authorization layer for LangGraph agents — here’s what a blocked decision actually looks like

7. #7 Kijai Ports Int8 ConvRot VAE Support for MiniMax H3 Into ComfyUItool

   Kijai's int8-quantized ConvRot VAE conversion for MiniMax H3's video VAE landed in ComfyUI, cutting the compute/VRAM cost of the decode step.

   ComfyUI · video VAE

   Int8 ConvRot VAE lands for MiniMax H3 — drop-in decode swap

   MiniMax H3 ConvRot VAE (int8)

   model int8 quantized

   MiniMax H3 video VAE → ConvRot conversion by KijaiAvailable now

   Kijai HuggingFace repoComfyUI VAE loaderExisting H3 workflow

   runSwap the VAE node — no other workflow changes

   Cuts compute and VRAM on the decode step, the local video bottleneck.

   Why it matters: VAE decode is a real bottleneck for local video generation; an official int8-quantized VAE path is a straightforward win for anyone running MiniMax H3 locally on consumer GPUs.

   How to apply: Pull the converted VAE from Kijai's HuggingFace repo and swap it into your existing MiniMax H3 ComfyUI workflow for a decode speed/VRAM improvement with no other workflow changes.

   video-generationcomfyuiquantizationopen-source

   Read more: Int8 convrot VAE support in Comfy

8. #8 First MiniMax H3 Turbo LoRA Lands for ComfyUI (T2V/I2V, Beta)repo

   A beta, undertrained Turbo LoRA for MiniMax H3 speeds up text-to-video and image-to-video generation, packaged for ComfyUI on HuggingFace.

   MiniMax H3 · ComfyUI

   The beta Turbo LoRA covers two of four H3 modes

   2 of 4 supported

   Text-to-video (T2V)

   Image-to-video (I2V)

   Video editing

   Reference-to-video (ref2v)

   pass warn fail

   Beta and undertrained — keep the base checkpoint loaded for the uncovered modes.

   Why it matters: Step-distillation LoRAs are one of the highest-leverage speedups for local video generation; even an early beta materially cuts render time, and the community is already organizing a shared prompt/output dataset to improve future versions.

   How to apply: Download the ComfyUI-compatible Turbo LoRA and drop it into existing H3 T2V/I2V workflows; it doesn't cover video-editing/ref2v modes yet, so keep the base checkpoint for those.

   video-generationcomfyuiopen-sourcelora

   Read more: Minimax H3 Turbo Lora · What if we created our own big dataset for video editing?

9. #9 Claude Can Now Search Back Through All Your Past Conversationstip

   Claude on paid plans (Pro/Max/Team/Enterprise) can now retrieve context from previous, unrelated conversations automatically, no setup required.

   Why it matters: Saves the recurring cost of re-explaining project context every new chat, and works retroactively across your existing conversation history.

   How to apply: Ask naturally — 'What did we discuss about X?' or 'find our conversation about Y' — and Claude will search prior sessions and pull in relevant context; confirm it's enabled on your account/plan.

   claudeproductivitytips

   Read more: claude can now search back through every past conversation you've ever had with it and pull the relevant one into what you're doing right now. didn't know it was tracking that much until i asked

10. #10 Gmail Threads on References + Matching Subject — Not References Alonetip

    Gmail silently starts a new thread if your auto-reply changes the subject line, even with correct In-Reply-To/References headers, unlike other mail clients.

    Email threading

    Gmail threads on headers AND a matching subject

    vs

    Gmail / Workspace

    Most mail clients

    In-Reply-To / References

    Required

    Required

    Subject must match

    Required

    Ignored

    Model rewrites subject

    Starts new thread

    Stays threaded

    Error on break

    None — silent

    Not applicable

    Gmail / Workspace wins the row Most mail clients wins the row

    Copy the original subject verbatim with a 'Re:' prefix — correct headers alone won't hold the thread.

    Why it matters: If you're building an LLM agent that drafts email replies for Workspace users, a model-generated subject line will quietly break threading with no visible error.

    How to apply: Don't let the model rewrite reply subject lines — copy the original subject verbatim (with 'Re:' prefix) alongside correct References/In-Reply-To chains whenever Gmail/Workspace is in your audience.

    agentsemailtips

    Read more: Gmail threads on References plus a matching subject, not References alone. What other email quirks should an LLM reply pipeline know about?

11. #11 Qwen3.8-2.4T-A95B ('Qwen3.8-Max') Open-Weight Release Slated for Next Wednesdaytool

    Alibaba is reportedly opening the weights of its largest Qwen3.8 MoE model (2.4T total / 95B active params) next week.

    architecture

    Qwen3.8-Max: 2.4T parameters on paper, 95B awake per token

    router

    top-3 gate

    Expert grid (illustrative)Routed active per token

    weighted

    merge

    2.4T

    Total params

    95B

    Active params

    ~4%

    Share activated

    Wed

    Reported open-weight date

    Why it matters: A frontier-scale open-weight release expands local/self-hosted options even though most teams will need multi-GPU or cloud rental to run it at full precision.

    How to apply: If you track open-weight releases for self-hosting or fine-tuning, watch for GGUF/quant conversions, which typically follow within days of a Qwen open release.

    open-sourcelocal-llmqwen

    Read more: Qwen3.8-2.4T-A95B (aka Qwen3.8-Max) open release time: next wednesday