MoE Memory Tricks and a Wave of Open-Weight Coding Models

Today's findings

1. #1 VRAM-as-cache MoE offloading trick unlocks trillion-parameter models on a single DGX Sparktechnique

   A llama.cpp CUDA offload strategy treats VRAM as a cache over disk-resident MoE experts, hitting 340 pp/s and 9.6 tg/s for a 1T-param Kimi K2.7 model on one DGX Spark.

   llama.cpp MoE offload

   VRAM as a cache: hot experts resident, cold experts streamed from disk

   HOT

   VRAM Active MoE experts pinned via -ot regex

   WARM

   System RAM Unified-memory spillover (CUDA unified flag)

   COLD

   Disk Idle experts of 1T-param Kimi K2.7 GGUF

   Single DGX Spark reaches 340 pp/s and 9.6 tg/s on a 1T-param MoE model

   Why it matters: Teams without multi-GPU racks can run trillion-parameter MoE models locally by exploiting llama.cpp's offload regex and unified-memory flags instead of being hard-blocked by VRAM size.

   How to apply: Use llama.cpp with an `-ot` regex targeting MoE expert tensors plus `GGML_CUDA_ENABLE_UNIFIED_MEMORY` and `GGML_OP_OFFLOAD_MIN_BATCH` env flags on GGUF MoE checkpoints (e.g. Kimi-K2.7-Code IQ_S) to keep hot experts resident in VRAM while cold ones stream from disk.

   moellama.cpplocal-llmquantization

   Read more: VRAM disk cache of MoE makes 340 pp/s 9.6 tg/s for Kimi 2.7 on a single dgx spark

2. #2 Poolside open-sources Laguna S 2.1, a strong open-weight coding model now in llama.cpprepo

   Poolside released Laguna S 2.1 (118B open-weight agentic coding model) that punches above its weight on SWE-Bench Multilingual, and llama.cpp added day-one support.

   Open-weight coding model

   Laguna S 2.1 lands, self-hostable via llama.cpp

   Laguna S 2.1

   model 118B agentic

   Poolside · open weights2026

   Hugging Face: poolside/Laguna-S-2.1llama.cpp day-one (PR #25165, b10087)GGUF for local self-host

   runhuggingface.co/poolside/Laguna-S-2.1

   Punches above its weight on SWE-Bench Multilingual; no code leaves your box.

   Why it matters: A capable open-weight coding model you can self-host via llama.cpp/GGUF gives teams a Claude Code-style agentic coding option without sending code to a third party.

   How to apply: Pull the llama.cpp build with PR #25165 support (release b10087) and grab weights from huggingface.co/poolside/Laguna-S-2.1; for coding/benchmarking use, patch the chat template to force a newline after `<think>` so it reliably reasons on medium-hard tasks instead of skipping straight to an answer.

   open-weightsllama.cppcoding-agentlocal-llm

   Read more: Poolside Releases Laguna S 2.1, an Open-Weight Agentic Coding Model Punching Above Its Weight Class on SWE-Bench Multilingual · Add support for Laguna XS.2 & M.1 by joerowell · Pull Request #25165 · ggml-org/llama.cpp · Llama.cpp just added support for Laguna XS.2 & M.1 · Force <thinking> in Laguna-S-2.1

3. #3 SkewAdam cuts MoE optimizer memory 97%, fits a 6.7B MoE fine-tune on 40GBpaper

   A tiered-optimizer-state technique (SkewAdam) slashes Adam's memory footprint for MoE fine-tuning by 97%, making a 6.7B MoE trainable on a single 40GB GPU.

   SkewAdam · MoE fine-tuning

   Tiered optimizer state slashes Adam's memory footprint

   97%

   less optimizer memory for MoE fine-tuning

   6.7B MoE now fits on a single 40GB GPU

   6.7B

   MoE model fine-tuned

   40GB

   one GPU, no A100/H100 cluster

   drop-in

   replaces full Adam optimizer

   Optimizer state, not weights, was the real ceiling on local MoE tuning.

   Why it matters: Optimizer state, not weights, usually blocks fine-tuning MoE models locally — this removes that ceiling for teams without A100/H100 clusters.

   How to apply: Read the paper (arxiv.org/abs/2607.19058) and reference implementation on GitHub before your next MoE fine-tuning run, and swap in SkewAdam as a drop-in optimizer where full Adam state currently forces you to rent bigger GPUs.

   fine-tuningmoeoptimizerpaper

   Read more: SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]

4. #4 A ~200-question probe can verify a Claude API endpoint is actually Claudepaper

   A paper shows LLMs have strongly biased 'random number' preferences, and that signature can fingerprint which model is actually behind an API endpoint.

   Model fingerprinting · arXiv 2607.10252

   A ~200-question probe reveals which model is really behind a Claude API endpoint

   1

   Batch prompts

   ~200 'pick 1-100'

   2

   Measure spread

   response entropy

   3

   Match fingerprint

   vs known Claude

   Biased random picks are a per-model signature

   4

   Verdict

   genuine or swapped

   Detects a silent swap to a cheaper model on relays or aggregators.

   Why it matters: Anyone routing Claude API calls through a third-party relay or aggregator has no easy way to confirm they aren't being silently swapped to a cheaper model — this gives a cheap verification test.

   How to apply: Send a suspect endpoint a batch of 'pick a random number 1-100' style prompts, measure response entropy/distribution, and compare against known Claude fingerprints from the paper (arXiv 2607.10252) before trusting production traffic to it.

   claudeverificationpaper

   Read more: is your Claude API actually Claude? a paper shows you can verify it with ~200 tiny questions

5. #5 In-place tokenizer expansion recipe upgrades a pretrained model's vocab without retrainingtechnique

   Liquid AI published the recipe used to upgrade LFM2.5-8B-A1B's tokenizer in place, avoiding a full retrain from scratch.

   Tokenizer upgrade

   Adding vocab without a from-scratch retrain

   Full retrain

   • New tokenizer = train from zero
   • Discards pretrained checkpoint
   • Costly, slow

   vs

   In-place expansion

   • Warm-start new embeddings
   • Aligns to existing model
   • Add domain tokens on a live checkpoint

   Liquid AI's recipe upgraded LFM2.5-8B-A1B's vocab in place

   Reusable for customizing open-weight models to domain-specific vocabularies

   Why it matters: Swapping tokenizers normally means a from-scratch retrain; a working in-place upgrade path is directly reusable when customizing open-weight models for a domain-specific vocabulary.

   How to apply: Follow the embedding-alignment/warm-start steps in the writeup when you need to add domain tokens (code symbols, jargon) to an existing open-weight checkpoint instead of retraining from zero.

   tokenizerfine-tuningopen-weights

   Read more: Tokenizer Expansion: Upgrading a Model's Tokenizer in Place - LFM2.5-8B-A1B

6. #6 A third of popular MCP servers grade D or F on agent usabilitytip

   An independent audit graded 36 popular MCP servers on how well agents can actually use them, finding a third scored D or F.

   Why it matters: Tool descriptions and schemas that read fine to humans often confuse agents — this quantifies which common MCP servers are worth trusting before you wire them into Claude Code or another harness.

   How to apply: Check the grading criteria and results before adopting a new MCP server, and apply the same usability checklist (clear schemas, useful error messages, unambiguous params) when writing internal MCP servers.

   mcpagentstooling

   Read more: I graded 36 popular MCP servers on agent usability. A third got a D or F

7. #7 whatbroke: a trajectory-diff CLI that catches silent agent regressions after a model swaptool

   An open-source CLI diffs recorded agent tool-call trajectories, not just text output, so a model swap that silently drops a critical tool call gets caught before production.

   whatbroke · trajectory diff

   Same text output, one tool call silently gone after a model swap

   Before swap (model A)

   • lookup_order()
   • check_policy()
   • cancel_order()
   • reply to user

   → cancel_order() dropped — text still reads fine, vibe check passes

   After swap (model B)

   • lookup_order()
   • check_policy()
   • reply to user

   Diffs recorded tool-call traces (JSONL), not text; non-zero exit blocks the regression in CI.

   Why it matters: Text-level evals and 'vibe checks' can pass while an agent quietly stops calling something like a cancellation tool — this closes a real gap for anyone swapping models behind a coding or support agent.

   How to apply: Record a JSONL trace of tool calls/args/costs before and after a model or prompt change, run `whatbroke diff before.jsonl after.jsonl`, and wire the non-zero exit code into CI to block silent behavioral regressions.

   agentstestingtool-calling

   Read more: A model swap silently broke my agent's cancellations, so I built a diff for agent behavior · A model swap silently broke my agent's cancellations, so I built a diff for agent behavior

8. #8 Frugal plugin routes Claude Code sub-tasks to the cheapest model that can do themtool

   An open-source Claude Code plugin turns the main model into a router, sending mechanical sub-tasks to shell commands, Haiku, or Sonnet and reserving Opus/Fable for design and debugging.

   Frugal · Claude Code plugin

   Route each sub-task to the cheapest model that can do it

   Fable

   Ceiling — hardest reasoning

   Main model

   Design & debugging

   Sonnet

   Mid sub-tasks

   Haiku

   Cheap sub-tasks

   Shell

   No model — locate & grep

   Mechanical work drops to the bottom tiers, cutting token spend per session

   Why it matters: Most of what a coding agent does in a session — locating files, grepping logs, mechanical edits — doesn't need frontier-model reasoning, so this directly cuts token spend on Claude Code sessions.

   How to apply: Install the Frugal plugin in Claude Code and let it tier sub-tasks (no-model shell command → Haiku → Sonnet → main model → Fable as ceiling) instead of paying top-tier rates for locate-and-extract work.

   claude-codecost-optimizationagents

   Read more: I built Frugal: a plugin that routes Claude Code work to the cheapest model that can do it

9. #9 Check your hybrid-search score column before trusting a retrieval-quality alarmtip

   A team nearly declared a production retrieval outage after 62% of searches showed 'low' scores — the real bug was RRF fusion scores and cosine-similarity scores sharing one monitoring column with incompatible ranges.

   Why it matters: Anyone running hybrid vector+BM25 retrieval with RRF fusion and a single quality threshold is at risk of the same false alarm, or worse, of missing a real regression because the threshold means nothing.

   How to apply: Audit your retrieval monitoring pipeline to confirm RRF-fused scores and raw cosine-similarity scores aren't logged to the same column/threshold, and store the fusion method alongside each score so a 'top_score < 0.3' alert means the same thing every time.

   ragretrievalmonitoring

   Read more: Our monitoring said 62% of retrievals were failing. The real bug: RRF scores stored in the same column as cosine similarities

10. #10 Atlas-Coder-2-0.5B: a top-5 sub-1B coding model fine-tuned on a free Kaggle GPUrepo

    A solo dev fine-tuned Qwen2.5-Coder-0.5B on 50K execution-verified Python samples using a free Kaggle T4 and beat Qwen2.5-Coder and a larger DeepSeek-Coder on EvalPlus.

    Why it matters: It's a concrete, reproducible proof that meaningful coding-model gains are achievable with free-tier compute and a curated, verified dataset rather than scale.

    How to apply: Grab the open model, dataset, and GGUF files from Hugging Face to run locally, or replicate the recipe — execution-verified sample filtering plus fine-tuning on a free Kaggle T4 — for your own narrow, task-specific coding model.

    fine-tuningcoding-modelopen-weights

    Read more: Atlas-Coder-2-0.5B: I built a Top 5 Sub-1B coding model on a free Kaggle GPU. It beats Qwen2.5 and DeepSeek on EvalPlus. · Atlas-Coder-2-0.5B: I built a Top 5 Sub-1B coding model on a free Kaggle GPU. It beats Qwen2.5 and DeepSeek on EvalPlus.

11. #11 Upstage releases Solar Open 2, a 250B-A15B open-weight MoE competitive with DeepSeek V4 Flashrepo

    Upstage's Solar Open 2 (250B-A15B, hybrid-attention MoE with linear attention) targets agentic office/coding work and benchmarks on par with DeepSeek V4 Flash.

    architecture

    Solar Open 2: 250B total, only ~15B active per token

    router

    top-2 gate

    Expert pool (sparse MoE)Activated per token

    weighted

    merge

    250B

    total params

    15B

    active / token

    ≈ DeepSeek V4 Flash

    benchmark parity

    Why it matters: Another strong open-weight MoE option widens the field for teams choosing a self-hostable agentic model instead of defaulting to a closed API.

    How to apply: Pull the weights from Hugging Face (upstage/Solar-Open2-250B) and benchmark it against your current self-hosted model on your actual agentic/document workloads before committing infrastructure to it.

    open-weightsmoelocal-llm

    Read more: upstage/Solar-Open2-250B · Hugging Face · Upstage 'Solar open2' release. performance on par with DeepSeek V4 Flash.