Local MoE Tuning, Agent 'Receipts,' and Claude Context Packs

Today's findings

1. #1 Run 35B MoE models fast on a 12GB GPU by forcing experts onto CPUtip

   A shared LM Studio config gets Qwen3.6-35B-A3B running at 30 tok/s on a 3060 12GB + 32GB RAM by keeping attention/KV cache on GPU and MoE expert FFN weights on CPU.

   LM Studio Config

   35B MoE at 30 tok/s on 12GB GPU

   Model = Qwen3.6-35B-A3B MoE, ~3B active params

   Context = 262K full

   GPU offload = Attention layers

   CPU offload = MoE expert weights

   Hardware = RTX 3060 12GB + 32GB RAM

   Speed = 30 tok/s

   Replicate in LM Studio or llama.cpp with these settings.

   Why it matters: Since only ~3B params activate per token in this MoE model, CPU-side matmuls stay cheap, making a 35B-class model usable on hardware most teams already have instead of requiring a cloud GPU.

   How to apply: In LM Studio (or llama.cpp directly), set full 262K context, offload all attention layers to GPU, but force the MoE expert weights to CPU/system RAM — replicate the exact settings shared in the post for similarly-sized MoE checkpoints.

   moequantizationlocal-llm

   Read more: Qwen3.6-35B-A3B (MoE) running at 30 t/s on a 3060 12GB + 32GB DDR5, full 262K context, here's the LM Studio config

2. #2 Verify tool calls actually fired — don't trust agent narrationtechnique

   Agents will confidently report 'done' on a tool call that silently never executed, and re-asking them to confirm just produces the same false yes.

   Agent Reliability

   Don't trust agent narration — verify tool call side effects

   Trust narration

   • Agent says 'done' even if tool never ran
   • Re-asking produces same false yes

   vs

   Verify side effects

   • Check DB row, file, API status directly
   • Source of truth is side effect, not model

   Always verify tool call side effects directly.

   Failure mode reported same day across four communities (AI_Agents, LangChain, LLMDevs, PromptEngineering).

   Why it matters: This failure mode was independently reported the same day across four separate communities (AI_Agents, LangChain, LLMDevs, PromptEngineering), signaling a widespread, production-biting problem that gets worse with bigger, more articulate models.

   How to apply: Treat the model's self-report as untrustworthy; verify tool-call side effects directly (DB row exists, file was written, API returned the expected status) as your source of truth instead of prompting the agent to double-check itself.

   agentsreliabilityobservability

   Read more: Stopped trusting what my agent says it did. Started trusting receipts. · Stopped trusting what my agent says it did. Started trusting receipts. · Stopped trusting what my agent says it did. Started trusting receipts. · Silent tool failures don't care how good your model is. Stopped trusting narration, started trusting receipts

3. #3 Give Claude Code a 'context pack' instead of letting it grep coldtechnique

   A CLAUDE.md + docs/claude/INDEX.md decision-table setup stops Claude Code from re-discovering architecture and repeating the same mistakes every session on a large production app.

   CONTEXT PACK

   Two files that stop cold grepping

   CLAUDE.md

   Rules of engagement

   INDEX.md

   Decision table: task → doc

   Claude loads only the relevant architecture doc before editing.

   Why it matters: On large codebases, every fresh session previously burned time and tokens on blind exploration; a structured pre-read context pack fixes that at the source.

   How to apply: Add a CLAUDE.md with rules of engagement and a docs/claude/ folder with an INDEX.md mapping 'task about X → read flows/X.md,' so Claude loads only the relevant architecture doc before editing.

   claude-codecontext-managementagents

   Read more: My CLAUDE.md + docs/claude "context pack" setup — how I got Claude Code to stop grepping blindly through a large production codebase

4. #4 Same-family self-grading inflates eval scores by ~18 pointstip

   Grading a pipeline's output with a judge model from the same family as the model under test scored 96%, while an independent-family judge scored the identical outputs at 78.4%.

   LLM-as-Judge Bias

   Same-family self-grading inflates scores by ~18 points

   vs

   Same-family judge

   Independent judge

   Eval score

   96%

   78.4%

   Same-family judge wins the row Independent judge wins the row

   Using a judge from the same model family as the system under test overstates quality by a wide margin.

   Why it matters: Teams that self-grade for convenience are likely overstating quality by a wide margin — enough to change a ship/no-ship decision.

   How to apply: Use a judge model from a different provider/family than the system under test, and pre-register your scoring rubric and pass/fail bar before running the eval to avoid post-hoc goalpost-moving.

   evalsllm-judging

   Read more: Same-family self-grading inflated my eval by 17.6 points (96% to 78.4%). Pre-registered the retest, published the floor that barely held, grep-only repro in under a second, no key

5. #5 Open-sourced pipeline shrinks Qwen2.5-Coder-7B to a 1.9GB mobile GGUFtool

   A published quantization script compresses Qwen2.5-Coder-7B-Instruct from 15.2GB to a 1.9GB IQ1_S GGUF that runs natively on phones via PocketPal, weights hosted on Hugging Face.

   Open-Source Quantization Pipeline

   Qwen2.5-Coder-7B-Instruct compressed 87% for mobile

   Original (FP16)

   • 15.2 GB

   → 13.3 GB saved

   Mobile GGUF (IQ1_S)

   • 1.9 GB

   Runs natively on phones via PocketPal; weights on Hugging Face.

   Why it matters: Puts a genuinely capable offline coding model on hardware as small as a phone, useful for air-gapped, field, or low-resource deployment scenarios.

   How to apply: Download the released GGUF for immediate mobile/offline use, or reuse the open-sourced script to compress your own fine-tuned coder models to similar sizes.

   quantizationggufmobile

   Read more: I managed to squeeze Qwen2.5-Coder-7B into a 1.9GB GGUF (IQ1_S) so it runs natively on Mobile! · I managed to squeeze Qwen2.5-Coder-7B into a 1.9GB GGUF (IQ1_S) so it runs natively on Mobile! · I managed to squeeze Qwen2.5-Coder-7B into a 1.9GB GGUF (IQ1_S) so it runs natively on Mobile! 📱

6. #6 MiniMax-M3-uncensored NVFP4 cuts weights from 795GiB to 242GiBtool

   An NVFP4 quantization pass makes MiniMax-M3-uncensored practical on a single 4x96GB Blackwell node instead of requiring a multi-node cluster.

   Why it matters: Roughly a 3x size reduction moves a frontier-scale open-weight MoE model from cluster-only territory into single-workstation reach.

   How to apply: If you have Blackwell-class GPUs, grab the NVFP4 release instead of full precision to cut hardware requirements substantially for the same model.

   quantizationmoelocal-llm

   Read more: MiniMax-M3-uncensored NVFP4 update: 795.5 GiB to 242.4 GiB, now practical on one 4x 96 GB Blackwell node

7. #7 llama.cpp CUDA patch gives ~5% speedup for Q1_0 quantized modelsrepo

   A merged llama.cpp PR extracts Q1_0 quantized elements via __byte_perm, yielding roughly a 5% token-generation speedup for Bonsai-class models on CUDA.

   llama.cpp CUDA patch

   Free speedup for ultra-low-bit models

   ~5%

   faster token generation

   No accuracy tradeoff

   Same model quality

   Q1_0 quantized only

   Bonsai-class models on CUDA

   PR #25628

   Merged into llama.cpp

   Update to a build including PR #25628 if serving Q1_0 on NVIDIA GPUs.

   Why it matters: A free inference speedup with no accuracy tradeoff for anyone already running ultra-low-bit quantized models on NVIDIA GPUs.

   How to apply: Update to a llama.cpp build that includes PR #25628 if you're serving Q1_0-quantized models on CUDA hardware.

   llama.cppquantizationperformance

   Read more: cuda: extract Q1_0 elements via __byte_perm by dfriehs · Pull Request #25628 · ggml-org/llama.cpp

8. #8 skills-evals: CI regression testing for Claude/Copilot skills and Cursor rulesrepo

   A zero-dependency CLI validates, trigger-tests, and regression-tests every Claude/Copilot skill or Cursor rule in a repo directly in CI.

   skills-evals

   Three-stage CI testing for skills

   1

   Validate

   Check syntax and structure

   2

   Trigger test

   Verify skill fires on cues

   3

   Regression test

   Compare against baseline

   Catches silent degradation

   Runs in CI to catch breakage before production.

   Why it matters: Skill and instruction files silently degrade as a codebase or its APIs evolve, and this catches breakage before it causes bad agent behavior in production.

   How to apply: Add skills-evals to your CI pipeline so builds fail when a code change causes a skill to stop triggering or start behaving incorrectly.

   claude-codecitesting

   Read more: Skills evals

9. #9 DocMind: open-source, self-hosted NotebookLM alternativetool

   DocMind is a self-hosted RAG app with PDF/DOCX chat, source citations, multi-org knowledge bases, and one-command Docker deploy, built to avoid locking documents into Google's ecosystem.

   Open-source RAG tool

   Self-hosted NotebookLM alternative

   DocMind

   tool open-source

   RAG app with PDF/DOCX chatAvailable now

   Docker

   runOne-command deploy

   Keeps documents on your own infrastructure, avoiding cloud lock-in.

   Why it matters: Gives teams a citable document-chat tool that stays entirely on their own infrastructure, relevant wherever data residency or IP concerns rule out cloud-hosted RAG.

   How to apply: Deploy via the provided Docker setup for an internal, embeddable document-chat widget with a public API for integration into existing tools.

   ragopen-sourceself-hosted

   Read more: I built an open-source, self-hosted NotebookLM alternative

10. #10 Guard against agent 'slopsquatting' before postinstall runstip

    A coding agent hallucinated and installed 'loadash' instead of 'lodash' — in agent mode a hallucinated package name isn't a harmless typo, it's a real npm install with a live postinstall script.

    Why it matters: Bigger, more capable models don't reduce this risk; the failure looks completely clean until a malicious or nonexistent package's install script has already run.

    How to apply: Diff-review dependency changes before allowing installs to run, pin against known lockfiles, and consider grounding agents with real package-doc tools (e.g. Context7 MCP) instead of relying on model recall for package names.

    agentssecuritysupply-chain

    Read more: My coding agent installed loadash. how do you hard-block fake packages before postinstall runs

11. #11 Abliterated Qwen3-VL-4B text encoder unblocks Krea 2 prompt refusalstool

    An abliterated version of Krea 2's Qwen3-VL-4B text encoder hits 100% HarmBench compliance (up from 30.8%) with GSM8K/MMLU essentially unchanged, packaged as drop-in ComfyUI checkpoints.

    Why it matters: Demonstrates abliteration can strip refusal behavior from a vision-language text encoder with minimal capability loss, and clarifies the encoder mainly affects prompt handling, not image-output censorship.

    How to apply: Swap the released checkpoint into existing Krea 2 ComfyUI workflows where legitimate prompts are being blocked by refusals.

    comfyuiopen-weightsfine-tuning

    Read more: Uncensored Qwen3-VL-4B text encoder for Krea 2 · Qwen3-VL-4B-Instruct Heretic for ComfyUI

12. #12 SPROG-9M: a 9M-parameter, LLM-free model for grade-school mathpaper

    SPROG-9M tests how far a tiny, non-LLM specialized architecture gets on grade-school math problems without any language model in the loop.

    Why it matters: A useful data point for deciding when a narrow, well-defined subtask in a pipeline needs a full LLM call versus a dedicated model that's orders of magnitude cheaper to run.

    How to apply: Evaluate SPROG-9M-style specialized models before defaulting to an LLM call for narrow numeric/reasoning subtasks embedded in a larger agent pipeline.

    efficiencysmall-modelsmath

    Read more: SPROG-9M: how far a 9-million-parameter, LLM-free model gets on grade-school math · SPROG-9M: how far a 9-million-parameter, LLM-free model gets on grade-school math