Gradient-Free Weight Surgery Steals the Show, Plus Qwen Decode Tricks and Agent Restraint Skills

Today's findings

1. #1 Closed-Form Weight Surgery Transfers 4B Capabilities into 0.8B Without Backproppaper

   A 0.8B student beats a 2B baseline on ARC-Challenge after ~12 minutes of closed-form trajectory matching on consumer hardware — no gradients, no billions of tokens.

   Gradient-free distillation

   0.8B student tops a 2B baseline on ARC-Challenge

   ~12 min

   closed-form weight surgery on consumer hardware

   no gradients · no billions of tokens

   4B→0.8B

   teacher capabilities transferred via hidden-state trajectory matching

   0

   gradient steps — regularized least-squares solve

   1

   consumer machine, no GPU cluster

   Capture teacher/student trajectories on a small calib set, solve for the MLP updates, lm-eval, then merge.

   Why it matters: Standard distillation is expensive; this shows you can transfer capability by solving for MLP weight updates directly from layer-to-layer hidden-state trajectories, making small-model specialization accessible without a GPU cluster.

   How to apply: Capture hidden-state trajectories from teacher and student on a small calibration set, solve regularized least squares with spectral projection for the student's MLP blocks, and validate with lm_eval before merging.

   fine-tuningdistillationlocal-llmquantization

   Read more: Closed-form trajectory weight surgery: transferring 4B capabilities into 0.8B without billions of tokens (why middle layers break, and how 4 anchor blocks fixed it) · A 0.8B model just beat a 2B model on ARC-Challenge (42.15%): Closed-form weight surgery beat multi-GPU SFT with 0 backprop (Independently verified on NVIDIA L4) · What if you don't need backprop? Our 0.8B model just hit 42.15% on ARC-Challenge, beating stock Qwen3.5-2B and crushing 25k SFT distillation (Verified on NVIDIA L4) · What if you don't need backprop? Our 0.8B model just hit 42.15% on ARC-Challenge, beating stock Qwen3.5-2B and crushing 25k SFT distillation (Verified on NVIDIA L4)

2. #2 NInfer Fork Hits 400 Tok/s Decode on Qwen 3.8 Flash Next with 8-Bit LM Head Drafttechnique

   Switching the LM head to 8-bit with MTP3 draft decoding lifts decode throughput 32-39% over 16-bit on an RTX 6000.

   Why it matters: Shows a concrete, low-effort inference optimization: quantizing the draft head and enabling multi-token prediction can nearly double throughput at long context without retraining.

   How to apply: Fork NInfer, enable --lm-head-draft with MTP3, and benchmark 8-bit vs 16-bit LM head at 512/8K/32K context on your own hardware before committing.

   inferencequantizationlocal-llmllama-cpp

   Read more: NInfer6000 - Qwen 3.8 Flash Next @ 400 tg/s & 13K pp/s · NInfer6000 - Qwen 3.8 Flash Next @ 400 tg/s & 13K pp/s

3. #3 Ponytail Skill Stops Local Coding Agents from Over-Buildingtip

   A ~1,070-word SKILL.md (or 438-word AGENTS.md variant) tells the agent to check whether code needs to exist, reuse stdlib/native features, and only then write the minimum.

   Coding agents · Ponytail skill

   Three gates before a line of code

   1

   Need check

   Does this code need to exist at all?

   2

   Reuse first

   Stdlib & native features before new deps

   3

   Write minimum

   Smallest viable diff — nothing more

   Code is the last resort, not the default

   A ~1,070-word SKILL.md, ~1.4K tokens of prefill, zero extra VRAM.

   Why it matters: Local coding agents default to dependency-heavy solutions; a tiny prompt file costs ~1.4k tokens of prefill and no extra VRAM, and directly reduces review churn.

   How to apply: Drop the SKILL.md into your agent's skills directory (or the AGENTS.md variant for small-context models) and use the review command to audit diffs for unnecessary abstractions.

   agentspromptinglocal-llmcoding

   Read more: Ponytail Review: The Lazy Senior Dev Skill

4. #4 Tuning llama.cpp on an 8 GB Gaming PC Yields 2x-9x Speedupstechnique

   Same RTX 4060 Ti, same models — only config changed — produced 2x to 9x throughput gains over download defaults, with headless Linux beating Windows.

   Why it matters: Most local-LLM performance is left on the table by default settings; a systematic one-change-at-a-time tuning pass is cheaper than buying new hardware.

   How to apply: Benchmark your current model/quant/context, then vary one llama.cpp flag at a time (threads, batch, KV cache, offload) on both Windows and headless Linux, keeping a results table.

   local-llmllama-cppinferenceperformance

   Read more: I turned my gaming PC into a inference machine and got 2x to 9x over default llama.cpp on an 8 GB card · I turned my gaming PC into a inference machine and got 2x to 9x over default llama.cpp on an 8 GB card

5. #5 Core Pinning on RK3588 Gives +318% Ollama Throughputtechnique

   Pinning Ollama to the four A76 big cores instead of the default eight threads avoids stalls on slower A55 little cores, delivering up to a 3.2x speedup.

   EDGE LLM · RK3588

   Fewer threads, up to 3.2× the throughput

   Default: 8 threads

   • Work spreads across all cores
   • Slow A55 little cores stall
   • Little cores bog down A76s

   → +318% throughput

   Pinned: 4 A76 cores

   • Only the 4 A76 big cores run
   • Little cores idle, no stalls
   • Big cores stay busy

   Verify with taskset/cgroup pinning — and watch thermals on sustained load.

   Why it matters: Edge and SBC deployments often leave huge performance on the table by letting the scheduler spread work across heterogeneous cores.

   How to apply: On RK3588 boards, set Ollama to 4 threads (big cores only), verify with taskset/cgroup pinning, and monitor thermals since sustained load changes the picture.

   local-llmollamaedgeperformance

   Read more: [Benchmark] Running Local LLMs on Orange Pi 5 Plus (RK3588, 16GB): Ollama Tok/s, NPU Offloading, Core Pinning & Thermals

6. #6 Audit Finds 13 Cases Where LLM Eval Tools Score Things They Didn't Measurepaper

   Reading the actual scoring code of seven eval tools (NVIDIA SkillEvaluator, MLflow, LangSmith, DSPy, DeepEval, Harbor, agent-skills) surfaced 13 reproducible cases where pass/fail wasn't backed by what was checked.

   Why it matters: If you gate releases on eval scores, silent scoring bugs can let regressions through; this is a concrete checklist of failure modes to audit for.

   How to apply: Read the scoring code of your eval stack, reproduce the reported cases, and add your own assertions that the metric actually measures the behavior you care about.

   evalsagentstestingclaude

   Read more: I read the scoring code of 7 tools people use to grade Claude Code skills and LLM apps (NVIDIA, MLflow, LangSmith, DSPy, DeepEval...). Found 13 cases where the result wasn't backed by what the tool actually measured. 3 were in my own Claude Code plugin.

7. #7 Ternary-Bonsai-4B Powers a Local 'System One' Coding-Agent Judgetechnique

   A frozen 2-bit 4B model answers multiple-choice, rating, and yes/no questions about a record in one forward pass using a tree attention mask — no adapter or fine-tuning.

   Ternary-Bonsai-4B

   Three question types, one forward pass

   Frozen 2-bit 4B weights: a tree attention mask shares the record across all branches, and answers are read straight from

   Why it matters: Gives you a cheap, local, single-pass judge for agent outputs, useful for routing, scoring, and gating without calling a hosted model.

   How to apply: Adapt the inference path to share the record prefix across questions with a tree attention mask, read answer probabilities from the existing LM head, and plug the MLX 2-bit build into your agent loop.

   local-llmagentsquantizationmlx

   Read more: I built a lightweight, local Jev-like System One with Ternary-Bonsai-4B — and used it as a coding-agent judge

8. #8 DeepSeek V4.1 Flash Runs on One DGX Spark via 113.6 GB VQ Base + 40 MB Domain Sidecarstechnique

   A native C/CUDA engine compresses 510 GB of weights into a 113.6 GB universal GGUF plus tiny per-domain sidecars, retaining 74-82% top-1 agreement.

   Why it matters: Shows a practical recipe for running very large MoE models on a single 128 GB unified-memory box by separating universal quantization from domain-specific scaling.

   How to apply: Quantize the base once, solve a small sidecar per domain, and keep the post-training file deletable so you can roll back; validate top-1 agreement against the original before shipping.

   quantizationlocal-llmggufinference

   Read more: DeepSeek V4.1 Flash on a single DGX Spark: 113.6 GB VQ base + 40 MB domain sidecars, 74–82% top-1 agreement vs original

9. #9 Tencent Open-Sources Octop, a Self-Hosted Multi-Agent Assistanttool

   Octop is an open-source, self-hosted AI assistant with a multi-agent architecture aimed at teams, families, and individuals.

   Open-source release

   Tencent ships Octop, a self-hosted multi-agent assistant

   Octop

   tool Open source

   From Tencent — built for teams, families, and individualsNewly open-sourced

   runClone the repo → point it at your local model endpoint

   A reference multi-agent architecture without vendor lock-in.

   Why it matters: A big-lab open-source release in the self-hosted assistant space gives teams a reference architecture for multi-agent collaboration without vendor lock-in.

   How to apply: Clone the repo, run it against your local model endpoint, and study its agent orchestration patterns before rolling your own.

   agentsopen-sourceself-hostedlocal-llm

   Read more: Tencent releases Octop, a self-hosted AI assistant

10. #10 Forge Is an MIT-Licensed, Self-Hosted Visual Builder for AI Agentstool

    Forge lets you wire agents, tools, RAG knowledge, and routing logic on a canvas, then test and ship without a hosted platform.

    Why it matters: Gives teams a self-hosted alternative to closed agent builders, with full control over data and deployment.

    How to apply: Self-host Forge, model your existing agent workflow on the canvas, and export or run it against your own model endpoints.

    agentsopen-sourceself-hostedrag

    Read more: I built Forge, an open-source, self-hosted visual builder for AI agents (MIT)

11. #11 Prism Preview Generates Native 2K Video and Audio in One Pass Under MITtool

    Tencent Hunyuan and Fudan's Prism preview checkpoints produce synchronized video and audio together at native 2K, released under MIT.

    Why it matters: Joint audio-video generation under a permissive license is a big deal for teams building media pipelines without proprietary APIs.

    How to apply: Pull the preview checkpoints from Hugging Face, wire them into ComfyUI, and test short clips before committing to a production pipeline.

    videoopen-sourcecomfyuimultimodal

    Read more: Prism (Tencent Hunyuan + Fudan) just dropped a preview: native 2K video + audio, MIT license · Prism (Tencent Hunyuan + Fudan) just dropped a preview: native 2K video + audio, MIT license

12. #12 Claude Project Memory Toggle Is On by Default and Retroactivetip

    Canary-code testing shows the 'Use account memory' toggle is enabled by default and switching it on pulls in prior project memory retroactively.

    Why it matters: Teams using Claude across projects may be leaking context between projects without realizing it; this is a privacy and data-hygiene issue worth checking today.

    How to apply: Audit the memory toggle per project, decide what should be shared, and use canary strings to verify what actually crosses project boundaries.

    claudeprivacymemoryenterprise

    Read more: Tested Claude's new project memory with canary codes: the "Use account memory" toggle is on by default, and switching it on is retroactive