Harness Over Weights: Same Model Jumps 0% to 36%, Plus Plan-Once Tradeoffs and Reading Costs

Today's findings

1. #1 Same Local Model, Three Agent Harnesses: SWE-bench Lite 0% to 36%technique

   Swapping the coding-agent harness around an unchanged local model moved SWE-bench Lite from 0% to 36%, dwarfing most model-choice gains.

   Agent harness ablation

   The harness was the lever, not the model

   36%

   SWE-bench Lite resolve rate under the best harness

   0% → 36% · model unchanged

   3

   harnesses tested around one untouched local model

   0%

   resolve rate under the weakest harness

   0

   new GPUs needed — the fix is free

   Tool loop, file editing, and test-feedback wiring dwarfed most model-choice gains.

   Why it matters: Teams often blame the local model when an agent fails, then buy GPUs; this shows the tool loop, file editing, and test-feedback wiring are the larger lever and are free to fix.

   How to apply: Run the same model under two or three harnesses on a small slice of your own repo tasks, log edit/run-test/retry steps, and adopt the loop that actually closes failures rather than the one with the best demo.

   agentslocal-llmevaluationharness

   Read more: Same local model, three coding agents: the harness moved SWE-bench Lite from 0% to 36%

2. #2 A 441-Example Weekend Fine-Tune That Kept Refusals and Visiontechnique

   Fine-tuning Qwen3.5-4B on 441 examples from one person's own work reduced lecturing without breaking safety refusals or vision.

   Why it matters: It shows the smallest useful data scale for behavior shaping is now a few hundred curated examples, which a single engineer can produce in a weekend instead of a data team.

   How to apply: Curate 300-500 examples from real transcripts, strip boilerplate, fine-tune a LoRA on a small Qwen variant, and diff behavior on a held-out set that includes refusal and multimodal cases before shipping.

   fine-tuninglocal-llmqwenlora

   Read more: Fine-tuned Qwen3.5-4B on 441 examples from my own work over a weekend. Less lecturing, same refusals, vision still works. Notes · Fine-tuned Qwen3.5-4B on 441 examples from my own work over a weekend. Less lecturing, same refusals, vision still works. Notes

3. #3 RAGFlow 1.0 Paper: Retrieval Is a Harness Problem, Reading Is the Costpaper

   Once retrieval becomes an agent loop, chunking and Recall@k stop being the bottleneck; the cost of reading what was retrieved takes over.

   RAGFlow 1.0 paper

   The bottleneck moves: from finding to reading

   Retrieval-bound

   • Tuned chunk size
   • Better embeddings
   • Recall@k races

   → Cost focus: retrieval → reading

   Reading-bound

   • Retrieval in a loop
   • Tokens read per query
   • Compiled summaries

   The agent already finds the right document — budget belongs in compilation, not chunking.

   Why it matters: Most RAG teams still optimize embeddings and chunk sizes long after their agent can already find the right document, burning budget on context instead of on compilation.

   How to apply: Instrument tokens read per answered query, compile documents into dense structured summaries or contracts the agent can consume in fewer passes, and let the agent decide when it has enough rather than fixing top-k.

   ragagentscontextpaper

   Read more: RAGFlow 1.0 preview: moving from RAG engine to agent-native database · RAGFlow 1.0 preview: moving from RAG engine to agent-native database

4. #4 AgentOrch: Plan-Once Workflows Trail Step-by-Step by 8.5 Points but Use 18x Fewer Callspaper

   Across 650 enterprise procedures, letting the model write the whole workflow upfront scored 8.5 points lower than a step-by-step agent while making 18x fewer LLM calls.

   AGENT ORCHESTRATION

   Buy accuracy with calls, or buy cost with planning — not both

   vs

   Plan-once

   Step-by-step agent

   Task accuracy

   8.5 pts lower

   8.5 pts higher

   LLM calls per task

   18× fewer

   18× more

   Branch decisions

   All fixed upfront

   Made as it goes

   Best fit

   High-volume, fixed recipes

   Ambiguous, branchy work

   Plan-once wins the row Step-by-step agent wins the row

   Across 650 enterprise procedures — each strategy wins on a different axis.

   Why it matters: It turns the ReAct-versus-planner argument into a budget decision: you can buy back accuracy with calls, or buy back cost with planning, but not silently get both.

   How to apply: Classify your procedures by ambiguity — use plan-once for high-volume, deterministic recipes and reserve step-by-step loops for tasks where branch conditions are unknown at authoring time.

   agentsorchestrationcostpaper

   Read more: Writing the whole workflow up front now trails a step-by-step agent by 8.5 points, with 18x fewer LLM calls: notes on “AgentOrch” (UNC Chapel Hill + Workato)

5. #5 Kanzo: One Tiny MCP Kanban Board Shared Across Machines and Agentstool

   Kanzo is a 4.5 MB binary using about 4.5 MB of RAM that gives you and every coding agent one shared kanban board over MCP.

   Why it matters: Agent coordination usually degenerates into copy-pasting task lists between sessions; a shared, near-zero-overhead board makes handoffs observable across machines.

   How to apply: Run the binary on one host, point each agent's MCP config at it, and use the board as the single source of task state before adding any heavier orchestration layer.

   mcpagentstoolingself-hosted

   Read more: Kanzo update: one kanban board shared across all your machines and agents, still a 4 MB binary using 4.5 MB of RAM · Kanzo update: one kanban board shared across all your machines and agents, still a 4.5 MB binary using 4.5 MB of RAM

6. #6 CLAUDE.md Is Context, Not Config — Move Mandatory Rules Into Hookstip

   Anthropic's docs treat CLAUDE.md and auto-memory as loaded context, so rules that must always fire belong in a PreToolUse hook that can block an action.

   Agent guardrails

   CLAUDE.md is context, not enforcement

   Rules in CLAUDE.md

   • Loaded as context, like any doc
   • Advisory — the model can still skip it
   • Fades under context pressure

   vs

   Rules in a PreToolUse hook

   • Runs before every tool call
   • Can return a block, with a reason
   • Fires every time, not when convenient

   Mandatory rules belong where they can block — leave CLAUDE.md as a short map to deeper docs.

   Anthropic's docs treat CLAUDE.md and auto-memory as loaded context, not enforcement.

   Why it matters: Most teams still write mandatory constraints as polite markdown sentences and then wonder why the agent skips them under context pressure.

   How to apply: List your never-do rules, move each into a PreToolUse hook that returns a block with the reason, and leave CLAUDE.md as a short map pointing at deeper docs.

   claude-codehooksguardrailsagents

   Read more: Claude Code's docs say CLAUDE.md is context, not enforced config. What have you moved into hooks because Claude kept skipping it?

7. #7 Subagent Briefs Must Name the Output Path and the Area to Exploretip

   In a six-for-six test, subagents whose briefs named no file path returned prose and wrote nothing, and unassigned parallel explorers duplicated coverage.

   Subagent briefs

   Three checks before every spawn

   6-for-6: pathless briefs wrote nothing

   Names the exact report path

   Assigns an owned directory

   Relying on system prompts

   pass warn fail

   No path → prose only, no files; no area → duplicated coverage. Refuse to spawn if either line is missing.

   Why it matters: Agent definitions and system prompts did not override this default, so the instruction has to live in each spawn brief or the work silently evaporates.

   How to apply: Template every spawn with two mandatory lines: the exact report path to write and the directory or module the subagent owns; refuse to spawn if either is missing.

   agentspromptingclaude-codetip

   Read more: 6 of 6 subagents wrote no file until the brief itself named the path

8. #8 digline 1.0: An Open-Source Regression Gate That Measures Judge Noise Firstrepo

   digline compares an LLM app against an approved reference and quantifies the judge's own variance before calling a score drop a regression.

   digline 1.0 · open-source eval gate

   Measure the judge's noise before calling a regression

   1

   Pin reference

   approved baseline run

   2

   Measure noise

   judge variance over repeated samples

   the order is the point: noise first, verdict second

   3

   Set threshold

   above the noise band

   4

   Gate CI

   required merge check

   Only score drops outside the judge's noise band block a merge.

   Why it matters: Most eval gates fire on noise, so teams learn to ignore them; separating judge variance from real regressions is what makes a gate trustworthy enough to block merges.

   How to apply: Pin an approved reference run, let digline compute judge noise on repeated samples, then set the regression threshold above that band and wire it into CI as a required check.

   evaluationciopen-sourcellm-devs

   Read more: Shipped 1.0 of my LLM regression gate before the work I’d planned for it was done. Here’s what 1.0 means instead.

9. #9 1-Bit Vectors: 52% Recall Alone, 99.3% After a 40x Float Rescoretechnique

   Binary-quantized vectors across Qdrant, pgvector and OpenSearch recovered 99.3% recall@10 only when the top candidates were rescored with full-precision vectors.

   Why it matters: It gives a concrete recipe for running vector search on a server too small for float32 vectors without quietly tanking retrieval quality.

   How to apply: Store 1-bit or half-precision vectors for candidate generation, over-fetch roughly 40x the final top-k, then rescore those candidates against retained float vectors before returning results.

   vector-searchquantizationragperformance

   Read more: 1-bit vectors on Qdrant, pgvector and OpenSearch, on a server too small for the full vectors: 52% recall alone, 99.3% with a 40x rescore.

10. #10 BEAM 100K: Open-Source Memory Systems Match Full Context at 24-35x Less Textpaper

    EverOS (70.1%) and Hindsight (67.8%) essentially matched feeding the whole conversation (69.5%) while reading 24-35x fewer tokens.

    Why it matters: It is rare to see open-source memory systems benchmarked head-to-head with a full-context baseline, and only those two clearly beat plain keyword search.

    How to apply: Adopt dated-note memory for facts that change over time and an exact-detail store for verbatim recall, and validate against your own 60-question set since ranks 3-8 were not clearly separated.

    memoryagentsevaluationopen-source

    Read more: I benchmarked 10 open-source AI memory systems on BEAM (100K) with Claude. Two matched full context while reading 24–35x less text

11. #11 Disable MTP When the Model Spills Past VRAMtip

    Testers report 20-30% tok/s gains simply by turning off multi-token prediction on large models that do not fit fully in GPU memory.

    Why it matters: Many local engines enable MTP by default, so people are quietly paying a speed penalty on exactly the oversized-model setups where throughput already hurts.

    How to apply: Benchmark your usual prompt with MTP on and off under your engine's Strata-style offload path, then pin the faster setting in your launch config rather than trusting the default.

    local-llminferenceperformancequantization

    Read more: If you are using big model compared to ram try and disable mtp

12. #12 Isolation Options for Agent-Written Code, Cheapest Firsttip

    Keep agent-generated code off the host holding your credentials, starting with a no-network Docker scratch container and escalating to microVM sandboxes as trust requirements rise.

    Why it matters: Coding agents now execute code routinely, and containers share the host kernel, so the isolation choice is a real security boundary decision rather than a convenience.

    How to apply: Default to a network-disabled local container with a mounted scratch folder, then move to E2B, Modal, Daytona or Vercel Sandbox when you need GPU access, hosted SDKs, or microVM-level separation.

    securityagentssandboxingtooling

    Read more: Where to run the code your agent writes (6 options, least to most isolation)