Prompt-Injection Gateways, Free KV-Cache Headroom, and MCP Tooling for Local Coding Agents

Today's findings

1. #1 Separate trusted instructions from untrusted tool data at the system leveltechnique

   A middleware gateway enforces a hard channel split between runtime instructions and external tool/API data so prompt injection can't cross into the instruction channel.

   Security

   Separate trusted instructions from untrusted tool data at the system level

   Mixed Channels

   • Instructions & data in same channel
   • Prompt injection crosses

   → Injection risk eliminated

   Hard Split

   • Instructions in trusted channel
   • Data in untrusted channel
   • Injection blocked

   A middleware gateway enforces a hard channel split so untrusted tool data cannot be interpreted as instructions.

   Why it matters: Prompt injection via web pages, files, or API responses is a system boundary problem, not something better prompting or a smarter model fixes on its own — this is directly relevant to anyone wiring MCP tools into agents.

   How to apply: If your agents call external tools (web search, file reads, third-party APIs), put a gateway or middleware layer between the agent and its tools that tags data by trust level and blocks the untrusted channel from being interpreted as instructions.

   mcpsecurityprompt-injectionagents

   Read more: A system-level approach to prompt injection: separating instruction and data channels in LLM agents

2. #2 Local, offline MCP server indexes your repo for coding agentsrepo

   basemind builds a local code map, git history/blame, and document RAG index and serves it over MCP so agents get signatures and line numbers instead of whole files.

   MCP code-context server

   Whole files vs. structural slices for coding agents

   Send whole files

   • Dumps entire source into context
   • Burns token budget fast
   • Strains tight local-model windows

   vs

   Signatures + line numbers

   • Returns just signatures and line refs
   • Expand-on-demand pulls full code only when needed
   • Local, offline; 300+ languages plus git blame

   basemind serves a local code map over MCP so agents get structure, not raw files

   Cuts token spend for local models with tight context budgets

   Why it matters: Feeding full files to a model for structural questions burns context fast; returning just signatures and line numbers (with an expand-on-demand tool) cuts token spend meaningfully when running local models with tight context budgets.

   How to apply: Run it as an MCP server or Claude Code plugin against your repo to give any coding agent structural search across 300+ languages plus git blame, without sending whole files or hitting the cloud.

   mcpcoding-agentslocal-llmtooling

   Read more: A fully local, self-hosted repo index for coding agents (Rust, MIT, runs offline)

3. #3 Quantize the KV cache, not just the weights, for free context headroomtip

   Dropping the KV cache from Q8 to a lower quant barely moves output quality but frees up a meaningful chunk of extra context on the same GPU.

   Why it matters: Most local-LLM tuning focuses on weight quantization; the KV cache is a separate, often-ignored memory sink that directly caps usable context length on consumer cards.

   How to apply: In your local inference server (llama.cpp, vLLM, etc.), enable KV-cache quantization below Q8 and re-test your actual context ceiling before assuming you need more VRAM.

   quantizationlocal-llmllama.cpp

   Read more: KV cache quantization is the free lunch nobody talks about

4. #4 Generate CLAUDE.md and every other tool's config from one sourcetool

   ai-rulez writes rules once and regenerates CLAUDE.md plus other assistants' native config files on every commit via a pre-commit hook.

   ai-rulez

   One rule source regenerates every AI tool's native config on each commit

   Pre-commit hook keeps all files in sync; 33 builtin rule domains ready to pull

   Why it matters: Teams running Claude Code alongside other AI tools inevitably let per-tool config files drift out of sync when rules are updated in only one place.

   How to apply: Define rules, agents, and commands once under .ai-rulez/, wire the generate step into a pre-commit hook, and pull from its 33 builtin rule domains (security, testing, git-workflow) instead of writing boilerplate from scratch.

   claudecoding-agentstooling

   Read more: I generate my CLAUDE.md (and every other tool's config) from one source, rebuilt on each commit

5. #5 Rotation-based INT8 text encoders cut VRAM needs for Qwen3.5tool

   ConvRot applies Hadamard-rotation outlier suppression before INT8 quantization, giving Qwen3.5 2B/4B/9B encoders better fidelity than plain INT8 while fitting on 8GB VRAM.

   Why it matters: Outlier suppression before quantization is a general technique that improves accuracy retention at low bit-widths, applicable beyond this specific encoder release.

   How to apply: Swap in the ConvRot INT8 encoder as a drop-in replacement in ComfyUI (or adapt the rotation technique to your own quantization pipeline) if you're VRAM-constrained on text-encoder-heavy workflows.

   quantizationvramlocal-llm

   Read more: [Release] Qwen3.5 INT8 + ConvRot text encoders for ComfyUI (2B/4B/9B, runs on 8GB VRAM)

6. #6 A shared task board stops parallel coding agents from colliding on the same repotechnique

   Agents register on a local task board, claim a sub-task, and heartbeat a lease, keeping parallel Claude Code/Aider sessions out of each other's files.

   Coordinating parallel coding agents

   Claim-and-lease keeps agents off each other's files

   1

   Register

   join board

   2

   Claim

   lock sub-task

   3

   Heartbeat

   renew lease

   4

   Edit

   own files only

   5

   Release

   mark done

   renew while working

   Local file or SQLite table; expired lease frees the task for others.

   Why it matters: Running multiple agent sessions concurrently on one codebase is an increasingly common way to speed up work, but naive parallelism causes git merge conflicts and rewritten files.

   How to apply: Build or adopt a lightweight task-claim-and-lease mechanism (even a local file or SQLite table) so each agent instance only touches sub-tasks it has explicitly claimed.

   agentscoding-agentstooling

   Read more: Managing parallel agent sessions without context-switching collisions

7. #7 A trust-kernel gates every action of a self-modifying agent before executiontool

   Chimera routes each shell/tool action through deterministic allow/warn/review/block rules before letting an agent with shell access and self-editing skills act.

   Chimera trust-kernel

   Every agent action hits one of four deterministic verdicts

   Allow Safe action runs

   Warn Runs, flagged risky

   Review Held for approval

   Block Refused pre-exec

   Permitted Caution Gated Denied

   Rules classify each shell/tool call before an agent with shell + self-edit access can run it.

   Why it matters: Giving an agent shell access and the ability to rewrite its own skills is powerful but one bad token away from rm -rf or a leaked key without a hard policy layer in front of execution.

   How to apply: If you're building an agent with broad execution privileges, add a deterministic pre-execution policy kernel (not just prompted guardrails) that classifies actions before they run, and review the open-source implementation for the rule design.

   agentssecuritygovernance

   Read more: I gave my open-source agent shell access and the ability to rewrite its own skills. Here's the governance kernel that keeps it from doing something catastrophic.

8. #8 Audit your RAG faithfulness judge before trusting its hallucination scorestip

   A 14% "contradicted claims" rate turned out to be entirely judge bugs — chunk truncation at 600 chars and a second flaw — not real hallucination.

   RAG faithfulness judge

   A 14% hallucination rate that was entirely judge bugs

   14%

   "contradicted claims" flagged

   0% real — all artifact

   600 chars

   silent chunk truncation fed to judge

   2 flaws

   truncation plus a second judge bug

   spot-check

   verify judge got full context before trusting

   LLM-as-judge scores are only as good as the context you feed the judge.

   Why it matters: LLM-as-judge faithfulness scoring is only as good as what you feed the judge; silent truncation or context-window mismatches in the judge produce false hallucination signals that waste debugging time on the wrong system.

   How to apply: Before trusting a faithfulness/hallucination score, manually spot-check a sample of flagged cases and verify the judge actually received the full supporting context, not a truncated slice.

   ragtooling

   Read more: RAG faithfulness

9. #9 Pick self-hosted LLMs by active-parameter fit, not total parameter counttip

   A 35B MoE model with 3B active params fits one H100 and can beat API-class small models, while a 745B MoE model still needs multi-GPU regardless of its open license.

   Self-hosting fit

   Active params, not total, decide what you can actually run

   vs

   35B MoE · 3B active

   745B MoE

   Fits one H100

   Yes

   No

   GPUs needed

   1

   Multi-GPU

   Vs API small models

   Can beat

   —

   Open license

   Yes

   Yes

   35B MoE · 3B active wins the row 745B MoE wins the row

   Filter by active-parameter fit to your GPU memory first, then check quality.

   Why it matters: Total parameter count and benchmark leaderboards are the wrong lens for a self-hosting decision — active parameters and hardware fit determine actual deployability and cost.

   How to apply: When evaluating an open-weight model for internal hosting, filter first by active-parameter count against your available GPU memory, then check quality bar, rather than starting from a leaderboard rank.

   self-hostinglocal-llmquantization

   Read more: How would you select a LLM model for internal hosting

10. #10 TUI manager for named llama-server profilestool

    llmctl stores per-model llama-server flag profiles, runs them as detached processes, and shows live health/token-rate/GPU memory.

    Why it matters: Anyone juggling multiple local-model configs (fast-draft vs quality, different context sizes) currently reconstructs long command lines from memory with no record of what's running or why something crashed.

    How to apply: Import your GGUF files, save a named profile per configuration, and use llmctl to launch, monitor, and switch between them instead of hand-editing llama-server invocations.

    llama.cpplocal-llmtooling

    Read more: llmctl - TUI llama cpp management tool

11. #11 Context length has a speed cliff well before the advertised ceilingtip

    A model spec'd for 262K context can hit a sharp generation-speed drop around 160K on consumer hardware, long before the actual limit.

    CONTEXT vs SPEED

    Speed cliffs near 160K — not at the 262K spec

    On consumer hardware, usable context runs well short of the spec sheet.

    Why it matters: Sizing a local deployment off the advertised max-context number alone will produce a nasty surprise in production; the usable context is often much smaller than the spec sheet implies.

    How to apply: Benchmark your actual token/sec at the context lengths you plan to use in production, not just at small prompts, before committing hardware or context-window budgets to a model.

    context-windowlocal-llm

    Read more: Context length has a speed cliff, and it's earlier than the spec sheet implies