Local LLM speedups, agent safety tools, and a RAG insight

Today's findings

1. #1 pg-dry-run: preview AI agent SQL writes before they hit the databasetool

   Open-source TypeScript library that runs INSERT/UPDATE/DELETE in a read-only transaction and returns a JSON proposal of affected rows, before/after values, and cascades.

   AGENT SAFETY

   Preview AI agent SQL writes before they hit the database

   bounded capability

   SQL writes to Postgres

   scope Preview diff

   limit Approval gate

   Dry-run returns affected rows, before/after values, and cascades — review then approve.

   Why it matters: Agents writing to Postgres can silently affect far more rows than intended; this gives you a reviewable diff before any data changes.

   How to apply: Add pg-dry-run to your agent toolchain and gate any write operation behind a human or policy approval step that inspects the proposal.

   agentspostgressafetyopen-source

   Read more: We open-sourced pg-dry-run: preview AI agent-generated Postgres writes before they change data

2. #2 octocode: an MCP server that gives Claude Code an import/call graph of your repotool

   Free Apache-2.0 MCP server that indexes your repo once and answers structural questions (semantic search, call graph, signatures) so Claude stops grepping blindly.

   MCP server

   Stop grepping — ask the repo

   vs

   Blind grep

   octocode

   Find usages

   regex search

   semantic search

   Call graph

   trace manually

   answer directly

   Signatures

   read source

   instant lookup

   Context cost

   many turns

   one query

   Blind grep wins the row octocode wins the row

   One-time index; Claude answers structural questions in a single call.

   Why it matters: On large repos, Claude Code wastes turns and context navigating files; a structural index cuts that overhead and keeps focus on the task.

   How to apply: Install octocode as an MCP server in Claude Code and point it at your repo; it will answer 'where is auth handled' and walk the call graph directly.

   mcpclaudecode-navigationopen-source

   Read more: We gave Claude Code an import/call graph of the repo instead of letting it grep around

3. #3 Tencent ContextPilot: proactive context management for long-horizon agentsrepo

   Open-weight Qwen3-based models (14B/8B/E4B) that teach agents to plan, maintain long-term memory, and offload less useful context while reasoning.

   ContextPilot

   Proactive context management for long-horizon agents

   Standard agents

   • Context fills
   • Performance drops

   → Managed context budget

   ContextPilot

   • Plans needs
   • Offloads unused

   Open-weight Qwen3-based models (14B/8B/E4B) reduce context bloat.

   Why it matters: Long-horizon agents degrade as context fills; ContextPilot gives you a local, fine-tuned model that manages its own context budget.

   How to apply: Swap your agent's base model for a ContextPilot checkpoint and evaluate on tasks with long tool-use sequences; it's designed to reduce context bloat.

   agentscontext-managementopen-weightslocal-llm

   Read more: tencent/ContextPilot 14B/8B/E4B

4. #4 PhoneLLM Alpha-1: a voice-agent model with GPT-5.6 Terra-level performance at 1/3 latency and 1/18 costrepo

   Open-source voice LLM from pipecat-ai that's optimized for typical voice agent tasks, running locally with low latency.

   Why it matters: Voice agents need fast, cheap inference; this model is built for that and can run on local hardware, reducing API costs.

   How to apply: Check the pipecat-ai/phonellm-alpha-1 repo and integrate it into your voice pipeline; it's designed for real-time interaction.

   voiceagentsopen-sourcelocal-llm

   Read more: pipecat-ai/phonellm-alpha-1: GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost

5. #5 llama.cpp PR #27621: extend MOE fusion to speculative decoding and earlier GLU/topk-router fusionrepo

   CUDA optimization that fuses MOE kernels for speculative decoding and removes the 1-token restriction, improving throughput on MoE models.

   Why it matters: MoE models are common for local LLMs; this PR can significantly speed up inference, especially with speculative decoding.

   How to apply: If you build llama.cpp from source, pull this PR and benchmark your MoE models; it's a drop-in performance boost.

   llama.cppmoecudaperformance

   Read more: CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token by ynankani · Pull Request #27621 · ggml-org/llama.cpp

6. #6 Getting Qwen 3.8 27B to 75-240 t/s on consumer GPUstechnique

   Community-tested recipes (hybrid IQ4_XS quant, patched sglang, MTP) push Qwen 3.8 to 75 t/s on a 16GB RTX 5080 and 240 t/s on an RTX 6000 Pro.

   LOCAL LLM SPEED

   Qwen 3.8 27B hits 240 t/s on a consumer GPU

   240

   t/s · RTX 6000 Pro

   75 t/s on RTX 5080

   27B

   params

   IQ4_XS

   hybrid quant

   MTP

   speculative tokens

   Community recipes make real-time coding feasible on single GPUs.

   Why it matters: Qwen 3.8 is a strong open model; these optimizations make it usable for real-time coding and chat on single GPUs.

   How to apply: Use the jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller quant with llama.cpp and enable MTP; for even higher speed, try the patched sglang from jpezzulli.

   quantizationqwenlocal-llmperformance

   Read more: How I got Qwen 3.8 27b running at ~75t/s decode on 16GB RTX 5080 · Qwen3.8-Next-Flash up to 240t/s on single rtx 6000 pro

7. #7 AURA Harness: an offline agent harness that wraps your existing loop instead of replacing ittool

   Thin membrane that records and optionally gates agent actions across any runtime (Ollama, LangGraph, etc.) without forcing a rewrite.

   Why it matters: Most agent frameworks want you to rebuild around them; AURA lets you keep your loop and add observability/control.

   How to apply: Add AURA to your existing agent loop as a middleware layer; it works offline and integrates with Ollama via stdlib HTTP.

   agentsharnessollamaopen-source

   Read more: Built an offline harness that conforms to your agent loop, not the other way around

8. #8 Sygal: local-first document-to-Markdown/JSON converter for RAG pipelinestool

   Converts PDFs, scans, Word, etc. to clean Markdown and structured JSON locally, with OCR routed to your own API key.

   Why it matters: RAG pipelines suffer from layout noise; Sygal gives you clean, structured text without sending documents to the cloud.

   How to apply: Run Sygal on your document corpus before chunking; it outputs Markdown for embedding and JSON for metadata.

   ragdocument-processinglocal-firstopen-source

   Read more: Built a local-first tool to convert documents/scans to clean Markdown + JSON for RAG pipelines (no cloud, own OCR key)

9. #9 Safe self-improvement: agents propose, never applytechnique

   Design pattern where an agent's memory updates are proposed by the model but applied only after deterministic analyzers and human approval.

   Design pattern

   Propose, never apply: the safe memory-update pipeline

   1

   Propose

   model writes recommendations

   2

   Analyze

   13 deterministic checks

   3

   Approve

   human sign-off

   The gate: no auto-apply

   4

   Apply

   memory mutation

   Proposals are never self-applied; analyzers and human approval are the only path to mutation.

   Why it matters: Self-improving agents fail security review when they rewrite their own memory; this pattern answers 'what changed, on what evidence, on whose authority'.

   How to apply: Implement a proposal/apply split: have the agent write recommendations, run 13 deterministic analyzers, and require explicit sign-off before any memory mutation.

   agentssafetymemorydesign-pattern

   Read more: Everyone wants a self improving agent. Almost nobody ships one.

10. #10 When the answer is a relation between documents, retrieval isn't the bottleneckpaper

    A study showing that even with full evidence in context, models fail to answer relational questions (0/38) but succeed when facts are structured (28/38).

    Why it matters: For questions that require connecting facts across documents, RAG needs to output structured relations, not just passages.

    How to apply: If your RAG queries involve temporal or causal relations, consider adding a structured extraction step that turns retrieved passages into a knowledge graph or relation tuples.

    ragresearchreasoningknowledge-graph

    Read more: [R] When the answer is a relation between documents, retrieval isn't the bottleneck: 0/38 with full evidence, 28/38 with the same facts as structure