Local Decision Engines Go Mainstream, Plus Qwen3.8 Speed Tricks and RAG Build Notes

Today's findings

1. #1 Logprob-based decision engines (Jev-style) replace generate-then-parsetechnique

   Scoring a single token's logprobs instead of generating and parsing text cuts latency roughly 2-3x and improves structured-output accuracy in reported benchmarks.

   Technique · agent decision steps

   Don't generate text for a choice you can simply score

   Generate-then-parse

   • Writes full text for a yes/no
   • Parse, validate, retry on bad format
   • Latency grows with every token
   • No confidence, just a string

   vs

   Score the logprobs

   • Read one token's logprobs
   • Decision is a ranked enum
   • Confidence comes for free
   • Any OpenAI-compatible endpoint

   Reported benchmarks: ~2–3× lower latency and better structured-output accuracy

   Fits binary or small-enum steps: keep searching? which passage? which model? — notjev runs on vLLM, llama.cpp or Ollama.

   Why it matters: Most agent and RAG pipelines burn tokens generating full text for what's really a classification or ranking decision (continue searching? which passage? which model?); this pattern is directly applicable and works with any local OpenAI-compatible server.

   How to apply: Try the open-source "notjev" library against your existing vLLM/llama.cpp/Ollama endpoint for any binary or small-enum decision in your agent loop; the jevmlx benchmark shows the approach also beats generate-then-parse for JSON field extraction on local models.

   agentslocal-llmragdecision-engines

   Read more: notjev – Turn any local LLM into a Jev-style decision engine (one token + logprobs) · jevmlx update: a 7B model on a Mac beats generate-then-parse by 14 points on TypeSafe's public eval, in 0.6 s per case · notjev – Turn any local LLM into a Jev-style decision engine (one token + logprobs) · What I figured out about Jev so far. · Spent the weekend stress-testing Jev on RAG pipelines. Here's what I found. · I tested Jev for search stopping, memory reranking, and graph retrieval. The results were mixed. · Jev + coding agents: what I learned from making semantic code search fail in batches, not all at once · Jev to autoselect Claude model - co-creator of chatgpt's project · jevals: locally runnable evals for agents using Jev-style decisions

2. #2 Qwen3.8-27B hits >70 tok/s single-stream on stock vLLM with 2x3090stip

   A tuned vLLM 0.29 config gets full 262k context, >70 tok/s decode, and >10k tok/s prefill on consumer 24GB cards.

   Qwen3.8-27B · 2× RTX 3090 · vLLM 0.29

   A tuned config, not bigger GPUs, doubles local decode speed

   70

   tok/s decode, single stream

   ≈2× what others report on identical hardware

   262K

   full context, no truncation

   10K

   tok/s prefill

   2×24GB

   consumer cards, stock vLLM

   The RedHatAI quant plus the right vLLM version and flags is the whole difference.

   Why it matters: Many people report half this throughput on the same hardware; the specific quant and serving settings are the difference between a sluggish and a genuinely usable local coding assistant.

   How to apply: Match the poster's quant (RedHatAI build) and vLLM version/flags before assuming you need more or better GPUs for this model.

   local-llmquantizationvllmqwen

   Read more: Qwen3.8-27B: >70 tok/s (>160 tok/s concurrent), 10k tok/s prefill, full context on 2x3090 (or and 48GB or larger on ampere or higher), vanilla vllm

3. #3 Production RAG: fix parsing and chunking before touching the LLMtip

   Bad parsing and weak retrieval break RAG systems far more often than prompt quality does.

   Production RAG

   RAG breaks at the front of the chain — not at the prompt

   1

   Parse

   Tables, scans

   Most failures originate here

   2

   Chunk

   Split boundaries

   3

   Retrieve

   Top-k search

   4

   Rerank

   Reorder hits

   5

   Generate

   Prompt + LLM

   Teams tune the last box; fix extraction first — Docling self-hosted, LlamaParse or Unstructured managed.

   Why it matters: Teams often tune prompts or swap models to fix RAG quality problems that actually originate in table/scan extraction — a wasted-effort trap this post explicitly warns against.

   How to apply: Start any new RAG build with Docling (self-hosted) or LlamaParse/Unstructured (managed) for parsing before investing in reranking or agentic retrieval logic.

   ragopen-source

   Read more: If I had to build a production RAG agent from scratch today, here's what I'd actually spend time on.

4. #4 Open-source OCR models tested as Textract replacementstool

   Docling, PaddleOCR-VL, MinerU and GLM-OCR each win on different document types, letting you self-host OCR instead of paying per-page Textract fees.

   Self-hosted OCR

   Four open OCR models, four document types

   MinerU Formula-heavy science PDFs

   PaddleOCR-VL Multilingual, modest GPU

   Docling Clean digital PDFs

   GLM-OCR General baseline

   Scientific / formulas Multilingual Born-digital text Catch-all default

   Route by document type to replace per-page managed OCR with a self-hosted mix.

   Why it matters: Managed OCR costs scale badly once tables and forms are involved; a self-hosted mix can cut that cost while staying open-source end to end.

   How to apply: Route by document type: MinerU for formula-heavy scientific PDFs, PaddleOCR-VL for multilingual docs on a modest GPU, Docling for clean digital PDFs, GLM-OCR as a general baseline.

   ocropen-sourcerag

   Read more: Best open source OCR models to replace Textract (tested a bunch)

5. #5 Co-locating your coding agent with the LLM server is ~33% fastertip

   Running the agent harness on the same machine that serves the model beat running it from a separate laptop by about a third in a controlled benchmark, even though the model itself wasn't faster.

   Why it matters: It's a free, one-time infrastructure change for anyone running a local coding agent against a local model — pure overhead reduction, no accuracy tradeoff.

   How to apply: If your agent harness runs on a laptop talking to a local inference box, move the harness onto the inference box itself and compare wall-clock time on a real task.

   local-llmagentsperformance

   Read more: Benchmarked where to run my AI coding agent: laptop vs. co-located with the LLM server. Co-location was ~33% faster

6. #6 Swift-Qwen3.8-27B-Splash: reverse-engineered Mac quantization formatrepo

   A 4-bit Qwen3.8-27B build for Apple's newer "Splash" inference engine gets 79-129 tok/s on an M5 Max, beating MLX by more than 2x at 64k context.

   Why it matters: The engine's package format wasn't documented, so this is a working reference for anyone targeting Splash on Apple Silicon rather than sticking with MLX.

   How to apply: Pull the quantized weights from Hugging Face if you're on a recent Mac and want faster long-context decode than stock MLX currently offers.

   quantizationmaclocal-llm

   Read more: SiliconSpecies/Swift-Qwen3.8-27B-Splash · SiliconSpecies/Swift-Qwen3.8-27B-Splash

7. #7 Model grafting: retrofit a causal encoder-decoder onto an existing decoder-only modeltechnique

   Cut an existing model at depth, feed lower layers the prompt, route upper-layer residuals as prefix KV via identity-init adapters, then heal with self-distillation — no training from scratch required.

   Why it matters: DeepSeek-V4.1-Flash showed this architecture works but trained it from scratch; this technique gets similar benefits on models you already have, at a fraction of the cost.

   How to apply: Follow the described recipe (cut point, identity-init adapters, self-distillation from the unmodified parent) if you want encoder-decoder behavior from an existing open-weight decoder model like Qwen3.5-4B.

   architecturefine-tuningqwen

   Read more: Model grafting: turning Qwen3.5-4B into a causal encoder-decoder after the fact

8. #8 Cache-friendly context-compacting plugin for OpenCodetool

   A plugin keeps conversation history intact and summarizes only when needed, instead of stripping tokens from the start of context and invalidating the local KV cache on every compaction.

   Why it matters: OpenCode's default compaction is fine for hosted APIs but forces a full re-prefill on local models every time it trims context, which this plugin avoids.

   How to apply: Install the plugin if you run OpenCode against a local model and notice repeated slow prefills after context compaction.

   local-llmcontextagents

   Read more: I built a cache-friendly context compacting plugin for OpenCode

9. #9 Understanding and Enhancing Kimi Delta Attentionpaper

   KDA's full diagonal gate lets it perform 2D rotations in a single step that Gated DeltaNet cannot express, explaining its higher expressivity.

   Why it matters: If you're choosing or fine-tuning a linear-attention architecture for long-context local inference, this clarifies a real capability gap between two popular delta-rule variants rather than just reporting benchmark deltas.

   How to apply: Read the analysis before picking between KDA- and GDN-based open models for long-context or state-tracking-heavy workloads.

   attentionarchitectureresearch

   Read more: Understanding and Enhancing Kimi Delta Attention [R]

10. #10 Open-source pooling layer for local models, APIs, and machinesrepo

    A new open-source project unifies workstations, VPSes, shared hosting, and mixed local/API models behind one interface to avoid rebuilding the same plumbing per project.

    Open-source infra

    Mixed machines and models behind one inference interface

    One reusable pool layer instead of per-project routing glue.

    Why it matters: Teams juggling several local LLM boxes plus API fallbacks often hand-roll this routing layer repeatedly; a reusable open-source version saves that work.

    How to apply: Evaluate it if you're maintaining more than one inference target (local + API) and want a single pool abstraction instead of custom glue code per project.

    infralocal-llmopen-source

    Read more: I got tired of rebuilding the same AI plumbing, so I built an open-source pool for local models, APIs and machines