Everything Fits: Tiny OCR, Token Pruning, and MoE on One GPU

Today's findings

1. #1 OvisOCR3 Brings Structured OCR and Cross-Page Tables to a 0.8B Local Modeltool

   A small Apache-2.0 OCR model now handles structured output and cross-page table reconstruction, making local document extraction much cheaper.

   OPEN OCR

   Cross-page tables from a model this small

   0.8B

   parameters — small enough to run locally

   much cheaper than cloud OCR

   Apache-2.0

   commercial-friendly open license

   vLLM

   serving examples included

   Multi-page

   tables spanning pages, rebuilt

   Targets the cloud-OCR workload — PDFs, forms and multi-page tables — extracted locally.

   Why it matters: Teams can replace or supplement cloud OCR for PDFs, forms, and tables with a model small enough to run locally and integrate via vLLM.

   How to apply: Run OvisOCR3 with the provided vLLM examples, test it on your own multi-page tables and forms, and wire the structured output into your ingestion pipeline.

   ocrlocal-llmdocument-parsingvllm

   Read more: OvisOCR3: a 0.8B OCR model with structured output and cross-page table reconstruction · OvisOCR3: a 0.8B OCR model with structured output and cross-page table reconstruction

2. #2 TokenGate Prunes Prefill Tokens for Faster, Leaner Long-Context Inferencepaper

   TokenGate learns which input tokens to skip during prefill, cutting compute and memory for long documents and codebases.

   LONG-CONTEXT INFERENCE

   TokenGate: prefill only the tokens that matter

   Standard prefill

   • All tokens processed
   • Prefill is the bottleneck
   • Slow and memory-heavy on long docs

   → Less compute + memory, model untouched

   Gated prefill

   • Gate learns token salience
   • Low-value tokens skipped
   • No architecture change

   A learned gate trims compute and memory for long documents and codebases.

   Why it matters: Prefill is often the bottleneck for long-context local inference; learned token pruning can improve latency and memory without changing the model architecture.

   How to apply: Read the paper, then benchmark a token-gating layer or equivalent pruning policy against your long-context RAG and code-review workloads before committing to a serving change.

   inferencelong-contextpruningpapers

   Read more: [Paper] TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning

3. #3 Specter Targets MoE Offloading with Quantization and Multi-Step Speculationpaper

   Specter combines quantized experts with multi-step speculative offloading to reduce the PCIe bottleneck when MoE models exceed GPU memory.

   Why it matters: Local MoE inference is increasingly practical, but expert offloading can be painfully slow; this paper points at a concrete way to keep large MoEs usable on smaller GPUs.

   How to apply: Use it as a design checklist for MoE serving: quantize experts, batch speculative steps, and measure PCIe utilization and tok/s on your own offloaded model.

   moequantizationinferenceoffloading

   Read more: [Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

4. #4 Qwen3.8-Flash-Next Fits Under 36GB at 1.66bpw EXL3tip

   A sub-2bpw EXL3 quant of Qwen3.8-Flash-Next runs stably under 36GB with no offloaded experts, fitting a single 5090.

   Why it matters: It gives local teams a way to run a strong MoE-class model on one high-end consumer GPU instead of a multi-GPU or CPU-offload setup.

   How to apply: Try the EXL3 quant in your local stack, verify stability on your prompts, and compare tok/s and quality against your current 27B or smaller model.

   quantizationlocal-llmmoegpu

   Read more: Qwen3.8-Flash-Next @ 1.66bpw-exl3 STABLE under 36gb

5. #5 Ornith-1.5-122B Runs on a Single RTX 3090 with Experts in System RAMtip

   A 125B MoE can hit 15.5 tok/s at 32K with Q4 and 26 tok/s with Q3 on one 24GB 3090 by keeping routed experts in system RAM.

   Ornith-1.5-122B · llama.cpp

   122B MoE, one RTX 3090 — the routed experts live in system RAM

   HOT

   RTX 3090 · 24GB VRAM Non-expert weights + KV cache

   COLD

   System RAM Routed experts (most of the 122B params)

   Sparse routing: only a few experts fire per token, so slow RAM still hits 26 tok/s (Q3) and 15.5 tok/s at 32K (Q4).

   Why it matters: It shows a practical path to running very large MoE models on commodity hardware without buying a datacenter GPU.

   How to apply: Follow the llama.cpp recipe, start with Q3 for speed or Q4 for quality, and tune context length and expert offload to your RAM bandwidth.

   local-llmmoellama.cppquantization

   Read more: Recipe: Ornith-1.5-122B on a single RTX 3090: 15.5 tok/s at 32K (Q4), 26 tok/s (Q3), 128K fits.

6. #6 W4A16 GPTQ vs MXFP4 Benchmark on a 35B MoE Intel Arc Handheldtip

   A 35B MoE was benchmarked with W4A16 GPTQ and MXFP4 on an Intel Arc 140V, reaching 25 tok/s with 128K context on a 17W handheld.

   Why it matters: If you are evaluating Intel Arc or low-power local inference, this gives concrete quant-format tradeoffs rather than vendor claims.

   How to apply: Use the benchmark as a starting point for Arc deployments, then test both formats on your model and context length before standardizing.

   quantizationintel-arclocal-llmbenchmark

   Read more: [Benchmarking] W4A16 GPTQ vs. MXFP4 on a 35B MoE: HumanEval+, 128k Context, and 25 tok/s on an Intel Arc 140V (Lunar Lake 17W Handheld)

7. #7 Claude Code’s Spring Regression Was a Harness Bug, Not a Model Bugtechnique

   A comparison of Claude Code’s harness with 11 other production agents argues that context management and tool-loop design caused the regression.

   DIAGNOSIS

   Not a model bug — a harness bug

   Model bug

   • Spring regression blamed on the model itself
   • Implied fix: wait for a newer model

   vs

   Harness bug

   • Discarded reasoning history
   • Overly tight response caps between tool calls
   • Context-management policies at fault
   • Compared against 11 production agents

   Same model — context and tool-loop design caused the dip, and teams can fix it without a new model.

   A harness-by-harness comparison pinned Claude Code's spring regression on surrounding code, not model weights.

   Why it matters: Agent quality depends heavily on the code around the model; teams can fix reliability and latency issues without waiting for a new model.

   How to apply: Audit your harness for discarded reasoning history, overly tight response caps between tool calls, and context-management policies; compare against the published patterns.

   agentsclaudeharnesscontext

   Read more: Claude Code's spring regression was a harness bug, not a model bug. I compared Claude Code's harness with 11 other production agents.

8. #8 Use Claude to Reduce Attack Surface Before Shipping Codetechnique

   Anthropic’s research shows Claude can help identify and remove unused or risky code paths, shrinking the attack surface.

   Attack-surface reduction

   First pass by Claude, final cut by humans

   1

   Run Claude

   prompt over branch

   2

   Map reachable

   live code only

   3

   Flag risky

   unused, over-privileged

   4

   Human review

   test, then remove

   human sign-off

   5

   Ship lean

   smaller attack surface

   Flags dead code, exposed endpoints, and over-privileged paths — humans test every removal before it ships.

   Why it matters: Security review is often under-resourced; using an LLM as a first-pass attack-surface reducer can catch dead code, exposed endpoints, and risky patterns earlier.

   How to apply: Run Claude over a branch with a focused prompt to map reachable code and flag unused or over-privileged paths, then have a human review and test the removals.

   claudesecuritycode-reviewagents

   Read more: Using Claude to reduce the attack surface of code

9. #9 Make “Done” Mean Done with Per-Action Evidence Gatestechnique

   Agent completion should require evidence—receipt, state check, or outcome verification—rather than trusting the model’s “task completed” message.

   Why it matters: The most expensive agent failure is a false success that forces a human to re-check the work; evidence gates catch it before handoff.

   How to apply: For each tool action, define the cheapest proof of success, check it in code, and block the agent from marking the task complete until the evidence passes.

   agentsverificationreliabilitytool-use

   Read more: Your agent said 'done' — here's how to make that word mean something

10. #10 Treat Tool-Result Text as a Prompt, Not Ground Truthtip

    A Claude sales agent invented an email bounce because a tool result said “NOT SENT,” showing that tool output text can steer the model.

    Why it matters: Agent security and correctness depend on validating tool results in code; a well-written prompt is not a substitute for a DNS check or schema validation.

    How to apply: Validate critical tool outputs in code before the model sees them, structure results as data rather than prose, and fail open or closed deliberately on timeouts.

    agentssecuritytool-useclaude

    Read more: Claude invented an email "bounce" that never happened. Lesson: tool-result text is a prompt too

11. #11 SA-Solver with Eta 1.7 Improves Low-Step ComfyUI Image Generationtechnique

    ComfyUI’s built-in sa_solver, run with stochastic strength eta=1.7, beats Euler at 10 steps for texture without the anatomy drift of ancestral samplers.

    ComfyUI samplers

    sa_solver at eta 1.7 — texture without the drift

    10-step test: ancestral-grade texture with Euler-stable anatomy — A/B at 8–14 steps vs your baseline.

    Why it matters: Local diffusion users can get better images at low step counts, saving time and GPU cycles without changing models.

    How to apply: Switch your ComfyUI workflow to sa_solver, set eta around 1.7, and A/B test at 8–14 steps against your current Euler or euler_a baseline.

    comfyuidiffusionimage-generationsamplers

    Read more: [Resource] We unlocked SA-Solver in ComfyUI: why stochastic strength eta=1.7 beats Euler at 10 steps (and what didn't work) · [Resource] We unlocked SA-Solver in ComfyUI: why stochastic strength eta=1.7 beats Euler at 10 steps (and what didn't work)

12. #12 llama.cpp-xdna2 Turns AMD’s XDNA2 NPU into a Local LLM Backendrepo

    A community fork treats the XDNA2 NPU as a llama.cpp backend, accelerating some attention operations for local models.

    Local LLM inference

    AMD's XDNA2 NPU becomes a llama.cpp backend

    llama.cpp-xdna2

    runtime Community fork

    Fork of llama.cpp targeting AMD Ryzen AI (XDNA2) NPUs

    runClone the fork, build on supported XDNA2 hardware, then benchmark tok/s and power vs your current GPU/CPU backend

    Accelerates some attention ops for local models — a third compute path beyond GPU and CPU.

    Why it matters: It gives AMD laptop and handheld users another local inference path beyond GPU or CPU, potentially improving power efficiency.

    How to apply: Try the repo on supported XDNA2 hardware, check which attention paths are accelerated for your model, and benchmark tok/s and power against your current backend.

    llama.cppnpuamdlocal-llm

    Read more: Making XDNA2 NPU less useless