Six 3090s Serve Seven 262K Users, Plus QLoRA on Quants and Evidence-First Search

Today's findings

1. #1 Qwen3.8-Flash-Next Serves Seven 262K Users on Six 3090stechnique

   vLLM fixes let Qwen3.8-Flash-Next run on six 3090s with PLE on SSD, int8 KV, and seven concurrent 262K-token users.

   Local LLM serving

   Seven 262K-token users on six 3090s

   262K

   tokens of context per user

   vs upstream vLLM: wouldn't start at all

   7

   concurrent users

   6

   RTX 3090 GPUs

   int8

   KV cache; PLE weights on SSD

   Community vLLM fixes turn an unstartable model into a practical multi-GPU local setup.

   Why it matters: It turns a model that upstream vLLM could not start into a practical multi-GPU local serving setup.

   How to apply: Use the published serve command and fixes, move PLE to SSD, enable int8 KV, and benchmark concurrent long-context users.

   local-llmvllmquantizationinference

   Read more: Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

2. #2 EXL3-QLoRA Trains Directly on ExLlamaV3 2-8 bpw Quantsrepo

   A native ExLlamaV3 fork adds QLoRA training on quantized weights without a separate dequantized pipeline.

   EXL3-QLORA

   Fine-tune the same quant you already serve

   Dequant pipeline

   • Dequantize to FP16
   • Train on the copy
   • Re-quantize to serve
   • Two artifacts, drift risk

   → Dequant detour removed — one artifact for train + serve

   Direct training

   • Load 2–8 bpw quant
   • Train QLoRA in place
   • Serve the same file
   • One artifact

   ExLlamaV3 fork trains QLoRA adapters on quantized weights in plain PyTorch; validate against your serving quant.

   Why it matters: It lets teams fine-tune the same quantized artifacts they already serve locally.

   How to apply: Use the EXL3-QLoRA fork with plain PyTorch, train at 2-8 bpw, and validate against your serving quant.

   fine-tuningqloraquantizationexllamav3

   Read more: QLoRA on ExLlamaV3 Quants - Train @ 2-8bpw

3. #3 Mixed-Precision Qwen3.8 Pack Targets 128 GB Apple Silicontechnique

   A U-shaped oQ4e mix6 pack puts 6-bit experts on edge layers and 4-bit elsewhere for 128 GB Macs.

   Quantization recipe

   oQ4e mix6 for Qwen3.8-Flash-Next

   edge_experts = 6-bit outer layers, both ends

   everything_else = 4-bit the bulk of the model

   target_memory = 128 GB Apple Silicon unified RAM

   Spend bits at the edges, save through the middle.

   Why it matters: It offers a quality-oriented quantization recipe for running Qwen3.8-Flash-Next locally on Apple Silicon.

   How to apply: Download the oQ4e mix6 pack, run it with your Mac inference stack, and compare against uniform 4-bit baselines.

   quantizationapple-siliconlocal-llmqwen

   Read more: Built a U-shaped mixed-precision pack: Qwen3.8-Flash-Next oQ4e mix6 (6-bit experts on the 4+4 edge layers) For 128G Apple Silicon · Built a U-shaped mixed-precision pack: Qwen3.8-Flash-Next oQ4e mix6 (6-bit experts on the 4+4 edge layers) For 128G Apple Silicon

4. #4 Evidence-First Local Search Keeps Agent Answers Verifiabletool

   sift-light 1.1.0 keeps exact text evidence authoritative, makes semantic candidates opt-in, and lets agents inspect source hashes.

   sift-light 1.1.0

   Plausible isn't proven

   Exact text

   vs

   Semantic hits

   Exact evidence stays authoritative; a semantic hit is only a lead until it matches the source.

   Evidence-first defaults keep agent answers verifiable.

   Why it matters: It addresses the common failure where an agent finds a plausible file but never proves the excerpt still matches disk.

   How to apply: Run the local MCP search tool, prefer exact search first, and require source inspection before accepting semantic candidates.

   mcpragagentslocal-search

   Read more: Shipping notes from an evidence-first local retrieval layer for agents · What I learned making an evidence-first local search layer for coding agents

5. #5 shad0w Caches Repeated Local LLM Classifications in Microsecondstool

   A proxy learns high-confidence repeated classifications and serves them from a tiny CPU table.

   LOCAL-LLM CACHING

   Never ask the model the same question twice

   vs

   Plain local LLM

   With shad0w in front

   Repeated ask

   Full model pass, again

   µs from a tiny CPU table

   Compute on repeats

   Paid every time

   Table lookup only

   New questions

   Model answers

   Model answers, then logged

   Setup cost

   None

   Proxy + ~1,000 logged answers

   Plain local LLM wins the row With shad0w in front wins the row

   Only high-confidence repeats are cached — anything new still reaches the model.

   Why it matters: It cuts latency and compute for local classification workloads that repeatedly ask the same questions.

   How to apply: Run shad0w in front of an OpenAI-compatible server such as Ollama, let it log about 1,000 answers, then route repeats to the table.

   local-llmcachingclassificationollama

   Read more: I built a tiny table that learns your local LLM's classification answers and serves the repeats in microseconds [D]

6. #6 Open-Source Dataset Prep Scores, Dedupes, and Leak-Safe Splits Imagestool

   An MIT Python tool scores image quality, removes near-duplicates, and exports Kohya-ready captioned splits.

   Open-source dataset prep

   From raw images to Kohya-ready splits

   1

   Score

   image quality

   2

   Dedupe

   near-duplicates out

   dedupe runs before the split — near-duplicates can't leak across train/val/test

   3

   Split

   train / val / test

   4

   Caption

   WD14 or Florence-2

   5

   Export

   Kohya folders

   Replaces repetitive hand-rolled pre-LoRA and Dreambooth prep scripts.

   Why it matters: It removes repetitive pre-LoRA/Dreambooth scripting and prevents near-duplicate leakage across train/val/test.

   How to apply: Use the plugin weights for your domain, caption with WD14 or Florence-2, and export the Kohya folder structure.

   datasetsloracomputer-visionopen-source

   Read more: Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT) · Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT) · Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT)

7. #7 GLM-5.3-Flash Hits 61 tok/s on Mac Studio via oMLXtechnique

   oMLX runs GLM-5.3-Flash at 61 tok/s and processes a 29K-token prompt in 15 seconds on a Mac Studio.

   Why it matters: It gives Mac-based teams a concrete local serving data point for a strong open model.

   How to apply: Use oMLX on Apple Silicon, test long prompts, and compare against your current local model/harness.

   local-llmapple-siliconomlxinference

   Read more: GLM-5.3-Flash on a Mac Studio with oMLX: 61 tok/s and a 29K-token prompt in 15 seconds

8. #8 Decision 3.0 Ships Open Multimodal Decision Models from 0.6B to 27Brepo

   The d3 family targets routing and judgment inside agent loops and ranks first on both Jev Decision Index boards.

   Why it matters: Small decision models can replace expensive general LLM calls for bounded choices like routing and keep/drop.

   How to apply: Evaluate d3 sizes with vLLM Semantic Router and use them for narrow agent decisions rather than long-form generation.

   agentsdecision-modelsopen-weightsrouting

   Read more: Decision 3.0: open, multimodal decision models from 0.6B to 27B. Ranks #1 on both Jev Decision Index boards

9. #9 Silicon Hippocampus Builds Persistent Memory from KV Cartridges and SSDpaper

   The paper demonstrates persistent AI memory using physical KV cartridges, a RAM recall engine, and SSD storage on a frozen Qwen2.5-7B.

   Persistent LLM memory

   Three-tier memory for a frozen model

   HOT

   KV cartridges Active context plugged into Qwen2.5-7B

   WARM

   RAM recall engine Recalls and merges cartridges on demand

   COLD

   SSD store Persistent memory across sessions

   Long-term memory without retraining the base model

   Why it matters: It offers a concrete architecture for long-term memory without retraining the base model.

   How to apply: Read the DOI, replicate the KV-cartridge and recall-engine design, and test against your current RAG memory layer.

   memorykv-cachelocal-llmpaper

   Read more: The Silicon Hippocampus: We Built Persistent AI Memory with Physical KV Cartridges, a RAM Recall Engine, and SSD Storage [Frozen Qwen2.5-7B] · The Silicon Hippocampus: We Built Persistent AI Memory with Physical KV Cartridges, a RAM Recall Engine, and SSD Storage [Frozen Qwen2.5-7B]

10. #10 Read Back Every Agent Write Before Trusting a 200 OKtip

    Tool success responses only mean the request was accepted, so fetch and compare the written object.

    Why it matters: It catches silent failures like emails stuck in drafts, overwritten CRM fields, or dropped calendar attendees.

    How to apply: Add a read-back verification step after every write and compare persisted state with the intended action.

    agentsreliabilityverificationtool-use

    Read more: A 200 from a tool call isn't proof your agent did the thing. How are you verifying actions?

11. #11 CLI and Skill Cut Claude Code Refactor Tokenstool

    A CLI and Claude Code skill route symbol renames and reference updates through IDE-aware operations.

    Why it matters: It reduces token-heavy refactors and lets Claude Code use precise code intelligence instead of rewriting files.

    How to apply: Install the CLI/skill and invoke it for symbol renames, reference updates, and other mechanical refactors.

    claude-coderefactoringtokenstooling

    Read more: I built a CLI and Skill to help Claude Code refactor faster with fewer tokens

12. #12 Library-First Engineering Locks Claude Code to Docs and Hookstechnique

    LFE makes docs the source of truth, tool-locks personas, and enforces gates with hooks instead of reminders.

    Why it matters: It prevents long Claude Code projects from re-litigating settled decisions and sprawling into spaghetti.

    How to apply: Adopt the scaffold, separate planner/inspector/coder roles, and enforce transitions with mechanical hooks.

    claude-codeagentsworkflowguardrails

    Read more: LFE - Library First Engineering · LFE - Library First Engineering