I gave my RTX 5090 a smaller job: reading what my coding agents skip

I had a Qwen 27B model running locally as a general implementation worker. It was useful enough to keep experimenting with, but not dependable enough for the implementation work I wanted to hand it. I moved those jobs to Haiku. That left the RTX 5090 free, and raised a more specific question: could a small, very fast local model read the bulky material that a better parent agent should not have to carry in its own context?

The material is familiar if you use coding agents: a long test log where the meaningful failure sits between thousands of passing lines; a search result that lists paths without showing what the files say; a systemd journal that mixes a current fault with old, resolved ones; or a past conversation that explains why a decision changed. I wanted a read-only helper that could digest this material, return a compact answer with source references, and leave the parent free to reason about the actual task. I did not want another autonomous worker with shell access.

My first sketch was “summarize the big text.” That description hides the hard part. A summary can sound excellent while quietly omitting the one fact I needed, or treating a filename as proof of the file’s contents. For this job, an honest “the search hit only gives me a path” is better than a persuasive invented explanation.

The initial investigation happened in one evening: exploring the use cases, running the model comparisons, and putting together the reader. Then I integrated it into my agent harness and brought back a round of feedback from real worker activity. That second part of the story matters as much as the speed charts: a promising experiment still has to survive contact with the workflow it is supposed to help.

What the helper would actually do

The parent keeps the authority to choose the question. A shell’s output becomes the reader model’s input: the bulky side is the log, while the answer stays small. A small frontend reads bytes from stdin or a local file, preserves line and byte locations, sends bounded chunks to a model over a local API, and reduces the notes to a short final answer. The source and intermediate notes stay in a local artifact. A future tool hook could keep the first few lines and last few lines verbatim, put a cited middle summary between them, and preserve the complete original for follow-up reads. That hook is a design, not a deployed part of this experiment. Only the final answer needs to enter the parent’s context. No model tools are exposed, and text inside the source is treated as data rather than instructions.

Chunking gives the GPU several independent requests to batch. It does not mean launching several copies of the model. I care about total useful input processed per wall-clock second, at the worker count that maximizes the whole job’s throughput. A blazing single-request decode rate is less interesting when the source is long and the output is deliberately short. Equally, a very fast answer that invents its evidence is unusable.

I kept a deterministic baseline in the picture. Sometimes a good grep, a few lines around the hit, and the beginning and end of a file answer the question immediately. Those methods have no model latency, no hallucination risk, and exact source text. A model earns its place when the decisive fact is outside those delivered windows, when it must connect several real lines, or when the question cannot be reduced to a search string. Even then, the parent should be able to open the cited raw lines.

Here is a synthetic example of the distinction. A path-only search may return reports/example-run.log. That proves the path appeared in the search result. It does not prove the log contains a restart, an error, or a successful recovery. The helper needs to say what it actually saw. This sounds mundane until a model confidently cites the path as support for a detailed story about the file it never read.

Reading versus writing

A token is a small piece of text, often a word fragment. Models split text differently, so tokens per second are useful serving measurements rather than a perfectly shared unit across model families. Prefill is the initial read of the prompt; decode is the step-by-step writing of the answer. My prompts can contain thousands of tokens while the answer is limited to 1,024. Reading speed matters, but a reasoning model may write thousands of hidden tokens too. I count those rather than treating them as free work.

Models I was curious about

The 5090 has enough memory to make this more than a tiny-model contest, but memory still matters because many concurrent requests need a KV cache: space for the tokens each request has already read. The model’s advertised maximum context, the server’s configured hard limit, and the shared KV pool are different numbers. A model that can accept one long request may fit far fewer such requests at once. I recorded those separately instead of treating “128K context” as a throughput specification.

I tried dense 4B and 9B models, larger 12B and 27B quantized models, and mixture-of-experts models whose active computation is smaller than their total parameter count. That last distinction matters: fewer active parameters can lower compute per token, but all the resident weights still consume VRAM, leaving less room for KV cache and batching. A 35B model with only a few billion active parameters can therefore be quick at inference while limiting concurrency in a way a small dense model does not.

The B in a name such as 4B means billions of learned parameters. BF16 stores values in a 16-bit floating-point representation; FP8 and FP4 use fewer bits, with additional scaling machinery. W4A16 means four-bit weights and sixteen-bit activations; W4A4 reduces both. These are different execution paths, not just smaller download files.

Quantization adds another tradeoff. A four-bit model moves less weight data and may leave more memory for context; whether it keeps the exact source-reading behavior I need is an empirical question. Quantization-aware training, or QAT, trains the model to tolerate its later low-precision representation. It can make a compact checkpoint more robust than a naive conversion, but the label alone does not establish that it will cite the right line in my logs. I also looked at speculative decoding, where a draft model proposes tokens for a larger model to verify. MTP uses a model’s multi-token prediction head; DSpark is a trained speculator design, not a universal speed switch. These were not enabled in the normalized round. They target output generation; I would first measure whether input reading or decoding dominates the chosen workload. LFM’s reasoning shows why a short visible answer can still be decode-heavy. A compatible trained speculator is also needed; adding one can consume memory otherwise available for context. A technique that accelerates decode may have little effect on whole-job time if prefill and chunk orchestration dominate.

I used existing local inference engines and isolated experimental profiles. I did not rebuild the host or turn the experiment into a packaging project. The model cards and engine settings explain why each candidate was plausible; the local measurements decide whether it was useful here. The vLLM batching guide explains the serving tradeoffs. The checkpoints below are the exact representations tested here, rather than interchangeable model-family names.

Finding the job before picking the model

The test machine has an RTX 5090 with 32 GB of VRAM, 192 GB of system RAM, and four PCIe 5.0 SSDs in a striped array. That is plenty of machinery for a reading assistant; it still cannot make a misleading answer useful. I started by inspecting recent real worker histories rather than searching for a benchmark that happened to flatter the hardware. Smoke tests and duplicate recordings were excluded.

A small corpus with a sharp question

I skimmed recent worker material and made eight compact tasks drawn from long searches, path inventories, test failures, history, and a journal fault. The normalized tasks read complete captured tool outputs directly; they are not all 1 MB archives. Chunked archives are a separate workload, discussed below. The source files were fixture data: no code or instructions inside them were executed. I marked 18 specific facts to inspect, then separated facts already visible in the deterministic head/tail envelope from facts truly hidden in the middle. Only four cases were eligible to demonstrate a helper gain, and only six subclaims were genuinely middle-only. That prevents an answer from getting retrieval credit for repeating information the parent already had.

For the normalized comparison, each new model receives the same input, base prompt, temperature setting, and a fixed 1,024-token visible answer cap. I test single-request and three-request arrivals, with a common 32,768-token context preflight and 90-second deadline. Temperature is zero and prefix caches are flushed between batches; the native engine disables prompt reuse. A 1,024-token cap is an allowance, not a demand to pad every answer to that length. The same prompt encourages concise output. Non-reasoning profiles have a 1,024-token backend cap; LFM has the exception below. Liquid documents LFM2.5-24B-A2B as an always-thinking model. Its tested FP8 profile receives an 8,192-token total completion allowance because reasoning consumes tokens before the visible answer; I keep the visible cap the same and report that exception rather than pretending the budgets are identical. The eight cases total 154,552 bytes of captured output. These were already used in exploratory tests: freezing a new protocol makes the comparison consistent, but it does not make this a fresh held-out set. The source review is model-assisted, with predeclared facts and source-line checks, not a blind human panel. A reviewer checks whether the answer actually supports its claims with the cited source, not merely whether a regex matches a phrase. Truncated or failed envelopes count as unusable for the delivered-answer scores; that is separate from proving a claim wrong. Extra unsupported assertions are recorded as well. A higher fact count cannot cancel invented evidence.

What survived normalization

The small QUASAR 4B was the serial speed leader: 9,271 prompt tok/s and 220 generated tok/s, with a 1.08-second median on the four eligible cases. It finished all eight serial answers and recovered four of the six omitted-middle facts. It also made four material unsupported or misattributed assertions. Fast, yes; ready to replace the source, no.

The Unsloth 35B-A3B Fast had the strongest serial combination in this sample: 6,929 prompt tok/s, a 1.44-second eligible median, eight usable answers, and five of six middle facts. One source-type mistake remained: a report document became a log. NVIDIA’s other 35B quant also found five middle facts, but had three material errors. This is a useful shortlist result, not an accuracy certificate for the model family.

The 27B IQ3_S reference was careful when it finished. Its serial usable answers had no material unsupported assertion under this rubric, and it found a genuine test-log path in the omitted middle. But only six of eight envelopes were usable: long citation lists consumed the allowance in the owner case and a history control. Its middle score was three of six, with an 8.05-second eligible median. Giving a model more answer room did not guarantee a better delivered answer.

The 9B distill finished every serial and burst case, recovered four middle facts each time, and took a 2.96-second serial median. It still added unsupported details. Compactor’s task-specific training did not win this extraction task; Gemma’s QAT checkpoint also struggled with the middle facts. Neither result is a verdict on conversation compaction or those families’ general capability.

LFM deserved the larger allowance. With 8,192 total tokens, it recovered four middle facts solo and five in bursts. Seven of eight answers were usable in each run. One solo case and one burst case exhausted reasoning without a visible final. Its eligible medians were 16.28 seconds solo and 25.04 seconds in bursts, and source errors persisted. Its generated-tok/s figure counts the reasoning work too. Here, a small visible answer was not the same thing as a small generation job.

Nobody met the declared gate: all six omitted-middle facts, without material unsupported attribution. That matters more than finding a winner-shaped row in the table. Also, these are single passes: observed serial/burst differences do not prove that concurrency caused the quality changes.

Batching changed the speed ranking

I ran a separate worker-count sweep on three representative configurations: the fast 35B, QUASAR 4B, and the dense 27B. Each point is the same eight requests with the same 1,024-token allowance. Counts of one, two, four and eight test the whole request batches; the native engine still has only four execution slots, so eight clients also create a queue.

Eight clients had the highest measured prompt throughput for all three in this small workload: 14,585 tok/s for Fast 35B, 12,619 for QUASAR, and 1,748 for 27B. The larger MoE overtook the smaller dense model under this load. These are best tested counts, not global optima. QUASAR completed only six of eight usable envelopes at that point; fast repetition still contributes to a raw token-rate number. I did not give these extra load runs a new semantic quality score.

What I would actually choose

My default on this 5090 is Unsloth Qwen3.6 35B-A3B Fast NVFP4 on vLLM. It gave the best balance of useful retrieval, delivery, and speed in the normalized round. I would start with one request for an occasional tool output, and feed it concurrent chunks when reading an archive. I keep the released reader in its exact-quote mode: the model picks evidence, and the parent decides what it means.

Here is the selection I would make from the configurations actually tested:

For history and policy questions, I would also start with Fast 35B, asking for dated quotes with the neighboring conditions and negations. I would not switch to the 27B simply because the subject feels more serious: it missed historical facts and contradicted source conditions in the burst run. If the selected lines do not establish the answer, read more raw source. A second model’s agreement is not a substitute for evidence.

I would leave Compactor 4B, Gemma 12B QAT, and LFM2.5 24B-A2B off this particular shortlist. Compactor and Gemma did not justify their speed/quality tradeoff for this extraction job. LFM improved with room to think, but its 16–25-second eligible medians, source errors, and occasional absent final make it a poor inline hook choice here. I would pick the Unsloth Fast quant over the NVIDIA 35B control for this workload, and keep Qwen3.5 4B BF16 as a useful baseline rather than my deployment choice.

These are my next-deployment choices from a small, reused corpus, not a claim that any model passed the quality gate. The normalized model comparison used prose summaries; the released default returns validated source spans instead. Both the 27B and, during deployment, Fast 35B passed a live extractive-hook canary. The other recommended backends still need that integration check before activation. Exact copying prevents fabricated quote text, but it cannot guarantee that the model selected the right evidence.

How those choices change with the job

Test failures and short journal faults: use the exact failure, status, and nearby lines first. In this pilot several needed facts were already in the retained edges. A model does not deserve retrieval credit for retelling what the parent already sees. The hook leaves reported failed or interrupted commands unchanged; an explicit reader can still inspect a saved failing log.

Path inventories and large searches: be almost pedantic about provenance. The 27B found a genuine omitted test log where quicker configurations often chose an edge example or called a report a log. A real filename is not proof of the file’s contents. This is a strong case for returning quoted evidence instead of a polished explanation.

History and policy changes: keep the date, negation, and neighboring conditions together. Models missed paired exclusions, invented causal details, or supplied relative chronology that the source did not establish. The helper was not given a clock, so an unsolicited “future report” claim is unsupported rather than a meaningful test of calendar reasoning.

Large archives: parallel chunks can make a reading job much faster, but a correct answer about a late event does not prove that the whole archive survived reduction. The earlier 1 MB experiment was a different output-budget condition, described below. I would use this route for a targeted, inspectable question before attempting an exhaustive narrative of everything that happened.

An earlier exploratory sweep already taught me why that order matters. A very fast 4B quant could finish a long transcript in seconds by pushing many chunks through the 5090. It also produced path substitutions and attractive citations that did not support the claim. A native 27B quant was much slower but recovered the omitted evidence across the eligible short cases more faithfully. Neither observation by itself chooses a production helper: the fast candidate needs better grounding, and the careful candidate needs a tighter latency and throughput check on new material. Prompt changes made after reviewing the corpus are useful experiments, but they are not fresh held-out validation.

In the earlier exploratory round, with a 512-token final allowance, a separate roughly 1 MB transcript showed how chunked parallelism made an enormous wall-time difference. The fastest tested 4B configuration handled the source far sooner than the 27B reference, but many map notes reached their caps. The 27B final recovered three late facts with precise anchors despite some capped maps. Those facts happened to be near the end of the source, so this is evidence of late retrieval, not proof that every middle section survived. A model can return the right final answer on one question while having dropped evidence another question would need.

The original workload audit suggested a sparse scenario of about 38 eligible calls per hour across 12 workers. That is a planning proxy, not measured live helper traffic, and it should not be presented as GPU utilization. Removing duplicate tool calls left eight unique runs and about 4.09 tool-hours in that audit. It is enough to motivate the experiment; it is not enough to extrapolate a fleet-wide win from a tiny evaluation corpus.

The implementation I was willing to release

The benchmark pushed the interface toward a smaller promise. In the default released mode, the model selects line and byte spans; Python validates their ranges and copies the original bytes. It does not put the model’s invented wording in the replacement. The result looks like head / selected evidence / tail, with the complete source retained locally.

The portable reader, skill, and hook Gist includes a generic JSON adapter and a Claude Code adapter. Observation is the default; replacement is explicit. A missing server, malformed selection, timeout, busy queue, or oversized prompt leaves the original output intact. Even a successful selection replaces stdout only when the complete envelope is smaller in both characters and UTF-8 bytes. A head/tail-only fallback can hide a failure in the middle of an otherwise successful aggregate command. Missing proof of full capture leaves the original untouched. A post-tool hook cannot recover bytes that the platform already truncated.

Claude’s successful Bash hook can replace structured tool output. OpenCode has a mutable after-tool result, but its installed version needs an integration fixture. Codex does not expose the same arbitrary rewrite contract: use a wrapper/custom tool that owns the returned output. Adding a summary beside an unchanged original does not save context. The Gist’s platform guide links the current official contracts.

The runtime is Python standard library on POSIX systems; the harness-facing JSON contract is platform-neutral. The skill is installed for my Codex environment; harness adoption is versioned separately from this portable release. Optional model-written prose is labeled unverified. The bounded hook reads directly; larger inputs can use the separate chunk-and-reduce frontend.

Twenty-five offline tests and an independent synthetic forward test checked the mechanics. A live synthetic canary on the 27B selected line 180 from 19,809 characters and returned a 3,845-character envelope, preserving stderr, metadata, and a byte-exact 0600 raw file. That is one functional example with a longer test deadline, not proof of general retrieval quality or worker throughput. A later deployment canary on Fast 35B found the same middle line in about 0.8 seconds, within the active caller deadline, and returned a 3,813-character envelope. This is another functional fixture, not a new quality benchmark. Exact quotes can still be irrelevant, incomplete, or superseded; the parent keeps the judgment.

The operational surprise, two days later

After that evening’s experiment, I integrated the reader into my harness and used a round of operational feedback to check whether workers were getting any benefit. The service was healthy. Workers were invoking the wrapper. The GPU was doing almost nothing. An operational review on October 16 found 63 worker calls and zero that reached the model, across about 7.6 hours after installation. The two inference requests it found were synthetic owner canaries. That is a surprisingly effective way to build a fast summarizer that saves no context at all.

The instructions and admission rules disagreed. Workers wrapped test and gate launchers as “logs,” but the capture classifier accepted journal and status commands rather than arbitrary test runners. Often the launcher printed only a short summary anyway; the useful output was already in a saved log. A second rule treated a literal alternation pipe in a directly executed search argument as shell syntax. The wrapper used an argument list without a shell, so rg 'FAIL|Error' was being rejected for punctuation that was just data.

The corrected release supports argument lists attested by the actual capture wrapper, keeps raw shell strings conservative, and treats git grep as search. For test evidence, the useful instruction is read the actual saved log, one relevant file at a time. It is not “wrap every test command.” I also changed failed selection to return the original stdout and required every replacement to shrink in both characters and bytes. Seeing “OK” at both ends of an oversized multi-log dump is no comfort if the failure in the middle has disappeared.

The review also caught a measurement trap: the report’s long “reader latency” included the wrapped test command itself, although no worker call reached inference. The next report needs to distinguish wrapper calls, actual model requests, selection success, replacements, and bytes saved. Selected line ranges must remain in private receipts so a later review can inspect the evidence. The reader Gist documents these corrections and the separate harness rollout requirement.

These are observations from that review window, not a fresh traffic measurement or new model-quality score. The repaired portable release does not automatically update a harness’s pinned copy. The next useful experiment is to verify real adoption and inspect the selected evidence, rather than rerun the speed contest while the model receives no work.

The lesson so far is pleasantly narrow. A read-only context reader is a better target for spare local compute than another unsupervised implementer. Its value comes from returning small, checkable evidence while preserving the raw source. Speed matters because it may sit on the critical path of an agent turn, but source precision is the admission ticket. The normalized round identifies models worth another source-checked test, not a universal winner. After the comparison, I chose Fast 35B as the persistent 5090 reader, with boot startup, automatic process recovery, and the existing desktop power toggle. The retired implementation-worker service stays retired. Making workers use the reader effectively is a separate piece of engineering, and the first operational review made that very clear.

Appendix: reproduce the method on your own data

Two public capsules are available:

• Reader, head/evidence/tail hook, skill, tests, and installation helper.
• Benchmark runners, pinned model inventory, protocol, aggregate results, and worker-count sweep.

The measured inputs are private worker histories. They are not in the Gists. The supplied invented fixtures reproduce the pipeline and scoring mechanics, not the published numerical results. There were 240 model attempts across quality and load checks, but only eight unique source cases; 144 answers received the source review. Model, engine, quantization, and tokenizer differences also mean this is a configuration comparison, not an isolated experiment proving one quantization method superior.

1. Assemble and test the reader

git clone https://gist.github.com/550271f704cd9545daca6ee736fc7a7e.git reader-download
cd reader-download
python3 assemble.py --dest ../context-reader
cd ../context-reader/reader-runtime
python3 -m unittest -q test_reader_runtime.py
python3 demo.py --output ../demo
python3 hook.py --config ../demo/policy.json --platform generic \
  < ../demo/event.json > ../demo/result.json

This first run needs no model: it demonstrates unchanged-output fallback while retaining a real raw reference. The assembler reconstructs the sibling skill and runtime directories from flat Gist files. Keep the package in a source checkout; link the skill directory into your agent’s skill directory if desired. No live hook is installed by these commands.

2. Bind one local model explicitly

Use the benchmark capsule’s pinned model inventory and measured launch flags. Serve one checkpoint under the local-reader name on a loopback or private LAN OpenAI-compatible endpoint. Set up the model on the inference host; run capture and the hook on the execution host so the raw path is actually readable by its worker. Do not silently choose another provider if the configured endpoint fails.

python3 demo.py --output ../model-demo \
  --base-url http://127.0.0.1:8000/v1 --model local-reader
python3 hook.py --config ../model-demo/policy.json --platform generic \
  < ../model-demo/event.json > ../model-demo/result.json

The demo creates only synthetic files and uses an explicit replacement policy with a longer deadline. The shipped policy remains observation mode with the endpoint unset. Before any real adapter activation, verify complete capture, unchanged status/stderr/metadata, usable raw references, and actual replacement of the model-visible output. Run observation first. The model never executes the command text in its input.

3. Repeat the normalized comparison

git clone https://gist.github.com/3ca8cf9136d06ef9874677caef7206fc.git benchmark
cd benchmark
python3 make_synthetic_fixture.py
python3 worker_eval.py --manifest synthetic/benchmark.json \
  --profiles public_profiles.json --profile r4-vllm-qwen36-35b-fast \
  --policy normalized-policy.json --base-url http://127.0.0.1:8000/v1 \
  --tag my-1024-c1 --visible-budgets 1024 --completion-budget 1024 \
  --context-tokens 32768 --concurrency 1 --flush-cache vllm \
  --request-deadline 90 --timeout 90 --run-deadline 300

Use a new tag for each run, repeat with three clients, then tune two/four/eight where useful. LFM uses the documented reasoning exception: --allow-thinking --completion-budget 8192, with a larger whole-run bound and the same visible allowance. Native llama.cpp uses --extra-json '{"cache_prompt":false}' instead of a vLLM cache flush. The capsule also contains the chunk-and-reduce frontend for sources beyond direct context capacity.

Replace the invented manifest with captured data from your own work. Mark facts before reading model answers, distinguish edge facts from omitted-middle facts, and check whether cited text supports the claim. Keep the answer allowance, task, cache policy, and deadline fixed. Report total tokens including reasoning, whole-job wall time, usable envelopes, shared KV capacity and per-worker limits. Failed or truncated attempts belong in the report; matching a regex is not the same as finding valid evidence.