Self-hosted LLM orchestration means running the entire coordination layer for language models — sampling, pooling, merging, serving — on hardware you own instead of renting an API call for every step. On a single consumer GPU it works, but only if the architecture is designed around one hard constraint: VRAM. These are field notes, not theory. Moamao’s production orchestration API serves every run from one 8GB RTX 3070 Ti, and most of what follows was learned by measuring the wrong architecture first.
The setup: one card, one resident model
The production box is a single consumer GPU with 8GB of VRAM. One model — qwen3:8b, quantized — stays resident at all times. A single API call runs the full dual-pool mixture server-side: candidates are sampled from the resident model into two pools, and deterministic code merges them. The whole mixture runs server-side, on hardware Moamao owns. End-to-end, a task takes roughly ten seconds.
That description sounds almost too small to be a product, which is the point. The interesting result is not that an 8B model runs on an 8GB card — anyone with a laptop knows that. It is what happened when we tried to make the mixture “bigger” and the measurements said no.
Every second model costs a VRAM swap
An 8B model at a practical quantization occupies most of an 8GB card once the KV cache is accounted for. There is no room for a second resident model. So a two-model architecture on this hardware does not mean two models cooperating — it means unloading one set of weights and loading another, over and over, inside a single task.
We measured it. Our old two-model tier averaged 36 seconds per task. The single-model architecture that replaced it averages 7.8 seconds on the same timing harness — 4.6x faster. The swap was the task, essentially: the card spent more wall-clock time moving weights than sampling tokens.
| Configuration | Resident weights | Measured latency | Bottleneck |
|---|---|---|---|
| Two-model tier (old) | Swapped per step | 36s/task | VRAM load/unload |
| Single resident 8B (current) | Always hot | 7.8s/task | Token sampling |
On multi-GPU hardware the second model is a parallelism question. On one consumer card it is a serialization tax, and the tax is large enough to dominate every other optimization.
The lesson: orchestration beats scale
Dropping the second model would have been a pure loss if quality had fallen. It did not. On our frozen ten-task tier-1 reasoning/context/memory suite, the single-model dual-pool architecture scored 10/10, replicated eight consecutive times — the same score claude-opus-5 posts on that suite under the same harness. Across all frozen suites the composite is 14.75/17, 87%, against Opus at 17/17. Those numbers are scoped to our benchmark suite and nothing wider, but the direction was unambiguous: a single well-orchestrated 8B matched a two-model panel at 4.6x the speed.
Why did the panel add nothing? Our published research points at selection. In Never Let a Small Model Choose we tested six mixture architectures where the correct answer was provably present in the candidate pool — and every one of them scored at or below its own best member when a small model did the judging. The pool was fine; the chooser was the bottleneck. The external literature rhymes with this: Self-MoA reports that mixing samples from one strong model can outperform mixing different models, a sharp caveat on the original mixture-of-agents result.
Our fix was to remove the judge entirely. The two pools are merged by one bit — did deterministic code substitute a value — and no model ever compares answers. That decision is what made the single-card constraint survivable: judge-free merging costs no VRAM, no second model, and no extra forward pass.
Practical guidance for an 8GB card
Quantize for headroom, not for fit. A quant that technically loads but leaves no room for KV cache will throttle context length and force truncation under load. Pick the quantization level that leaves comfortable cache headroom at your real context sizes, then benchmark quality at that quant — not at the full-precision weights you will never serve.
Pin the model resident. Configure keep-alive so weights never unload between requests. Cold loads are the same tax as the two-model swap, just paid at request boundaries instead of step boundaries. A resident model turns worst-case latency into steady-state latency.
Sample sequentially; queue requests. On one card, parallel in-flight generations compete for the same compute and cache. Sampling candidates sequentially from the resident model — varying sampling parameters per pool — keeps the card saturated with zero contention. Across requests, a simple queue beats fan-out for the same reason. Parallelism is a multi-GPU luxury; on one card, order is throughput.
Let deterministic code do the free work. Merging, substitution checks, and formatting cost microseconds of CPU. Every step moved out of the model and into code is a step that no longer competes for VRAM.
When not to self-host
Self-hosting is the right call less often than the enthusiasm around it suggests. Rent instead when any of these hold: the latency budget is under a couple of seconds end-to-end; traffic is bursty and needs real concurrency, which one card cannot fake; workloads need long contexts that blow past an 8GB KV-cache budget; uptime requirements imply redundancy you do not have; or — most common — nobody on the team will own the box, the quant testing, and the regression discipline. The choice between a single model and a panel has its own trade-offs, which we walk through in one model vs. a panel.
Honest costs
The hardware is the cheap part: one consumer card, amortized, plus the modest power draw of a single-GPU box. The real cost is engineering time — quantization trials, keep-alive tuning, and above all measurement discipline. Our benchmark files are hash-locked, our predictions were pre-registered, and we still published three retractions, same-day, when earlier claims failed re-testing. That overhead is the price of trusting your own numbers. If you are not prepared to pay it, use someone else’s orchestration layer and hold them to the same standard.
For comparison, this is exactly the work Moamao sells as a product: the free tier is $0 with a quota, Scale is $20/mo, Max is $79/mo, and bringing your own key is unmetered — details on the pricing page. Self-hosting competes with those numbers, not with API list prices.
FAQ
Can a mixture-of-agents system run on an 8GB consumer GPU? Yes. Moamao’s production mixture runs on a single 8GB RTX 3070 Ti with one resident qwen3:8b model, sampling dual pools sequentially and merging deterministically, at roughly ten seconds per task. On our frozen ten-task tier-1 suite it scored 10/10, replicated eight consecutive times.
Does adding a second model make a self-hosted mixture better? Not in our measurements. The second model cost a VRAM swap on nearly every step — 36s/task versus 7.8s single-model — and on our frozen benchmark suite a single well-orchestrated 8B matched the two-model panel at 4.6x the speed.
When is self-hosting the wrong choice? Sub-second latency budgets, bursty concurrent traffic, long-context workloads, redundancy requirements, or no owner for the box and its regression discipline. Renting the layer is then the honest answer.
Related: what is mixture-of-agents, BYOK cost math, orchestration platforms compared, and the research note never let a small model choose.