I keep seeing the same argument on HackerNews: “Self-host your LLMs to save money”, but it’s just not true if you run the numbers. Utility bills and hardware depreciation just cost too much, unless you’re willing to pay for the data security.
Numbers
Suppose you run a standard, quantized 8B open-weight model locally on a high-end consumer workstation. Under active inference load, an RTX 4090-5090 system typically draws around 500-700W
With vLLM, you will easily hit 100 tokens per second because generation is bound strictly by the GPU’s memory bandwidth.
- The Active Cost: Generating 18k tokens at 100 tps takes 3 minutes. At 500W for an RTX 4090 setup, those 3 minutes consume 0.025 kWh of electricity, costing roughly $0.0095 USD in pure power
- The Centralized Cost: Calling an equivalent lightweight model like DeepSeek V4 Flash costs roughly 0.0038 USD.
- Even running your local GPU at absolute maximum efficiency, active local electricity costs are ~2.5x to 3.5x higher than centralized APIs
The Idle Tax
In practice, local systems spend most of their time sitting idle between user queries.
If a high-end Nvidia dual-GPU desktop system idles at 90W, keeping it powered 24/7 consumes roughly 788 kWh annually. That results in a three-year idle tax of $898 USD
Even if you switch to a more efficient Apple Mac Studio M4 Max (which idles at around 10W), the upfront hardware cost still creates a barrier. Let’s compare the three-year total cost of ownership (assuming 1M tokens/day, compared against a baseline API cost of $230):
- Nvidia PC (Dual RTX 4090): 898 idle + 5,976 total cost** (a 26.0x multiple over the API cost)
- Mac Studio M4 Max (128GB): 100 idle + 3,931 total cost** (a 17.1x multiple over the API cost)
The Memory
Why can’t local hardware compete?
During the sequential decode phase of inference, the system must load all model weights from memory into the processor’s compute cores for every single token generated. At batch size 1, the arithmetic intensity drops to roughly 1 FLOP per byte, leaving compute cores idle while waiting on memory.
This forces consumer hardware into two bad trade-offs:
- The VRAM (Nvidia GPUs): Graphics cards like the RTX 5090 have good bandwidth (~1,792 GB/s), but their capacity is 32 GB. Running a 70B parameter model at 4-bit precision requires splitting it across multiple cards or offloading layers to the CPU
- The Bandwidth (Apple UMA & AMD Strix Halo): Unified memory architectures let you fit bigger models (up to 128GB or 512GB) in memory, but their bandwidth is restricted (546 GB/s for M4 Max). A dense 70B model runs at only 8 to 15 tokens per second.
Cloud providers benefit from massive multi-tenancy by using continuous batching, and amortize the memory energy tax across concurrent users
What Needs to Be Developed?
- Native Ternary Silicon Quantizing models to ternary values () like Ternary-Bonsai-27B reduces arithmetic energy by order of magnitude by replacing multiplications with additions. We need chipmakers to build native ternary tensor kernels into hardware.
- Smart-Sleep Runtime Management: We need runtime engines that can transition an inference block from a 0W deep-sleep state to active execution the moment a query comes in