How Much Does It Cost to Self-Host Open Models on AWS?

Your AI bill tripled last quarter. Your CTO forwarded you an article about companies saving 70% by switching to open models. Now someone is asking you to figure out what that would actually look like. I spent the last few weeks digging into this. The numbers, the hardware, the real trade-offs. Here's what I found, with enough specifics that you can actually make a decision rather than just nodding along to another "open source is the future" think piece. What "Open Models" Actually Means When someone says "open model" they mean an AI model where the weights (the learned parameters that make the model work) are publicly downloadable. You grab the file, run it on your hardware, and you don't pay anyone per request. The big names right now: Meta's Llama 4, DeepSeek V4, Zhipu's GLM-5.2, Moonshot's Kimi K3, Alibaba's Qwen 3.5, and Google's Gemma 4. These aren't toys. Some of them genuinely compete with the frontier models on real benchmarks. Chinese open models now handle over 30% of enterprise traffic on OpenRouter, up from 4.5% in early 2025. That's a massive shift in barely a year. The Architecture: What You Actually Need You want your team to use an open model. Here's the stack, from bottom to top. Hardware (The Expensive Part) A model is a giant file. We're talking anywhere from 4 GB (a small 7B model, quantized) to 1.5 TB (Kimi K3, full weights). That entire file needs to sit in GPU memory to run fast. Why GPU memory specifically? Because generating each word in a response requires billions of multiply-and-add operations. GPUs do thousands of these in parallel. A CPU does them one at a time. The practical difference: a 7B model on a CPU generates 2-5 tokens per second (painfully slow for interactive use). The same model on a GPU generates 30-80 tokens per second (feels instant). For one person on a CPU, it might be tolerable. For a team of 10 all hitting the same endpoint? Unusable. Requests queue up and everyone waits 30-60 seconds for responses. Think of it like a highway. A CPU is a single lane with a high speed limit. A GPU is 4,000 lanes at a moderate speed. Language model inference is a traffic problem, not a speed problem. You need lanes, not a faster car. The Serving Software (The Free Part) Good news: the software stack is mature, open-source, and works today. No custom code required. vLLM for the inference engine. It loads the model, handles concurrent requests, optimizes GPU utilization, and exposes an OpenAI-compatible API. Industry standard for production use. Open WebUI for a ChatGPT-like browser interface. User accounts, conversation history, file uploads. Your team won't know the difference from a commercial product. nginx or Caddy in front for authentication, TLS termination, and rate limiting. The setup: install vLLM, run vllm serve meta-llama/Llama-4-Maverick, point Open WebUI at it, hand your team the URL. A day of work for someone comfortable with Linux. The vLLM API is OpenAI-compatible, which means any tool, extension, or script that works with the OpenAI API works here with zero code changes. Just swap the endpoint URL. The Model Download from Hugging Face with a single command. Models come in different quantization levels (compression trade-offs). A 4-bit quantized version is roughly 4x smaller than the full-precision version, with minor quality loss. For most team use cases, the quantized versions are the practical choice because they fit in less GPU memory. The Cost Breakdown This is where it gets real. I'm using AWS on-demand pricing as of August 2026. Scenario 1: Team of 10 For 10 people, you want a single inference server that everyone hits via API or web UI. The sweet spot model is Llama 4 Maverick (400B parameters, MoE architecture, but only ~17B active per request). It's a Meta model (US-origin, community license), strong all-rounder, and runs on a single node with 4 GPUs. Setup AWS Instance Monthly (business hours) Monthly (24/7) Budget (Qwen 3.5-27B) g5.2xlarge (1x A10G) ~$440 ~$1,460 Sweet spot (Maverick) g5.12xlarge (4x A10G) ~$1,250 ~$4,100 Always-on + reserved g5.12xlarge (1-yr RI) ~$2,900 - Frontier-class (Kimi K3 self-hosted) p6-b300.48xlarge (8x B300) ~$25,000-30,000 ~$71,000+ The business-hours trick is the key cost saver. If your team works 10 hours a day on weekdays, you pay for ~220 hours/month instead of 730. Set up a Lambda or EventBridge scheduler to stop the instance at night and start it each morning. That single optimization cuts 70% off the bill. Why 4 GPUs for Maverick? The model has 400B total parameters. Even though only 17B are active per request, all 400B must sit in memory. Each A10G has 24 GB of VRAM. Four of them give you 96 GB total, enough to hold the quantized Maverick model comfortably while leaving headroom for request batching. Scenario 2: Company of 500 At 500 people, the bottleneck is concurrent requests. If 10-15% of your company is hitting the model simultaneously, that's 50-75 concurrent requests. One server won't keep up. You need 3-4 replicas behind a load balancer. Setup Cost/Month Budget (business hours, reserved, some queuing at peak) $3,500-5,000 Standard (always-on, savings plan, good response times) $9,000-12,000 Premium (autoscaling, always fast, redundant) $12,000-16,000 For comparison: 500 ChatGPT Enterprise seats would run ~$30,000/month. 500 people calling the Claude API at moderate usage (50 requests/person/day) would cost $6,000-12,000/month. The self-hosted route is competitive at this scale, and you get data sovereignty on top. Don't forget the hidden costs: someone (or a small team) needs to keep this running. Model updates, instance reboots, monitoring, scaling adjustments. At 500 users that's justified. At 10, it might be more hassle than it's worth. (Pricing based on AWS on-demand rates, August 2026. Check the EC2 pricing page for current numbers.) The Models People Are Switching To Here's the part that makes this conversation timely. The gap between open and closed models has collapsed. At the end of 2023, it was 17.5 percentage points on standard benchmarks. By mid-2026, it's single digits on most tasks and effectively zero on knowledge benchmarks. The top contenders right now: DeepSeek V4 Pro matches frontier on coding tasks with an 80.6 score on SWE-Bench Verified. MIT license. From a Chinese lab. GLM-5.2 (Zhipu AI) beats Claude Opus on some coding benchmarks at 46% of the cost. MIT license. Fastest throughput of the top-tier open models. Kimi K3 (Moonshot) is a 2.8 trillion parameter beast. Near-frontier quality. Available via API at $3/1M input tokens. Self-hosting it requires 8+ NVIDIA B300 GPUs and costs $70,000+/month, which makes zero sense for most organizations. Use the API instead. That said, if you do want to self-host Kimi K3 on AWS, it's now documented. AWS published a step-by-step guide for deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS. The infrastructure: a p6-b300 instance (8x NVIDIA B300 Blackwell Ultra GPUs) using vLLM as the serving engine, with Flexible Training Plans or Capacity Blocks for GPU reservation. Enterprise-grade, not a weekend project, but at least the path is documented. Llama 4 Maverick (Meta) is the US-origin workhorse. Roughly 90% of frontier quality on general tasks. The most practical model to self-host because it balances capability with reasonable hardware requirements. Qwen 3.5 (Alibaba) has an Apache 2.0 license and the 27B model is surprisingly capable for coding and structured tasks. Runs on a single GPU. The budget pick. The Honest Trade-offs Should you actually do this? Here's my framework. Self-host if: You have 200+ users (the economics start working in your favor) Data privacy is non-negotiable (nothing leaves your infrastructure) You want to fine-tune on proprietary data You can't stomach unpredictable per-token billing You have someone who can maintain the infrastructure Stick with API providers if: Your team is small (under 50 people) You need absolute top-tier reasoning quality for most tasks You don't have anyone to maintain GPU infrastructure Your usage is bursty and unpredictable The middle ground (what most teams should actually do): Route your traffic. Use open models for the 80% of tasks that don't need frontier quality: summarization, drafting, code completion, internal Q&A. Keep Claude or GPT for the 20% that does: complex reasoning, high-stakes decisions, nuanced analysis. This alone cuts your AI bill by 60-80% without sacrificing quality where it matters. The Geopolitical Angle I'd be dishonest if I didn't mention this. Almost all the leading open models are coming out of Chinese labs. DeepSeek, Zhipu, Moonshot, Alibaba. They're outpacing Meta's Llama on most benchmarks. Depending on your compliance posture, this might not matter. The weights are MIT-licensed, you self-host, no data leaves your infrastructure. Or it might be a hard blocker if your security team won't allow Chinese-origin model code on company infrastructure. If you're in the latter camp, your practical options narrow to Llama 4 Maverick and whatever Google releases next as Gemma. Both are capable. Neither is the best open model available. That's the current state of things. If you're running agentic AI workflows on self-hosted models, the security surface is different from API-hosted ones. I covered that in The OWASP Agentic AI Top 10: What Builders on AWS Need to Know. Worth reading if you're planning to give these models tool access. What to Do Monday Morning If this is new territory for you, here's the least risky way to start: Spin up a g5.xlarge on AWS (~$1/hour). Install Ollama. Download Llama 4 Scout or Qwen 3.5-27B. Point Open WebUI at it. Give 3-4 people on your team access. Run it for two weeks. See if the quality meets your needs for your actual workloads. Measure the gap. Compare responses against what you get from Claude or GPT. For many tasks, you won't notice the difference. Then decide whether to scale up to Maverick and roll it out broadly. Total cost of this experiment: about $200. The question isn't "open vs closed" anymore. That debate is over. The question is: which tasks go where? And if your architecture assumes your AI provider will always be there, you're running on hope. Self-hosting gives you a hedge. Whether that hedge is worth the operational cost depends on your team, your usage, and your risk tolerance. But $200 to find out? That's not a bet. That's a rounding error. I would be very interested to hear your thoughts or comments, so please feel free to ping me on Twitter or LinkedIn or leave me a comment below.