How to Scale a Backend From 1 User to 1 Million Users

System Design from Developer to Architect --- Part 1 In Part 0, we started with a simple rule: don't add architecture because it looks scalable. Add it when a real problem makes it necessary. In this part, we'll start with one user, keep adding traffic, and change the architecture only when something gives us a reason to. Previous: Part 0 --- How to Think About System Design Without Just Drawing Boxes A million users sounds like a completely different engineering problem from one user. And eventually, it is. But the interesting part isn't the final architecture. It's how we get there. We won't begin with CDN, Redis, replicas, event brokers, and a cluster of services. We'll begin here: User → Application → Database Then we'll keep increasing traffic. Every time something becomes a real constraint, we'll ask: What broke, why did it break, and what is the smallest architectural change that solves it? That's the scaling journey. Stage 1 --- 1 User: Keep It Boring Imagine we've just launched a service-booking application. One user opens the application, searches for a professional, checks a slot, and creates a booking. Our architecture can be extremely simple: User │ ▼ Application │ ▼ Database And that's enough. No Redis. No Kafka. No Kubernetes. No read replicas. There is nothing wrong with this architecture. In fact, adding distributed infrastructure now would probably make the system harder to build, deploy, debug, and operate without solving a real problem. The cheapest scaling problem is the one you don't have yet. Stage 2 --- 100 Users: The Simple Architecture Still Works Now we have 100 users. Do we need microservices? Probably not. Do we need Kafka? Probably not. At this stage, the highest-value improvements are often much less exciting: Add the right database indexes Fix inefficient queries Avoid N+1 queries Use database connection pooling Compress HTTP responses Handle static assets efficiently Add basic metrics and logs Measure latency before optimizing it A surprisingly large amount of scale can come from simply making the existing system efficient. Sometimes: Slow API ↓ Fix query / index ↓ Fast API is a better scaling strategy than adding another server. Before distributing the system, make the simple system efficient. Stage 3 --- 10,000 Users: Scale Up Before Scaling Out Traffic keeps growing. Now we start seeing resource pressure: CPU 78% Memory 82% p95 650ms Traffic ↑ The first response doesn't always need to be horizontal scaling. We might simply give the machine more resources: 2 CPU → 8 CPU 4 GB → 32 GB RAM That's vertical scaling. It's simple because our architecture barely changes. Vertical scaling can take us surprisingly far. But machines don't grow forever, and a single server remains a single failure domain. Eventually, traffic may exceed what one instance can comfortably handle. That's when the second server becomes interesting. Stage 4 --- 50,000 Users: The Second Server Changes Everything Eventually one application instance isn't enough. So we add another. Now we need something to decide where incoming requests should go. Enter the load balancer. Users │ ▼ Load Balancer │ ┌──────┴──────┐ ▼ ▼ App 1 App 2 │ │ └──────┬──────┘ ▼ Database Now we can scale horizontally by adding application instances. But this works cleanly only when those instances are interchangeable. And that creates our next problem. The State Problem Suppose the user logs in through App 1. Their session is stored in that server's memory. Their next request reaches App 2. Login → App 1 Session stored locally Next request → App 2 Session = ? The load balancer did exactly what we asked. Our application architecture didn't. This is why horizontal scaling often pushes us toward stateless application servers. For shared session state, Redis is one possible solution: Load Balancer │ ┌────────┴────────┐ ▼ ▼ App 1 App 2 │ │ └────────┬────────┘ ▼ Redis │ ▼ Database Notice the sequence: More traffic ↓ More instances ↓ Requests move between instances ↓ Local state becomes a problem ↓ Shared/stateless state becomes useful Redis didn't appear because Redis is fashionable. It appeared because the architecture developed a state problem. Stage 5 --- 100,000 Users: Repeated Reads Start to Hurt Some requests repeatedly fetch the same information: GET /services GET /categories GET /professionals/123 If that data changes infrequently, sending every request to the database is wasteful. Now caching has a concrete job. Request │ ▼ Application │ ▼ Cache / \ HIT MISS │ │ ▼ ▼ Return DB │ ▼ Cache │ ▼ Return A cache hit avoids an unnecessary database read. A cache miss falls back to the source of truth. This reduces latency and database pressure. But caching introduces new concerns: TTL Invalidation Eviction Stale data Cache stampedes Failure behavior For a service description, some staleness may be acceptable. For the last available booking slot, it may not be. Caching isn't just a performance decision. It's also a correctness decision. Stage 6 --- 250,000 Users: The Bottleneck Moves The API tier looks healthy. Then monitoring shows: API CPU 35% ✓ API Memory 42% ✓ DB CPU 92% ⚠ DB Connections 95% ⚠ p95 Query Latency 780ms ⚠ Should we add another API server? No. The API isn't the bottleneck anymore. The bottleneck moved. App 1 ──┐ App 2 ──┤ App 3 ──┼────> DATABASE 🔥 App 4 ──┤ App 5 ──┘ A struggling database does not immediately mean: "We need sharding." Start with evidence: Database pressure │ ├── Inspect slow queries ├── Verify indexes ├── Remove N+1 access ├── Reduce unnecessary reads ├── Cache appropriate data ├── Review connection usage └── Then consider infrastructure scaling A bad query executed across ten replicas is still a bad query. Scale the bottleneck only after understanding the bottleneck. Stage 7 --- 500,000 Users: Separate Reads From Writes Suppose our workload is heavily read-oriented. Users browse much more often than they modify data. We may introduce read replicas: Application │ ┌──────────┴──────────┐ ▼ ▼ Writes Reads │ │ ▼ ▼ Primary Read Replicas │ │ ▼ ▼ Replica Replica Writes continue to go to the primary. Read-heavy traffic can be distributed across replicas. But we've bought a new problem: Replication Lag Replication is not always instantaneous. A user may create a booking on the primary and immediately read from a replica that hasn't received the update yet. The system scaled. Consistency became more complicated. Problem ↓ Solution ↓ New Trade-off Stage 8 --- 750,000 Users: Stop Sending Everything Through the Backend Images, JavaScript bundles, CSS, downloads, and other static assets don't necessarily need to travel through application servers on every request. A CDN can move cacheable content closer to users. Users │ ┌───────┴────────┐ ▼ ▼ CDN Load Balancer │ │ Static Content API Servers This reduces: Origin traffic Backend bandwidth Static-content latency Unnecessary application-server work Again, the CDN has a reason to exist. Stage 9 --- 1 Million Users: Stop Doing Everything Synchronously Consider what happens when someone creates a booking. Our API might perform: Create Booking ↓ Send Email ↓ Send Push Notification ↓ Update Analytics ↓ Award Loyalty Points ↓ Notify Professional ↓ Return Response The customer is waiting for work that doesn't necessarily need to finish before the booking is acknowledged. The core path may only require: Validate ↓ Reserve ↓ Persist ↓ Confirm Other work can happen asynchronously: Booking Service │ ▼ BookingCreated │ ▼ Event Broker │ ┌───────────┼───────────┐ ▼ ▼ ▼ Notify Analytics Loyalty This improves responsiveness and decouples downstream work. But now we need to think about: Duplicate events Consumer failures Ordering Retries Dead-letter queues Database/event consistency For example: Database COMMIT ✓ Event publish ✗ That's not just a messaging problem. It's a consistency problem. We'll explore that later in the series. What Did We Actually Build? We started here: User → Application → Database And ended somewhere closer to this: Users │ ▼ CDN │ ▼ Load Balancer │ ┌──────────┼──────────┐ ▼ ▼ ▼ API 1 API 2 API 3 │ │ │ └──────────┼──────────┘ │ ┌───────┴────────┐ ▼ ▼ Cache Database │ ┌────────┴────────┐ ▼ ▼ Replica 1 Replica 2 │ ▼ Event Broker │ ┌─────────┼─────────┐ ▼ ▼ ▼ Worker Notify Analytics That's a much more sophisticated architecture. But here's the important part: We didn't start by designing this architecture. We arrived at it. Every component earned its place. One Million Users Didn't Create One Scaling Problem It created a sequence of different problems. Growth Stage What Starts Hurting Architectural Response 1--100 Nothing significant Keep it simple 100--10K Inefficient Optimize and scale code/queries, resource vertically pressure 10K--50K Single-server capacity Horizontal scaling + load balancing 50K--100K Instance-local state Stateless services / shared state 100K--250K Repeated expensive Caching reads 250K--500K Database pressure Query optimization + read scaling 500K--750K Origin/static-content CDN / edge delivery load 750K--1M Too much synchronous Async processing / work event-driven workflows These user counts are illustrative, not universal thresholds. A read-heavy application may hit database limits much earlier. A compute-heavy application may hit CPU limits first. A media platform may need a CDN almost immediately. A financial system may prioritize consistency and transaction boundaries long before raw traffic becomes interesting. Scale is workload-specific. Don't Scale Users. Scale Bottlenecks. You don't actually scale because you reached: 100,000 users You scale because something measurable changed: CPU ↑ Memory ↑ Connections ↑ Queue depth ↑ Database latency ↑ Error rate ↑ p95 / p99 ↑ User count is context. Resource pressure and system behavior tell you what needs to change. Two applications with one million users can require completely different architectures. The Scaling Loop The mental model from Part 0 still applies: Traffic grows ↓ Observe ↓ Find bottleneck ↓ Understand cause ↓ Choose solution ↓ Measure again ↓ Repeat Not: Traffic grows ↓ Add every technology we know That difference is where architecture starts becoming engineering. The Real Goal The goal isn't to build a one-million-user architecture on day one. The goal is to build a system that: Works for today's requirements Is observable enough to tell you what is breaking Has clear boundaries where change is likely Can evolve when the next constraint appears A three-box architecture that correctly serves your current workload is better than a fifteen-service architecture whose complexity you don't need. Start simple. Measure. Find the bottleneck. Solve that bottleneck. Then repeat. Don't design for one million users on day one. Design a system that gives you a clear path to the next stage. Up Next We've spent this article scaling application infrastructure. But eventually, almost every system-design discussion reaches another question: What database should we actually use? SQL? NoSQL? Document? Key-value? Graph? More importantly: How do we choose without starting from technology hype? Next → Part 2 How to Choose Between SQL, NoSQL, and Everything in Between This is **Part 1* of System Design from Developer to Architect --- a practical series about scalability, databases, APIs, distributed systems, reliability, and the engineering decisions behind production architecture.* Previous: Part 0 --- How to Think About System Design Without Just Drawing Boxes