Adaptive Concurrency Limits in Go: Dynamically Protecting Dependencies Under Load

Most services eventually need a concurrency limit. You might start with something simple: const maxConcurrentRequests = 100 A semaphore ensures that no more than 100 requests execute simultaneously. It works. Until the conditions change. Your service gets more CPU resources after scaling. A downstream database becomes slower. A new deployment changes request processing time. Traffic patterns shift throughout the day. The limit that worked yesterday might be too restrictive today. Or worse, it might be too high and allow a dependency to become overloaded. Static concurrency limits are easy to implement, but they require continuous tuning. What if a service could adjust its concurrency limit automatically based on how the dependency behaves? That's the idea behind adaptive concurrency limiting. Instead of relying entirely on a predefined number, the system observes request latency and failure signals, estimates when queueing is happening, and adjusts the number of requests it allows in flight. In this article, we'll explore: Why fixed concurrency limits eventually become difficult to maintain The relationship between concurrency, throughput, and latency How TCP congestion control inspired adaptive concurrency algorithms How to implement a latency-aware concurrency limiter in Go How to integrate it into an HTTP client How adaptive limits interact with backpressure, bulkheads, and retry budgets What to monitor and test before using adaptive concurrency in production 1. The Problem with Fixed Concurrency Limits Imagine a Go service that calls an external market data API. You initially configure a concurrency limit of 50. limiter := NewFixedLimiter(50) Under normal traffic, the dependency responds in 40ms. Everything looks healthy. Then traffic increases. The dependency starts processing requests more slowly. Its response time grows to 150ms, then 300ms. Your service still allows 50 concurrent requests. The problem is that concurrency and latency are connected. When requests take longer to complete, existing requests occupy concurrency slots for longer. New requests accumulate, queues grow, and latency increases further. A fixed limit cannot automatically distinguish between: A dependency that has additional capacity A dependency approaching saturation A dependency already experiencing queueing A dependency recovering from an overload You can increase the limit manually, but that risks overloading the dependency. You can decrease it, but that might unnecessarily reduce throughput. The challenge is finding a concurrency level that allows high throughput without creating excessive queueing. 2. Why Concurrency Matters More Than RPS Alone Rate limiting and concurrency limiting solve different problems. A rate limiter controls how many requests can start during a period. A concurrency limiter controls how many requests can be in progress simultaneously. Consider two situations: Scenario Request rate Average latency Approx. concurrency Healthy dependency 500 req/s 40ms 20 Slow dependency 500 req/s 200ms 100 Both scenarios have the same incoming request rate. But the second requires five times as many concurrent requests to sustain that throughput. This relationship is described by Little's Law: [ L = \lambda W ] Where: (L) is the average number of items in the system (\lambda) is the average arrival rate (W) is the average time spent in the system For request processing: [ Concurrency \approx Throughput \times Latency ] This explains why a fixed RPS limit is not always sufficient to protect a dependency. A service can receive an acceptable number of requests per second while still accumulating too many in-flight operations because response times have increased. Adaptive concurrency focuses on this relationship. 3. The Idea Behind Adaptive Concurrency The central idea is straightforward: Measure request latency. Maintain an estimate of normal, unloaded latency. Compare current latency against that baseline. Increase concurrency when the dependency has spare capacity. Reduce concurrency when latency indicates queueing or failures. This resembles congestion control in TCP. TCP does not blindly send unlimited data. It maintains a congestion window and adjusts how much data can be in flight based on network conditions. Adaptive concurrency applies a similar idea to service requests. The concurrency limit becomes a dynamic window. There are several algorithms worth understanding. AIMD Additive Increase, Multiplicative Decrease. Increase the limit gradually when requests succeed. Reduce it significantly when failures or overload signals appear. For example: Current limit: 50 Successful samples: 50 → 51 → 52 → 53 Overload detected: 53 → 37 AIMD is relatively simple and works well when the system has reliable failure signals. However, it may react only after a dependency starts rejecting requests or timing out. Vegas Vegas uses latency as an early signal of congestion. It estimates queueing using the difference between observed latency and a no-load baseline. A simplified queue estimate is: [ Q = L \times \left(1 - \frac{RTT_{baseline}}{RTT_{current}}\right) ] Where: (Q) is estimated queue size (L) is the current concurrency limit (RTT_{baseline}) is the observed no-load latency (RTT_{current}) is the current measured latency When latency stays close to the baseline, the algorithm can increase concurrency. When latency grows significantly, it reduces the limit. Gradient2 Gradient2 compares short-term and long-term latency averages to detect changes in the latency trend. It is useful when latency baselines drift over time or workloads are bursty. Instead of relying only on a minimum latency measurement, it uses smoothed averages to identify growing queueing pressure. For a deeper implementation reference, Netflix's concurrency-limits project is worth exploring: https://github.com/Netflix/concurrency-limits For this article, we'll build a smaller Vegas-inspired controller. It is an educational implementation, not a complete reproduction of Netflix's Vegas or Gradient2 algorithms. 4. Building an Adaptive Limiter in Go Let's implement a limiter that: Tracks in-flight requests Enforces a dynamic concurrency limit Measures request duration Collects latency samples in windows Estimates queueing using a baseline latency Increases or decreases the limit based on observed behavior We'll use a fail-fast admission strategy. When the current concurrency limit is reached, the limiter rejects the new request instead of allowing an unbounded queue to form. Step 1: Define the limiter package adaptive import ( "errors" "sync" "time" ) var ErrLimitExceeded = errors.New( "adaptive concurrency limit exceeded", ) type Limiter struct { mu sync.Mutex limit int minLimit int maxLimit int inflight int peak int baseline time.Duration sampleCount int sampleTotal time.Duration dropped bool windowSize int alpha float64 beta float64 } func New( initialLimit int, minLimit int, maxLimit int, ) *Limiter { if minLimit < 1 || initialLimit < minLimit || maxLimit < initialLimit { panic("invalid concurrency limits") } return &Limiter{ limit: initialLimit, minLimit: minLimit, maxLimit: maxLimit, windowSize: 20, alpha: 2, beta: 4, } } The limiter maintains two important pieces of information: Current limit: The maximum number of requests allowed simultaneously. In-flight count: The number of requests currently executing. We also maintain latency statistics and a baseline. The windowSize determines how many completed requests we collect before updating the concurrency limit. Using a window helps avoid changing the limit after every individual request. Step 2: Acquire a permit The limiter needs to decide whether a new request can execute. func (l *Limiter) Acquire() (func(bool), error) { l.mu.Lock() if l.inflight >= l.limit { l.mu.Unlock() return nil, ErrLimitExceeded } l.inflight++ if l.inflight > l.peak { l.peak = l.inflight } l.mu.Unlock() startedAt := time.Now() var once sync.Once done := func(dropped bool) { once.Do(func() { elapsed := time.Since(startedAt) l.mu.Lock() defer l.mu.Unlock() l.inflight-- l.recordSample(elapsed, dropped) }) } return done, nil } The returned function releases the permit and reports the outcome. For example: done, err := limiter.Acquire() if err != nil { return err } defer done(false) return callDependency() If the operation experiences a dependency-level overload signal, we can report that separately: done, err := limiter.Acquire() if err != nil { return err } defer func() { done(isOverloadError(err)) }() err = callDependency() return err The sync.Once protects against accidentally releasing the same permit twice. The caller must invoke the returned function exactly once in normal control flow. The sync.Once makes repeated invocations harmless. Step 3: Record latency samples Now we need to collect completed request samples. func (l *Limiter) recordSample( latency time.Duration, dropped bool, ) { l.sampleCount++ l.sampleTotal += latency if dropped { l.dropped = true } if l.sampleCount < l.windowSize { return } average := l.sampleTotal / time.Duration(l.sampleCount) l.updateLimit(average) l.sampleCount = 0 l.sampleTotal = 0 l.dropped = false l.peak = 0 } Instead of adjusting concurrency after every request, we wait until a complete sample window is collected. This reduces sensitivity to individual latency spikes. A production implementation may use more sophisticated rolling windows, exponential moving averages, or percentile-based measurements. Step 4: Implement the control algorithm Now comes the interesting part. We compare the current average latency against the baseline. func (l *Limiter) updateLimit( currentLatency time.Duration, ) { if currentLatency l.beta: l.decrease() } } The queue estimate is an approximation of the queueing pressure observed at the dependency. When current latency is close to the baseline, the estimated queue remains small. When current latency increases, the estimated queue grows. We use two thresholds: alpha: Below this value, the dependency appears to have spare capacity. beta: Above this value, queueing pressure is considered too high. The gap between the thresholds creates a stability region where the limit does not change. This helps avoid unnecessary oscillation. Step 5: Increase and decrease concurrency The final methods enforce the configured boundaries. func (l *Limiter) increase() { if l.limit < l.maxLimit { l.limit++ } } func (l *Limiter) decrease() { next := int(float64(l.limit) * 0.7) if next < l.minLimit { next = l.minLimit } l.limit = next } The limiter uses: Additive increase: Grow gradually. Multiplicative decrease: Reduce more aggressively when congestion appears. This is intentionally conservative. Increasing the limit by one allows the dependency to demonstrate additional capacity without creating a large traffic spike. Reducing the limit by 30% provides a faster response to overload. These values are configuration choices, not universal production defaults. Step 6: Expose the current limit We should expose the current state for observability. func (l *Limiter) Stats() ( limit int, inflight int, baseline time.Duration, ) { l.mu.Lock() defer l.mu.Unlock() return l.limit, l.inflight, l.baseline } Now we can inspect the limiter without exposing its internal synchronization. For example: limit, inflight, baseline := limiter.Stats() log.Printf( "adaptive limiter: limit=%d inflight=%d baseline=%s", limit, inflight, baseline, ) 5. Integrating the Limiter with an HTTP Client Let's apply the limiter to an external dependency. type MarketDataClient struct { httpClient *http.Client limiter *adaptive.Limiter } func (c *MarketDataClient) Fetch( ctx context.Context, symbol string, ) error { done, err := c.limiter.Acquire() if err != nil { return err } var dropped bool defer func() { done(dropped) }() req, err := http.NewRequestWithContext( ctx, http.MethodGet, "https://api.example.com/market/"+symbol, nil, ) if err != nil { return err } resp, err := c.httpClient.Do(req) if err != nil { dropped = isDependencyOverload(err) return err } defer resp.Body.Close() if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= http.StatusServiceUnavailable { dropped = true return fmt.Errorf( "market data dependency returned %d", resp.StatusCode, ) } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf( "unexpected status: %d", resp.StatusCode, ) } return nil } The example classifies overload responses separately from other errors. This distinction matters. A validation error, for example, should not necessarily cause the adaptive limiter to reduce its concurrency limit. Likewise, a network timeout might be an overload signal in one system but a network infrastructure issue in another. The failure classifier should reflect the dependency's actual behavior. The HTTP client should also have appropriate timeouts and connection settings. An adaptive concurrency limiter is not a replacement for transport-level protection. 6. What Happens During an Overload? Let's walk through a simplified scenario. Assume the limiter starts with a concurrency limit of 50. The dependency initially responds in 40ms. As traffic increases, the limiter observes the following: Stage Average latency Concurrency limit Initial 40ms 50 Increased traffic 45ms 51 Higher utilization 55ms 52 Queueing begins 120ms 36 Dependency recovers 45ms Gradually increases The important behavior is not the exact numbers. The limiter detects that latency is increasing relative to the baseline and reduces the amount of work it allows in flight. As the dependency recovers, the limiter can gradually explore additional capacity again. This is a feedback control loop. However, it is important to understand that the limiter does not instantly discover the perfect concurrency limit. It continuously estimates a useful operating point based on observed signals. 7. Adaptive Concurrency vs. Other Resilience Patterns Adaptive concurrency is not a replacement for the patterns we've discussed in previous articles. It complements them. Mechanism Responsibility Backpressure Controls incoming work Load shedding Rejects work when capacity is exceeded Bulkhead Isolates resources and dependencies Retry budget Limits additional retry traffic Adaptive concurrency Adjusts the concurrency ceiling based on observed conditions Consider a service that communicates with three dependencies: Market Data API Portfolio Database Reporting Service Each dependency has different latency characteristics and capacity. Using one global concurrency limit would couple their behavior. A better approach is to create an independent adaptive limiter for each dependency. type DependencyLimiters struct { MarketData *adaptive.Limiter Portfolio *adaptive.Limiter Reporting *adaptive.Limiter } This gives each dependency its own feedback loop. If the reporting service becomes slow, its concurrency limit can decrease without automatically reducing the capacity available to market data requests. This is where adaptive concurrency and bulkheads work particularly well together. 8. Important Production Considerations The example implementation demonstrates the core idea, but a production adaptive limiter needs additional safeguards. Baseline latency can become stale A latency baseline is only useful when it represents a reasonably unloaded system. If the baseline is established while the dependency is already overloaded, the algorithm may incorrectly treat unhealthy latency as normal. Production implementations should carefully manage baseline updates and resets. Traffic must be representative A dependency may expose different latency characteristics for different operations. For example: Reading a cached market price Fetching historical prices Calculating a portfolio Executing a database aggregation Combining all these operations into one latency signal can produce misleading results. Consider separate limiters or traffic partitions when workloads have significantly different behavior. The algorithm needs enough samples Adaptive control depends on observations. A low-traffic service might not generate enough samples to update its limit reliably. A new instance also needs a warm-up strategy. Slow start, conservative initial limits, and minimum sample requirements can help. Avoid uncontrolled oscillation A limiter that constantly increases and decreases its capacity can introduce instability. Useful techniques include: Sampling windows Exponential smoothing Separate increase and decrease thresholds Bounded increase and decrease rates Minimum and maximum concurrency limits Jitter when multiple independent clients share a dependency Local limits multiply across replicas This is especially important in distributed deployments. Imagine ten replicas, each with an adaptive concurrency limit of 100. The dependency may receive up to 1,000 concurrent requests. Each instance is making decisions based on its own observations. The aggregate behavior is not automatically coordinated. Local adaptive concurrency is often sufficient for protecting individual instances and reacting to local conditions. But strict global dependency quotas may require a shared dispatcher, centralized coordination, or provider-side quota enforcement. Do not confuse concurrency with connection limits An HTTP transport's connection pool and an application's concurrency limiter protect different resources. A request may hold an application permit while waiting for a connection. Conversely, HTTP/2 can multiplex multiple requests over one connection. Configure both based on the actual workload. 9. Testing Adaptive Concurrency Adaptive algorithms should be tested as control systems, not only as ordinary utility functions. At minimum, test these behaviors: The initial concurrency limit is respected. Requests are rejected when the current limit is reached. Permits are released exactly once. Successful low-latency windows allow gradual growth. Rising latency causes the limit to decrease. Overload signals trigger a decrease. The limit never exceeds its configured maximum. The limit never falls below its configured minimum. Concurrent acquisitions are race-safe. Insufficient sample windows do not trigger premature changes. Run the race detector: go test -race ./... Also consider deterministic algorithm tests. Instead of using real HTTP requests and sleeping for arbitrary durations, separate the control algorithm from the admission mechanism. Feed known latency samples into the controller and verify the resulting limit. For example: Baseline: 40ms Window 1: 42ms → increase Window 2: 45ms → increase Window 3: 120ms → decrease Window 4: 130ms → decrease Window 5: 45ms → gradual recovery This makes algorithm behavior easier to reproduce and debug. Finally, run load tests with changing dependency latency. A useful test should simulate: A healthy dependency. Gradually increasing latency. Sudden overload. Partial recovery. A complete dependency outage. Measure throughput, p95/p99 latency, rejection rate, and concurrency limit changes. The goal is not simply to maximize throughput. The goal is to maintain useful throughput without allowing latency and queueing to grow uncontrollably. 10. Observability An adaptive limiter introduces a new operational component. You need visibility into how it makes decisions. Useful metrics include: Metric Type Purpose adaptive_concurrency_limit Gauge Current concurrency ceiling adaptive_inflight_requests Gauge Active requests adaptive_request_latency_seconds Histogram Observed dependency latency adaptive_rejected_total Counter Requests rejected at admission adaptive_limit_increase_total Counter Limit increase events adaptive_limit_decrease_total Counter Limit decrease events adaptive_baseline_latency_seconds Gauge Estimated unloaded latency These metrics help answer important questions: Is the limiter constantly operating at its maximum? Is the concurrency limit frequently decreasing? Is the dependency latency increasing? Are requests being rejected before reaching the dependency? Is the limiter recovering too slowly? A particularly useful dashboard plots three signals together: Current concurrency limit In-flight requests Dependency latency Looking at them independently can hide the relationship between capacity and latency. Avoid high-cardinality labels such as raw URLs, user IDs, and request IDs. 11. When Should You Use Adaptive Concurrency? Adaptive concurrency is not necessary for every service. A fixed limit is often the better choice when: Dependency capacity is stable and well understood. Traffic patterns are predictable. The service has low request volume. The cost of an adaptive control loop is not justified. A simple semaphore already provides sufficient protection. Adaptive concurrency becomes more attractive when: Dependency capacity changes frequently. Traffic is bursty or unpredictable. Services scale horizontally. Latency is a reliable indicator of queueing. Manual concurrency tuning is difficult. You need to protect dependencies before they start rejecting requests. Start with a fixed limit if it is sufficient. Introduce adaptive behavior when operational evidence shows that static configuration is becoming a limitation. Conclusion A fixed concurrency limit assumes that system capacity is relatively predictable. In distributed systems, that assumption does not always hold. Dependencies slow down, traffic changes, instances scale, and resource availability evolves. Adaptive concurrency provides a feedback loop that adjusts the amount of work allowed in flight based on observed latency and failure signals. In Go, the core mechanism is straightforward: track active requests, collect latency samples, estimate congestion, and adjust the concurrency ceiling. The difficult part is not implementing a semaphore. It is building a stable controller that reacts to real overload without constantly oscillating or making incorrect decisions from noisy measurements. Don't just limit how much work your service can handle. Let it learn when it should slow down.