A load balancer gives clients a stable destination while distributing connections or requests among eligible backends. That sounds like a capacity feature, but production behavior is dominated by lifecycle and failure questions: when is a backend eligible, what happens to in-flight work, how does the algorithm react to uneven requests, and what happens when the balancing layer itself is impaired?
A balancer can move work. It cannot create backend capacity, make non-idempotent retries safe, or turn a superficial health endpoint into evidence that a user journey works.
Choose the layer deliberately
Layer 4 load balancing makes decisions using transport information such as source and destination addresses, ports, and connection state. It can preserve protocol transparency and avoid application parsing, but its unit of distribution is commonly a connection. Long-lived connections can therefore keep traffic uneven even when the connection counts originally looked balanced.
Layer 7 balancing understands an application protocol such as HTTP. It can route by host, path, headers, or other request attributes; terminate TLS; apply request-level policies; and produce richer application telemetry. It also becomes responsible for more protocol behavior and may add configuration, processing, and failure surface.
Many systems use both: a network-level service distributes connections to application-aware proxies, which route requests to services. The right arrangement depends on protocol needs, trust boundaries, encryption, observability, and operations—not on one layer being universally superior.
Hardware, software, and managed load balancers
Dedicated hardware appliances package traffic processing and network features in vendor-managed platforms. They can fit environments with established network operations, specific throughput or compliance requirements, and existing appliance investments.
Software load balancers run on general-purpose infrastructure and can be deployed as proxies, ingress tiers, sidecars, or host-level components. They offer flexible automation and application-aware routing but require teams to operate their capacity, configuration, upgrades, and high availability.
Managed cloud load balancers transfer much of the infrastructure operation to a provider, while applications still own listener configuration, health semantics, quotas, observability, and downstream capacity. “Managed” does not remove architecture decisions; it changes who operates parts of the implementation.
Evaluate these options against failure domains, required protocols, scale characteristics, change process, available skills, and total operational responsibility. Product category alone does not determine reliability.
Health is not one boolean
A process can be alive but not ready. A backend can be ready for cheap reads but unable to complete writes because a dependency is unavailable. Model the states required by the workload.
- Startup indicates that initialization is still in progress and prevents premature failure handling.
- Liveness asks whether the process is making enough progress to remain running.
- Readiness asks whether the target should receive new traffic.
- Degraded capability may be useful when some routes can still succeed.
Checks should be cheap, bounded, and representative enough to guide routing. A check that always returns success from memory can leave a broken backend in rotation. A check that synchronously tests every dependency can create synchronized load or remove all backends during a shared dependency incident.
Avoid flapping by requiring appropriate evidence before changing state. Check intervals, timeouts, success and failure thresholds, and the number of independent observers all affect detection time. Document these intervals when making recovery claims.
Passive health signals can complement active checks. Connection failures, resets, or repeated qualifying responses may tell the balancer that a target is impaired. They also need thresholds so one client-specific error does not eject a healthy backend.
Drain before removal
Deployments, scaling, maintenance, and host shutdown all remove targets. Immediate removal from discovery prevents new selection but does not complete existing work.
Draining should:
- mark the target unready for new work;
- allow in-flight requests or connections a bounded period to finish;
- close or migrate long-lived sessions according to protocol behavior;
- terminate only after the grace period or completion condition.
Coordinate the application’s shutdown timeout with proxy and orchestrator timeouts. If the process exits before the balancer learns it is unavailable, requests can still be sent to a dead target. If draining waits forever, a stuck connection can block deployments.
Connection-oriented protocols need special thought. A target with a few long-lived, high-volume connections may carry more work than one with many idle connections. Draining also may not redistribute those sessions unless clients reconnect.
Match the algorithm to the workload
Round robin cycles through eligible targets. It is understandable and works best when requests and backend capacities are reasonably similar.
Weighted round robin represents unequal target capacity or a staged rollout, but static weights do not automatically adapt to changing request cost or local impairment.
Least connections favors the target with fewer active connections. This can help when connection duration varies, but connection count is only a proxy for work. One connection can be idle while another performs expensive streaming.
Least outstanding requests or latency-aware choices use more application-relevant signals. They can respond to uneven work, but feedback can be noisy: routing away from a temporarily slow target can overload others, while measured latency includes factors beyond the target.
Hash-based routing keeps a key, user, or session associated with a target. It can improve locality or support applications that still rely on local state. It also complicates rebalancing and failover; a changed target set remaps keys unless the scheme limits movement.
No algorithm repairs a capacity shortage. Test with realistic request cost, connection lifetime, cache state, and target heterogeneity.
Treat retries as additional load
When a target fails before a response, a proxy may retry another target. That can hide transient faults, but the original operation may already have taken effect. Retrying a non-idempotent write can duplicate work.
Use retry budgets, deadlines, and operation-level idempotency where retries are permitted. Do not let every layer independently retry; multiplicative attempts can amplify an incident. Prefer a small number of attempts within the original request’s time budget, and stop when overload or a non-retryable response is clear.
Design overload and balancer failure
Queues must be bounded. During overload, unlimited waiting consumes connections, memory, and caller time until the system fails less predictably. Admission control, load shedding, and explicit rejection can preserve a useful subset of traffic and let callers make informed retry decisions.
The balancing tier needs its own capacity model and redundancy. Avoid a single proxy process hidden behind an otherwise redundant application. Understand control-plane failure separately from data-plane failure: existing routing may continue when configuration management is unavailable, or a bad configuration may affect every proxy at once.
Retain spare backend capacity during maintenance, and test what happens when a zone, proxy tier, or health-check dependency fails. DNS-based routing can move clients between balancer endpoints, but cached answers and established connections introduce delay, as explained in DNS and Traffic Routing.
An architecture review should trace this whole path: client resolution, balancing layers, health semantics, backend state, retries, and overload controls. The key outcome is not a preferred algorithm or product. It is a routing system whose behavior remains understandable when targets start, stop, slow down, and fail.
