Scalability is the ability to meet defined throughput, latency, and error objectives as workload changes by adding capacity or changing the design. It is not unlimited growth, and it is not proved by running more application instances.
Define the workload
A bottleneck exists relative to a workload. Before changing infrastructure, describe:
- operations and their proportions;
- arrival rate and concurrent work;
- payload and response sizes;
- data volume and access distribution;
- read-to-write ratio;
- burst shape and duration;
- background work competing for resources;
- expected growth and geographic path.
A test dominated by cached reads says little about a peak dominated by writes to a small set of records. Uniform synthetic keys may miss hot tenants or popular products. A short test may not expose connection leaks, queue growth, compaction, or cache eviction. State assumptions and model the request mix that matters.
Separate offered load from completed throughput. A system may appear stable because it rejects, times out, or sheds work. Count successful outcomes, errors, and unfinished backlog together.
Read latency as a distribution
Average latency compresses unlike experiences into one number. Percentiles expose the distribution:
- p50 is the median: half of observations are at or below it.
- p95 is the value at or below which 95% of observations fall.
- p99 describes the slower tail experienced by roughly one in a hundred observations.
Percentiles need context: the operation, population, environment, and time window. Combining unrelated endpoints can hide a slow critical workflow. Aggregating percentiles from separate instances incorrectly can also mislead; calculate from a suitable combined distribution or histogram.
Tail latency matters because one user action may call several services or fetch several resources. Its experience can be dominated by the slowest component. But p99 from a tiny sample is unstable, so report sample volume and avoid implying precision the data cannot support.
Observe latency with throughput, errors, and saturation. Low latency during low traffic does not demonstrate capacity. Rising queue time with busy workers is different from slow database execution with idle application CPU.
Build a latency budget
Start from a user-facing objective grounded in workflow needs or measured expectations. Divide the request path into segments:
client/network -> gateway and queue -> application
-> database -> downstream dependency -> response
An illustrative budget might assign portions to these segments, but those allocations are hypotheses, not production facts. Use traces, application timing, database plans, dependency telemetry, and queue metrics to discover where time is spent. Beware of missing time caused by instrumentation gaps or clocks.
Distinguish service time from queue time. A handler may take little CPU once scheduled while requests wait on a thread pool, event loop, database connection, lock, or downstream rate limit. Scaling code that is not executing cannot remove the queue’s constraint.
Latency budgets clarify tradeoffs. If a downstream dependency consumes most of the path, micro-optimizing serialization is unlikely to change the outcome. If database acquisition dominates, inspect pool sizing and query duration before adding application nodes.
Find saturation and contention
For each resource, examine utilization, saturation, and errors:
- CPU run queues and throttling;
- memory pressure, allocation, and garbage collection;
- event-loop delay;
- database connections, locks, query plans, and I/O;
- worker-pool occupancy and queue age;
- network throughput and connection limits;
- downstream rate limits and timeout categories.
High utilization is not always a problem; sustained saturation with growing queues is. A database can be constrained by lock contention while CPU remains moderate. An application can be limited by a small connection pool while host resources look idle.
Use profiling for expensive code paths and execution plans for expensive queries. Correlate the evidence with the same time window and workload. A top CPU function observed during a maintenance job may not explain interactive latency.
Choose scale-up or scale-out deliberately
| Decision factor | Scale up | Scale out |
|---|---|---|
| Change | Add CPU, memory, or faster storage to a node | Add nodes and distribute work |
| Best fit | Work limited by one node where larger capacity is available | Partitionable or stateless work with a distribution strategy |
| Advantages | Simpler topology and fewer distributed coordination concerns | Incremental capacity and potential fault isolation |
| Limits | Hardware ceiling, larger failure domain, possible downtime or cost jumps | Load balancing, coordination, consistency, and operational complexity |
| Does not fix | Lock contention, inefficient algorithms, external limits | A saturated shared database, hot key, or serial critical section |
This table is a decision aid, not a universal preference. Scale up can be the safer immediate step. Scale out can provide flexibility when the workload partitions cleanly. Both fail when the constrained resource is elsewhere.
Application scale-out usually requires requests to run on multiple nodes safely. Move necessary session state to a shared store or use appropriately designed signed claims; do not assume affinity is recovery. Background workers need safe message handling and visibility into backlog. Databases may scale reads with replicas, but replication delay and write concentration remain design concerns.
Evaluate common interventions against evidence
Caching reduces repeated computation or data access when reuse exists. Define keying, expiry, invalidation, staleness tolerance, and stampede protection. A low hit rate or highly dynamic workload may add complexity without removing the constraint.
CDNs reduce delivery distance and origin work for cacheable content. They do not fix slow writes, database contention, or non-cacheable application logic.
Queues absorb bursts and decouple acceptance from completion. They shift latency into backlog and require capacity, retry, idempotency, and poison-message design.
Database indexes can create better access paths for specific queries while adding write and storage cost. Start with the query and execution plan, as described in PostgreSQL Indexes: Start with the Query and Execution Plan.
Partitioning or sharding can distribute data or work, but introduces routing, cross-partition operations, rebalancing, and hotspot risks. Use it when evidence shows a boundary that simpler capacity changes cannot address.
Concurrency increases can raise throughput until another resource saturates. Increasing pool sizes without capacity downstream can deepen queues and worsen tail latency.
Test one hypothesis at a time
Form a statement: “Database connection wait is the dominant contributor during this workload; reducing query duration or changing pool behavior should reduce queue time.” Make the smallest safe change that tests it.
Run representative tests with a warm-up period, steady stage, and—when relevant—a controlled burst. Record configuration, dataset shape, cache state, and background activity. Compare the same percentiles, throughput, errors, and saturation before and after. A result from synthetic infrastructure is evidence about that setup, not a promise for production.
After improvement, look for the bottleneck to move. Removing database wait may expose CPU, a downstream dependency, or network transfer. Capacity work is iterative because the limiting component changes.
Production validation should use a staged rollout where possible. Watch user-visible indicators and rollback conditions. Avoid declaring success from one dashboard panel while errors or backlog increase elsewhere.
For a structured investigation across application, database, and operating signals, see performance and reliability consulting.
Finding the real bottleneck is disciplined elimination: define the workload, inspect the latency distribution, decompose the path, identify saturation, change the constrained part, and repeat under the same conditions.
