Autoscaling adds and removes instances according to a metric. It is genuinely useful for workloads whose load varies on a schedule, and it is frequently deployed for reasons it cannot address.
What the application must already be
Autoscaling is a consequence of statelessness, not a substitute for it. Before it can work:
- Any instance must be able to serve any request. No local session state, no local uploads.
- Instances must start and become ready quickly and without manual steps.
- A load balancer with a meaningful health check must sit in front.
- The shared dependencies — database, cache, queue — must tolerate the added connections.
That last point is where autoscaling most often fails in practice. Doubling the application tier can simply move the bottleneck to the database, which does not autoscale, and now the outage arrives faster.
It is slower than people expect
From metric breach to a new instance serving traffic is typically several minutes: the metric window, the scaling decision, instance boot, application start, health checks passing. For a spike that arrives in thirty seconds, autoscaling responds after it is over.
The practical implications:
- For daily and weekly patterns, autoscaling works well, and scheduled scaling ahead of a known peak works even better.
- For flash traffic, headroom and caching are the answer. A CDN in front of cacheable content absorbs far more, far faster, than any scaling policy.
Choosing the metric
CPU utilisation is the default and is often the wrong signal — a memory-bound or I/O-bound application saturates long before its CPU does. Better options, where the platform supports them:
- Request count or concurrency per instance.
- Queue depth, for worker fleets. This is the cleanest autoscaling signal there is.
- Response latency at a percentile, as a scaling trigger of last resort.
Scale down carefully
Aggressive scale-in causes thrashing: remove an instance, load per instance rises, add it back. Set a cooldown, scale in more slowly than out, and always drain connections before terminating so in-flight requests complete.
What it hides
- A memory leak. Instances that would eventually fail are replaced before anyone notices the pattern.
- An inefficiency. A query that got ten times slower is absorbed by more instances and shows up only on the bill.
- An unbounded cost. A retry storm or a scraper looks exactly like legitimate demand to a scaling policy. Set a maximum instance count, always, and pair it with a budget alert.
That maximum is not a limitation to work around. It is the difference between a degraded service and an unbounded invoice.
Sometimes a bigger machine is the answer
Vertical scaling is unfashionable and frequently correct. One larger instance has no distribution problem, no session problem and no cold start, and modern instances are very large. Scale out when you need redundancy or exceed what one machine can do — not as a default.
Where to go next
- Load balancers: when you need one
- Serverless vs virtual machines
- Cutting cloud costs without downtime
Questions
Frequently asked
How fast does autoscaling react?
Typically several minutes from metric breach to a healthy instance serving traffic. That suits daily traffic patterns and is too slow for a spike that arrives in seconds.
What metric should trigger scaling?
Queue depth for worker fleets, request concurrency for web tiers. CPU utilisation is the default and is misleading for memory-bound or I/O-bound applications.
Can autoscaling cause a large bill?
Yes. A retry storm or scraper looks like legitimate demand. Always set a maximum instance count and pair the scaling policy with a budget alert.