GPU instances are the most expensive commodity a cloud account rents by the hour, and the most frequently left running by accident. Both facts are worth understanding before the first one is launched.
What they are actually for
A GPU is good at doing the same arithmetic to enormous numbers of values simultaneously. That maps onto a narrow set of workloads:
- Training neural networks.
- Inference for models too large or too slow on a CPU — language models, image generation, speech.
- Scientific simulation and other dense linear algebra.
- Video transcoding, using the dedicated encode blocks rather than the general compute units.
- Rendering.
Everything else — web serving, databases, ordinary batch processing — runs no faster on a GPU instance and costs considerably more. Renting one “in case it helps” is the most common way to waste money on an AI project.
VRAM is the binding constraint
The number that decides whether a workload runs at all is the accelerator’s memory, not its speed. A model that does not fit in VRAM either fails outright or falls back to a slower path that negates the reason for the GPU.
Practical implications:
- Size by memory first, then by throughput.
- Quantisation and reduced precision are memory strategies before they are speed strategies.
- Splitting a model across multiple accelerators adds interconnect as a new bottleneck — and interconnect bandwidth varies enormously between instance families.
Availability is a real constraint
GPU capacity is finite and regionally uneven in a way ordinary compute is not. Two consequences:
- The instance type you want may be unavailable in your preferred region, and the answer is to use another region rather than to wait.
- New accounts frequently have a GPU quota of zero and must request an increase explicitly. Do this days before you need it — see account age, verification and service limits.
The data has to get there
Training data and model weights are large, and moving them is neither instant nor free. Keep the dataset in object storage in the same region as the accelerator, and be aware that pulling it repeatedly across regions is billed as egress every time.
A frequent and avoidable pattern: an expensive accelerator sitting idle while data copies over a slow path. Stage the data first, then start the instance.
Billing granularity
GPU instances bill for the whole instance whenever it is running, whether or not the accelerator is busy. An idle GPU instance costs the same as a saturated one, so the discipline is entirely about lifetime:
- Stop instances the moment a job finishes; script it rather than relying on memory.
- Use spot or preemptible capacity for training that checkpoints.
- Separate development from training — write and debug code on a cheap CPU instance with a tiny sample, then run the real job.
- Set a budget alert before the first launch, not after the first invoice.
Where to go next
- Running LLM inference on your own cloud account
- Cloud accounts for AI training
- Choosing a cloud for AI workloads
Questions
Frequently asked
Do I need a GPU for every AI workload?
No. Small models, classical machine learning and low-volume inference frequently run acceptably on CPU. GPUs matter for training and for models whose size or latency requirements exceed what a CPU can deliver.
Why can I not launch a GPU instance on a new account?
GPU quotas commonly default to zero on new accounts and must be requested. Capacity is also regionally uneven, so the type you want may exist only in some regions.
Does an idle GPU instance cost less?
No. Billing is for the instance running, not for accelerator utilisation. Stopping instances promptly is the main cost control.