A training run is a long-lived job on expensive hardware, and the things that end it early are almost never the model. They are quotas, disks, interruptions and the absence of a checkpoint.
Before the first run
- Confirm the quota for the exact accelerator type in the exact region. New accounts frequently start at zero — see account age, verification and service limits.
- Confirm capacity, which is a separate question from quota. Popular types are regionally scarce.
- Stage the data into object storage in the same region as the accelerator.
- Do a full dry run on a small sample and a cheap instance, end to end, including the checkpoint write and a restart from that checkpoint.
- Set a budget alert at the number you expect the run to cost.
Step four is the one that separates a run that finishes from a run that is abandoned at 70%.
Checkpointing is the whole strategy
Checkpoint frequently enough that losing the interval is acceptable, and write checkpoints somewhere outside the instance. With that in place, three things become true at once:
- An interruption costs the interval, not the run.
- Spot and preemptible capacity becomes usable, which is where the substantial saving on long jobs comes from.
- You can stop deliberately — to change a hyperparameter, to move regions, to wait for cheaper capacity — without losing progress.
Checkpoints are also large. Set a retention policy when you set the schedule, or storage becomes the surprise line item.
Handling interruption properly
Preemptible instances give a short termination warning. Handle it: catch the signal, flush a checkpoint, exit cleanly. Then have the job resubmitted automatically. A training pipeline that restarts itself is the difference between spot capacity being a saving and being a source of frustration.
What to watch while it runs
- Accelerator utilisation. Persistently low utilisation means the data pipeline is the bottleneck and the expensive hardware is waiting. Fix the pipeline before renting more accelerators.
- Disk on the instance. Logs and local checkpoints fill it, and the job dies at 100%.
- Loss curves, so a diverging run is stopped in hour two rather than hour twenty.
- Spend to date against the budget.
Controlling the cost of a long job
- Develop on CPU with a tiny sample; reserve the accelerator for real runs.
- Profile one epoch before committing to fifty.
- Prefer one larger instance to several small ones where the model fits, because multi-node adds interconnect overhead and complexity.
- Shut down automatically at the end of the job — scripted, not remembered.
- Delete intermediate artefacts on a schedule.
Where to go next
Questions
Frequently asked
How often should a training job checkpoint?
Often enough that losing the interval is acceptable, written to storage outside the instance. That single practice is what makes cheaper interruptible capacity usable.
Why is my GPU utilisation low during training?
Almost always the data pipeline. If the accelerator is waiting for input, adding more accelerators makes the bill larger and the run no faster.
Can I train on spot instances?
Yes, provided the job checkpoints, handles the termination signal cleanly and is resubmitted automatically. That combination is where most of the saving on long runs comes from.