Every GPU machine on Cloudj shows an "available until" date. When that date arrives, the instance and its disk are removed, along with anything you left on it. Add the fact that each machine is an independent host with no hardware SLA, and the conclusion is simple: a GPU instance backup is your job, and it should be running long before the date gets close. This guide explains what exactly is at risk, where to copy data, and gives a script you can schedule.
The general principles of backups, such as the 3-2-1 rule and testing restores, are covered in the VPS backup strategy guide. Here we apply them to GPU instances.
Two ways an instance can end
The availability date. Each listing shows "available until" with a date, and a related max duration in days. You see it before you deploy. On that date the instance and its disk are removed, so plan your work and your final copy to finish before it.
The host goes offline. Every listing is an independent machine, and the console says it directly: "Machines are independent hosts with no hardware SLA. Keep backups of important data." If a host goes offline, billing pauses while it is offline, but the instance is not moved to another machine. Whether and when your data becomes reachable again depends on that host. Your backup is what lets you continue elsewhere.
The datacenter vs independent GPU hosts article explains what the Datacenter badge does and does not change here. In short: it describes where the host is; it is not an SLA.
What to back up
Not everything on the disk is worth copying. Sort it:
| Item | Back up? | Notes |
|---|---|---|
| Training checkpoints, LoRA adapters, final weights | Yes, regularly | The output you paid GPU time for |
| Logs, metrics, evaluation results | Yes | Small and hard to recreate |
| Your code and configs | Yes, but via git | Push to a repository rather than copying files |
| Generated outputs (images, audio, video, embeddings) | Yes | Download in batches |
| Datasets you uploaded | Only if no other copy exists | Usually you already have the original |
| Base model weights from a public source | No | Download again on a new instance |
| Python environment, pip cache, container layers | No | Rebuild from a requirements file or a custom image |
The less you need to copy, the faster your backup runs, and the lower your bandwidth charges, which are billed per GB downloaded and uploaded at the machine's listed price.
Where to copy it
Choose a target you control that outlives any single GPU:
- A Cloudj VPS. Always on, fixed address, its own disk. A good place to pull checkpoints into and to serve them to the next GPU instance. Put it in or near the country where your GPUs usually run.
- Your own storage. An office server, a NAS, or object storage you already use.
- Your workstation, for small projects, as long as it is on when the backup runs.
Pulling from the target side is usually simpler than pushing from the GPU, because the target holds the schedule and the credentials.
A GPU instance backup script
Run this on your VPS or other target. It pulls the important folders from the GPU instance, keeps a dated log and fails loudly if something goes wrong. Fill in the host and port from the instance's Connection tab.
#!/usr/bin/env bash
set -euo pipefail
HOST="<host>"
PORT="<port>"
DEST="/backup/gpu-run"
SSH="ssh -p ${PORT} -o ConnectTimeout=15"
mkdir -p "$DEST"
for dir in checkpoints adapters outputs logs; do
rsync -az --partial -e "$SSH" "root@${HOST}:/workspace/${dir}/" "${DEST}/${dir}/"
done
echo "$(date -Is) backup ok" >> "${DEST}/backup.log"Schedule it with cron, for example every hour:
0 * * * * /usr/local/bin/gpu-backup.sh >> /var/log/gpu-backup.log 2>&1Match the interval to how much work you could accept losing. For a multi-day training run, the AI training GPU USA checklist covers checkpoint cadence in more detail.
Verify, then trust
A backup you have never restored is a hope, not a backup. At least once per project:
- Compare sizes or checksums between source and copy (
sha256sumon both sides for the key files). - Load a backed-up checkpoint on a different machine and run a quick evaluation.
- Check that the backup log shows recent successful runs.
Before the date: a final checklist
A few days before "available until":
- Run the backup script by hand and confirm it succeeds.
- Copy anything else you need, such as notes in the home directory or local configuration.
- If the work continues, deploy the next instance now, while both are running, and copy data across. The guide to moving models and data between GPU regions covers instance-to-instance copies.
- Destroy the old instance once you are sure. Destroy deletes it permanently and stops charges; the deposit held at deploy returns afterwards.
Put the date in your calendar on the day you deploy. It is easy to forget during a long run.
FAQ
Can I extend a machine past its "available until" date?
Plan as if you cannot. The instance and its disk are removed on that date, so copy your data off and deploy a new instance if work continues.
What happens to my data if the host goes offline?
Billing pauses while it is offline, and the instance does not move. Your off-machine backup is what lets you continue on another machine.
Does stopping the instance protect my data?
Stopping keeps the disk until the machine's date, but it does not protect against the host going offline, and the disk is still removed on the date.
Do I need to back up base model weights?
Usually not. Download them again from their official source on the new instance.
Protect your next run
Pick your backup target, schedule the script on the day you deploy, and check each listing's available-until date on the GPU deploy page. Live prices for every model are on GPU cloud.