# AI training GPU USA checklist: planning multi-day rented runs

URL: https://cloudj.net/blog/ai-training-gpu-usa
Published: 2026-11-12 · Updated: 2026-11-12

> An AI training GPU USA checklist for multi-day runs: choose 1x to 8x machines, check the availability date, set a checkpoint cadence and track per-minute cost.

Booking an AI training GPU USA run for an afternoon is easy. Running one for three or four days without losing work or money takes planning. On Cloudj, US GPUs come from a live inventory of independent host machines, each with its own reliability score, maximum duration and "available until" date, and none with a hardware SLA. That is fine for training, as long as you plan for it. This checklist walks through the run in order: before you deploy, at deploy, during the run, if something goes wrong, and after.

For general advice on renting in the United States, see the [GPU cloud USA guide](https://cloudj.net/blog/gpu-cloud-usa).

## Before you deploy

**Estimate the job on a short test.** Run a few hundred steps on the target card and time them. Multiply out to the full run, then add a margin. That number drives every later decision. Per-minute billing makes the test cheap.

**Choose the card class.**

| Card | Memory | Good for |
|---|---|---|
| [H100 SXM](https://cloudj.net/gpu-cloud/h100-sxm) | 80 GB | Large models, full fine-tunes, multi-GPU training with heavy communication |
| [A100 SXM4](https://cloudj.net/gpu-cloud/a100-sxm4) | 40 GB version listed | Mid-size fine-tunes and multi-GPU training on a tighter budget |
| [RTX 4090](https://cloudj.net/gpu-cloud/rtx-4090) | 24 GB | LoRA, vision models, diffusion training, sweeps that fit in 24 GB |

H100 and A100 are datacenter GPUs and need a higher account level than new accounts have. If the console says "This GPU class is not available at your account level", contact Cloudj support from the console well before your start date.

**Choose the GPU count.** Listings come as 1x, 2x, 4x or 8x GPUs in the same machine. More GPUs help when your code scales across them (data parallel, sharded training) and when communication between cards is efficient. The [single vs multi-GPU guide](https://cloudj.net/blog/single-gpu-vs-multi-gpu-cloud) helps you decide whether the speed-up is worth the complexity.

**Size the disk.** Base weights, dataset, several checkpoints and logs, plus headroom. The disk cannot be resized after deploy.

## At deploy

Open the [H100 SXM deploy page](https://app.cloudj.net/deploy/gpu?model=h100-sxm), or switch to the model you chose, and enter `US` in the **Country** field. Results are sorted by total price, lowest first. Location is shown at country level only.

Before clicking deploy, check three fields on the listing:

1. **Available until.** When this date arrives, the instance and its disk are removed. Your estimated run, plus margin, plus time to copy results off must end before it. If it does not, pick another machine.
2. **Max duration.** Same idea, in days.
3. **Reliability.** For a multi-day job, prefer higher values.

If no US machine fits, change the GPU count, try the other card class, loosen filters, add `CA`, or search again later; stock changes during the day. Training rarely cares about distance, so clearing the Country field is also reasonable.

Write down the locked price shown at deploy. It stays fixed for this instance, which makes cost tracking simple.

## During the run

**Checkpoint on a cadence you can afford to lose.** Ask: if the machine disappeared right now, how much training could I accept redoing? Save at least that often, including optimizer state if you want an exact resume.

**Copy checkpoints off the machine.** A checkpoint on the instance disk is lost with the instance. Pull them to storage you control, for example a VPS, on a schedule:

```bash
while true; do
  rsync -avP -e "ssh -p <port>" root@<host>:/workspace/checkpoints/ /backup/run-42/
  sleep 3600
done
```

The [GPU instance backup guide](https://cloudj.net/blog/backup-gpu-instance-before-contract-ends) covers targets and scripts in more detail.

**Watch logs and utilisation.** The instance page shows status, GPU utilisation and the last lines of the container logs. Also write your training logs to a file and include it in the sync, so you keep the full history.

**Track cost per minute.** Cost so far is the locked rate multiplied by minutes running, plus storage, plus bandwidth at the machine's listed price per GB. Check your balance once a day against your estimate. If the run is slower than the test predicted, find out why early: low GPU utilisation usually points to data loading.

**Mind the deposit.** Deploying holds a deposit (about an hour of GPU, some storage hours and a bandwidth reserve). Keep enough balance for the whole run on top of it.

## If a host goes offline

Every listing is an independent machine without a hardware SLA. If a host goes offline:

- billing pauses while it is offline;
- the instance is not moved to another machine automatically;
- your work resumes only if that host comes back.

Your recovery plan is your latest off-machine checkpoint. Deploy a new instance, copy the checkpoint and data over, and resume. A run that cannot afford this delay is one you should split into shorter segments or run on infrastructure with a contractual SLA.

## After the run

1. Copy final weights, adapters, metrics and logs off the machine.
2. Verify the copy (sizes, checksums, a quick load test).
3. **Destroy** the instance. Stopping still charges storage and releases the GPU anyway, so there is little reason to keep a finished training box.
4. Compare actual cost with your estimate, so the next estimate is better.

If you mostly train on H100s in the US, the [H100 cloud USA](https://cloudj.net/blog/h100-cloud-usa) article covers that card specifically.

## FAQ

### Can I extend a machine past its "available until" date?

Plan as if you cannot. The instance and disk are removed on that date, so finish the run and copy your data off before it.

### What happens to billing if the host goes offline?

Billing pauses while it is offline. The instance does not move to another machine.

### How often should I checkpoint?

As often as the amount of training you can afford to redo. Copy checkpoints off the machine each time.

### Is there a monthly price for long runs?

No. GPU instances are billed per minute at the price locked when you deploy.

## Plan your next US run

Open the [H100 SXM deploy page](https://app.cloudj.net/deploy/gpu?model=h100-sxm), enter `US` and check the available-until date before you commit. Live "from" prices for every card are on [GPU cloud](https://cloudj.net/gpu-cloud).