# Batch inference GPU cost: when the region does not matter

URL: https://cloudj.net/blog/batch-inference-cheapest-region
Published: 2026-11-12 · Updated: 2026-11-12

> Cut batch inference GPU cost for embeddings, captions and transcripts: ignore distance, measure cost per item, sort by price and watch bandwidth.

Batch inference GPU cost behaves differently from serving cost. When you embed a million documents, caption an image archive or transcribe a year of call recordings, nobody is waiting on a single response. Distance to users stops mattering, and the only number that counts is what each finished item costs. That frees you to search every country, take the lowest suitable price and design the job around it. This article shows how.

The underlying trade-off, near users versus cheapest, is covered in [GPU near users vs the cheapest GPU](https://cloudj.net/blog/gpu-near-users-vs-cheapest-gpu).

## Which jobs qualify

A job is a good fit for "region does not matter" when:

- inputs can be uploaded in bulk and outputs downloaded in bulk;
- no person waits on an individual result;
- the job can be split into shards and resumed;
- your data policy allows processing in any country (if not, filter to the countries you are allowed to use and apply the rest of this article within them).

Typical examples: generating embeddings for search, captioning or tagging images, transcribing audio, translating documents, classifying support tickets, scoring datasets with a model. For transcription specifically, see [Whisper transcription on a GPU for Vietnamese audio](https://cloudj.net/blog/whisper-transcription-gpu-vietnamese).

## Batch inference GPU cost: optimise cost per item

For a batch job, a cheaper hourly price is not automatically cheaper. A faster card at a higher price can finish sooner and cost less overall. Work it out as:

```text
cost per item = (GPU minutes x locked price per minute
                 + storage for the run
                 + GB downloaded and uploaded x bandwidth price)
                / items processed
```

Everything on the right side is visible before or during the run. The price shown at deploy is locked for that instance, billing is per minute, and each listing shows its storage price per GB-month and bandwidth price per TB. The [GPU cost estimation guide](https://cloudj.net/blog/estimate-gpu-cloud-cost) walks through each term.

## Step 1: measure throughput on a sample

Before renting for the full job, run a representative sample, for example a few thousand items, on each candidate card. Record items per minute with your real batch size and precision.

Typical candidates for batch inference:

| Card | Memory | Why consider it |
|---|---|---|
| [RTX 3090](https://cloudj.net/gpu-cloud/rtx-3090) | 24 GB | Plenty of memory for embedding models, captioners and speech models at a modest hourly price |
| [RTX 4090](https://cloudj.net/gpu-cloud/rtx-4090) | 24 GB | Same memory, newer architecture; may finish enough sooner to win on cost per item |
| [A100 PCIe](https://cloudj.net/gpu-cloud/a100-pcie) | 40 GB version listed | Larger batches or larger models; datacenter card, needs a higher account level |

Do not trust general benchmarks for this. Your model, batch size and input lengths decide the result. Per-minute billing makes a sample run on three cards cheap.

## Step 2: search every country, sorted by price

Open the [RTX 3090 deploy page](https://app.cloudj.net/deploy/gpu?model=rtx-3090) and leave the **Country** field empty. Cloudj GPUs come from a live inventory of independent host machines in many countries, and the results are sorted by total price, lowest first, up to 50 machines. For a batch job, that sorted list is exactly what you want.

Then check three things on the top candidates:

- **Max duration and "available until"**: the whole job, plus time to download results, must finish before the instance and disk are removed.
- **Bandwidth price per TB**: see step 3.
- **Download speed**: a slow link makes pulling the model and uploading inputs take longer, while the GPU sits idle.

If your policy limits countries, enter those codes instead. If the card you want is not listed, compare the next card on your cost-per-item sheet; stock changes during the day.

## Step 3: watch the bandwidth price

Bandwidth is the term people forget. It is charged per GB downloaded and uploaded at each machine's listed price, and those prices differ between machines. A job that uploads large audio files or downloads millions of embeddings can spend a meaningful share of its budget on transfer.

- Use the **maximum bandwidth $/TB** filters for download and upload when the job moves a lot of data.
- Compress inputs and outputs (`zstd`, `gzip`, compact formats such as Parquet for embeddings).
- Pull models from their official source directly onto the instance instead of uploading them yourself.
- Download only final outputs, not intermediate files.

## Step 4: shard, checkpoint, resume

Every listing is an independent host with no hardware SLA. If a host goes offline, billing pauses but the instance is not moved. Design the batch so losing a machine costs you minutes, not the job:

1. Split inputs into numbered shards.
2. Write each shard's output to its own file, then mark it done.
3. On start, skip shards already marked done.
4. Copy finished outputs off the instance regularly:

```bash
rsync -avP -e "ssh -p <port>" root@<host>:/workspace/out/ ./out/
```

With that in place, you can also split a big job across several cheap machines in different countries and merge the results.

## Step 5: destroy promptly

When the last shard is downloaded and verified, destroy the instance. Stopping still charges storage, and a finished batch box has no reason to stay. The deposit held at deploy returns after you destroy, sometimes after up to 72 hours while bandwidth charges settle.

The same thinking applies to training jobs that do not need to be near users; [LoRA training on GPUs in the USA](https://cloudj.net/blog/lora-training-gpu-usa) is one example.

## FAQ

### Should I always take the cheapest listing?

Take the lowest cost per item, not the lowest hourly price. Check that duration, bandwidth price and download speed suit the job.

### Does the GPU's country affect batch results?

No. It only affects transfer times and, if you have one, your data policy.

### What if a machine disappears mid-job?

With sharded outputs copied off regularly, deploy another machine and resume from the next unfinished shard.

### Are bandwidth prices the same on every machine?

No. Each listing shows its own download and upload prices per TB.

## Price your batch

Open the [RTX 3090 deploy page](https://app.cloudj.net/deploy/gpu?model=rtx-3090), leave the country empty and compare the top results against your cost-per-item sheet. Live "from" prices for every model are on [GPU cloud](https://cloudj.net/gpu-cloud).