The single vs multi GPU cloud decision shows up as a small selector on the Cloudj deploy page: 1x, 2x, 4x or 8x. All GPUs in an instance sit in the same machine and are the same model. Picking a bigger number feels like buying speed, but more GPUs only help when your software splits the work across them, and sometimes one larger card is the better answer. This guide explains what each option gives you, the two main ways to use several GPUs, and what to expect when you search for multi-GPU machines in a particular country.
If your question is specifically about large language models around 70B, the 70B multi-GPU guide does the memory arithmetic in detail.
What the GPU count selector means
When you choose 2x, 4x or 8x, the console only lists machines that have that many GPUs of the selected model. You rent the whole set together, in one container, on one host. You cannot add GPUs to a running instance; to change the count, you deploy a new machine.
Because every GPU is in the same machine, they share its CPU, RAM, disk and network. How fast the GPUs talk to each other depends on the hardware: consumer cards usually communicate over PCIe, while SXM-based datacenter machines such as the A100 SXM4 or H100 SXM usually have high-bandwidth NVLink between GPUs.
Four questions that decide it
Work through these in order.
- Does your model and its working memory fit on one GPU? If yes, a single GPU is the simplest option. Move on to question 3.
- If not, is there a single card large enough? One 80 GB or 96 GB card is often simpler and faster to set up than splitting a model across smaller cards. If such a card is listed and your account level allows it, prefer it.
- Is your job limited by throughput rather than memory? Training on a large dataset, or serving many users, can go faster with several GPUs working in parallel.
- Does your code support multi-GPU? If your training script or serving stack is single-GPU only, extra GPUs will sit idle while you pay for them.
The two ways to use several GPUs
Data parallelism. Each GPU holds a full copy of the model and processes a different slice of the batch. Gradients are averaged between GPUs each step. This is the standard way to speed up training when the model fits on one card. In PyTorch, it is usually launched with torchrun:
torchrun --nproc_per_node=4 train.py --config config.yamlFor inference, the equivalent is simpler: run one model server per GPU and put a load balancer in front, or let your serving stack handle replicas.
Tensor (or model) parallelism. The model itself is split across GPUs, so each holds part of every layer. This is how you serve or train a model that does not fit on one card. It needs fast links between GPUs, because they exchange data at every layer. With vLLM, for example:
vllm serve <hf-model-id> --tensor-parallel-size 2 --host 127.0.0.1 --port 8000The vLLM inference server guide covers this setup end to end.
Quick comparison
| Choice | Best for | Watch out for |
|---|---|---|
| 1x | Most inference, fine-tuning small and mid-size models, development | Memory ceiling of the card |
| 2x | A model slightly too big for one card; doubling training throughput | Communication overhead on PCIe cards |
| 4x | Larger training runs; serving a large model with tensor parallelism | Code must use all GPUs; disk and RAM must scale too |
| 8x | Large-scale training, very large models | Fewer listings; higher account level for datacenter cards |
A worked example: four RTX 4090 cards give 96 GB in total, the same as one 96 GB workstation card. The four-card machine may offer more raw compute for data-parallel training, but for serving one large model with tensor parallelism, the single card avoids inter-GPU traffic entirely. Test both if both are listed.
Multi-GPU listings by region
On Cloudj, location is a country filter over a live list of independent machines. Machines with 8 GPUs are a small part of that list, and 4x and 8x datacenter machines are often rented for long blocks. In a narrow country search, the 8x option may return nothing even when 1x machines of the same model are listed. Stock changes during the day.
If a multi-GPU search is empty:
- widen the country list, especially for training, where location rarely matters;
- try a different count (two 4x machines are not the same as one 8x, but 4x may be enough);
- try a sibling model with more memory per card;
- check again later.
For long training runs on US hosts, the AI training on US GPUs guide covers checkpointing and data transfer.
Cost, without numbers
Multi-GPU machines cost more per hour, and you pay for every GPU whether it is busy or not. Before a long run, confirm that all GPUs are working:
nvidia-smi --query-gpu=index,utilization.gpu,memory.used --format=csv -l 5Also size the disk and check the deposit: deploying holds roughly an hour of GPU from your balance, which grows with the GPU count. Live prices are on GPU cloud.
Consumer GPUs are available to new accounts in any count that is listed. Datacenter cards need a higher account level. The consumer vs datacenter comparison explains the differences.
FAQ
Are the GPUs in a 4x instance in the same machine?
Yes. All GPUs of an instance are in one host and are the same model.
Can I add a second GPU later?
No. The count is fixed at deploy. Deploy a new machine with the count you need.
Is 2x RTX 4090 the same as one 48 GB card?
Not quite. Total memory is similar, but splitting a model across two cards adds communication overhead.
Why can't I find an 8x machine in my country?
8x listings are rarer and often rented for long periods. Widen the country filter or try 4x.
Next step
Answer the four questions above, then open the deploy page with the RTX 4090 and compare 1x, 2x and 4x listings. Confirm every GPU is busy before a long run, and keep checkpoints off the machine.