# vLLM GPU server in the US: an inference API on a rented GPU

URL: https://cloudj.net/blog/vllm-inference-server-usa
Published: 2026-11-12 · Updated: 2026-11-12

> Deploy a vLLM GPU server for US users: pip or a public Docker image, an OpenAI-compatible API, tensor parallel on multi-GPU machines and a US VPS in front.

vLLM has become the default way many teams serve open-weight language models: it batches requests efficiently, manages KV cache memory well and exposes an OpenAI-compatible API, so existing client code works with a changed base URL. This guide sets up a vLLM GPU server on a rented Cloudj GPU for users in the United States, from choosing a card and a US host to securing the API behind a VPS. It assumes you are comfortable with SSH and Python.

## Step 1: size the GPU for your model

vLLM needs room for model weights plus KV cache. Rough, general guidance:

| Model | Precision | Starting point |
|---|---|---|
| 7B to 8B | 16-bit | [RTX 4090](https://cloudj.net/gpu-cloud/rtx-4090), 24 GB |
| 14B | 4-bit or 8-bit quantized | RTX 4090, 24 GB |
| 30B class | Quantized | [A100 PCIe](https://cloudj.net/gpu-cloud/a100-pcie), listed in the 40 GB version, or 2× 24 GB |
| 70B class | 8-bit or 4-bit | [H100 SXM](https://cloudj.net/gpu-cloud/h100-sxm), 80 GB, or several GPUs in one machine |

The RTX 4090 is available at the starting account level. The A100 and H100 are datacenter GPUs and need a higher account level; the console tells you if a choice is not available. Live "from" prices are on [GPU cloud](https://cloudj.net/gpu-cloud).

## Step 2: find a US host

If your users are in the United States, they wait on every token, so a US host makes sense.

1. Open the [deploy page with the RTX 4090 preselected](https://app.cloudj.net/deploy/gpu?model=rtx-4090), or choose another model.
2. Pick the GPU count. For tensor parallel, choose 2× or 4×; all GPUs are in the same machine.
3. Enter `US` in the Country field. The console shows country only, not a state or city.
4. Check reliability, max duration, download speed (weights are large) and bandwidth price.
5. Choose the disk size with room for weights and cache; it cannot be resized later.

If nothing suitable is listed, try later, choose a different model with enough VRAM, or add `CA`. [GPU cloud in the USA](https://cloudj.net/blog/gpu-cloud-usa) covers US searches in more depth.

## Step 3: install vLLM

Cloudj does not ship vLLM as a template. You have two routes.

### Route A: pip on the base template

Deploy with the Ubuntu 22.04 + CUDA template and install over SSH. Use the host and port from the Connection tab:

```bash
ssh -p PORT root@HOST
nvidia-smi
python3 -m venv /opt/vllm && . /opt/vllm/bin/activate
pip install vllm
```

This is the simplest route and works at any account level.

### Route B: a public vLLM Docker image

vLLM publishes official images on Docker Hub. With the Custom image template you can reference a public image and pin its version, which avoids install time on every redeploy. Custom images need a higher account level, and private registries are not supported. [Using a custom Docker image on a cloud GPU](https://cloudj.net/blog/custom-docker-image-gpu) explains how custom images behave on Cloudj, including CUDA versions and the host driver.

## Step 4: start the OpenAI-compatible server

```bash
tmux new -s vllm
export VLLM_API_KEY=change-me
vllm serve MODEL_ID --host 127.0.0.1 --port 8000 --api-key "$VLLM_API_KEY" --max-model-len 8192
```

Replace `MODEL_ID` with the Hugging Face model you want, and check its licence. Binding to `127.0.0.1` keeps the API off the public internet. Test it from the GPU host:

```bash
curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
```

The [vLLM documentation](https://docs.vllm.ai) lists every server option.

## Step 5: tensor parallel on 2× or 4× machines

If the model does not fit on one card, or you want more throughput, deploy a 2× or 4× machine and split the model:

```bash
vllm serve MODEL_ID --tensor-parallel-size 2 --host 127.0.0.1 --port 8000 --api-key "$VLLM_API_KEY"
```

The tensor parallel size should match the number of GPUs you want to use. Consumer cards communicate over PCIe, which is slower than the interconnect on datacenter boards, so test throughput with your own traffic. Sometimes running one model copy per GPU behind a load balancer serves more users than splitting one copy. [Single GPU vs multi-GPU in the cloud](https://cloudj.net/blog/single-gpu-vs-multi-gpu-cloud) covers this choice.

## Step 6: put a US VPS in front

A GPU host is an independent machine with no hardware SLA and an "available until" date. If the host goes offline, billing pauses, but the instance does not move automatically. Do not point users at it directly. Instead:

- Run a Cloudj [VPS in the United States](https://cloudj.net/cloud-servers/united-states) with your domain, TLS, authentication, rate limiting and logging.
- On the VPS, open a persistent SSH tunnel to the GPU host: `ssh -N -L 8000:127.0.0.1:8000 -p PORT root@HOST`, managed by systemd or autossh.
- Reverse-proxy `https://api.example.com` on the VPS to `127.0.0.1:8000` with nginx.

When you need to replace the GPU, deploy a new one, start vLLM, and update the tunnel. Clients keep the same URL. [Running an AI agent on a US VPS](https://cloudj.net/blog/ai-agent-vps-united-states) shows a related VPS setup.

## Operating notes

- Watch `nvidia-smi` and the vLLM logs for memory pressure; lower `--max-model-len` or `--gpu-memory-utilization` if you hit out-of-memory errors.
- Keep a script that installs vLLM and starts the server with your flags, so a redeploy takes minutes.
- Stopping an instance releases the GPU; for a live API, prefer keeping it running or switching to a fresh deployment.

## FAQ

### Does Cloudj offer vLLM preinstalled?

No. Install vLLM with pip on the base template, or use a public vLLM image with the Custom image template.

### Can I expose the vLLM port directly to the internet?

Keep it on localhost and reach it through an SSH tunnel from a VPS that handles TLS and authentication.

### Do I need an A100 or H100?

Only for larger models or heavy traffic. Many 7B to 14B deployments run well on an RTX 4090.

### What if no US host is listed for my model?

Try later, pick another model with enough VRAM, or add `CA` and measure.

## Next step

Open the [RTX 4090 deploy page](https://app.cloudj.net/deploy/gpu?model=rtx-4090), set the Country field to `US`, and have an OpenAI-compatible endpoint running behind your VPS today.