Running a VPS in front of GPU server backends solves a problem every team meets once their model goes public: the GPU is the part most likely to change. On Cloudj, a GPU instance runs on an independent host machine with an "available until" date, there is no automatic move if the host goes offline, and you may simply find a better card next week. A small VPS that never moves gives your users one stable address while the GPU behind it comes and goes. This guide shows the architecture, the SSH tunnel that connects the two, an Nginx configuration, and the routine for swapping GPUs without touching DNS.
If you are still deciding whether you need a GPU at all, the GPU instance vs VPS for AI guide helps.
The architecture
users -> DNS -> VPS (Nginx: TLS, auth, rate limit, queue) -> SSH tunnel -> GPU instance (model server on 127.0.0.1)The VPS holds everything that should be stable: the domain, TLS certificate, API keys or login, rate limits, logs, a job queue and perhaps a database. The GPU instance runs only the model server, bound to localhost so it is never exposed directly.
Why an SSH tunnel? GPU instances are reached by SSH with a key, on the host and port shown in the instance's Connection tab. The port is not always 22. A tunnel reuses that access, encrypts the traffic and needs no extra open ports on the GPU side.
Choosing where the VPS lives
Put the VPS close to your users, because every request passes through it. Cloudj VPS locations have fixed cities, unlike GPUs, which are reported at country level only. Common choices:
- a Singapore VPS for Southeast Asian users,
- a VPS in Germany (Frankfurt) for European users,
- a VPS in the United States, in one of eight cities, for North American users.
The GPU can then be anywhere that suits the job. For interactive services, pick a GPU in the same region as the VPS to keep the extra hop short; the LLM inference on a Singapore GPU guide shows a matching setup for Asia. Search GPUs on the deploy page with the country codes for that region; stock changes during the day.
Step 1: prepare the VPS
Start from a hardened server: a non-root user, SSH keys, a firewall that only allows SSH, HTTP and HTTPS, and automatic security updates. The secure new VPS checklist covers each step. Then install Nginx and obtain a TLS certificate; the Nginx reverse proxy with SSL guide walks through it.
Create a dedicated SSH key on the VPS for the tunnel and add its public key to your Cloudj account, so you can select it when deploying GPU instances. A newly added key can take up to three minutes to be accepted after an instance starts.
Step 2: run the model server on the GPU
On the GPU instance, start your server bound to localhost, for example on port 8000:
vllm serve <hf-model-id> --host 127.0.0.1 --port 8000Any server works the same way: an OpenAI-compatible API, an image generation backend, a speech service. Keep it in tmux or a process manager so it survives a dropped session.
Step 3: keep the tunnel alive with systemd
On the VPS, create a service that holds the tunnel open and restarts it if it drops:
[Unit]
Description=SSH tunnel to current GPU
After=network-online.target
[Service]
User=tunnel
EnvironmentFile=/etc/gpu-tunnel.env
ExecStart=/usr/bin/ssh -N -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o ExitOnForwardFailure=yes -o StrictHostKeyChecking=accept-new -i /home/tunnel/.ssh/gpu -p ${GPU_PORT} -L 127.0.0.1:9000:127.0.0.1:8000 root@${GPU_HOST}
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.targetSave it as /etc/systemd/system/gpu-tunnel.service, put GPU_HOST and GPU_PORT in /etc/gpu-tunnel.env, then run systemctl enable --now gpu-tunnel. Port 9000 on the VPS now reaches port 8000 on the GPU.
Step 4: point Nginx at the tunnel
location /v1/ {
auth_request /auth;
proxy_pass http://127.0.0.1:9000;
proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 300s;
}proxy_buffering off lets streamed tokens reach clients as they are generated, and the long read timeout covers slow generations. Use whatever authentication fits: an auth_request to your app, basic auth, or API keys checked by a small service. The Nginx proxy module documentation lists every directive.
Swapping GPUs without DNS changes
When a host's "available until" date approaches, a host goes offline, or you want a different card:
- Deploy the new GPU instance and start the model server on it.
- Test it directly through a temporary tunnel from the VPS.
- Edit
/etc/gpu-tunnel.envwith the new host and port, thensystemctl restart gpu-tunnel. - Check a request through the public URL.
- Copy anything you need off the old instance, then destroy it.
Users see, at most, a few seconds of retries. If you keep a queue on the VPS, requests can wait instead of failing. For extra resilience, run two GPU instances and two tunnels, and list both in an Nginx upstream block.
Limits to keep in mind
Every GPU host is independent, with no hardware SLA, so this pattern reduces disruption but does not eliminate it. Keep backups of weights and configuration, and script the GPU setup so a new instance is ready in minutes. Live GPU prices are on GPU cloud.
FAQ
Why not point DNS straight at the GPU?
GPU hosts change. Each change would mean a DNS update, a new certificate and downtime. The VPS keeps one stable address.
Does the tunnel add much delay?
It adds the network distance between VPS and GPU. Keep both in the same region for interactive services.
Can the VPS handle the model itself?
Not for GPU workloads. The VPS handles TLS, auth and queuing; the GPU runs the model.
Which VPS location should I choose?
The one closest to most of your users: Singapore, Frankfurt and the US cities are common choices.
Next step
Set up a VPS near your users, then deploy a GPU from the deploy page in the same region. Connect them with the tunnel service above, and practise one GPU swap before you go live.