# How to test latency to a GPU server before a long job URL: https://cloudj.net/blog/measure-latency-to-gpu-host Published: 2026-11-12 · Updated: 2026-11-12 > Test latency to a GPU server before you commit: deploy briefly, measure with ping, mtr and curl from your users' side, time the first token and destroy if bad. Before you commit a GPU to a week of serving users, spend a few minutes finding out how it feels from where those users are. Country-level location is a good first filter, but two hosts in the same country can behave differently from your users' networks, and only a measurement tells you which. On Cloudj, GPU billing is per minute, so a test deployment that turns out badly costs very little: you measure, decide, and destroy if needed. This guide shows how to test latency to a GPU server properly, from the network layer up to the time users wait for the first token. For how latency fits into the wider choice of region, start with the [GPU cloud by location guide](https://cloudj.net/blog/gpu-cloud-by-location-guide). ## Step 1: deploy a short test instance Pick the candidate from the live list on the [GPU deploy page](https://app.cloudj.net/deploy/gpu). Filter by the country codes near your users, then deploy the machine you are considering. You only need it long enough to measure, usually well under an hour. - Use the "Ubuntu 22.04 + CUDA (SSH)" template unless you need your full stack for the end-to-end test. - Choose a small disk if this is purely a network test; disk cannot be resized later, but you will destroy this instance anyway. - Select your SSH key. A newly added key can take up to 3 minutes to be accepted after the instance starts. When it is running, open the instance's **Connection** tab and note the host and the SSH port. The port is often not 22. ## Step 2: test the network path from your users' side Run these from a machine on your users' network, not from your office if your users are elsewhere. For a Vietnamese audience, that could be a laptop on a Viettel, VNPT or FPT line; for an API consumed by your own VPS, run them on that VPS. **Round-trip time with ping.** It may or may not answer, because some hosts drop ICMP: ```bash ping -c 20 ``` **The path and where delay or loss appears, with mtr:** ```bash mtr -rwc 50 ``` Look at the last hops. Loss that appears only in the middle and disappears later is usually a router de-prioritising replies, not real loss. Loss or a jump in delay that persists to the final hop is what users will feel. **TCP connect time to the SSH port**, which works even when ping does not: ```bash for i in $(seq 1 10); do curl -s -o /dev/null -w "%{time_connect}\n" telnet://: --max-time 3 done ``` If you already measure VPS locations this way, the method is the same as in [testing VPS latency with ping and mtr](https://cloudj.net/blog/test-vps-latency-ping-mtr). ## Step 3: measure what users actually wait for Network round-trip time is only part of the story. For AI apps, users wait for the first token or the first result, which also includes queueing, prompt processing and the model itself. Measure end to end. Start your model server on the GPU bound to `localhost`, and reach it the way production will, ideally through your front-end VPS. For a quick test from your own machine, a tunnel works: ```bash ssh -N -L 8000:localhost:8000 -p root@ ``` Then time a streaming request. For an OpenAI-compatible endpoint: ```bash curl -N -s -o /dev/null \ -w "first_byte=%{time_starttransfer}s total=%{time_total}s\n" \ http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{"model":"my-model","prompt":"Xin chào","max_tokens":64,"stream":true}' ``` `time_starttransfer` approximates the time to first token as seen by the client. Repeat it 20 or 30 times, with realistic prompts, and look at the median and the worst results, not just the best. The [real-time AI latency guide](https://cloudj.net/blog/real-time-ai-latency-gpu) explains how to split this number into network and compute parts. Note that an SSH tunnel from a home laptop adds its own overhead. For a decision about production, measure through the same path production will use. ## Step 4: test at the times that matter International routes can behave differently at peak hours. If your users are most active in the evening, measure in the evening too. One run at a quiet hour can make a host look better than it will feel on a busy night. Your test instance costs per minute while it runs, so plan two short sessions rather than leaving one running all day. ## Step 5: decide, then keep or destroy Compare what you measured with your latency budget: | Result | Action | |---|---| | Path is clean and first-token time fits your budget | Keep it, or deploy your full stack on it | | Network is fine but first-token time is too high | The problem is compute: try a faster card or a smaller model | | Network delay or loss is high from users' side | Destroy and try a host in a nearer country | | Results are unstable between runs | Test at another hour or try another listing | Destroy deletes the instance and its disk, and charges stop. The deposit held at deploy returns after destroy, sometimes after up to 72 hours while final bandwidth charges settle. ## Measure the VPS side too If a VPS sits in front of the GPU, the user's path ends at the VPS, and the GPU only needs to be close to the VPS. Measure users to VPS and VPS to GPU separately. The [VPS location latency guide](https://cloudj.net/blog/choose-vps-location-latency) covers the first half of that path. ## FAQ ### Why does ping to my GPU host time out? Some hosts drop ICMP. Use TCP connect time to the SSH port or an HTTP request through your service instead. ### Can I see the city of the GPU host before testing? No. GPU locations are shown at country level only, which is why measuring matters. ### Is it expensive to test several hosts? Billing is per minute, so short tests are cheap. Destroy each test instance as soon as you have your numbers. ### Should I measure from my office? Only if your users are there. Measure from networks your users actually use. ## Run your first test Open the [GPU deploy page](https://app.cloudj.net/deploy/gpu), deploy a short instance near your users and run the commands above before committing. Live prices for every model are on [GPU cloud](https://cloudj.net/gpu-cloud).