# Python requests proxy guide: requests, httpx and Scrapy

URL: https://cloudj.net/blog/python-scraping-proxy
Published: 2026-11-11 · Updated: 2026-11-11

> Set up a Python requests proxy with authentication, SOCKS5, httpx and Scrapy, plus timeouts, retries and error handling that hold up in real scraping jobs.

Configuring a Python requests proxy takes one dictionary, but getting it right in a real job takes a little more: credentials with special characters, SOCKS5 support, timeouts, retries and clear errors when the proxy itself is the problem. This guide walks through working examples for `requests`, `httpx` and Scrapy using a dedicated proxy with username and password authentication. Every snippet uses placeholders (`HOST`, `PORT`, `USERNAME`, `PASSWORD`) that you replace with the values from your proxy provider.

## Before you write code: scrape responsibly

A proxy changes where your requests come from. It does not change what you are allowed to do. Before pointing any script at a website:

- **Read the terms of service.** Many sites restrict automated access. If the site offers an API or a data feed, use it.
- **Check `robots.txt`** and respect the paths it disallows for your user agent.
- **Rate-limit yourself.** Keep concurrency low per domain, add delays, and back off when the server returns `429 Too Many Requests` or `503`.
- **Identify yourself** with an honest `User-Agent` that includes a way to contact you, where appropriate.
- **Avoid personal data** unless you have a lawful basis to collect it.

A dedicated proxy is useful here for stable reasons: a fixed exit IP that a partner can allowlist, a country-specific view of public pages, or keeping your server's own IP separate from crawling traffic. The [dedicated proxy guide](https://cloudj.net/blog/dedicated-proxy-guide) explains how dedicated IPs differ from shared and rotating ones.

## Step 1: install the libraries

```bash
pip install "requests[socks]" "httpx[socks]" scrapy
```

The `[socks]` extras pull in the SOCKS support each library needs. You only need them if you plan to use SOCKS5; HTTP proxies work without them.

## Step 2: a Python requests proxy with authentication

`requests` takes a `proxies` dictionary keyed by the scheme of the target URL. The value is the proxy URL. Note that the proxy URL starts with `http://` even for HTTPS targets: the client opens a tunnel to the proxy with the `CONNECT` method and TLS runs end to end inside it.

```python
import requests
from urllib.parse import quote

USERNAME = "USERNAME"
PASSWORD = quote("PASSWORD", safe="")  # encode characters such as @ : /
PROXY = f"http://{USERNAME}:{PASSWORD}@HOST:PORT"

proxies = {"http": PROXY, "https": PROXY}

resp = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxies,
    timeout=(5, 20),  # (connect, read) seconds
)
resp.raise_for_status()
print(resp.json())  # should print the proxy IP, not yours
```

Two details save hours of debugging. First, URL-encode the password; an `@` or `:` inside it breaks the URL silently. Second, always set a timeout. Without one, `requests` can wait indefinitely on a stalled connection.

### Using a session

For many requests, reuse a `Session` so connections are kept alive:

```python
session = requests.Session()
session.trust_env = False  # ignore HTTP_PROXY / HTTPS_PROXY from the environment
session.proxies.update(proxies)
session.headers["User-Agent"] = "example-bot/1.0 (+https://example.com/contact)"

resp = session.get("https://example.com/", timeout=(5, 20))
```

Setting `trust_env = False` matters because proxy variables in the environment can otherwise take precedence over `session.proxies`. It also stops `requests` from reading `.netrc` and CA bundle variables, so leave it on only if you do not rely on those.

### SOCKS5

If your proxy order is set to SOCKS5, change the scheme. Use `socks5h` so DNS names are resolved by the proxy rather than on your machine:

```python
PROXY = f"socks5h://{USERNAME}:{PASSWORD}@HOST:PORT"
proxies = {"http": PROXY, "https": PROXY}
```

The trade-offs between the two protocols are covered in our [SOCKS5 vs HTTP proxy comparison](https://cloudj.net/blog/socks5-vs-http-proxy).

## Step 3: error handling and polite retries

Separate proxy failures from website failures. A `ProxyError` usually means wrong credentials, a client IP that is not on the whitelist, or an expired order; retrying will not fix it. Timeouts and `5xx` responses are often temporary and worth a limited retry with backoff.

```python
import time
import requests

def fetch(url, proxies, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            resp = requests.get(url, proxies=proxies, timeout=(5, 20))
        except requests.exceptions.ProxyError as exc:
            raise RuntimeError("Proxy rejected the connection: check credentials or whitelist") from exc
        except (requests.exceptions.Timeout, requests.exceptions.ConnectionError):
            time.sleep(2 ** attempt)
            continue

        if resp.status_code == 429 or resp.status_code >= 500:
            retry_after = resp.headers.get("Retry-After", "")
            time.sleep(int(retry_after) if retry_after.isdigit() else 2 ** attempt)
            continue

        resp.raise_for_status()
        return resp

    raise RuntimeError(f"Giving up on {url} after {attempts} attempts")
```

`ProxyError` is a subclass of `ConnectionError`, so it must be caught first. Honour `Retry-After` when the server sends it, and stop after a few attempts instead of hammering a site that is asking you to slow down.

## Step 4: the same setup in httpx

`httpx` takes a single `proxy` argument on the client (the older `proxies` argument was removed in recent versions). It also supports async, which is handy when you need a handful of concurrent requests.

```python
import asyncio
import httpx

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

async def main(urls):
    limits = httpx.Limits(max_connections=2)
    async with httpx.AsyncClient(proxy=PROXY, timeout=20.0, limits=limits) as client:
        for url in urls:
            try:
                r = await client.get(url)
                r.raise_for_status()
                print(url, len(r.text))
            except httpx.ProxyError as exc:
                print("Proxy problem:", exc)
                break
            except httpx.HTTPError as exc:
                print("Request failed:", url, exc)
            await asyncio.sleep(1)

asyncio.run(main(["https://api.ipify.org?format=json"]))
```

For SOCKS5, install `httpx[socks]` and use a `socks5://` proxy URL. Keep `max_connections` small; async makes it easy to send far more traffic than a site wants.

## Step 5: Scrapy

Scrapy's built-in `HttpProxyMiddleware` reads the proxy from `request.meta["proxy"]` and turns credentials in the URL into a `Proxy-Authorization` header. Combine it with Scrapy's politeness settings:

```python
import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 2,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 1,
        "AUTOTHROTTLE_ENABLED": True,
        "USER_AGENT": "example-bot/1.0 (+https://example.com/contact)",
    }

    def start_requests(self):
        yield scrapy.Request("https://example.com/catalog", meta={"proxy": PROXY})

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item, meta={"proxy": PROXY})

    def parse_item(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}
```

Setting `meta["proxy"]` covers the requests you create. To route every request through the proxy without touching each one, `HttpProxyMiddleware` also reads the standard `http_proxy` and `https_proxy` environment variables. Scrapy does not support SOCKS proxies out of the box, so use an HTTP proxy with it.

## Troubleshooting quick reference

| Symptom | Likely cause | Fix |
|---|---|---|
| `407 Proxy Authentication Required` or `ProxyError` | Wrong username or password, or unencoded special characters | Copy credentials again; URL-encode the password |
| Works on your laptop, fails on the server | Order uses IP whitelist and the server IP is not on it | Add the server IP to the whitelist |
| `Missing dependencies for SOCKS support` | SOCKS extra not installed | Install `requests[socks]` or `httpx[socks]` |
| Exit IP is still your own | Environment variables override session proxies | Pass `proxies=` per request or set `trust_env = False` |
| Many `429` responses | Crawling too fast | Lower concurrency, add delay, honour `Retry-After` |

## FAQ

### Should the proxy URL start with http or https for HTTPS sites?

Use `http://` for a standard HTTP proxy. The HTTPS connection to the site is tunnelled through the proxy with `CONNECT`, so your traffic is still encrypted end to end.

### How do I use an IP whitelist instead of a password?

Add the public IP of the machine running your script to the whitelist in the console (up to 20 addresses), then drop the credentials from the proxy URL: `http://HOST:PORT`.

### Can Scrapy use SOCKS5?

Not natively. Use the HTTP protocol for your proxy order when running Scrapy.

### Does a proxy stop me from being blocked?

No. A proxy does not prevent blocks. Polite rates, honest identification and following a site's rules matter far more.

## Next steps

Cloudj sells dedicated IPv4 proxies in seven countries with username and password or whitelist authentication; see the [proxy overview](https://cloudj.net/proxy-ipv4) and live prices on the [pricing page](https://cloudj.net/pricing). If you also manage store or ad accounts, the guides on [e-commerce seller proxies](https://cloudj.net/blog/ecommerce-seller-proxy-vietnam) and [Facebook Ads proxies](https://cloudj.net/blog/facebook-ads-proxy) cover those workflows, and you can order IPs directly in the [console](https://app.cloudj.net).