Python requests proxy guide: requests, httpx and Scrapy

Configuring a Python requests proxy takes one dictionary, but getting it right in a real job takes a little more: credentials with special characters, SOCKS5 support, timeouts, retries and clear errors when the proxy itself is the problem. This guide walks through working examples for requests, httpx and Scrapy using a dedicated proxy with username and password authentication. Every snippet uses placeholders (HOST, PORT, USERNAME, PASSWORD) that you replace with the values from your proxy provider.

Before you write code: scrape responsibly

A proxy changes where your requests come from. It does not change what you are allowed to do. Before pointing any script at a website:

• Read the terms of service. Many sites restrict automated access. If the site offers an API or a data feed, use it.
• Check robots.txt and respect the paths it disallows for your user agent.
• Rate-limit yourself. Keep concurrency low per domain, add delays, and back off when the server returns 429 Too Many Requests or 503.
• Identify yourself with an honest User-Agent that includes a way to contact you, where appropriate.
• Avoid personal data unless you have a lawful basis to collect it.

A dedicated proxy is useful here for stable reasons: a fixed exit IP that a partner can allowlist, a country-specific view of public pages, or keeping your server's own IP separate from crawling traffic. The dedicated proxy guide explains how dedicated IPs differ from shared and rotating ones.

Step 1: install the libraries

pip install "requests[socks]" "httpx[socks]" scrapy

The [socks] extras pull in the SOCKS support each library needs. You only need them if you plan to use SOCKS5; HTTP proxies work without them.

Step 2: a Python requests proxy with authentication

requests takes a proxies dictionary keyed by the scheme of the target URL. The value is the proxy URL. Note that the proxy URL starts with http:// even for HTTPS targets: the client opens a tunnel to the proxy with the CONNECT method and TLS runs end to end inside it.

import requests
from urllib.parse import quote

USERNAME = "USERNAME"
PASSWORD = quote("PASSWORD", safe="")  # encode characters such as @ : /
PROXY = f"http://{USERNAME}:{PASSWORD}@HOST:PORT"

proxies = {"http": PROXY, "https": PROXY}

resp = requests.get(
    "https://api.ipify.org?format=json",
    proxies=proxies,
    timeout=(5, 20),  # (connect, read) seconds
)
resp.raise_for_status()
print(resp.json())  # should print the proxy IP, not yours

Two details save hours of debugging. First, URL-encode the password; an @ or : inside it breaks the URL silently. Second, always set a timeout. Without one, requests can wait indefinitely on a stalled connection.

Using a session

For many requests, reuse a Session so connections are kept alive:

session = requests.Session()
session.trust_env = False  # ignore HTTP_PROXY / HTTPS_PROXY from the environment
session.proxies.update(proxies)
session.headers["User-Agent"] = "example-bot/1.0 (+https://example.com/contact)"

resp = session.get("https://example.com/", timeout=(5, 20))

Setting trust_env = False matters because proxy variables in the environment can otherwise take precedence over session.proxies. It also stops requests from reading .netrc and CA bundle variables, so leave it on only if you do not rely on those.

SOCKS5

If your proxy order is set to SOCKS5, change the scheme. Use socks5h so DNS names are resolved by the proxy rather than on your machine:

PROXY = f"socks5h://{USERNAME}:{PASSWORD}@HOST:PORT"
proxies = {"http": PROXY, "https": PROXY}

The trade-offs between the two protocols are covered in our SOCKS5 vs HTTP proxy comparison.

Step 3: error handling and polite retries

Separate proxy failures from website failures. A ProxyError usually means wrong credentials, a client IP that is not on the whitelist, or an expired order; retrying will not fix it. Timeouts and 5xx responses are often temporary and worth a limited retry with backoff.

import time
import requests

def fetch(url, proxies, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            resp = requests.get(url, proxies=proxies, timeout=(5, 20))
        except requests.exceptions.ProxyError as exc:
            raise RuntimeError("Proxy rejected the connection: check credentials or whitelist") from exc
        except (requests.exceptions.Timeout, requests.exceptions.ConnectionError):
            time.sleep(2 ** attempt)
            continue

        if resp.status_code == 429 or resp.status_code >= 500:
            retry_after = resp.headers.get("Retry-After", "")
            time.sleep(int(retry_after) if retry_after.isdigit() else 2 ** attempt)
            continue

        resp.raise_for_status()
        return resp

    raise RuntimeError(f"Giving up on {url} after {attempts} attempts")

ProxyError is a subclass of ConnectionError, so it must be caught first. Honour Retry-After when the server sends it, and stop after a few attempts instead of hammering a site that is asking you to slow down.

Step 4: the same setup in httpx

httpx takes a single proxy argument on the client (the older proxies argument was removed in recent versions). It also supports async, which is handy when you need a handful of concurrent requests.

import asyncio
import httpx

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

async def main(urls):
    limits = httpx.Limits(max_connections=2)
    async with httpx.AsyncClient(proxy=PROXY, timeout=20.0, limits=limits) as client:
        for url in urls:
            try:
                r = await client.get(url)
                r.raise_for_status()
                print(url, len(r.text))
            except httpx.ProxyError as exc:
                print("Proxy problem:", exc)
                break
            except httpx.HTTPError as exc:
                print("Request failed:", url, exc)
            await asyncio.sleep(1)

asyncio.run(main(["https://api.ipify.org?format=json"]))

For SOCKS5, install httpx[socks] and use a socks5:// proxy URL. Keep max_connections small; async makes it easy to send far more traffic than a site wants.

Step 5: Scrapy

Scrapy's built-in HttpProxyMiddleware reads the proxy from request.meta["proxy"] and turns credentials in the URL into a Proxy-Authorization header. Combine it with Scrapy's politeness settings:

import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

class CatalogSpider(scrapy.Spider):
    name = "catalog"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 2,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 1,
        "AUTOTHROTTLE_ENABLED": True,
        "USER_AGENT": "example-bot/1.0 (+https://example.com/contact)",
    }

    def start_requests(self):
        yield scrapy.Request("https://example.com/catalog", meta={"proxy": PROXY})

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            yield response.follow(href, callback=self.parse_item, meta={"proxy": PROXY})

    def parse_item(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}

Setting meta["proxy"] covers the requests you create. To route every request through the proxy without touching each one, HttpProxyMiddleware also reads the standard http_proxy and https_proxy environment variables. Scrapy does not support SOCKS proxies out of the box, so use an HTTP proxy with it.

Troubleshooting quick reference

FAQ

Should the proxy URL start with http or https for HTTPS sites?

Use http:// for a standard HTTP proxy. The HTTPS connection to the site is tunnelled through the proxy with CONNECT, so your traffic is still encrypted end to end.

How do I use an IP whitelist instead of a password?

Add the public IP of the machine running your script to the whitelist in the console (up to 20 addresses), then drop the credentials from the proxy URL: http://HOST:PORT.

Can Scrapy use SOCKS5?

Not natively. Use the HTTP protocol for your proxy order when running Scrapy.

Does a proxy stop me from being blocked?

No. A proxy does not prevent blocks. Polite rates, honest identification and following a site's rules matter far more.

Next steps

Cloudj sells dedicated IPv4 proxies in seven countries with username and password or whitelist authentication; see the proxy overview and live prices on the pricing page. If you also manage store or ad accounts, the guides on e-commerce seller proxies and Facebook Ads proxies cover those workflows, and you can order IPs directly in the console.