Redis Session Persistence for Distributed Scrapers #

Run one scraper and its cookies, its set of already-seen URLs, and its work queue all live comfortably in process memory. Run twenty scrapers across five machines and that in-memory state becomes the problem: worker B re-crawls what worker A already fetched, a session that solved a challenge on worker C is useless to worker D, and a deploy wipes the frontier and restarts the crawl from zero. The fix is to move shared crawl state out of the workers and into Redis, where every worker reads and writes the same session store, dedup set, and request frontier atomically. This guide is a runnable recipe for exactly that. It sits under the parent guide on async data pipelines with Python & Scrapy, and pairs with optimizing Scrapy for 10k SKUs per hour, whose throughput targets assume workers are not duplicating each other’s work.

Prerequisites & Input Contract #

You need Redis 6+ (for GETDEL and reliable SCAN), redis-py>=5.0 (pip install redis), and a stable way to identify a domain’s session. The recipe maps three crawl concerns onto three Redis data structures:

Every worker is stateless: it holds a Redis connection and nothing else that must survive a restart. The canonical URL used as a dedup key must be normalized first (lowercased host, sorted query params, stripped tracking tokens) so that two spellings of the same page collapse to one entry.

import redis

# One shared client per worker; decode_responses keeps values as str, not bytes.
r = redis.Redis(host="redis-1", port=6379, db=0, decode_responses=True)

CRAWL_ID = "prices-2026-07-15"  # namespaces this crawl's frontier and seen-set

The three structures and how every worker shares them is shown below.

Step-by-Step Implementation #

The design principle running through every step is the same: any state two workers might touch at once must be mutated with a single atomic Redis command, never a read-then-write in application code. Redis is single-threaded per shard, so each command runs to completion before the next begins — which is exactly why SADD, ZPOPMAX, and HSET can act as lock-free coordination primitives. Whenever you feel tempted to GET a value, decide something in Python, and SET it back, stop: that gap between read and write is where two workers race and both crawl the same page.

Step 1 — Persist session and cookie state in a hash #

Store each domain’s cookies and any session tokens as fields of a hash keyed by domain. A hash lets you update a single cookie without rewriting the blob, and a per-key TTL expires stale sessions automatically so a rotated challenge cookie does not linger.

import json, time

def save_session(domain: str, cookies: dict, ttl_s: int = 1800):
    """Write a domain session to Redis with a sliding 30-minute TTL."""
    key = f"sess:{domain}"
    r.hset(key, mapping={
        "cookies": json.dumps(cookies),   # serialized cookie jar
        "updated_at": int(time.time()),
    })
    r.expire(key, ttl_s)                   # session self-expires if unused

def load_session(domain: str) -> dict:
    """Return the shared cookie jar for a domain, or {} if none/expired."""
    raw = r.hget(f"sess:{domain}", "cookies")
    return json.loads(raw) if raw else {}

save_session("shop.example.com", {"cf_clearance": "abc123", "sid": "xy"})
print(load_session("shop.example.com"))
# -> {'cf_clearance': 'abc123', 'sid': 'xy'}

Now any worker that picks up a shop.example.com request reuses the clearance cookie a different worker earned, instead of re-solving the challenge and burning a proxy IP.

Step 2 — Deduplicate URLs with an atomic set #

Use SADD, whose return value is the number of new members added. That single integer is an atomic test-and-set: if it returns 1, this worker is the first to see the URL and owns crawling it; if 0, another worker already claimed it. No read-then-write race is possible.

def claim_url(url: str) -> bool:
    """Return True if THIS worker is the first to see `url`, else False."""
    # SADD returns 1 for a newly-added member, 0 if it already existed —
    # atomic, so two workers cannot both get True for the same URL.
    return r.sadd(f"seen:{CRAWL_ID}", url) == 1

print(claim_url("https://shop.example.com/p/1"))  # -> True  (first sighting)
print(claim_url("https://shop.example.com/p/1"))  # -> False (already claimed)

For very large crawls where an exact set would grow past memory budget, swap the set for a Bloom filter via the RedisBloom BF.ADD command — same atomic add-if-absent semantics, constant memory, at the cost of a tunable false-positive rate that occasionally skips a genuinely-new URL.

Step 3 — Drive the request frontier with a sorted set #

A sorted set gives you a priority queue: each URL is a member, its priority is the score, and ZPOPMAX atomically removes and returns the highest-priority URL. Because the pop is atomic, twenty workers can share one frontier and no two ever receive the same URL.

def push_request(url: str, priority: float):
    """Enqueue a URL at a given priority (higher = crawled sooner)."""
    r.zadd(f"frontier:{CRAWL_ID}", {url: priority})

def next_request() -> str | None:
    """Atomically claim the highest-priority URL, or None if the frontier is empty."""
    hit = r.zpopmax(f"frontier:{CRAWL_ID}")  # -> [(url, score)] or []
    return hit[0][0] if hit else None

push_request("https://shop.example.com/p/hot", priority=10)   # flash-sale SKU
push_request("https://shop.example.com/p/cold", priority=1)   # low-velocity SKU
print(next_request())  # -> 'https://shop.example.com/p/hot'  (highest score first)

Prioritize high-velocity SKUs, flash-sale pages, and recently-changed competitor listings with larger scores so they are refreshed first — the same prioritization logic your throughput tuning in optimizing Scrapy for 10k SKUs per hour depends on.

Step 4 — Make claim-and-enqueue atomic and restart-safe #

Steps 2 and 3 are individually atomic, but a worker that crashes between popping a URL and finishing it drops that work silently. Wrap the discovery path so a URL is only enqueued if it is newly seen, and use a short-lived processing set (an “in-flight” list with a reaper) so an interrupted URL is recovered rather than lost.

def discover(url: str, priority: float = 5) -> bool:
    """Claim a URL and enqueue it exactly once. Returns True if enqueued."""
    if claim_url(url):                 # atomic first-sighting check
        push_request(url, priority)    # only new URLs enter the frontier
        return True
    return False

def process_next(worker_id: str):
    """Pop a URL, track it in-flight, and ack on success (restart-safe)."""
    url = next_request()
    if url is None:
        return None
    # Record in-flight with a timestamp so a reaper can requeue if we die.
    r.hset(f"inflight:{CRAWL_ID}", url, f"{worker_id}:{int(time.time())}")
    try:
        # ... fetch(url), parse, yield price ...
        r.hdel(f"inflight:{CRAWL_ID}", url)   # ack: work completed
        return url
    except Exception:
        # Leave it in-flight; the reaper requeues it after a timeout.
        raise

for _ in range(2):
    print(process_next("w-1"))
# -> https://shop.example.com/p/hot
# -> https://shop.example.com/p/cold

A tiny reaper — a periodic job that scans inflight:{CRAWL_ID} for entries older than, say, 120 seconds and ZADDs them back onto the frontier — closes the crash-recovery gap. This is what makes the crawl survive a rolling deploy: kill every worker, and the URLs they were mid-fetch reappear in the frontier within two minutes.

Verification & Testing #

Assert the two properties that matter most — dedup exclusivity and frontier exclusivity — because both are silent when broken.

def test_shared_state():
    r.delete(f"seen:{CRAWL_ID}", f"frontier:{CRAWL_ID}")
    # 1. Only one claim succeeds for the same URL (dedup).
    assert claim_url("https://x/1") is True
    assert claim_url("https://x/1") is False

    # 2. Priority ordering is respected and each URL pops once (frontier).
    push_request("https://x/a", 1)
    push_request("https://x/b", 9)
    assert next_request() == "https://x/b"   # higher score first
    assert next_request() == "https://x/a"
    assert next_request() is None            # each URL delivered exactly once

    # 3. Session survives across "workers" (separate clients, same Redis).
    save_session("d.com", {"sid": "1"})
    other = redis.Redis(host="redis-1", decode_responses=True)
    assert json.loads(other.hget("sess:d.com", "cookies"))["sid"] == "1"
    print("shared state verified")

test_shared_state()

For a realistic check, run the workers concurrently: launch N processes that each loop discover/process_next against a seeded frontier and assert that the union of URLs each processed is disjoint and equals the full set — no duplicates, no drops.

Edge Cases & Gotchas #

• Session TTL vs long crawls. A 30-minute session TTL expires mid-crawl for a slow domain, forcing a re-challenge. Refresh the TTL on every successful use (EXPIRE on read) so active sessions slide forward and only genuinely-idle ones die.
• Seen-set unbounded growth. An exact seen: set grows with every unique URL and never shrinks during a crawl. Set a TTL on the whole set keyed to the crawl’s lifetime, or switch to RedisBloom once the set passes a few million members, or the key silently becomes your largest memory consumer.
• Lost work on ZPOPMAX. ZPOPMAX removes the URL immediately, so a crash before completion loses it unless you use the in-flight tracking from Step 4. Never treat a bare pop as durable delivery — pair it with the reaper.
• Cookie serialization drift. Storing a live cookie-jar object instead of a plain dict couples the store to a library version and breaks on upgrade. Serialize to a plain JSON dict of name/value/domain, exactly as in Step 1, so any worker on any version can rehydrate it.

Performance Notes #

Every operation here is O(1) or O(log N): SADD, HSET, and HGET are constant-time, and ZADD/ZPOPMAX are O(log N) in the frontier size — so a single Redis node comfortably serves tens of thousands of claim-and-pop cycles per second, well above what a scraper fleet bounded by network and proxy latency will ask of it. Keep the hot path to one round trip: batch discovery with a pipeline (r.pipeline()) when a page yields many links, and colocate Redis in the same region as the workers so round-trip time stays sub-millisecond. The frontier sorted set is the one structure to watch — if it grows into the millions while workers stall, that is backpressure telling you the crawl is producing URLs faster than it consumes them, and the fix is more workers or fewer discovered links per page, not a bigger Redis. When a single node’s throughput or memory becomes the ceiling, shard by crawl_id across instances before reaching for Redis Cluster, since these keys partition cleanly by crawl.

Frequently Asked Questions #

Why Redis instead of the database for shared scraper state? Crawl state is high-churn, short-lived, and read-write on every request — exactly what an in-memory store with atomic primitives is built for. A relational database adds latency and lock contention on the hottest path of the crawl, and you would be reimplementing SADD and ZPOPMAX with rows and transactions. Persist the extracted prices to the database; keep the coordination in Redis.

How do I stop two workers crawling the same URL? Rely on the atomic return values: SADD returns 1 only for the worker that first inserts a URL, and ZPOPMAX hands a URL to exactly one caller. Combined, they guarantee single-ownership without any application-level locking, which is the whole reason to centralize this state in Redis.

What happens to in-flight work when a worker crashes? With the in-flight hash and reaper from Step 4, a URL a crashed worker was fetching is detected as stale after a timeout and requeued onto the frontier, so it is retried rather than lost. Without that pattern, a bare ZPOPMAX silently drops the URL — which is why the recipe tracks in-flight work explicitly.

Related #

• Async Data Pipelines with Python & Scrapy — the parent guide describing the distributed crawl this shared state coordinates.
• Optimizing Scrapy for 10k SKUs per Hour — throughput tuning that assumes workers dedup and share a frontier rather than duplicating work.
• Configuring Headless Browsers for Dynamic Pricing — where the shared session store pays off most, reusing hard-won challenge cookies across browser workers.