Price Monitoring Competitor Intelligence Workflows

Parsing robots.txt in Python

A price scraper that does not consult robots.txt before each fetch is making a decision it cannot defend: it is fetching paths the target has explicitly asked automated agents to leave alone. Honoring the Robots Exclusion Protocol is the first and cheapest control layer in compliance and legal boundaries of price scraping, and this recipe implements it end to end — fetching the file robustly, parsing per-user-agent rules and Crawl-delay, discovering sitemaps, caching the result, and exposing a single can_fetch check that runs before every request. The output plugs directly into the async fetch loop described in async data pipelines with Python & Scrapy.

This is engineering guidance, not legal advice. Honoring robots.txt is well-established good practice and a strong signal of good faith, but it is not by itself a complete compliance program; pair it with the ToS and personal-data controls in the parent guide.

Prerequisites & Input Contract

The recipe uses only the standard library plus requests for the fetch (swap in httpx or an async client without changing the logic). Python 3.11+ is assumed. The parser consumes a domain and produces a rules object; the crawler then calls can_fetch(user_agent, url) before dispatching.

# Input: the domain (scheme + host) whose robots.txt governs the crawl.
# Output contract of the parsed rules object:
#   can_fetch(user_agent: str, url: str) -> bool   # honor before every fetch
#   crawl_delay(user_agent: str)        -> float | None
#   sitemaps                            -> list[str]
#   fetched_ok                          -> bool     # False if the file was unreachable

One design decision governs everything below: when robots.txt cannot be fetched or parsed, the safe default for a well-behaved price scraper is to disallow, not allow. Standards leave the unreachable case ambiguous, and being cautious costs a little coverage while a permissive default risks fetching paths you had no basis to touch.

Step-by-Step Implementation

Step 1 — Fetch robots.txt robustly with a bounded response

The fetch must survive a missing file, a redirect, a slow server, and a hostile 10 MB response. A 404 conventionally means “no restrictions”; a 5xx or timeout is treated as fetch-failed, which we later resolve as disallow.

import requests

ROBOTS_MAX_BYTES = 512 * 1024   # cap: real robots.txt files are tiny; refuse giants

def fetch_robots(domain: str, user_agent: str, timeout: float = 5.0):
    """Fetch <domain>/robots.txt. Returns (text, fetched_ok).

    fetched_ok is True only when we have an authoritative answer:
    a 200 with a body, or a 404 (meaning 'no rules, allow all').
    """
    url = domain.rstrip("/") + "/robots.txt"
    try:
        resp = requests.get(url, headers={"User-Agent": user_agent},
                            timeout=timeout, stream=True)
    except requests.RequestException:
        return "", False                     # network error -> fail closed later
    if resp.status_code == 404:
        return "", True                      # no robots.txt = crawling permitted
    if resp.status_code != 200:
        return "", False                     # 5xx / 401 / 403 -> not authoritative
    body = resp.raw.read(ROBOTS_MAX_BYTES, decode_content=True)
    return body.decode("utf-8", errors="replace"), True

Step 2 — Parse per-user-agent groups, crawl-delay, and sitemaps

urllib.robotparser handles the core User-agent/Disallow/Allow grammar and longest-match precedence correctly, so we build on it rather than reinventing path matching. It does not expose sitemaps and older versions ignore Crawl-delay, so we scan those lines ourselves.

from urllib.robotparser import RobotFileParser

def parse_robots(text: str, fetched_ok: bool):
    rp = RobotFileParser()
    rp.parse(text.splitlines())              # builds per-agent Allow/Disallow groups

    # Sitemaps and Crawl-delay: scan raw lines (case-insensitive directives).
    sitemaps, delays, current_agents = [], {}, []
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()  # strip inline comments
        if not line or ":" not in line:
            continue
        field, value = (p.strip() for p in line.split(":", 1))
        field = field.lower()
        if field == "user-agent":
            current_agents.append(value.lower())
        elif field == "sitemap":
            sitemaps.append(value)
        elif field == "crawl-delay":
            for agent in current_agents or ["*"]:
                try:
                    delays[agent] = float(value)
                except ValueError:
                    pass
        elif field in ("allow", "disallow"):
            current_agents = []              # a rule line closes the agent group
    return rp, sitemaps, delays, fetched_ok

Step 3 — Wrap it in a rules object with a fail-closed can_fetch

The crawler should never touch urllib internals. This wrapper is the contract: can_fetch returns False when the file was unreachable, and resolves Crawl-delay by falling back from the specific agent to the * group.

class RobotRules:
    def __init__(self, rp, sitemaps, delays, fetched_ok):
        self._rp = rp
        self.sitemaps = sitemaps
        self._delays = delays
        self.fetched_ok = fetched_ok

    def can_fetch(self, user_agent: str, url: str) -> bool:
        if not self.fetched_ok:
            return False                     # fail closed on unreachable robots.txt
        return self._rp.can_fetch(user_agent, url)

    def crawl_delay(self, user_agent: str):
        ua = user_agent.lower()
        # exact token match first, then the wildcard group
        for key in (ua, "*"):
            if key in self._delays:
                return self._delays[key]
        return None

def load_rules(domain, user_agent):
    text, ok = fetch_robots(domain, user_agent)
    return RobotRules(*parse_robots(text, ok))

Step 4 — Run it and read the output 

SAMPLE = """
User-agent: *
Disallow: /cart
Disallow: /account
Crawl-delay: 2

User-agent: PriceBot
Allow: /products
Disallow: /products/beta
Crawl-delay: 5

Sitemap: https://shop.example.com/sitemap.xml
"""

rules = RobotRules(*parse_robots(SAMPLE, fetched_ok=True))
print(rules.can_fetch("PriceBot", "https://shop.example.com/products/widget-1"))
print(rules.can_fetch("PriceBot", "https://shop.example.com/products/beta/x"))
print(rules.can_fetch("PriceBot", "https://shop.example.com/cart"))
print(rules.crawl_delay("PriceBot"), rules.crawl_delay("OtherBot"))
print(rules.sitemaps)

Expected output:

True
False
True
5.0 2.0
['https://shop.example.com/sitemap.xml']

Note PriceBot may fetch /cart — its own group has no such Disallow, and once a matching named group exists the * group no longer applies to it. That surprises people, and it is exactly why per-agent parsing matters: a rule written for everyone does not automatically bind a named agent.

Verification & Testing

Assert the behaviors that break silently in production — agent precedence, the fail-closed default, and longest-match on Allow/Disallow.

def test_robots():
    r = RobotRules(*parse_robots(SAMPLE, fetched_ok=True))
    assert r.can_fetch("PriceBot", "https://shop.example.com/products/widget") is True
    assert r.can_fetch("PriceBot", "https://shop.example.com/products/beta/z") is False
    assert r.crawl_delay("PriceBot") == 5.0
    assert r.crawl_delay("Anon") == 2.0            # falls back to '*'

    # Fail closed: an unreachable robots.txt disallows everything.
    blocked = RobotRules(*parse_robots("", fetched_ok=False))
    assert blocked.can_fetch("PriceBot", "https://shop.example.com/products/x") is False
    print("robots rules verified")

test_robots()
# robots rules verified

For a live smoke test, point load_rules at a domain you control, confirm fetched_ok is True, and diff the can_fetch verdict for a known-disallowed path against the file you published.

Edge Cases & Gotchas

  • Missing vs unreachable robots.txt. A clean 404 means crawl freely (fetched_ok=True, all paths allowed). A timeout or 5xx means you do not know the rules, so fetched_ok=False and can_fetch returns False. Collapsing these two into “allow” is the most common and most dangerous bug in homemade parsers.
  • Oversized or adversarial files. A misconfigured or hostile server can return a multi-megabyte body. The ROBOTS_MAX_BYTES cap in Step 1 bounds memory and parse time; truncation is acceptable because legitimate directives live at the top of the file.
  • Wildcards and end-anchors. Modern robots.txt uses * wildcards and $ end-of-path anchors (Disallow: /*.pdf$). urllib.robotparser on Python 3.11+ handles these; if you must support older runtimes, verify wildcard matching explicitly or use a third-party parser rather than assuming.
  • Conflicting or multiple matching agents. If two User-agent groups both match your token, the standard says the most specific one wins, and within a group the longest matching rule wins. Do not merge groups by hand; let the parser resolve precedence, and pass the exact UA string your fetcher sends — not a friendly display name.

Performance Notes

Parsing is a one-time cost per domain, not per URL. A price crawl of ten thousand product pages on one host does a single fetch-and-parse, then answers can_fetch from memory for every subsequent URL — a bounded path-prefix scan measured in microseconds. Cache the RobotRules object per domain with a short TTL (an hour is a reasonable default; shorter for cautious targets) so a target that tightens its rules is respected quickly, and refresh on TTL expiry rather than on every request. Because the object is immutable after construction, it is safe to share across the entire async worker pool without locking, which is what keeps the compliance check off the hot path even at high crawl concurrency.

Frequently Asked Questions

Does a missing robots.txt mean I can scrape everything? A genuine 404 conventionally means there are no restrictions, so crawling is permitted. But a timeout, a 5xx, or a 403 is not a 404 — you do not have an authoritative answer, so a careful price scraper treats those as disallow until the file can be fetched.

Should I obey Crawl-delay even though it is not in the original standard? Yes. Crawl-delay is not part of the original specification and some crawlers ignore it, but honoring it is a clear duty-of-care signal and costs you almost nothing. Treat the parsed value as a floor on the interval between requests to that host, falling back to the wildcard group when your agent has no specific delay.

Why parse per-user-agent instead of just using the wildcard rules? Because a named group overrides the wildcard group entirely for that agent. If a site publishes rules for your specific bot token, the * rules stop applying to you, and honoring only * could make you both over- and under-restrictive. Always evaluate with the exact user-agent string your fetcher sends.