ToS Risk Tiers for Price Scraping #

“Is it okay to scrape this site?” is the wrong question to answer once, informally, in a Slack thread. The right move is to score every target on a few observable signals — is the data public, is there an auth wall, does the Terms of Service explicitly prohibit automation, is personal data present, how aggressive is the planned request rate — and let that score assign a risk tier that carries a fixed set of engineering controls. This recipe builds that scoring function and its tier registry in Python. It is the second control layer in compliance and legal boundaries of price scraping, running after a target’s robots.txt has been parsed and honored.

This is engineering guidance, not legal advice. A risk tier is a way to operationalize a policy your legal team owns; the signals and weights below are a defensible default, but where the line sits between “caution” and “forbidden” for your business is a decision for counsel, not a scoring constant.

Prerequisites & Input Contract #

No third-party libraries are required — the scorer is standard-library Python 3.11+. The input is a small profile of observable facts about a target, gathered once during onboarding and refreshed when a target’s site changes.

from dataclasses import dataclass

@dataclass(frozen=True)
class TargetProfile:
    domain: str
    data_is_public: bool        # reachable without login?
    behind_auth_wall: bool      # is the price behind a login/paywall?
    tos_prohibits_scraping: bool  # does ToS explicitly forbid automated access?
    contains_personal_data: bool  # seller names of individuals, reviewer identities?
    planned_rate_hz: float      # intended per-domain request rate
    official_api_exists: bool   # is there a first-party data feed?

The scorer’s job is to turn that profile into a tier plus a control set. Everything downstream — the fetch method, the rate ceiling, the retention window — reads from the tier, so the classification is the single decision that governs how a target is treated.

Step-by-Step Implementation #

Step 1 — Score the risk signals #

Each signal contributes points; higher totals mean higher risk. An explicit ToS prohibition and the presence of personal data are the heaviest signals, because they carry the clearest legal exposure. Note the negative weight for an available official API — its existence lowers scraping risk because it means a compliant alternative is at hand.

def risk_score(p: TargetProfile) -> int:
    """Additive risk score. Higher = more caution required."""
    score = 0
    score += 0 if p.data_is_public else 40      # non-public data is a hard escalator
    score += 35 if p.behind_auth_wall else 0    # auth walls signal a contract you accept on login
    score += 30 if p.tos_prohibits_scraping else 0
    score += 30 if p.contains_personal_data else 0
    score += min(20, int(p.planned_rate_hz * 10))  # aggressive rate adds load risk
    score -= 10 if p.official_api_exists else 0  # a sanctioned feed lowers risk
    return max(0, score)

Step 2 — Map the score to a tier #

Tiers are bands over the score, but two signals override the band outright: an auth wall or non-public data forces the target to restricted or below regardless of the numeric total, because those are categorical, not gradual, concerns.

def assign_tier(p: TargetProfile) -> str:
    # Categorical overrides first — these are not negotiable by score.
    if p.behind_auth_wall or not p.data_is_public:
        return "forbidden" if p.contains_personal_data else "restricted"
    score = risk_score(p)
    if score >= 55:
        return "restricted"
    if score >= 30:
        return "caution"
    if score >= 10:
        return "standard"
    return "open"

Step 3 — Attach controls to each tier #

The tier is only useful if it resolves to concrete controls the fetch layer enforces. This mapping is the operational heart of the recipe: allowed methods, rate ceiling, PII policy, and retention all follow from the tier, so the crawler never improvises.

TIER_CONTROLS = {
    "open":       {"methods": ("get",),        "max_rate_hz": 1.0,  "pii": "drop",    "retention_months": 24, "scrape_allowed": True},
    "standard":   {"methods": ("get", "render"),"max_rate_hz": 0.5,  "pii": "drop",    "retention_months": 18, "scrape_allowed": True},
    "caution":    {"methods": ("get",),        "max_rate_hz": 0.2,  "pii": "forbid",  "retention_months": 12, "scrape_allowed": True},
    "restricted": {"methods": ("api",),        "max_rate_hz": 0.0,  "pii": "forbid",  "retention_months": 12, "scrape_allowed": False},
    "forbidden":  {"methods": (),              "max_rate_hz": 0.0,  "pii": "forbid",  "retention_months": 0,  "scrape_allowed": False},
}

def classify(p: TargetProfile) -> dict:
    tier = assign_tier(p)
    controls = TIER_CONTROLS[tier]
    return {"domain": p.domain, "tier": tier, "score": risk_score(p), **controls}

Step 4 — Persist assignments in a versioned registry #

Classifications must be stored, not recomputed ad hoc, so that a later audit can show which tier governed a scrape and when it changed. The registry is keyed by domain and versioned.

class TierRegistry:
    def __init__(self, version="registry-v1"):
        self.version = version
        self._by_domain: dict[str, dict] = {}

    def upsert(self, profile: TargetProfile) -> dict:
        record = {**classify(profile), "policy_version": self.version}
        self._by_domain[profile.domain] = record
        return record

    def get(self, domain: str) -> dict | None:
        return self._by_domain.get(domain)

Step 5 — Run it #

registry = TierRegistry()

public_shop = TargetProfile("shop.example.com", data_is_public=True,
    behind_auth_wall=False, tos_prohibits_scraping=False,
    contains_personal_data=False, planned_rate_hz=0.4, official_api_exists=False)

strict_market = TargetProfile("market.example.net", data_is_public=True,
    behind_auth_wall=False, tos_prohibits_scraping=True,
    contains_personal_data=True, planned_rate_hz=1.5, official_api_exists=True)

for prof in (public_shop, strict_market):
    print(registry.upsert(prof))

Expected output:

{'domain': 'shop.example.com', 'tier': 'standard', 'score': 14, 'methods': ('get', 'render'), 'max_rate_hz': 0.5, 'pii': 'drop', 'retention_months': 18, 'scrape_allowed': True, 'policy_version': 'registry-v1'}
{'domain': 'market.example.net', 'tier': 'restricted', 'score': 65, 'methods': ('api',), 'max_rate_hz': 0.0, 'pii': 'forbid', 'retention_months': 12, 'scrape_allowed': False, 'policy_version': 'registry-v1'}

The strict marketplace scores high (explicit prohibition + personal data + aggressive rate) and lands in restricted — no scraping, API only — even though its data is technically public. That is the intended behavior: a public page under a prohibiting ToS is not a green light.

The Tier Reference #

Verification & Testing #

Test the overrides explicitly — they are where a naive score-only classifier goes wrong.

def test_tiers():
    base = dict(data_is_public=True, behind_auth_wall=False,
                tos_prohibits_scraping=False, contains_personal_data=False,
                planned_rate_hz=0.1, official_api_exists=False)

    assert assign_tier(TargetProfile("a", **base)) == "open"

    # Auth wall forces restricted regardless of an otherwise-clean profile.
    walled = {**base, "behind_auth_wall": True}
    assert assign_tier(TargetProfile("b", **walled)) == "restricted"

    # Auth wall + personal data escalates to forbidden.
    pii_walled = {**walled, "contains_personal_data": True}
    assert assign_tier(TargetProfile("c", **pii_walled)) == "forbidden"

    # Explicit prohibition pushes a public target into caution or worse.
    prohibited = {**base, "tos_prohibits_scraping": True, "contains_personal_data": True}
    assert assign_tier(TargetProfile("d", **prohibited)) in {"caution", "restricted"}
    print("tier assignment verified")

test_tiers()
# tier assignment verified

Beyond unit tests, review the registry quarterly: re-fetch each target’s ToS and robots.txt, re-run classify, and diff against the stored tier. A target that moved from Standard to Restricted because it added an anti-scraping clause must have its crawler behavior change immediately, and the version bump records exactly when the policy shifted.

Edge Cases & Gotchas #

• Public data under a prohibiting ToS. Reachability without a login does not equal permission. The scorer weights an explicit prohibition heavily precisely so that public-but-prohibited targets escalate instead of sliding into Open. When in doubt, prefer the API fallback path.
• ToS changes silently. Terms are edited without notice. A stored tier can be stale within a day of a target updating its terms, so treat the classification as perishable and re-verify on a cadence tied to the tier — quarterly for Caution and above.
• Rate as a soft signal that hardens. A modest planned rate barely moves the score, but an aggressive one against a small site is a real duty-of-care problem the numeric weight understates. Always floor the rate ceiling well below anything that could degrade the target’s service, independent of the score.
• Personal data hides in “public” pages. A seller marketplace lists individual sellers by name; those pages are public but the data is personal. Set contains_personal_data from what the page actually renders, and route the downstream handling through GDPR and scraped pricing data.

Performance Notes #

Classification is cheap and rare: it runs once per target at onboarding and again only when a target’s profile changes, so its cost never touches the crawl hot path. At fetch time the crawler does a single dictionary lookup into the registry to read the tier’s controls — an O(1) operation with no computation. The registry itself is small (one record per target domain, typically hundreds to low thousands of entries) and lives entirely in memory, loaded at startup and refreshed on change. The only recurring cost is the periodic re-verification, which is a scheduled batch job, not an inline check.

Frequently Asked Questions #

If data is publicly visible, is it always safe to scrape? No. Public visibility and permission are different things. A page reachable without a login can still be governed by Terms of Service that prohibit automated collection, and it can contain personal data that triggers separate obligations. The scorer deliberately escalates public-but-prohibited and public-but-personal targets rather than treating reachability as consent.

What makes an auth wall an automatic escalation? Logging in typically means accepting terms as a condition of access, and data behind that wall is not public. The classifier treats any auth wall as a categorical override to Restricted or Forbidden regardless of the numeric score, because scraping behind a login is a materially different risk from scraping an open catalog.

How often should I re-check a target’s tier? Tie the cadence to the tier: Open and Standard targets can be reviewed semi-annually, while Caution and above deserve a quarterly re-fetch of the ToS and robots.txt. Any target whose classification changes should have its crawler controls updated immediately and the change recorded via a registry version bump.

Related #

• Compliance & Legal Boundaries of Price Scraping — the parent guide where this scorer is the second control layer.
• Parsing robots.txt in Python — the layer that runs before tier assignment.
• GDPR and Scraped Pricing Data — how targets flagged for personal data are handled downstream.
• API Fallback & Official Data Source Integration — the required method for Restricted-tier targets.