Price Monitoring Competitor Intelligence Workflows

GDPR and Scraped Pricing Data

Pricing data is not personal data — until a scraper carelessly staples a reviewer’s name, an individual seller’s identity, or a “sold by” contact to the price it captured. The moment that happens, your pricing warehouse holds personal data under the GDPR, and every obligation that follows — lawful basis, minimization, retention limits, erasure requests — now applies to tables that were never meant to carry it. The durable fix is not a cleanup job downstream; it is to make personal data structurally unable to enter the store. This recipe builds that guarantee at the parser, as the personal-data layer of compliance and legal boundaries of price scraping, and follows on from the ToS risk tiers that flag which targets carry personal data in the first place.

This is engineering guidance, not legal advice. Minimization, lawful basis, and retention are legal concepts your data-protection lead owns; what follows is how to implement a minimization-by-construction policy in code, not a ruling on what your obligations are.

Prerequisites & Input Contract

The recipe uses pydantic v2 for schema enforcement (pip install pydantic), on Python 3.11+. The input is a raw field dictionary as the parser extracts it from a page — deliberately messy, potentially containing personal fields. The output is a validated pricing record that cannot contain personal data, because the schema has no field to hold it.

# Raw extraction from a product page — may include personal data the parser saw.
raw = {
    "product_id": "SKU-4471",
    "price": "29.99",
    "currency": "EUR",
    "seller": "Jana Novak",            # an individual's name -> personal data
    "seller_type": "individual",
    "reviewer": "m.becker@email.com",  # personal data -> must never be stored
    "review_text": "great, fast ship", # free text may embed personal data
    "url": "https://market.example.net/p/4471",
    "scraped_at": "2026-07-15T09:00:00Z",
}

The guiding principle is data minimization: collect and retain only the fields the pricing use-case genuinely needs. A price index needs the price, the currency, the product identity, and lineage. It does not need who sold the item or who reviewed it, so those fields should never reach storage.

Step-by-Step Implementation

Step 1 — Detect personal data in the raw fields

Before minimizing, make the personal data visible so the pipeline can log and monitor it. A lightweight detector flags obvious identifiers — emails, phone numbers — and fields named as personal, plus individual-seller names.

import re

EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE = re.compile(r"(?:(?:\+|00)\d{1,3}[\s-]?)?(?:\d[\s-]?){7,12}\d")
PII_FIELD_NAMES = {"seller", "reviewer", "reviewer_name", "buyer", "contact", "email", "phone"}

def detect_pii(raw: dict) -> list[str]:
    """Return the field names that carry (likely) personal data."""
    flagged = []
    for key, value in raw.items():
        if key in PII_FIELD_NAMES and value:
            # A named business seller is not personal data; an individual is.
            if key == "seller" and raw.get("seller_type") == "business":
                continue
            flagged.append(key)
        elif isinstance(value, str) and (EMAIL.search(value) or PHONE.search(value)):
            flagged.append(key)          # free text carrying an email/phone
    return flagged

Step 2 — Enforce a PII-free schema at the parser boundary

The schema is the guarantee. It declares exactly the fields a pricing record may hold, and pydantic’s extra="forbid" rejects any attempt to smuggle in an undeclared field. There is simply no attribute named seller or reviewer on the model, so personal data has nowhere to land.

from pydantic import BaseModel, ConfigDict, field_validator
from decimal import Decimal

class PricingRecord(BaseModel):
    model_config = ConfigDict(extra="forbid")   # unknown fields raise, not silently pass

    product_id: str
    price: Decimal
    currency: str
    url: str
    scraped_at: str
    # NOTE: no seller/reviewer/contact fields exist. PII cannot be represented.

    @field_validator("currency")
    @classmethod
    def _iso_currency(cls, v: str) -> str:
        if len(v) != 3 or not v.isalpha():
            raise ValueError("currency must be an ISO 4217 code")
        return v.upper()

Step 3 — Scrub, then construct

The scrub step keeps only the declared fields and drops everything else, logging what it removed so the pipeline can prove minimization happened. Construction then validates the survivors.

ALLOWED = set(PricingRecord.model_fields)

def scrub_and_build(raw: dict) -> tuple[PricingRecord, dict]:
    """Drop non-allowed fields (incl. all PII), then validate. Returns record + audit."""
    flagged = detect_pii(raw)
    minimized = {k: v for k, v in raw.items() if k in ALLOWED}
    dropped = sorted(set(raw) - ALLOWED)
    record = PricingRecord(**minimized)         # raises if a required field is missing
    audit = {"url": raw.get("url"), "pii_detected": flagged,
             "fields_dropped": dropped, "scraped_at": raw.get("scraped_at")}
    return record, audit

Step 4 — Run it 

record, audit = scrub_and_build(raw)
print(record.model_dump())
print(audit)

Expected output:

{'product_id': 'SKU-4471', 'price': Decimal('29.99'), 'currency': 'EUR', 'url': 'https://market.example.net/p/4471', 'scraped_at': '2026-07-15T09:00:00Z'}
{'url': 'https://market.example.net/p/4471', 'pii_detected': ['seller', 'reviewer', 'review_text'], 'fields_dropped': ['review_text', 'reviewer', 'seller', 'seller_type'], 'scraped_at': '2026-07-15T09:00:00Z'}

The stored record contains only pricing and lineage. The seller name, the reviewer email, and the free-text review — all dropped before construction, all logged in the audit so you can demonstrate the minimization occurred. The prices themselves flow on to normalization, such as converting multi-currency prices to a base currency, carrying no personal data with them.

PII scrub at the parser boundary before pricing data is storedA raw extracted field set containing price, currency, seller name, and reviewer email enters a detector that flags the personal-data fields. The scrub step keeps only fields declared in the PII-free schema and drops the rest. A validated pricing record holding only price, currency, product id, url, and timestamp is written to the pricing store, while the list of dropped and flagged fields is written to an audit log. No personal data reaches the store.Raw fieldsprice · currencyseller · reviewerurl · scraped_atDetect PIIflag personalScrub +PII-free schemaextra = forbidDropped fieldslogged, not storedPricing storeprice · currencyno personal dataAuditlog

Verification & Testing

Assert the two invariants that matter: personal data never survives, and a required pricing field being absent is a hard failure rather than a silent gap.

from pydantic import ValidationError

def test_scrub():
    rec, audit = scrub_and_build(raw)
    stored = rec.model_dump()
    # 1. No personal field can appear in the stored record.
    assert not ({"seller", "reviewer", "review_text"} & set(stored))
    # 2. Minimization is evidenced in the audit trail.
    assert "reviewer" in audit["pii_detected"]
    assert "seller" in audit["fields_dropped"]
    # 3. Missing a required pricing field fails loudly.
    try:
        scrub_and_build({"product_id": "x", "currency": "EUR",
                        "url": "u", "scraped_at": "t"})   # no price
        assert False, "should have raised"
    except ValidationError:
        pass
    print("pii scrub verified")

test_scrub()
# pii scrub verified

Run the detector over a sample of stored records periodically as a canary: it should find nothing, because the schema already forbids it. A non-zero hit means a field was added to the model without review — exactly the regression this test guards against.

Edge Cases & Gotchas

  • Free-text fields as PII carriers. A review_text or description can embed a name, email, or phone number even when the field itself is not “personal.” The safest rule is to not store free text at all in a pricing pipeline; if you must, run the regex detector and reject or redact matches before storage.
  • Business vs individual sellers. A company name is generally not personal data; a sole trader’s own name is. The detector’s seller_type == "business" exemption encodes that distinction, but it is only as good as the source’s labeling — when in doubt, drop the field.
  • Lawful basis and retention are still required. Even PII-free pricing data has retention limits, and if any lawful-basis question arises you must be able to show minimization and purpose. Record the purpose (price monitoring) alongside the data and delete price observations past the retention window from the ToS tier controls automatically.
  • Erasure requests reach derived data. If any personal data ever did enter the store, an erasure request must reach not just the raw row but every derivative — indexes, backups, caches. The by-construction approach is what lets you answer such a request truthfully with “we do not hold personal data” rather than hunting through derived tables.

Performance Notes

The scrub is a per-record operation dominated by the regex scan over string values, which is linear in the total length of the free-text fields — negligible for typical pricing rows and easily tens of thousands of records per second on a single worker. The schema construction is a bounded validation over a handful of declared fields. Because both steps are pure and stateless, they parallelize across the ingestion worker pool with no coordination. The one place to watch is the regex detector on large free-text blobs: cap the length you scan (a few kilobytes is ample) so a pathological description cannot dominate parse time, consistent with the byte-cap discipline used elsewhere in the compliance layer.

Frequently Asked Questions

Is scraped pricing data personal data under GDPR? Price, currency, and product identity on their own are not personal data. It becomes personal data the moment it is linked to an identifiable individual — a named private seller, a reviewer’s email, a buyer’s contact. The whole point of minimizing at the parser is to keep that link from ever forming, so the pricing store stays outside the scope of personal-data obligations.

Why enforce a PII-free schema instead of just deleting personal fields later? Deleting later is a process you have to run correctly every time and prove afterward; a schema that has no field to hold personal data makes storing it structurally impossible. With extra fields forbidden, a parser change that tried to add a seller name would fail loudly at construction rather than quietly persisting personal data.

How does minimization by construction help with erasure requests? If you never store personal data, you can answer an erasure request truthfully and immediately — there is nothing linked to the individual to erase. Minimizing at the parser avoids the far harder task of tracing one person’s data across raw rows, indexes, caches, and backups after the fact.