Scrapy vs Playwright for Dynamic Pricing #

Every price-monitoring crawler eventually hits the same fork in the road: a target renders its price in the raw HTML, or it renders it in the browser after a burst of JavaScript, XHR calls, and A/B-test logic. Pick Scrapy for the second case and you get an empty price field; pick a headless browser for the first and you burn ten times the CPU for no reason. This guide compares Scrapy’s async HTTP engine against Playwright’s headless-browser automation specifically for scraping dynamic e-commerce prices, and lands on a hybrid routing design rather than a single winner. It sits under the parent guide on async data pipelines with Python & Scrapy, and leans on two neighbours: configuring headless browsers for dynamic pricing for the Playwright deep-dive, and optimizing Scrapy for 10k SKUs per hour for squeezing the async path.

The Core Difference: HTTP Client vs Browser #

Scrapy is an asynchronous HTTP client with a parsing and scheduling framework wrapped around it. It fetches the bytes a server returns and never executes JavaScript. If the price is present in the initial HTML — or in a JSON blob, a <script type="application/ld+json"> tag, or a documented XHR endpoint — Scrapy retrieves it in a single lightweight request and its Twisted-based reactor keeps hundreds of those requests in flight concurrently on one core.

Playwright drives a real Chromium, Firefox, or WebKit process. It downloads the HTML, executes every script, fires network requests, applies client-side rendering, and only then exposes a fully-hydrated DOM you can query. That is exactly what you need when a price is written by a React component or gated behind a “click to reveal” interaction — and it is also why a Playwright page costs 30–80× the memory and CPU of a Scrapy request.

The decision is therefore not ideological. It is a per-domain question: where does the price actually live in the response lifecycle? Answer that first, and the tool follows.

Where Scrapy Wins #

Scrapy is the correct default for the majority of price targets, and the volume gap is larger than most teams expect. Three properties dominate.

Throughput per core. A single Scrapy process comfortably holds 100–300 concurrent requests through its Twisted reactor, and with well-tuned CONCURRENT_REQUESTS and AUTOTHROTTLE it sustains hundreds of product pages per minute per core. There is no browser process, no DOM, no layout engine — just parse and yield. For the tuning specifics see optimizing Scrapy for 10k SKUs per hour.

Cost. Because each request is a few kilobytes of state rather than a 300 MB browser tab, you scrape tens of thousands of SKUs on hardware that would run a handful of Playwright workers. On a metered proxy plan, Scrapy also transfers only the bytes you request, whereas a browser pulls images, fonts, analytics, and ad scripts unless you aggressively block them.

Reaching the real data source. Many “JavaScript-rendered” prices are actually served by a clean JSON API that the front end calls. Open the network tab, find the XHR, and Scrapy hits that endpoint directly — faster and more stable than the rendered page, because APIs change their shape far less often than DOM markup. This “call the API the site calls” pattern retires a large fraction of would-be Playwright targets.

import scrapy

class PriceSpider(scrapy.Spider):
    name = "static_price"
    start_urls = ["https://shop.example.com/p/wh-1000xm5"]

    def parse(self, response):
        # Most retailers embed a Product JSON-LD block; the price is right there,
        # no browser required. This is the single highest-yield extraction path.
        for block in response.css('script[type="application/ld+json"]::text'):
            import json
            data = json.loads(block.get())
            offers = data.get("offers", {})
            if offers:
                yield {
                    "sku": data.get("sku"),
                    "price": offers.get("price"),           # -> "349.99"
                    "currency": offers.get("priceCurrency"), # -> "USD"
                    "availability": offers.get("availability"),
                }

For extracting these embedded blobs at scale, see extracting hidden price data from JSON-LD.

Where Playwright Wins #

Playwright earns its cost on the targets Scrapy structurally cannot reach.

Genuinely client-rendered prices. Single-page-app storefronts (React, Vue, Svelte) that compute the price in the browser from a GraphQL response merged with client-side pricing rules leave nothing useful in the initial HTML. Playwright runs that code and reads the final number.

Interaction-gated prices. “Add to cart to see price”, geo/currency selectors, size-variant dropdowns, and login walls all require real DOM interaction. Playwright clicks, selects, waits for the network to settle, and extracts.

Sophisticated anti-bot. Some targets fingerprint the client — checking for a real navigator, canvas rendering, and JS challenge execution. A headless browser passes challenges (including some Turnstile and JS-challenge flows) that a bare HTTP client cannot; see bypassing Cloudflare Turnstile with Playwright.

from playwright.sync_api import sync_playwright

def scrape_dynamic_price(url):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        # Block images/fonts/media so the browser only spends time on the
        # scripts that actually produce the price — cuts bandwidth ~70%.
        page.route("**/*", lambda r: r.abort()
                   if r.request.resource_type in {"image", "font", "media"}
                   else r.continue_())
        page.goto(url, wait_until="networkidle")
        page.wait_for_selector('[data-testid="product-price"]')  # SPA hydration
        price = page.inner_text('[data-testid="product-price"]')
        browser.close()
        return price  # -> "$349.99"

Head-to-Head Trade-off Table #

The maintenance column deserves emphasis: Playwright adds timing fragility on top of selector fragility. A spider that waits on networkidle can flake when a third-party analytics beacon hangs, so headless flows need explicit, price-specific wait_for_selector waits and generous retries — operational overhead Scrapy simply does not have.

Recommendation: Route, Don’t Choose #

Do not standardize the whole fleet on one tool. Standardize on a router that classifies each domain and sends it down the cheapest path that actually returns the price. The economics are decisive: if Playwright is 40× more expensive per page and 80% of your targets expose the price in HTML or a JSON endpoint, an all-browser fleet costs roughly 30× more than a hybrid one for identical coverage.

Concretely:

1. Probe once per domain. Fetch the raw HTML and check for the price in the markup, JSON-LD, or a discoverable XHR endpoint. Cache the verdict per domain.
2. Default to Scrapy. Route static-HTML, JSON-LD, and clean-API domains to the async engine. This is the bulk of the fleet.
3. Escalate to Playwright only on miss. Reserve browsers for domains where the probe found no server-side price and no usable API — the genuine SPA and interaction-gated cases.
4. Re-probe on a schedule. Sites migrate from server-rendered to client-rendered (and back). A weekly re-probe keeps a domain from silently sitting on the expensive path after it started serving JSON-LD.

def classify_target(html: str, has_json_api: bool) -> str:
    """Decide the crawl engine for a domain from one probe response.
    Returns 'scrapy' for cheap paths, 'playwright' for browser-only prices."""
    price_in_markup = ('itemprop="price"' in html
                       or 'application/ld+json' in html and '"price"' in html)
    if price_in_markup or has_json_api:
        return "scrapy"        # ~80% of targets land here
    return "playwright"        # reserve the expensive engine for real SPAs

print(classify_target('<span itemprop="price">349.99</span>', False))  # scrapy
print(classify_target('<div id="root"></div>', False))                  # playwright

Run the router at the front of the pipeline described in async data pipelines with Python & Scrapy, and you keep Scrapy’s throughput on the common case while paying for Playwright only where it is the sole option.

Verification & Testing #

Validate the classifier against fixtures that mirror your real target mix, and treat a wrong route as a defect — a false “scrapy” verdict silently drops prices, and a false “playwright” verdict quietly inflates cost.

def test_classifier():
    # Static price in markup -> cheap path
    assert classify_target('<span itemprop="price">19.99</span>', False) == "scrapy"
    # JSON-LD offer present -> cheap path
    ld = '<script type="application/ld+json">{"@type":"Product","price":"9"}</script>'
    assert classify_target(ld, False) == "scrapy"
    # Empty SPA shell, no API -> browser path
    assert classify_target('<div id="app"></div>', False) == "playwright"
    # Empty shell but a documented API exists -> cheap path
    assert classify_target('<div id="app"></div>', True) == "scrapy"
    print("router routes all fixtures correctly")

test_classifier()

Beyond unit tests, sample the live output: if a domain routed to Scrapy starts emitting null prices, the site has gone client-rendered and the domain must escalate. Alert on per-domain null-price rate, not just crawl success.

Edge Cases & Gotchas #

• Hybrid pages. Some templates render a stale server-side price that JavaScript then overwrites with a personalized or promotional one. Scrapy reads the stale number and looks successful. Spot-check a handful of Playwright renders per domain and compare — a systematic delta means the domain must move to the browser path despite having a price in the HTML.
• JSON-LD that lies. A minority of retailers ship a placeholder or list price in JSON-LD and show the real price only after client-side rendering. Trust JSON-LD by default, but validate it against occasional browser renders exactly as you would flag a fake discount in statistical outlier detection for price data.
• Playwright memory leaks. Long-running browser workers leak context and memory. Recycle the browser process every N pages and use one context per target with context.close(), rather than reusing a single page across thousands of goto calls.
• scrapy-playwright is not a free lunch. The integration middleware lets a Scrapy spider render specific requests in a browser, which is convenient — but each rendered request still carries full browser cost. Use it to render only the flagged requests, never as a blanket setting.

Performance Notes #

The whole argument reduces to one ratio. A Scrapy request costs on the order of single-digit milliseconds of CPU and a few kilobytes of memory; a Playwright page costs hundreds of milliseconds of CPU and hundreds of megabytes of memory. At 10,000 SKUs/hour, that is the difference between one modest worker and a small autoscaling cluster of browser pods. Keep Scrapy synchronous and horizontally cheap, run Playwright behind a bounded worker pool with a concurrency cap (browsers exhaust RAM long before they exhaust CPU), and put a queue between the router and the Playwright pool so a burst of SPA targets cannot starve the fast path. Block non-essential resource types on every browser page — images, fonts, media, and third-party analytics routinely account for 60–70% of a page’s bytes and none of its price.

Frequently Asked Questions #

Can Scrapy render JavaScript at all? Not on its own — Scrapy never executes JavaScript. You can bolt on a rendering backend such as the scrapy-playwright middleware or a Splash service, but that just runs a browser under Scrapy’s scheduler and carries full browser cost. If a page truly needs rendering, you are paying Playwright prices regardless of the wrapper.

Is Playwright always better at avoiding blocks? No. A real browser defeats fingerprint and JS-challenge defenses that a bare HTTP client fails, but it does nothing for IP-reputation or rate-based blocking — those need good proxies and pacing regardless of engine. For simple targets, a well-behaved Scrapy crawler with rotating proxies is both cheaper and less detectable than an under-configured headless browser.

Should I just use Playwright everywhere to keep one codebase? Only if your target set is tiny. The per-page cost gap of roughly 30–80× means an all-browser fleet is dramatically more expensive for the same coverage once you pass a few hundred SKUs. A router adds modest code but pays for itself immediately in compute and proxy bandwidth.

Related #

• Async Data Pipelines with Python & Scrapy — the parent guide describing the ingestion pipeline this router sits at the front of.
• Configuring Headless Browsers for Dynamic Pricing — the full Playwright setup for the targets the router escalates.
• Optimizing Scrapy for 10k SKUs per Hour — tuning the async path so the cheap route carries as much of the fleet as possible.
• Extracting Hidden Price Data from JSON-LD — the highest-yield extraction the Scrapy path relies on.