Kafka vs RabbitMQ for Price Pipelines #

Between the scrapers and the normalizer sits a broker, and the broker you pick shapes what your price pipeline can and cannot do. Choose Apache Kafka and you get a replayable, ordered log that lets you reprocess a week of competitor prices after a parser bug; choose RabbitMQ and you get flexible routing, per-message acknowledgement, and low-latency task dispatch that suits fan-out to heterogeneous workers. Neither is universally “faster” or “better” — they solve different problems, and price monitoring touches both. This guide compares them as the messaging backbone for a scraping and ingestion pipeline. It sits under the parent guide on async data pipelines with Python & Scrapy, and connects to redis session persistence for distributed scrapers, which handles the crawl-frontier coordination the broker deliberately does not.

The Fundamental Split: Log vs Queue #

RabbitMQ is a message queue. A producer publishes a message, the broker routes it to one or more queues via exchanges, a consumer acknowledges it, and the broker then deletes it. Messages are transient by design — once acked, they are gone. This is smart-broker, dumb-consumer: routing logic, retries, dead-lettering, and priorities all live in the broker.

Kafka is a distributed commit log. Producers append records to the end of a partitioned, append-only log; consumers track their own offset and read forward at their own pace; and records persist for a configured retention window regardless of who has read them. This is dumb-broker, smart-consumer: the broker just stores an ordered sequence, and consumers decide what “processed” means and where they are.

That single difference — delete-on-ack versus retain-and-offset — drives almost every downstream property that matters to a price pipeline: replay, ordering, backpressure, and operational cost.

Where Kafka Wins #

Replay after a bug. Price parsers break — a competitor changes its markup, a currency field shifts, promotional discount parsing mishandles a new bundle format. With Kafka, you fix the consumer, reset its offset to yesterday, and reprocess the retained raw events. With RabbitMQ, the acked messages are already deleted; the only recovery is re-scraping, which is slower, costlier, and produces different data because prices moved.

Per-key ordering. Kafka guarantees ordering within a partition. Key raw price events by product_id and every observation for a given SKU lands in the same partition in scrape order — exactly what a time-series price history and outlier detection need. RabbitMQ preserves order only within a single queue with a single consumer, which does not scale.

High sustained throughput and multi-consumer fan-out. One raw-price topic can feed the normalizer, a real-time alerting service, and a data-lake sink independently, each at its own offset, without duplicating the data. Kafka sustains very high append rates on modest hardware because it writes sequentially and lets the OS page cache do the work.

Where RabbitMQ Wins #

Complex routing and task dispatch. RabbitMQ’s exchanges (direct, topic, fanout, headers) route messages by rule without consumer-side logic. Dispatching scrape jobs to specialized workers — “send .co.uk domains to the UK proxy pool, JS-heavy domains to the browser pool” — is a natural fit for a topic exchange.

Per-message operations. Acknowledgements, negative-acks, redelivery, per-message TTL, priority queues, and delayed delivery are first-class. A scrape job that fails can be nacked and retried or dead-lettered with a few lines; in Kafka, retry/DLQ is a pattern you build yourself with extra topics.

Operational simplicity at small-to-mid scale. A single RabbitMQ node handles tens of thousands of messages per second and is quick to stand up. Kafka’s model — partitions, consumer-group rebalancing, retention tuning, and (historically) ZooKeeper or KRaft quorum — is more to learn and operate.

Config: A Kafka Producer and a RabbitMQ Publisher #

Kafka producer tuned for durable, ordered raw-price ingestion:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092",
    "acks": "all",                 # wait for all in-sync replicas -> no data loss
    "enable.idempotence": True,    # exactly-once append semantics per producer
    "compression.type": "zstd",    # price JSON compresses well; cuts network ~5x
    "linger.ms": 20,               # small batching window for throughput
})

def publish_price(event: dict):
    # Key by product_id so every observation for a SKU keeps scrape order
    # within one partition — required for a correct price time series.
    producer.produce(
        topic="raw-prices",
        key=event["product_id"].encode(),
        value=__import__("json").dumps(event).encode(),
    )

publish_price({"product_id": "p-44182", "price": "349.99", "source": "amzn"})
producer.flush()  # -> delivers batched records; blocks until acked by replicas

RabbitMQ publisher for routed, per-message scrape jobs:

import pika, json

conn = pika.BlockingConnection(pika.ConnectionParameters("rabbit-1"))
ch = conn.channel()
# Topic exchange routes jobs to worker pools by a dotted routing key.
ch.exchange_declare(exchange="scrape.jobs", exchange_type="topic", durable=True)

def dispatch(job: dict, routing_key: str):
    ch.basic_publish(
        exchange="scrape.jobs",
        routing_key=routing_key,               # e.g. "browser.uk" or "static.us"
        body=json.dumps(job).encode(),
        properties=pika.BasicProperties(
            delivery_mode=2,                   # persist message to disk
            priority=job.get("priority", 0),   # priority-queue support
        ),
    )

dispatch({"url": "https://shop.co.uk/p/1", "priority": 5}, "browser.uk")
conn.close()

Head-to-Head Trade-off Table #

A word on backpressure, because it trips teams up. RabbitMQ buffers in the broker and applies flow control (and per-message TTL) when consumers lag, so an unbounded queue can exhaust broker memory — you cap it with queue length limits and dead-lettering. Kafka never pushes; consumers pull at their own rate, so a slow consumer simply falls behind its offset. The failure mode is different: if the consumer lags past the retention window, unread records are deleted, so retention must exceed your worst-case recovery time.

Recommendation: Kafka for the Event Log, RabbitMQ for Job Dispatch #

For most price-monitoring platforms the honest answer is both, on different edges of the pipeline — but if you must pick one, pick by which property is load-bearing.

• Choose Kafka when the pipeline is fundamentally an event stream you want to store, order, and reprocess. The raw-price firehose is exactly this: you want replay after parser fixes, per-SKU ordering for the price history, and independent fan-out to the normalizer, alerting, and analytics. This is the backbone of the async ingestion pipeline, and Kafka is the right default there.
• Choose RabbitMQ when the problem is dispatching discrete tasks to heterogeneous workers with routing, priorities, and per-message retries — the crawl-job distribution layer. Pairing RabbitMQ for job hand-out with Redis for the shared crawl frontier and dedup is a clean, low-ops distributed-scraper design.
• A common production shape is RabbitMQ in front, Kafka behind: RabbitMQ dispatches scrape jobs to workers; the workers publish extracted raw prices onto a Kafka topic that the rest of the platform consumes and replays. Each broker does the job it is actually good at.

If operational simplicity dominates and your volume is modest (say, under a few million price observations a day) and you never need historical replay, a single RabbitMQ deployment can carry the whole pipeline — but you are trading away replay, and replay is the feature you will wish you had the first time a parser silently corrupts a day of prices.

Verification & Testing #

Prove the two properties that most often bite: Kafka replay and RabbitMQ redelivery.

# Kafka: verify replay by resetting a consumer group to the earliest offset.
from confluent_kafka import Consumer, TopicPartition

c = Consumer({
    "bootstrap.servers": "kafka-1:9092",
    "group.id": "reprocess-test",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
c.assign([TopicPartition("raw-prices", 0, 0)])  # rewind to offset 0
msg = c.poll(5.0)
assert msg is not None and msg.offset() == 0   # re-reads a message already consumed
print("kafka replay confirmed: re-read offset", msg.offset())
c.close()

# RabbitMQ: verify a nacked job is redelivered to another consumer.
def on_message(ch, method, props, body):
    ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)  # reject, requeue
    print("job requeued for redelivery:", body[:40])
# A second consumer on the same queue then receives the requeued message.

Load-test each broker at your real peak scrape rate and, critically, test the failure path: kill a Kafka consumer mid-batch and confirm it resumes from the last committed offset with no gaps; kill a RabbitMQ consumer mid-message and confirm the unacked message is redelivered, not lost.

Edge Cases & Gotchas #

• Kafka retention shorter than recovery time. If a bug goes unnoticed for eight days but retention is seven, the data you wanted to replay is gone. Size retention to your slowest plausible detect-and-fix loop, and consider tiered/long retention for the raw-price topic specifically.
• RabbitMQ unbounded queues. A consumer outage lets a queue grow until the broker hits its memory high-watermark and blocks publishers — stalling the scrapers. Always set x-max-length and a dead-letter exchange so overflow is shed deliberately, not catastrophically.
• Kafka key skew. Keying by product_id is correct for ordering but a handful of hot SKUs can overload one partition. Watch per-partition lag; if one partition dominates, add a low-cardinality salt to the key for the hottest products.
• “Exactly-once” is narrower than it sounds. Kafka’s exactly-once covers the produce-and-commit path within Kafka. The moment your consumer writes to an external price database, you are back to at-least-once unless that write is idempotent — so make the normalizer upsert on (product_id, scraped_at), never blind-insert.

Performance Notes #

Kafka’s throughput comes from sequential disk writes and zero-copy reads through the OS page cache; it scales by adding partitions and consumer instances, and a single modest cluster ingests millions of price events per second long before CPU is the bottleneck — network and retention disk are the real limits. RabbitMQ’s throughput is bounded by per-message bookkeeping (acks, routing, persistence), so it peaks in the tens of thousands of messages per second per node and scales by sharding queues rather than partitions. For the price firehose, favor larger Kafka batches (linger.ms, zstd compression) over tiny low-latency writes. For RabbitMQ job dispatch, keep messages small (publish a URL and job id, not the page body), enable publisher confirms, and bound prefetch so one slow worker cannot hoard the queue.

Frequently Asked Questions #

Can I get message replay with RabbitMQ? Not in the classic queue model — a message is deleted once acknowledged. RabbitMQ Streams (added in 3.9) do add an append-only, offset-based log similar to Kafka, so replay is possible there, but for a full event-log workload with high fan-out most teams still reach for Kafka. If replay of raw prices is a hard requirement, treat it as a Kafka-shaped problem.

Which one is lower latency for a real-time price alert? For a single hop, RabbitMQ typically delivers lower per-message latency because it pushes to consumers, while Kafka consumers poll. But Kafka latency is easily low enough for price alerting once you tune fetch.min.bytes and linger.ms, and its replay and fan-out usually outweigh a few milliseconds. Pick on semantics, not on raw latency.

Do I really need both brokers? No — plenty of pipelines run on one. Use only Kafka if your workload is essentially one big replayable event stream, or only RabbitMQ if it is essentially task dispatch with modest volume and no replay need. Run both when you genuinely have both shapes: RabbitMQ dispatching scrape jobs and Kafka carrying the raw-price log behind it.

Related #

• Async Data Pipelines with Python & Scrapy — the parent guide describing the ingestion pipeline this broker sits inside.
• Redis Session Persistence for Distributed Scrapers — the shared crawl-frontier and dedup layer that complements RabbitMQ job dispatch.
• Statistical Outlier Detection for Price Data — a downstream consumer that depends on the per-SKU ordering Kafka provides.
• Parsing Complex Promotional Discount Structures — the kind of parser whose bugs make Kafka replay worth the operational cost.