Price Anomaly Alerting & Monitoring #

A price-intelligence pipeline is only as trustworthy as the alarms that fire when it breaks. Two distinct failure surfaces need watching: the market (a competitor slashes a price, our own margin breaches a floor) and the pipeline (a feed goes stale, ingestion fails, coverage silently drops). This guide builds the operational alerting layer that sits over both — deciding what to alert on, how to route by severity, how to suppress duplicates before they become alert fatigue, and where a fixed threshold should give way to statistical anomaly detection. It sits under Competitive Price Intelligence & Repricing, consumes verdicts from MAP violation & price-war detection, and feeds the summaries that surface in automated price reporting & dashboards. Its statistical backbone reuses the statistical outlier detection for price data work rather than reinventing it.

Problem Framing & Prerequisites #

The purpose of the alerting layer is to convert a firehose of price observations and pipeline metrics into the smallest set of actionable notifications a human should see. That framing is the whole discipline. Alert on too little and a competitor’s midnight price cut goes unanswered until the morning stand-up; alert on too much and the on-call engineer mutes the channel, which is functionally the same as having no alerting at all. Every rule you add must justify itself against the question: if this fires at 3 a.m., is there an action a human can take?

Two prerequisites make the difference between signal and noise. First, a baseline for every market signal — a competitor price drop is only meaningful relative to that product’s recent price, so you need a rolling reference (trailing median or an exponentially weighted mean) already maintained per product_id and seller. Second, pipeline health metrics must be emitted as first-class data: feed age, per-source record counts, ingestion error rate, and match-coverage ratio. Without those, you can only alert on the market and will be blind to the far more dangerous failure where the pipeline quietly stops updating and every downstream price is stale.

There is a hierarchy of trust here that dictates ordering. Pipeline-health alerts must take precedence over market alerts, because a market signal computed on stale or partial data is not just useless — it is actively misleading. If a coverage gap has silently dropped 40% of a source’s SKUs, the “competitor raised prices” signal you see is an artifact of the missing cheap listings, not a real market move. For that reason the evaluator should tag every market alert with the health state of the data it was computed from, and a downstream consumer (a repricer, a report) should refuse to act on a market alert whose underlying feed is flagged stale or degraded. This is the operational embodiment of “garbage in, garbage out”: the cheapest way to prevent a bad automated price change is to know the input was untrustworthy before you act on it.

The categories worth alerting on, and why each earns its place:

• Competitor price drops — a rapid fall against baseline drives repricing and MAP checks. The runnable detector for this is in building price-drop alerts in Python.
• Our margin breaches — our own price (often set by an automated repricer) has fallen below its floor. This is a money-losing event and always at least a warning.
• Coverage gaps — a source that normally returns 10,000 SKUs returns 4,000. The prices did not change; our visibility collapsed, which silently biases every index and report.
• Stale feeds — a source’s newest record is hours older than its cadence promises. Stale data masquerading as fresh is worse than missing data.
• Ingestion failures — elevated scrape error rates, parser exceptions, or dead-letter growth from the scraping and ingestion workflows.

Algorithm or Architecture Detail #

An alert rule is a small, declarative object: a signal type, a condition, a severity, and a routing channel. Keeping rules as data rather than scattered if statements means they can be versioned, tested, and tuned without redeploying the evaluator. The evaluator walks the rules, tests each against the incoming signal, and emits candidate alerts.

from dataclasses import dataclass, field
from enum import IntEnum
from typing import Callable

class Severity(IntEnum):
    INFO = 1
    WARNING = 2
    CRITICAL = 3

@dataclass
class Rule:
    name: str
    signal: str                    # which signal type this rule inspects
    predicate: Callable[[dict], bool]
    severity: Severity
    channel: str                   # "page", "chat", "digest"
    cooldown_s: int = 3600         # min seconds between repeats of same alert

@dataclass
class Alert:
    rule: str
    severity: Severity
    channel: str
    key: str                       # dedup fingerprint (rule + entity)
    context: dict = field(default_factory=dict)

def evaluate(signal: dict, rules: list[Rule]) -> list[Alert]:
    """Return candidate alerts for one signal event (pre-dedup)."""
    out = []
    for r in rules:
        if r.signal != signal["type"]:
            continue
        if r.predicate(signal):
            key = f"{r.name}:{signal.get('entity_id', '-')}"
            out.append(Alert(rule=r.name, severity=r.severity,
                             channel=r.channel, key=key,
                             context=signal))
    return out

The predicate is where fixed thresholds and statistical detection diverge. A threshold rule (price < margin_floor) is right for hard business constraints that are known in advance — a margin floor is a policy number, not a distribution. A statistical rule is right when “normal” is data-defined and drifts: a competitor’s price is anomalously low relative to its own trailing distribution. For those, reuse a robust z-score on the trailing window rather than a hand-picked constant, exactly as in statistical outlier detection for price data.

import statistics

def robust_z(value: float, history: list[float]) -> float:
    """Median/MAD z-score; resistant to the outlier we're detecting."""
    if len(history) < 8:
        return 0.0                 # too little history to judge
    med = statistics.median(history)
    mad = statistics.median([abs(x - med) for x in history]) or 1e-9
    return 0.6745 * (value - med) / mad   # 0.6745 scales MAD to sigma

# A statistical drop rule: fire when today's price is >3.5 robust-sigma low.
drop_rule = Rule(
    name="competitor_price_anomaly",
    signal="competitor_price",
    predicate=lambda s: robust_z(s["price"], s["history"]) <= -3.5,
    severity=Severity.WARNING, channel="chat", cooldown_s=7200,
)

Candidate Generation & Compute Optimization #

The dedup stage is what keeps alerting usable, and it is pure compute optimization in the sense that its job is to collapse many near-identical events into one notification. Each alert carries a fingerprint key (rule name plus entity). The dispatcher consults a state store — Redis or a small table — for that key’s last-fired time and suppresses a repeat inside the cooldown. It also applies a flap guard: an alert that resolves and re-fires repeatedly within a short window is itself a signal (unstable data or a boundary-hugging price) and should be collapsed into a single “flapping” notification rather than paging on every oscillation.

import time

class AlertDispatcher:
    def __init__(self, state, channels, flap_window_s=1800, flap_max=4):
        self.state = state          # dict-like: key -> {"last": ts, "count": n}
        self.channels = channels    # {"page": fn, "chat": fn, "digest": fn}
        self.flap_window_s = flap_window_s
        self.flap_max = flap_max

    def dispatch(self, alert: Alert, now=None):
        now = now or time.time()
        st = self.state.get(alert.key, {"last": 0, "count": 0, "win": now})
        # Cooldown: suppress a repeat of the same alert too soon.
        rule_cd = alert.context.get("cooldown_s", 3600)
        if now - st["last"] < rule_cd:
            return "suppressed_cooldown"
        # Flap guard: too many fires in the window -> collapse to one notice.
        if now - st["win"] > self.flap_window_s:
            st["win"], st["count"] = now, 0
        st["count"] += 1
        if st["count"] > self.flap_max:
            st["last"] = now
            self.state[alert.key] = st
            self.channels["chat"](self._flap_notice(alert))
            return "flapping"
        st["last"] = now
        self.state[alert.key] = st
        self.channels[alert.channel](alert)     # actually notify
        return "sent"

    def _flap_notice(self, alert):
        return {"text": f"⚠ {alert.rule} flapping on {alert.key}"}

At scale, evaluate rules in the same streaming workers that already consume the price feed so alerting adds no separate crawl. Keep the state store hot (in-memory or Redis) because every candidate alert reads it; batch informational alerts into the digest queue and flush them on a schedule rather than dispatching each one, which is the single biggest lever against fatigue.

Configuration & Threshold Tuning #

Every rule is configuration, versioned alongside the detector. The table below is a starting rule set; tune the numbers against your own false-positive review, not as universal constants.

Two tuning principles keep this honest. First, map severity to human response, not to how bad the number looks — CRITICAL means “wake someone up,” so reserve it for money-losing or blinding failures (margin breach, ingestion down). Second, set cooldowns to the natural cadence of the underlying process: a competitor who reprices hourly does not need an alert every fifteen minutes. Track your alert-to-action ratio per rule; a rule that fires often and is acted on rarely is a fatigue source and should be demoted or deleted.

Failure Modes & Mitigations #

• Alert fatigue from duplicates. The same competitor drop re-detected every crawl cycle floods the channel. The fingerprint-plus-cooldown dedup above is the primary guard; without it, a single event becomes dozens of notifications and the channel gets muted.
• Flapping on boundary conditions. A price hovering at a threshold, or an intermittently failing source, fires and resolves repeatedly. The flap guard collapses these into one “flapping” notice — the instability itself is the thing to investigate, not each oscillation.
• Silent pipeline death. The most dangerous failure is no signal: a source stops updating and everything downstream looks calm because no anomaly fires. Guard it with a heartbeat / dead-man’s-switch rule that alerts on the absence of fresh data past its cadence, so silence itself pages someone.
• Missing baseline on new products. A newly tracked product has no history, so a robust z-score is undefined and a naive rule either never fires or fires on noise. The len(history) < 8 guard returns a neutral score; suppress statistical rules until enough history accrues and rely on hard thresholds in the interim.
• Threshold masquerading as anomaly detection. A fixed percentage cut is not statistical detection and will misfire across categories with different volatility. Use thresholds for policy constraints and z-scores for data-defined normal; do not conflate them.
• Seasonality and known events read as anomalies. A coordinated market-wide sale (Black Friday, Prime Day) trips every drop rule at once, burying the one alert that actually matters under hundreds of expected ones. Maintain a calendar of known pricing events and either widen thresholds or route those windows to the digest, so the on-call channel stays reserved for the unexpected. The alternative — muting the channel for a week — is how a real incident during a sale goes unnoticed.
• Notification-channel outage. The alerting path itself can fail: a webhook endpoint returns 500s, or a chat integration is rate-limited during a mass event. Treat a failed dispatch as its own alertable condition with a fallback channel, and never let a dropped notification pass silently — an alert that was computed but never delivered is indistinguishable, from the operator’s seat, from an alert that never fired.

Compliance & Auditability #

Alerting acts on the same competitive data as the rest of the pipeline, so every notification must be reconstructable. Persist each dispatched alert with its rule name and version, the signal payload that triggered it, the severity and channel, and the dedup decision (sent, suppressed, or flapping). This record answers “why did we / didn’t we get paged?” after an incident and proves that an automated repricing action taken in response to an alert had a documented trigger. Because alerts can drive price changes, keep the rule set version-controlled and change-reviewed: a loosened threshold that suppresses margin-breach pages is a financial control change, not a config tweak. Redact any PII from captured signal context, and set an SLO for the alerting path itself — for example, a critical alert must dispatch within 60 seconds of the triggering observation — then monitor that SLO the same way you monitor the pipeline it watches.

Deployment Checklist #

• Maintain a per-product, per-seller baseline so market signals are measured against a reference, not an absolute.
• Emit pipeline-health metrics (feed age, source counts, error rate, coverage) as first-class signals.
• Keep alert rules as versioned data, not inline conditionals; review threshold changes as financial controls.
• Use fixed thresholds for policy constraints and robust z-scores for data-defined anomalies; never conflate them.
• Deduplicate by fingerprint with a per-rule cooldown so one event yields one notification.
• Add a flap guard that collapses rapidly oscillating alerts into a single notice.
• Add a heartbeat / dead-man’s-switch rule that alerts on the absence of fresh data.
• Map severity to human response; reserve paging for money-losing or blinding failures.
• Batch informational alerts into a digest; page only on critical.
• Persist every dispatch decision and set (and monitor) an SLO for the alerting path itself.

Operational alerting is what turns a price pipeline from a dataset into a system someone can trust to run unattended. By separating market signals from pipeline health, keeping rules as tunable data, and treating deduplication and the dead-man’s-switch as first-class, teams keep the signal high and the on-call channel one people still read.

Related #

• Competitive Price Intelligence & Repricing — the parent guide that frames how alerts drive repricing and reporting decisions.
• MAP Violation & Price-War Detection — a primary source of high-severity verdicts this layer routes and dedups.
• Automated Price Reporting & Dashboards — where alert history and pipeline health roll up into scheduled reports.
• Statistical Outlier Detection for Price Data — the robust-statistics backbone reused by the anomaly predicates here.
• Building Price-Drop Alerts in Python — the runnable recipe for the competitor-drop signal that feeds this layer.