Price Monitoring Competitor Intelligence Workflows

Z-Score and IQR for Price Outliers

A scraped price feed is full of numbers that are technically valid and economically nonsense: a decimal shifted two places, a placeholder $1.00, a 90%-off “deal” that is really a data error. Left in, they poison your competitive averages and can trigger a self-inflicted price war. This guide is a focused recipe for detecting those outliers with three classical tools — the z-score, the modified z-score built on the median absolute deviation, and the interquartile-range fence — and, just as importantly, for knowing when each one is the wrong choice. It sits under the parent guide on statistical outlier detection for price data and pairs with filtering fake sale prices using historical averages, which handles the temporal case this recipe does not.

Prerequisites & Input Contract

The detector operates on a single, already-normalized numeric column: prices for one comparable product group, in one currency, with units standardized. Cross-marketplace and cross-currency normalization happen upstream in the data normalization and promo parsing pipelines; mixing a 1L and a 2L SKU in the same group will manufacture “outliers” that are really an apples-to-oranges error.

import pandas as pd
from decimal import Decimal

# One row per competitor offer for a single product group, base currency.
offers = pd.DataFrame({
    "source": ["a", "b", "c", "d", "e", "f", "g", "h"],
    "category": ["headphones"] * 8,
    "price": [199.0, 205.0, 189.0, 210.0, 195.0, 2.0, 201.0, 1990.0],
})
# Prices 2.0 (placeholder) and 1990.0 (decimal shift) are the target outliers.

Library versions: Python 3.11+, numpy>=1.26, pandas>=2.1. The three methods share one interface — given a price array they return a boolean mask of outliers — so you can swap or ensemble them without touching the pipeline around them.

Step-by-Step Implementation

The relationship between the three fences is easiest to see on the distribution itself. The z-score fence is symmetric around the mean and gets dragged outward by the very outliers it should catch; the IQR and MAD fences are anchored on quantiles and the median, so a lone extreme value barely moves them.

A right-skewed price distribution with IQR and z-score fencesA histogram of prices rises to a peak near the median around 200 and tails off to the right. A box marks the interquartile range from the first to the third quartile, with the median line inside it. The IQR upper fence sits at the third quartile plus 1.5 times the interquartile range and cleanly separates the bulk of the data from a lone bar far to the right representing a decimal-shift error near 1990. The z-score mean plus three sigma fence is drawn further right than the IQR fence because the outlier itself inflated the mean and standard deviation, illustrating why the mean-based fence is less robust to skew.priceerrorIQR (Q1–Q3), medianIQR fenceQ3 + 1.5·IQRz fencemean + 3σ

Step 1 — The classic z-score

The z-score measures how many standard deviations a value sits from the mean. It is the right tool only when the group is roughly symmetric and reasonably large, because both the mean and the standard deviation are themselves distorted by extreme values.

import numpy as np

def zscore_outliers(prices, threshold=3.0):
    """Boolean mask where |z| exceeds threshold. Assumes near-normal data."""
    x = np.asarray(prices, dtype=float)
    mu, sigma = x.mean(), x.std(ddof=0)
    if sigma == 0:                       # all identical -> nothing is an outlier
        return np.zeros_like(x, dtype=bool)
    z = (x - mu) / sigma
    return np.abs(z) > threshold

mask = zscore_outliers(offers["price"])
print(offers.loc[mask, ["source", "price"]].to_dict("records"))
# -> [{'source': 'h', 'price': 1990.0}]   (misses the 2.0 placeholder!)

Note the failure already visible here: the single huge value inflates sigma so much that the 2.0 placeholder no longer looks extreme. That is the classic weakness of the plain z-score on skewed price data.

Step 2 — The modified z-score (MAD), robust to skew

The modified z-score replaces the mean with the median and the standard deviation with the median absolute deviation (MAD), both of which shrug off a handful of extreme values. The 0.6745 constant rescales MAD so the threshold is comparable to a normal-distribution z.

def modified_zscore_outliers(prices, threshold=3.5):
    """Robust MAD-based mask. Recommended default for skewed price groups."""
    x = np.asarray(prices, dtype=float)
    med = np.median(x)
    mad = np.median(np.abs(x - med))
    if mad == 0:                         # fall back to mean abs dev if MAD collapses
        mad = np.mean(np.abs(x - med)) or 1e-9
    modified_z = 0.6745 * (x - med) / mad
    return np.abs(modified_z) > threshold

mask = modified_zscore_outliers(offers["price"])
print(sorted(offers.loc[mask, "price"].tolist()))
# -> [2.0, 1990.0]   (catches BOTH tails)

Step 3 — The IQR fence

The IQR method builds fences from the quartiles: anything below Q1 - k·IQR or above Q3 + k·IQR is flagged. With k=1.5 it is the standard Tukey fence; k=3.0 flags only “far out” points. Like MAD it is quantile-based and therefore robust.

def iqr_outliers(prices, k=1.5):
    """Tukey fence mask. k=1.5 is standard; raise k to be more permissive."""
    x = np.asarray(prices, dtype=float)
    q1, q3 = np.percentile(x, [25, 75])
    iqr = q3 - q1
    lower, upper = q1 - k * iqr, q3 + k * iqr
    return (x < lower) | (x > upper), (lower, upper)

mask, fences = iqr_outliers(offers["price"])
print(f"fences={tuple(round(f, 1) for f in fences)}")
print(sorted(offers.loc[mask, "price"].tolist()))
# -> fences=(170.5, 231.5)
# -> [2.0, 1990.0]

Step 4 — Per-category thresholds and flag-vs-drop

A single global threshold is wrong when categories have genuinely different spreads: a $5 spread is an outlier for USB cables and rounding error for laptops. Fit thresholds per category, and — critically — flag, do not drop. A dropped row is a silently lost competitor observation; a flagged row is auditable and reversible.

def flag_outliers(df, method="mad", by="category", price_col="price", **kw):
    """Return df with an is_outlier column, thresholds fit within each group."""
    detectors = {
        "z": zscore_outliers,
        "mad": modified_zscore_outliers,
        "iqr": lambda p, **k: iqr_outliers(p, **k)[0],
    }
    fn = detectors[method]
    out = df.copy()
    out["is_outlier"] = False
    for _, idx in out.groupby(by).groups.items():
        grp = out.loc[idx, price_col]
        if len(grp) < 5:                 # too few to fit a threshold; skip, don't guess
            continue
        out.loc[idx, "is_outlier"] = fn(grp.values, **kw)
    return out

flagged = flag_outliers(offers, method="mad")
print(flagged[["source", "price", "is_outlier"]].to_string(index=False))
# rows with price 2.0 and 1990.0 show is_outlier True; the rest False

Verification & Testing

Pin the behaviour that separates a robust method from a fragile one: MAD and IQR must catch both tails of a skewed group, while the plain z-score is allowed to miss the low tail. Also assert the safe degenerate cases.

def test_detectors():
    prices = np.array([199, 205, 189, 210, 195, 2, 201, 1990.0])

    # Robust methods catch both the placeholder and the decimal shift
    for fn in (modified_zscore_outliers, lambda p: iqr_outliers(p)[0]):
        flagged = set(prices[fn(prices)])
        assert {2.0, 1990.0} <= flagged, f"{fn} missed a tail"

    # Degenerate: all-identical prices -> no outliers, no divide-by-zero
    same = np.full(6, 50.0)
    assert not modified_zscore_outliers(same).any()
    assert not zscore_outliers(same).any()
    print("detectors verified")

Backtest the chosen method the same way you would any classifier: sample a few hundred flagged rows, have an analyst label true error versus legitimate price, and track precision and recall per category. If recall is low, tighten the threshold; if analysts keep overturning flags, loosen it or switch methods.

Edge Cases & Gotchas

  • Small samples. With fewer than a handful of observations, every method is unreliable — quartiles are ill-defined and one value can be 50% of the data. The flag_outliers guard skips groups under five rows rather than fabricating a threshold; route those to review or wait for more competitor coverage instead.
  • Multimodal and seasonal prices. A group that mixes a discounted and a full-price cluster is bimodal, and a global fence will flag the whole cheaper mode. Detect multimodality (a large gap between adjacent sorted prices, or a low silhouette on a two-cluster fit) and split the group before fencing, or defer to the time-aware method in filtering fake sale prices using historical averages.
  • A legitimate sale versus an error. A real clearance price and a decimal-shift bug can produce the same z-score. Static detection cannot tell them apart from one snapshot; corroborate a low-tail flag against the item’s own price history and promo flags before acting, and never let a single robust-fence hit trigger an automated reprice on its own.
  • Choosing the wrong method. Reach for the plain z-score only on symmetric, large groups; default to the modified z-score or IQR for the skewed, small groups that dominate real price data. Using the mean-based z-score on skewed data is the single most common reason a detector misses obvious errors, exactly as Step 1 demonstrates.

Performance Notes

All three detectors are O(n) up to the percentile and median computations, which are O(n log n) from the underlying sort — trivial for the tens-to-hundreds of offers in a typical product group. The realistic cost is the per-category groupby, and it vectorizes well: flag_outliers over a few million rows across tens of thousands of categories runs in seconds because each group’s arithmetic is pure NumPy. If you need streaming detection, keep a rolling median and MAD per category rather than recomputing over the full window, since the median is the expensive statistic to maintain incrementally. Store the fitted fences alongside the flags so a downstream reviewer can see why a row was flagged without recomputing anything.

Frequently Asked Questions

When should I use z-score versus IQR or the modified z-score? Use the plain z-score only for roughly symmetric, reasonably large price groups. For the skewed, small groups that dominate real price data, use the modified z-score (MAD) or the IQR fence, because both are anchored on the median and quartiles and are not dragged outward by the very outliers you are trying to catch.

Should I drop outlier prices or just flag them? Flag them. A dropped row is a silently lost competitor observation that biases your price index and cannot be audited later. A flagged row keeps the data, records the reason, and lets a downstream stage decide whether to exclude it from averages or send it to human review.

How do I tell a real sale from a data error? A single static snapshot cannot reliably distinguish them, because a legitimate clearance and a decimal-shift bug can share the same score. Corroborate a low-tail flag against the item’s own price history and promotion flags before acting, and never trigger an automated reprice from one outlier hit alone.