TF-IDF vs Transformer Embeddings for SKU Matching #

When a competitor lists “Sony WH1000XM5 Wireless NC Headphone - Blk” and your catalog holds “Sony WH-1000XM5 Noise Cancelling Headphones (Black)”, something has to decide they are the same product. Two families of technique dominate that decision: TF-IDF, a sparse lexical representation that scores overlapping tokens, and transformer sentence embeddings, a dense representation that scores semantic similarity. TF-IDF is fast, cheap, and interpretable but blind to synonyms; embeddings understand that “NC” means “noise cancelling” but cost more to compute and serve. This guide compares them for matching product titles and SKUs across catalogs and lands on a hybrid design. It sits under the parent guide on fuzzy matching algorithms for SKU alignment, and builds on implementing Levenshtein distance for product matching, which handles the character-level typo tier this comparison sits above.

Two Representations of a Product Title #

TF-IDF turns a title into a sparse vector over the vocabulary: each token gets a weight that rises with its frequency in the title and falls with its frequency across the whole catalog, so distinctive tokens like a model number dominate and filler like “the” is discounted. Similarity is the cosine of two such vectors — high when the titles literally share rare tokens. It is a bag of words: it has no notion that “headphone” and “headphones” are related, let alone “NC” and “noise cancelling”, unless the exact strings overlap.

A transformer embedding (from a sentence-transformer model) maps the whole title to a dense vector of a few hundred dimensions in which semantic neighbours sit close together. The model has learned from billions of sentences that “wireless” and “cordless” are near-synonyms and that “blk” abbreviates “black”, so two titles describing the same product land near each other even with zero shared tokens. The price is compute: every title must pass through a neural network, and matching at scale needs an approximate-nearest-neighbour (ANN) index rather than a sparse dot product.

Where TF-IDF Wins #

Cold-start and cost. TF-IDF needs no GPU, no model download, and no training beyond fitting the vectorizer on your catalog — a few seconds on hundreds of thousands of titles. It runs on a laptop, adds no serving infrastructure, and vectorizes new titles in microseconds. For a new price-monitoring build, it gets a matcher into production the same day.

Model numbers and identifiers. Product titles are dense with rare, discriminative tokens — WH-1000XM5, A2591, MQD83. TF-IDF weights these heavily precisely because they are rare across the catalog, so a shared model number produces a strong match. Embedding models, trained on natural language, can actually underweight an alphanumeric SKU that looks like noise to them. On identifier-heavy catalogs TF-IDF is not just cheaper — it is often more accurate.

Interpretability. A TF-IDF match is explainable: you can list the shared high-weight tokens that produced the score, which matters when an analyst reviews a borderline match or an auditor asks why two SKUs were aligned. An embedding cosine of 0.83 explains nothing on its own.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

catalog = [
    "Sony WH-1000XM5 Noise Cancelling Headphones Black",
    "Bose QuietComfort Ultra Headphones",
]
# char_wb n-grams tolerate hyphen/spacing noise in model numbers.
vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4)).fit(catalog)
cat_mat = vec.transform(catalog)

query = "Sony WH1000XM5 Wireless NC Headphone Blk"
sim = cosine_similarity(vec.transform([query]), cat_mat)[0]
print(sim.round(2))          # -> [0.62 0.13]  strong hit on the Sony row
print("match:", catalog[sim.argmax()])  # -> Sony WH-1000XM5 ...

Note the char_wb n-gram trick: character n-grams give TF-IDF partial credit for WH1000XM5 vs WH-1000XM5, recovering some of the robustness you would otherwise get from Levenshtein distance.

Where Transformer Embeddings Win #

Synonyms, abbreviations, and paraphrase. This is the whole reason to pay for embeddings. “Cordless drill” vs “wireless drill”, “5-pack” vs “pack of 5”, “NC” vs “noise cancelling”, localized titles in different phrasings — these share few tokens and defeat TF-IDF, but sit close in embedding space because the model learned they mean the same thing.

Cross-lingual and marketplace-style variation. Multilingual sentence-transformer models embed a German and an English title of the same product near each other, enabling cross-border matching that lexical methods cannot touch without a translation step.

Recall on messy, human-written titles. Marketplace sellers write wildly inconsistent titles stuffed with keywords. Embeddings capture the underlying product identity through that noise better than token overlap does, lifting recall on exactly the messy, low-overlap titles where TF-IDF quietly misses.

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")  # 384-dim, CPU-friendly
catalog = [
    "Sony WH-1000XM5 Noise Cancelling Headphones Black",
    "Bose QuietComfort Ultra Headphones",
]
cat_emb = model.encode(catalog, normalize_embeddings=True)

query = "Sony cordless over-ear headset with active noise reduction, black"
q_emb = model.encode(query, normalize_embeddings=True)
scores = util.cos_sim(q_emb, cat_emb)[0]
print(scores.round(2))                 # -> tensor([0.71, 0.44])
print("match:", catalog[int(scores.argmax())])  # -> Sony WH-1000XM5 ...

That query shares almost no tokens with the catalog title — TF-IDF scores it near zero — yet the embedding matches it correctly on meaning. For a large catalog, index cat_emb in a FAISS ANN index and query it in sub-millisecond time instead of the brute-force cos_sim above.

Head-to-Head Trade-off Table #

Recommendation: TF-IDF Recall, Embedding Rerank #

Do not treat this as either/or. The production-grade design is a two-stage retrieve-and-rerank cascade that uses each method for what it is best at:

1. Candidate generation with TF-IDF. For each incoming title, retrieve the top ~50 lexical candidates with a cheap TF-IDF query, blocked by brand or category so the comparison set stays small. This stage is fast, catches all the easy identifier and exact-token matches outright, and shrinks the problem from “the whole catalog” to a handful.
2. Rerank with embeddings. Embed the query and the ~50 candidates and rerank by cosine similarity. The expensive model runs on 50 titles, not the whole catalog, so you get its synonym-and-paraphrase understanding at a fraction of the cost of embedding-first retrieval.
3. Accept, review, or reject by threshold. Auto-accept above a high combined score, route the middle band to human review, reject below a floor — the same confidence-banding used throughout the fuzzy matching stage before matches drive any repricing.

Pick a single method only at the extremes. TF-IDF alone is the right call when titles are identifier-dominated, the catalog is monolingual, latency and cost budgets are tight, or you need explainable matches for audit — and it is always the correct first thing to ship. Embedding-first with a FAISS index earns its infrastructure when titles are natural-language and messy, cross-lingual matching is required, or TF-IDF recall has plateaued below your target and manual review volume is the bottleneck. For everything in between — which is most real catalogs — the hybrid cascade dominates both.

def match_hybrid(query, catalog, vec, cat_mat, model, cat_emb, top_k=50):
    """TF-IDF retrieves top_k candidates; embeddings rerank them."""
    # Stage 1: cheap lexical recall over the whole catalog.
    lex = cosine_similarity(vec.transform([query]), cat_mat)[0]
    cand_idx = lex.argsort()[::-1][:top_k]          # top_k lexical candidates
    # Stage 2: expensive semantic rerank on the shortlist only.
    q_emb = model.encode(query, normalize_embeddings=True)
    sem = util.cos_sim(q_emb, cat_emb[cand_idx])[0]
    best = cand_idx[int(sem.argmax())]
    return catalog[best], float(sem.max())          # (title, rerank_score)

Verification & Testing #

Measure both methods on a labeled set of known matches and non-matches, and report precision and recall separately — the whole point of the hybrid is that it should beat either method on both.

def evaluate(pairs, predict):
    """pairs: list of (query, gold_title, is_match). Returns precision/recall."""
    tp = fp = fn = 0
    for query, gold, is_match in pairs:
        pred, score = predict(query)
        hit = score >= 0.60                 # tune per method/threshold
        if hit and is_match and pred == gold: tp += 1
        elif hit and not is_match:          fp += 1
        elif not hit and is_match:          fn += 1
    prec = tp / (tp + fp) if tp + fp else 0.0
    rec = tp / (tp + fn) if tp + fn else 0.0
    print(f"precision={prec:.2f} recall={rec:.2f}")
    return prec, rec

Build the gold set from the categories you actually match — a model tuned on electronics titles behaves differently on apparel — and re-evaluate whenever you change the embedding model or the blocking key. Track precision and recall separately over time so a synonym-heavy new competitor feed does not silently tank recall while precision looks fine.

Edge Cases & Gotchas #

• Embeddings ignore critical size/color/pack distinctions. A dense model happily rates “iPhone 15 128GB” and “iPhone 15 256GB” as near-identical because they are semantically similar — but they are different SKUs at different prices. Extract and hard-match variant attributes (capacity, size, color, pack count) outside the embedding, then use similarity only for the residual product identity.
• TF-IDF vocabulary drift. A vectorizer fitted once goes stale as new brands and tokens appear; unseen tokens are simply dropped, silently weakening matches. Refit on a schedule or use char_wb n-grams, which degrade more gracefully on novel tokens.
• Cosine threshold is not transferable. A 0.60 cutoff that works for all-MiniLM-L6-v2 is meaningless for a different model or for TF-IDF. Recalibrate the threshold on labeled data every time you swap either representation.
• ANN recall is approximate by design. FAISS/HNSW trade a little recall for speed, so the “nearest” neighbour it returns is occasionally not the true nearest. For high-stakes matches, over-retrieve (larger top_k) and let the exact rerank correct for the ANN’s approximation.

Performance Notes #

TF-IDF is effectively free at query time: a sparse dot product against a compact matrix runs in microseconds and the whole index fits in a fraction of the memory a dense one needs. Embeddings invert that: encoding is the cost center (batch it, and prefer a small model like MiniLM on CPU before reaching for a GPU), and a 384-dimensional dense index consumes roughly 1.5 KB per title, so a million titles is about 1.5 GB in FAISS. The hybrid cascade is what makes embeddings affordable at scale — because the expensive encoder only ever runs on the ~50 candidates TF-IDF already shortlisted, per-query embedding cost is bounded regardless of catalog size, and total throughput stays dominated by the cheap lexical stage. Cache embeddings for the stable master catalog (it changes slowly) and only encode the incoming competitor titles live.

Frequently Asked Questions #

Is TF-IDF obsolete now that embeddings exist? No. On identifier-heavy product titles, TF-IDF is frequently more accurate than embeddings because it weights rare model numbers heavily while a language model can treat them as noise. It is also far cheaper, needs no serving infrastructure, and is interpretable. Most production matchers keep TF-IDF as the candidate-generation stage even after adding embeddings.

Which embedding model should I use for product titles? Start with a small, fast general model such as all-MiniLM-L6-v2 for the rerank stage; it runs on CPU and is strong enough for most catalogs. Move to a larger or e-commerce-fine-tuned model only when evaluation shows the small one is the bottleneck, and use a multilingual model if you match titles across languages. Always re-tune the similarity threshold after any model change.

Do I need a vector database like FAISS? Only if you do embedding-first retrieval over a large catalog. In the recommended hybrid, TF-IDF narrows each query to about fifty candidates, so you rerank with a brute-force cosine over that shortlist and need no ANN index at all until the catalog or query volume grows large enough that even candidate generation must be vectorized.

Related #

• Fuzzy Matching Algorithms for SKU Alignment — the parent guide that frames where lexical and semantic matching fit in the overall matching cascade.
• Implementing Levenshtein Distance for Product Matching — the character-level typo tier that sits beneath both methods compared here.
• Building a Unified Product Catalog Schema — the canonical catalog these methods match incoming titles against.
• Cross-Platform Category Taxonomy Mapping — the category blocking that keeps both methods’ candidate sets small and precise.