AI Review Summaries That Contradict the Reviews Underneath Them
Summarised review widgets invent consensus: 'runs large' over reviews saying true to size. Why summarisation drifts, and the sampling audit that catches it before shoppers do.
The widget at the top of the reviews says customers find the boots run large and recommend sizing down. The first six reviews under it say true to size. A shopper who reads both now trusts neither, and she is right not to: the summary was generated once, from a sample she cannot see, and nothing in the store's process has compared it against the review corpus since.
AI review summaries fail differently from human ones. A human summariser drifts toward the recent and the loud. A model drifts toward the average phrasing of product reviews in general, smoothing a split opinion into a consensus that never existed. This post covers where the contradictions come from and a sampling audit small enough to run monthly.
Split opinions are the first casualty
Sizing is the canonical case. On a product where a third of reviewers say runs small, a third say true to size, and a third say runs large, the truthful summary is that fit varies, ideally with a clue about why, foot width, intended sock, break-in. Models dislike that answer. Given a pile of conflicting text, summarisation tends to pick the most common phrasing pattern or blend it into a single confident direction, and the widget then states as consensus the one thing the data does not contain.
The same blending hits durability and quality splits: a product loved by light users and criticised by heavy users gets 'well made' as its headline, and the heavy users, often the highest value buyers, read their own experience edited out of the record.
Staleness is the second
A summary generated at launch reflects the first fifty reviews, often seeded by a launch promotion to existing fans. Two hundred reviews later, including the production batch with the changed lining, the summary still speaks for the first fifty. Unless the tool regenerates on a schedule and someone checks, the widget becomes a historical document presented as a live one.
Version changes are the sharp edge: a reformulated, resized, or re-sourced product inherits reviews and summary from its previous self. Shoppers are then reading testimony about a product that no longer exists, summarised.
The monthly sampling audit
Pick the ten highest traffic products with summaries. For each, read the twenty most recent reviews and the twenty most helpful, and score the summary on three questions: does its main claim match the majority view, does it mention the biggest split or complaint, and is it describing the current version. Two failures out of ten means the widget settings, regeneration cadence, source window, get changed, not just the offending summary edited, because the process produced it and will produce the next one.
Related reading: AI Commerce Watch, Answers, Glossary.
Frequently asked questions
Why do AI review summaries sound confident when reviews disagree?
Summarisation models compress text toward common phrasing, and a split opinion has no single common phrasing. The output resolves the conflict in one direction instead of reporting the split. Products with polarised feedback, especially on fit, are where confident wrong summaries appear first.
How often should review summaries regenerate?
On a cadence tied to review volume, not the calendar alone: regenerate when a meaningful block of new reviews has arrived, and always after a product change. A summary frozen at the launch sample misrepresents the product more with every month of new reviews.
Should summaries mention negative points at all?
Yes, the main recurring one. A summary that names the common complaint, stiff first week, narrow toe box, is more trusted and more useful than uniform praise, and it reduces returns by setting the expectation before purchase instead of after.
Can we just turn summaries off?
You can, and on products with split or sparse reviews you probably should. Summaries help most on high volume products with consistent feedback. Everywhere else, the honest review list, sorted well, does the job the widget is faking.