AI Visual Search Returns: Accuracy, Bias, and the Wrong Product Problem
A shopper photographs a chair and gets table lamps. Visual search fails in patterns worth measuring: colour drift, category confusion, and results that ignore stock and price.
Visual search demos always start with a clean photograph of a distinctive product on a white background. Shoppers do not take those photographs. They take a picture of a chair in a dim restaurant, a shoe on a pavement, a handbag half hidden under a coat, and then judge the feature entirely on what comes back from those messy, real attempts.
When it works, it is genuinely useful, because it accepts the query the shopper actually has, a thing they saw, rather than forcing them to guess the words a catalogue uses for it. When it fails, it fails confidently, returning a grid of products that share a colour or a silhouette and have nothing else in common with the photograph. Measuring that gap is the job most launches skip.
Score the returns like a search audit, not a demo
Build a small, fixed test set and keep it. Fifty to a hundred real photographs, taken in real conditions, covering your main categories, a range of colours and materials, and a few items you do not sell at all. Run them after every model or index change and grade the first five results for each: same product, same category and a plausible substitute, same colour but wrong category, or unrelated.
The items you do not sell are the most revealing part of the set. A good visual search says, in effect, nothing close, and returns near neighbours with some honesty about the distance. A weak one returns five confident, wrong products, because returning something is what it was built to do. Shoppers learn quickly whether the feature can be trusted, and they learn it from the failures.
Colour is doing too much of the work
The most common failure pattern is colour dominance. The photograph is analysed, the dominant colour and rough shape win, and the returns are everything in that colour across unrelated categories. A navy chair returns navy cushions, navy throws, and a navy lamp. That is a matching exercise, not a search, and it shows up most on furniture, fashion, and homeware, where colour ranges repeat across the whole catalogue.
Check material and pattern handling separately, because they fail differently. Stripes, checks, and textured fabrics are often flattened to their average colour, so a striped shirt search returns plain shirts in the stripe's main colour. If your catalogue leans on pattern as a selling point, that specific failure is worth a test subset of its own rather than a line in a general accuracy figure.
Stock, price, and the substitute problem
A visual match for a sold out product, offered with no indication that it is unavailable, wastes the shopper's strongest signal of intent. They showed you the exact thing they want. Returning it at the top, unbuyable, followed by in stock alternatives, is defensible if it is labelled. Returning it with no alternatives, or burying the in stock substitutes below unrelated in stock items, is a ranking choice that should be made on purpose and tested.
Price drift is the quieter issue. A photograph of a premium product tends to return premium priced results even when a closer visual match exists at a lower tier, and the reverse happens for budget items. Whether that is a bug or a merchandising choice depends on the store, but it should be a choice someone made and can explain, not an artefact nobody noticed.
Related reading: Glossary, About, When AI Search Returns Nothing: Synonyms, Slang, and Zero Results.
Frequently asked questions
How should a store test visual search accuracy?
With a fixed set of real world photographs, including poor lighting and partial views, graded consistently across the top results as exact match, plausible substitute, colour only match, or unrelated. Reusing the same set after each change makes the scores comparable.
Why include products the store does not sell in the test set?
To see how the feature behaves when there is no right answer. Confidently returning unrelated products teaches shoppers not to trust the feature. A good result set signals that the matches are near neighbours, not the photographed item.
What is the most common visual search failure?
Colour dominance, where the dominant colour and rough silhouette drive the results across unrelated categories. A navy chair returning navy cushions and lamps is matching, not searching.
Should sold out visual matches appear in results?
They can, if they are clearly marked unavailable and accompanied by in stock alternatives. Showing an unbuyable exact match with no substitute wastes a strong expression of what the shopper wants.