Amazon scores product page content with a VLM, OCR and CatBoost stack at 94% accuracy

A familiar pain for any marketplace: every element of a page is fine on its own, but together they make a mess. The image is sharp, the bullets are error-free, the photo is beautiful. But the text claims a 64-ounce pitcher while the photo shows a single-serve blender. Or a kitchen photo sits above the product itself. Element-by-element checks don't catch this. The Amazon Marketplace team described how it solves this for A+ Content, and the approach is applicable well beyond Amazon.

The page as a single snapshot

The idea is simple: render the whole page into one image and hand it to a vision-language model together with a rubric. The rubric covers four dimensions: Visual Storytelling, Feature Presentation, A+ Layout and Conversion Optimization. For each, the model returns a score from 0 to 10, a confidence level and a written rationale.

They chose not to fine-tune the VLM itself. There are only 5,478 labeled pages across 10 categories, and with a sample that size, fine-tuning a large model risks overfitting and requires GPUs. Instead, the VLM with its rubric stays unchanged, its scores become features, and CatBoost is trained on top using expert labels. The labeling was done by an ecommerce marketing expert and four annotators, with the final label decided by majority vote. Each annotator's agreement with the consensus, measured by Cohen's kappa, was 0.845–0.915.

Why OCR is needed

A VLM grasps meaning and composition well, but is worse at sensing text density and spacing. It also tends to praise long, detailed descriptions, even though shoppers need to skim quickly. So OCR on SageMaker computes four signals: text density, character density, spacing and overall legibility.

Results on a holdout set of roughly 1,100 pages:

• baseline (average of the four VLM scores plus a threshold): 80% accuracy, F1 0.84;
• OCR only: 82%;
• VLM plus CatBoost: 89%;
• VLM plus OCR plus CatBoost: 94%, F1 0.95.

So OCR does not replace the VLM but complements it. The gain over the VLM-only variant was 5 percentage points.

SHAP instead of a bare score

A "low quality" label tells a seller nothing. That is why SHAP is used: it shows each of the eight features' contribution to the decision on a common scale. On average, the strongest influences were character density (1.1), overall legibility (1.0) and Visual Storytelling (1.0). For each poor page, the most negative features, the product category, best practices and the VLM's rationales are collected, and an LLM turns this into a prioritized list of fixes. If layout has a SHAP of -1.2 and visual storytelling -0.3, the advice is to rework the layout first, then turn to the images.

The effect was tested in practice: after edits following these recommendations, the average score rose by 1.4 points out of 10.

Limitations and takeaways

This is an AWS blog, so keep in mind a promotional bias. The metrics were computed on internal data, and the article text cuts off at an experiment with a lighter VLM, so its results are not covered here. The binary "high/low" label is also somewhat coarse.

Practical takeaways for teams that evaluate composite content (landing pages, product cards, presentations, emails):

• evaluate the result as a whole, not in parts;
• don't fine-tune a large model until you've tried a small classifier on top of its structured outputs;
• add classic signals such as OCR where the VLM is blind;
• build in explainability from the start, because that is what turns a score into quality gains.