Property Finder Found a Flaw in Its Labels and Cut VLM Costs by 86%

Property Finder, a digital real estate platform, processes tens of thousands of photos a day across the MENA region. Every image is checked against 11 attributes, and each listing averages 15 pictures. The team wanted to move off an expensive proprietary vision model onto a cheaper one on Amazon Bedrock, but ran into an unexpected problem: 96.

8% of the reference labels were not the product of manual review. They were copies of the predictions of the very model the team planned to replace. Working with the AWS Generative AI Innovation Center, the team rebuilt its evaluation, tuned the prompts and switched to Llama 4 Maverick 17B Instruct: Micro F1 of 0.

80 against 0.56 for the previous model, and 86% savings.

Answers written by the model being replaced

Many teams will recognize the pattern. You need a dataset to compare models, you have a production system that nobody complains about, so you take its output as ground truth. The result is a closed loop: the candidate is scored on how closely it matches the incumbent, not on whether it is correct. Any alternative is structurally at a disadvantage before the first request is sent.

Property Finder's data also contained outright errors. In watermark text extraction, letter-level inaccuracies were marked as correct, including invented brand names and missing characters. Photo classification forced a single label even when a shot showed several room types. In the remaining 3.2% of labels, which were made by humans, annotators added categories outside the specification, such as Office.

How the reference set was built and 27 prompt versions were run

Experts manually labeled 356 images from the UAE independently, without access to the old model's outputs. The first 256 became the dev set for tuning, another 100 were added later, and all 356 went into the final comparison. A separate set of 230 images from Egypt served as a cross-regional check and was not included in the report. The labeling was done by an analyst with knowledge of listing quality standards, and disputed cases were worked through in sessions with the AWS team.

The key decision was moving from a single label to multi-label. A photo of a hallway with a bedroom in the doorway really does contain both rooms. The metrics were chosen to fit the task: Micro F1 as the primary one, Macro F1 for rare classes, per-label precision/recall, sample-level IoU, and, for the nine single-label attributes, exact-match accuracy as well.

The authors describe the optimization by analogy with RL: a prompt version plays the role of the agent, Micro F1 serves as the reward, and error analysis replaces the gradient. The idea is close to GEPA. Most versions came out of reviewing misclassified images with the help of AI assistants, and in v27 the team brought in MIPROv2 from DSPy for photo classification.

What worked, step by step:

• v0: a generic prompt classifying against a list of rooms • structured output and attribute definitions • a rule to classify by the room that takes up most of the frame • regional rules: decorative Arabic lettering does not count as a watermark, which eliminated an entire class of false positives • v21: multi-label for the photo and room category, a common starting point for all models; Llama 4 Maverick reached Micro F1 0.75 on the dev set • v24: fixing overestimated room sizes, multi-label, output format • v25: an auto-optimized version, with critical tasks moved to the beginning • v27: MIPROv2 rebuilt photo classification, with Micro F1 of 0.76 on that task • v27_PF: adds Property Finder rules for renders, frames and hallways, plus format normalization after every call

The failures were useful too. Splitting the work into two calls, watermarks separately and everything else separately, gave no noticeable gain, and framed-image detection dropped sharply: without the watermark context, the models mistook overlays for borders. Universal prompts did not work on any model.

Numbers, caveats and cost

On 356 images, Llama 4 Maverick 17B Instruct with v27_PF scored Micro F1 0.80, while proprietary model A with v22 scored 0.56. On the dev set, switching between models, each with its tuned prompts, raised the metric from 0.55 to 0.81, a 47% relative gain. Prompt tuning alone on Llama lifted it from 0.75 to 0.81. The authors candidly warn that the evaluation set includes the dev images, so the figures are optimistic. In addition, Llama got more tuning (v24–v27 plus the customer's rules), while model A got only one round, v22.

Exact-match on single-label attributes across 356 images:

• rotation detection: 100% • people in frame: 97% • framed image: 97% • watermarks: 94% • renders: 93% • furnishing status: 91% • room under construction: 81% • room size: 65% • watermark text extraction: 55%

The model scored 90% or higher on six of the nine attributes and matched or beat model A on all 11, with identical results on photo classification. The weak spots are obvious: room size and watermark OCR.

The cost is estimated at $2.35 per 1000 images (US East, Ohio, December 2025), 86% less than the previous setup. The calculation assumes 9000 input and 200 output tokens per image for a two-call analysis. The actual production prompt in a single call takes 7500–8000 input tokens, so the figure is an upper bound. The comparison uses the old configuration's cost estimated on v21, not v22, so its actual cost may differ. Inference latency, according to Property Finder's own monitoring, fell by 40–60%.

What to check in your own setup

Start by auditing your labels: if they grew out of a model's predictions, you have the same problem. You need a few hundred representative samples with independent expert labeling, and a separate test split that never takes part in tuning. For overlapping attributes, use multi-label metrics and look at Macro F1 alongside Micro. Revise prompts based on analysis of specific misclassified images, and keep one candidate prompt per model family. To expand the dataset, the authors suggest a hybrid approach: models from different families propose labels and flag disagreements, and humans give the verdict on each case.