KU-PCPE Benchmark: Models Find Composition Anchors at 0.664 Dice, Expert at 0.867

A composition classifier can say that a photo is "diagonal" or follows the "rule of thirds," but it cannot explain which parts of the frame are responsible. A team from the computer vision lab at TU Wien and UAS St. Pölten proposes to close that gap with a benchmark called KU-PCP Elements (KU-PCPE). The task works like this: the input is a photograph and a composition label already confirmed as fitting it. The output is a binary mask of the visible regions that explain why the label applies. The best working model reaches a Dice of 0.664 (a measure of overlap between the predicted and reference masks), while a held-out human expert scores 0.867 when compared against two colleagues.

What Was Annotated and How

KU-PCPE builds on the KU-PCP dataset. That dataset contains whole-image composition labels and per-class geometric elements: bounding boxes of main objects, leading lines, symmetry axes, triangles and sky regions. The new work adds a single human-annotated format covering all nine classes: horizontal, vertical, diagonal, curved, triangle, rule of thirds, center, symmetry and pattern. A mask can consist of several disconnected parts and can include objects, parts of objects, background and empty space. If an image has several labels, each label becomes a separate task.

The training split has 4093 masks on 3169 images, all annotated by a single person. The test split has 1327 tasks on 1073 images, with three masks per task. The test annotators are three experts with art or design training and more than five years of experience. They worked independently in an interface where SAM2 Hiera-L had pre-proposed candidate regions. Annotators selected the relevant regions, split or merged them and, when needed, drew boundaries by hand. They also judged which explains the label better: a region or an automatically detected semantic line.

Where Experts Disagree

Mean pairwise Dice between experts is 0.7428. For the held-out expert measured against the agreed consensus of the other two, the results were Dice 0.8666, IoU 0.8054, boundary F1 0.7541 and [email protected] of 0.9312. The last metric is the share of tasks where Dice is at least 0.5. Experts agree most on rule of thirds, center and triangle. Horizontal, diagonal and curved compositions, along with pattern, allow several reasonable readings. For example, with a single horizon, either the sky or the ground can count as the "anchor," so the authors build two references: a strict one and one that accounts for this ambiguity. The ranking of methods does not change on either reference.

The authors call their estimates of explanation form exploratory. Pairwise agreement across three groups (segment, mixed, line) is only 0.54. Horizontal composition is more often explained with a mix, diagonal and triangle with lines, and the remaining classes more often with regions. For 171 triangle tasks that have a detector result from KU-PCP, the geometry does overlap with the expert mask. But people also mark the objects, surfaces and background through which the triangle is perceived.

What the Baseline Models Showed

The comparison included random masks of the same area, class templates, a U-Net without and with the label, and a SegFormer-B0 conditioned on the label. All were trained on the single annotator's labels. Separately, the authors took an oracle: it knows the reference and greedily assembles candidate masks from SAM2.1 Hiera-L. Its Dice is 0.672, only slightly above the 0.664 of the best working model. This suggests the limit lies partly in the proposal set itself.

For VLM selectors, the models are shown proposals marked with numbers (up to 12), and the model returns up to three numbers. The union of those regions is taken as the answer. The set includes Qwen3-VL, InternVL3.5, MiniCPM-V 4.5 and Molmo2; the full list is in the appendix. The best of them reached a Dice of 0.395, noticeably behind the trained segmenters. The authors also tested ResNet-50, Grad-CAM and DINO ViT similarity maps to link the masks to class recognition.

Limitations

The training annotations come from one person, so they reflect that person's idea of composition. Agreement on the "segment or line" judgments is weak. VLMs choose from ready-made proposals rather than drawing a mask themselves, so their results are capped by the quality of the candidates. Full tables, confidence intervals and prompts are in the appendix, and the data, evaluation code and reconstruction tools are published on the project page. For anyone building photo assistants or aesthetic scoring, the takeaway is simple: current models are still noticeably worse than humans at explaining their labels through specific regions of the frame.