A 2015 CNN Beat GPT-6 Astra and Claude Fable 5.1 at Skin Cancer Diagnosis

A dermatologist ran 200 skin lesions with histologically confirmed diagnoses through GPT-6 Astra and Claude Fable 5.1, then compared the results with a fine-tuned Inception v3 convolutional network, which Google published in 2015. The frontier LLMs picked the correct diagnosis out of six options in 84 and 85 of 200 cases, or 42%. The old CNN answered correctly in 125 cases (62%). Fine-tuning took about twenty minutes on a single H100 GPU.

How the 200-lesion test was built

The data comes from MILK10k, an open dataset that the International Skin Imaging Collaboration released in 2025. Each lesion was photographed twice: a clinical close-up and a dermoscopic image, taken through an illuminated magnifying lens. The author randomly selected 200 cases and kept only those with a biopsy result.

The sample is split evenly between lesions that cannot be released without follow-up and lesions for which the patient can safely be sent home:

• 30 melanomas;
• 25 basal cell carcinomas;
• 25 squamous cell carcinomas, including Bowen's disease;
• 20 actinic keratoses;
• 50 benign nevi (moles);
• 50 seborrheic keratoses.

The catch is that everyone in the set was biopsied, including those with benign lesions. If a mole was cut out, something about it worried the doctor, which makes it harder than usual to tell apart from a dangerous one.

The LLMs received both images and nothing else. The questions were phrased the way a physician would ask them: which of the six diagnoses is most likely (or "not sure"), with what confidence, which option ranks second, and whether the patient can be sent home without treatment, a biopsy or follow-up. The prompt and default settings were identical for both models, and neither refused to answer.

The ear melanoma that was sent home

The most telling case is an invasive melanoma on the ear, less than 0.8 mm thick. GPT-6 Astra and Claude Fable 5.1 called it a seborrheic keratosis, a harmless warty growth, and said the patient could be sent home. The CNN, working from the dermoscopic image, diagnosed melanoma.

Another example is an almost non-pigmented invasive melanoma of the same thickness. GPT-6 Astra identified it as basal cell carcinoma with 95% confidence, while Claude Fable 5.1 and the CNN got it right. All three models recognized a basal cell carcinoma on the trunk of a 75-year-old man. Both LLMs mistook a seborrheic keratosis on a leg for an actinic keratosis, and the CNN mistook it for a squamous cell carcinoma. All three got it wrong, but none suggested leaving the lesion unattended.

An old architecture against large generalists

The CNN was deliberately not the most modern option. Inception v3 is pretrained on ordinary photographs, and the author then fine-tuned all of its layers on 68,011 dermoscopic and clinical images. He removed all 200 test lesions, plus every other image of those same lesions and the same patients, from the training data, including the validation set. The model never saw them, and neither the threshold nor the model selection was tuned on them. The CNN scored both images, combined the results with most of the weight on the dermoscopic image, and chose from the same six diagnoses.

The gap apparently comes down to specialization: the network has tens of thousands of labeled examples of exactly this task, while the LLMs have only general knowledge and a prompt. The model is also small enough to run entirely in a phone browser via WebGPU. The images stay on the device, and the random-example button draws from the same 200 lesions.

What to check before deployment

Final test numbers:

• GPT-6 Astra and Claude Fable 5.1: 84 and 85 correct diagnoses out of 200 (42%);
• Inception v3 (2015), fine-tuned on 68,011 images: 125 out of 200 (62%);
• CNN fine-tuning time: about twenty minutes on a single H100;
• dataset: MILK10k, ISIC, 2025.

For teams building multimodal LLMs into medical products, the takeaway is practical. A general-purpose model that states high confidence (such as the 95% on a wrong answer) and recommends sending the patient home is more dangerous than modest accuracy on paper. Before launch, these systems should be checked against a narrow, specialized baseline on an independent biopsy-confirmed sample. Teams should also measure the rate of false "can be sent home" calls separately and verify that patients do not overlap between training and test sets. A strong fine-tuned CNN that fits on a phone makes an honest baseline for any LLM in the clinic.