How a Three-Million-Case Labeled Training Set Changes What AI Can Actually Rule Out During Skin Assessment

AI-assisted teledermatology with mandatory clinician sign-off uses AI as an input to care. A three-million-case labeled training set can help a model weigh more competing conditions, so clinically supervised exclusion can be more useful than a simple warning flag. Establishing what is safe to rule out depends on dataset diversity, real-world validation, and a physician who carefully reviews the results.

Why AI skin assessment has historically been a flagging tool, not a reasoning tool

Early dermatology AI often reduced assessment to a binary question, such as whether a lesion was benign or malignant. That structure suited models trained on narrow datasets, but it left little room for the competing explanations a clinician considers. Psoriasis, atopic dermatitis, and rosacea can share erythema, scaling, and papules, while their appearance changes with skin tone, anatomical site, disease duration, and treatment history.

What could a model exclude if its training barely represented those alternatives? A narrow dataset gives it little to draw on when comparing unfamiliar conditions with familiar ones. It learns to recognize conditions within a restricted field, with little scope to compare them across a broad differential. Performance outside that field can deteriorate sharply. The ADAE model, for example, saw specificity collapse to 13.1% when tested on the Stanford/Cleveland Clinic MIDAS dataset. That result illustrates why success on one collection of images cannot establish reliable judgment on another, especially when image artifacts can confound the model.

What three million labeled cases actually represent as an engineering achievement

A three-million-case labeled training set represents an annotation undertaking. A useful clinical training case needs a diagnosis confirmed by a clinician and structured information on features such as skin tone, anatomical site, and disease stage, often with pathological confirmation. Labeling is harder than collecting the image itself. Engineers must connect each presentation to a dependable diagnosis while preserving the context that helps distinguish it from similar presentations.

Yan et al.’s PanDerm study, published in Nature Medicine in August 2025, makes the scale of this work concrete. PanDerm was pretrained on more than 2 million real-world skin images from 11 clinical institutions across 4 imaging modalities, a different unit of data from three million labeled clinical cases. The distinction prevents unlike data counts from being treated as equivalent, while the varied sourcing illustrates how training can include the natural differences a model must handle outside a single institution.

PanDerm achieved state-of-the-art performance across 28 diverse benchmarks using only a fraction of the labeled data required by prior approaches. That finding makes quality and diversity central to the scale argument. More examples can deepen coverage, but their usefulness depends on what they add. Three million cases of mostly familiar presentations would give a model less to learn from than three million carefully labeled cases that cover competing diagnoses and the ways they vary.

How scale enables the shift from pattern-matching to differential reasoning

A ranked differential is a list of possible diagnoses ordered by how well they fit the presentation. Producing a useful one requires more than recognizing the leading candidate. The model must also have learned the alternatives. Training that covers many conditions gives the model examples that resemble one another, while enough cases of each condition show how it varies. Scale can therefore support an output closer to clinical differential reasoning, although no case count by itself proves that this capability has been achieved.

The DermFlow validation study at Indiana University School of Medicine in 2025 illustrates why the output’s structure deserves attention. Among 68 biopsy-proven pigmented lesions, DermFlow’s ranked list included the correct diagnosis with 92.6% any-diagnosis accuracy. The correct answer appeared there far more often than it ranked first. That gap is clinically important without making the list infallible. A clinician can consider a correct lower-ranked possibility that a single-answer interface would have hidden, then use clinical judgment to assess its position.

Yet a ranked list is not a confidence certificate. The Dolphin & Knowles 2026 benchmarking study compared off-the-shelf multimodal AI models with certified dermatologists on patient-captured skin images and found that requested confidence ratings were poorly calibrated. A confident-sounding answer could not be treated as a dependable measure of correctness. The useful distinction is therefore between showing plausible alternatives and asserting certainty about them. A differential gives the clinician something to examine, while the model’s stated confidence does not establish how much trust that examination should place in it.

Why ruling out is a more clinically actionable output than flagging

A warning flag identifies a possibility, but it does not necessarily resolve what should happen next. A reliable exclusion can change triage by separating cases that need a specialist from those judged to have a low risk of the condition being assessed. This is a different kind of claim. Saying that a pigmented lesion resembles a benign condition is not equivalent to establishing that melanoma is very unlikely. The latter requires evidence about how reliably the system’s negative judgments hold up.

Negative predictive value, or NPV, expresses the probability that a condition is genuinely absent when the assessment says it is absent. Broad training coverage supports this task by exposing the model to true negatives for each condition, including other diseases that could be mistaken for it. High case counts for a single diagnosis cannot supply that range of contrasts. Nor does a low position in a ranked differential, on its own, establish a clinically safe exclusion.

The potential benefit is more focused use of specialist time. When melanoma can be reliably excluded, attention can shift toward cases where that exclusion is not possible. However, this clinical benefit requires rigorous validation and real-world evidence in the setting where the assessment will be used. The model’s negative result must be dependable for the patient receiving it.

Where the ruling-out logic breaks down: skin tone and dataset composition

A model trained predominantly on lighter skin tones cannot support the same confidence in exclusions for darker skin merely because its dataset is large. One tool’s accuracy fell to 17.0% when tested on patients with Fitzpatrick skin type VI. That figure concerns accuracy rather than NPV, but it exposes the weakness in treating overall performance as a promise to every patient. An exclusion learned from an uneven population can become unreliable for those poorly represented within it.

Daneshjou et al., from Stanford and Memorial Sloan Kettering, presented the Diverse Dermatology Images dataset at ML4H 2021. DDI was the first publicly available, expertly curated, and pathologically confirmed image dataset with diverse skin tones. The researchers found that state-of-the-art dermatology AI models performed substantially worse on DDI than in their original results. Dark skin tones and uncommon diseases were the primary drivers of the decline. Robust training methods alone could not correct those biases without diverse training data.

The problem also reaches into the labels themselves. Dermatologist-assigned training labels demonstrate reduced accuracy for darker skin, so adding cases can repeat an error. This creates an unequal clinical risk. Patients who get less dependable exclusions may also have less access to an in-person second opinion, so there are fewer chances to catch the mistake.

What deliberate dataset curation adds that raw volume cannot

Deliberate curation organizes cases by the differences an assessment must distinguish, including skin tone, condition rarity, anatomical site, and imaging modality. These features cannot simply be left to chance. A dataset may contain many images while leaving particular conditions or presentations thinly represented. Structured labeling makes those gaps something to address while building the dataset, preventing unpleasant discoveries after deployment. It also gives case volume a purpose, adding coverage where another familiar image would contribute little.

Derm1M, presented at ICCV 2025, uses a clinical ontology developed collaboratively by experts, covering more than 390 conditions across four hierarchical levels. An ontology is a framework that names conditions and shows how they relate to one another. That structure makes diverse labeling systematic. PanDerm’s multi-institution sourcing offers a complementary approach through data provenance, drawing on different care settings and the variation they introduce.

Fine-tuning on diverse data has been shown to remedy the disparity identified in the skin-tone disparity literature by closing the performance gap. Remedying it requires active construction of the training set. Uncommon diseases pose a related challenge because they are systematically underrepresented in most public datasets. If a model has not learned enough about a rare condition to exclude it reliably, its absence from the leading predictions cannot safely remove it from clinical consideration.

What does mandatory clinician sign-off add to AI-assisted teledermatology?

Mandatory sign-off should mean that a licensed clinician with sufficient dermatologic grounding reviews the differential and takes responsibility for the assessment. Approval alone is not meaningful oversight. The clinician must weigh the proposed diagnoses against history, prior treatments, and patient-reported symptoms, including context the model cannot see. The exclusion becomes usable through that judgment. This distinction also separates AI-assisted care from autonomous diagnosis.

Automation bias explains why a review step can fail. Clinicians may anchor to an algorithmic prediction and discount their own assessment, with trainee accuracy dropping by 12.2% when AI predictions were incorrect. Primary care providers may be particularly susceptible because they have less dermatologic training to draw on when the model errs. The Dolphin & Knowles study also supports the need for clinicians to interpret results for themselves. Providing patient metadata improved every measured metric for one model but degraded every metric for another, showing that additional context is not uniformly beneficial when processed by AI.

In asynchronous care, the patient submits photographs and information for later clinician review, often with AI pre-screening. Image quality and the completeness of the symptom description affect what the clinician can make of the differential, regardless of the training set’s size. A blurred photograph or incomplete history can limit a ruling-out decision even when the model performs well under suitable conditions. For sign-off to be meaningful, the clinician must judge the evidence for this case before approving the model’s list.

What this means for patients navigating AI-assisted skin assessment today

A ranked differential exposes competing explanations for clinical review, but not every low-ranked or omitted condition has been safely excluded. A 2025 cross-sectional survey at a US academic dermatology clinic found that 73.8% of patients trusted dermatologist-guided AI.

Does a real physician review the AI findings before any prescription is sent? Does the assessment provide a clinician-reviewed differential, explicit escalation criteria when serious conditions cannot be excluded, and an explanation of what images alone cannot determine? These are concrete criteria for judging the service. The studies discussed here do not show which commercial platforms require a physician to approve a prescription before it is sent, so they cannot be used to produce a verified vendor ranking.

Patients can also improve the evidence they submit. Clear, well-lit images from multiple angles, a complete description of symptom duration and changes over time, and relevant history give the clinician more to work with. Honest reporting of current treatments is important because treatment can alter the presentation. For many people facing a shortage of dermatologists, the realistic alternative may be no timely evaluation. AI-assisted care earns its place when it helps a responsible clinician distinguish what can reasonably be set aside from what still needs attention.