Seeking Advice on Hysteroscopy Lesion Classification with Transfer Learning

[akonst08]

I'm working on 9-class hysteroscopy lesion classification (lesion classes 0–7 + no_lesion) using the HS-CMU and HS-CMU-V2 datasets with patient-level cascading stratified splits (70/15/15).
Dataset: 5,675 images. Severe class imbalance:
Class 0: 228 images (17 patients)
Class 1: 145 images (11 patients)
Class 2: 1,403 images (152 patients)
Class 3: 379 images (43 patients)
Class 4: 199 images (14 patients)
Class 5: 240 images (24 patients)
Class 6: 396 images (36 patients)
Class 7: 95 images (7 patients)
Class 8 (no_lesion): 806 images (50 patients)

Models tried: DenseNet-121 and DINOv2-small, both pretrained.

My issue is that: Validation Macro-F1 stays around 0.30 across different setups. Training metrics improve but validation doesn't follow. Tried various optimizers, schedulers, and augmentation strategies.

Is this a domain gap issue or data limitation?
Should I switch to medical-pretrained encoders?
Is 1-3 patients per class in val/test too few for reliable metrics?
Any tips or suggestions would be greatly appreciated I want to solve this issue so as to train properly feature extractos!

[fffffgggg54]

Your best bet would probably be DINOv3 family weights, since they are trained to reconstruct image embeddings, not process semantic information on natural images (much larger domain gap). There are many huggingface spaces that demonstrate, which can help with exploratory analysis. You should try an see if you can get salient clustering results from the embeddings. Something like a kNN classifier may work better than a supervised probe since you're working with a small dataset. There are quite a few scikit-learn tools that are better in this sort of data-limited case compared to training/augmentation. If you want to train the backbone, you could try starting with a DINOv3 backbone, then train a supervised head along with LoRA adapters on different parts of the backbone (you'll have to experiment with what works, and if this leads to any advantage over sklearn methods).