This project compares supervised learning and LeJEPA-style self-supervised learning under the same visual classification setting.
The goal is not only to ask which model reaches the highest validation accuracy, but also to inspect what kind of internal visual representations are formed by each training strategy. For that reason, the project combines standard classification evaluation with post-training analysis based on PCA maps, saliency maps and a quantitative semantic alignment score.
The final comparison covers four models:
| Model | Architecture | Training strategy |
|---|---|---|
| ViT supervised | ViT-S/16 | Supervised ImageNet-pretrained backbone with a 30-class head |
| ViT LeJEPA | ViT-S/16 | LeJEPA-pretrained backbone with a 30-class linear head |
| CNN supervised | ResNet-50 | Supervised training from scratch |
| CNN LeJEPA | ResNet-50 | Self-supervised LeJEPA training from scratch, followed by a linear probe |
Supervised models are optimized directly to predict labels, so their final representations often become strongly tied to the target classes. Self-supervised models, instead, learn from the structure of the images before a classifier is attached. This can lead to representations that are less label-specific and sometimes more transferable or semantically broad.
This project studies that difference in a controlled setting:
- the same dataset subset is used for every model;
- the CNN models start from the same random initialization;
- the same evaluation images are used for accuracy, PCA, saliency and alignment;
- both transformer and convolutional architectures are included.
The central question is:
When two models solve the same classification task, do their internal representations focus on similar visual regions?
Dataset: ImageNet-100 on Kaggle
The experiments use a 30-class subset of ImageNet-100. The selected classes are the same across every training run and every analysis step.
The dataset is expected in this format:
data/
train.X1/<class_id>/*.JPEG
train.X2/<class_id>/*.JPEG
train.X3/<class_id>/*.JPEG
train.X4/<class_id>/*.JPEG
val.X/<class_id>/*.JPEG
The project does not create an artificial test set from the validation data.
val.X is used as the held-out evaluation split. For the representation
analysis, a fixed subset of 500 validation images is saved and reused across all
models so that every PCA/XAI comparison refers to the exact same images.
The supervised transformer branch uses a ViT-S/16 backbone with supervised ImageNet pretraining. A classifier is trained for the 30 selected classes.
The experiments first evaluate the frozen backbone with a linear head, then fine-tune only the last transformer blocks. This keeps the comparison focused: the model can adapt to the 30-class task without rewriting the whole backbone.
The LeJEPA transformer branch uses a pretrained LeJEPA ViT-S/16 backbone. Since LeJEPA does not produce class predictions by itself, a classifier head is added on top of the frozen backbone and then refined through limited fine-tuning.
This model is useful because it separates two questions:
- how good are the representations before task-specific training?
- how much can a small amount of supervised adaptation improve them?
The supervised CNN branch trains a ResNet-50 from scratch with cross-entropy. This gives a convolutional baseline that does not depend on a pretrained ImageNet classifier.
The CNN LeJEPA branch trains a ResNet-50 backbone with a self-supervised objective. After pretraining, the backbone is frozen and a linear classifier is trained on top of it.
The two ResNet-50 branches start from the same random weights. This matters because it makes the comparison cleaner: the main difference between them is the training objective, not a different initialization.
The project was developed in stages.
First, the ViT models were evaluated with frozen-backbone linear probes. This showed that the supervised ViT was already very strong on the 30-class subset, while the LeJEPA ViT needed more task-specific adaptation to close the gap.
Then, the last two transformer blocks were unfrozen and several learning rate/weight decay combinations were tested. Fine-tuning gave only a small gain for the supervised ViT, but it substantially improved the LeJEPA ViT.
For the CNN branch, both ResNet-50 models were trained from scratch. The supervised model learned the classification task more effectively, while the LeJEPA CNN required a separate linear probe to measure how useful the learned features were for classification.
Finally, the best checkpoints were fixed and used for post-training analysis. No further retraining is needed to reproduce the PCA/XAI outputs.
Best validation accuracy obtained during the experiments:
| Model | Best validation accuracy | Best epoch |
|---|---|---|
| ViT supervised | 90.47% | 4 |
| ViT LeJEPA | 89.13% | 23 |
| CNN supervised | 77.20% | 72 |
| CNN LeJEPA | 61.00% | 50 |
The ViT results are the strongest overall. The supervised ViT remains the best classifier, but the LeJEPA ViT becomes very competitive after fine-tuning. This is one of the most interesting outcomes of the project: the self-supervised transformer starts behind, but its representation can be adapted effectively.
The CNN results show a wider gap. The supervised ResNet-50 reaches a solid accuracy for a from-scratch run, while the LeJEPA ResNet-50 is weaker as a classifier. Even so, it remains useful for representation analysis because it shows how a self-supervised CNN organizes visual information before being directly optimized for labels.
Accuracy alone does not explain what the models are using internally. For that reason, the project extracts visual maps from the trained models and compares them across architectures and training strategies.
The analysis uses three kinds of outputs:
- PCA maps, which summarize the dominant spatial structure of internal features;
- XAI maps, which show regions that support the model prediction;
- N-SAS, a normalized semantic alignment score between PCA and XAI maps.
For the ViT models, saliency is computed with GMAR. For the CNN models, saliency is computed with Grad-CAM.
N-SAS is normalized against shuffled image pairs. A score around zero means that the PCA/XAI alignment is close to chance. A positive score means the internal representation and saliency map are more aligned than a shuffled baseline.
| Block | Supervised N-SAS | LeJEPA N-SAS |
|---|---|---|
| 3 | 1.61 | 0.90 |
| 6 | 0.61 | 0.69 |
| 9 | 0.69 | 0.43 |
| 11 | 0.89 | 0.48 |
The supervised ViT shows stronger alignment in the early and final selected blocks. The LeJEPA ViT is stronger at block 6, which suggests that some of its mid-level features remain more semantically aligned than the supervised counterpart at that stage.
This is a useful qualitative pattern: supervised training seems to push the representation toward class-discriminative evidence, while LeJEPA keeps a different internal organization that does not always peak in the final block.
| Layer | Supervised N-SAS | LeJEPA N-SAS |
|---|---|---|
| layer2 | 0.38 | 0.71 |
| layer3 | 0.25 | 0.42 |
| layer4 | 0.69 | 0.40 |
The CNN comparison follows a similar idea but with a different shape. The LeJEPA CNN has stronger alignment in the intermediate layers, while the supervised CNN has the strongest alignment in the final convolutional layer.
This fits the intuition that supervised classification makes the last layer more directly tied to the decision, while self-supervised training can preserve useful intermediate structure that is not fully reflected by final accuracy.
The repository includes presentation-ready examples under:
outputs/presentation/
The following example compares PCA maps for the same image across all four models. This is useful because it keeps the visual input fixed and makes the representation differences easier to inspect.
For the ViT models, the qualitative comparison uses block 11 and places PCA maps next to GMAR saliency maps. This makes it possible to see whether the dominant internal structure is also connected to the evidence used for prediction.
For the CNN models, the same idea is applied to the final ResNet block using Grad-CAM as the saliency method.
Additional examples are available in:
outputs/presentation/pca_comparison_examples/outputs/presentation/qualitative_cases/vit_block11/outputs/presentation/qualitative_cases/cnn_layer4/
The main result is not simply that one model wins. The more interesting picture is that accuracy and representation alignment do not tell the exact same story.
The supervised ViT is the best classifier, and the supervised CNN clearly beats the LeJEPA CNN in validation accuracy. However, the N-SAS analysis shows that self-supervised models can still produce meaningful internal structures, especially in intermediate layers.
For the ViT branch, LeJEPA becomes close to the supervised model after limited fine-tuning. For the CNN branch, the gap is larger, but the analysis still reveals non-trivial alignment between learned features and prediction evidence.
This makes the project a compact study of an important modern deep learning theme: supervised accuracy is only one view of representation quality.
configs/ Training and analysis configurations
data/ Local ImageNet-100 subset
outputs/
train/ Training histories and checkpoints
xai/ GMAR and Grad-CAM outputs
pca/ PCA maps and reports
sas/ Semantic alignment scores and summaries
presentation/ Lightweight figures and examples for reports/slides
scripts/ Utility scripts
src/
analysis/ PCA, XAI and SAS pipelines
data/ Dataset loading and transforms
network/ ViT and ResNet-50 model definitions
train/ Training and probing entry points
Large output folders are ignored by Git, while selected presentation figures, summary CSV files and reports stay small enough to version.
Create the environment:
venv\Scripts\Activate.ps1
python -m pip install -r requirements.txtThe main experiments are controlled through YAML files in configs/.
For the CNN branch, the shared ResNet initialization can be created with:
python scripts\init_shared_weights.py --seed 42 --output outputs\train\shared\resnet50_shared_init.ptExample training entry points:
python -m src.train.train --config configs\finetune_supervised.yaml
python -m src.train.train --config configs\finetune_lejepa.yaml
python -m src.train.train --config configs\resnet50_supervised.yaml
python -m src.train.train_lejepa_cnn --config configs\resnet50_lejepa.yamlThe final analysis uses fixed checkpoints and the fixed 500-image subset, so PCA, XAI and SAS can be recreated without retraining the models.