How To Set Trainable Layers

Note: I wrote an app to classify an object using transfer learning, and the model was trained on CIFAR-10 dataset, all by myself.

Here's the app:

Transfer Learning Experiments and Findings

Initial Model Comparison

I started by comparing several backbone architectures, including MobileNet V3, MobileNet V4, ResNet-18, and ResNet-34. For the initial experiment, I set TRAINABLE_LAYERS = 1 and ran training across all models.

The results were unexpected:
MobileNet achieved around 60% validation accuracy, while ResNet models struggled at less than 20%. Below is the corresponding configuration and training log:

# ======================
# Configuration
# ======================
NUM_CLASSES = 10
TRAINING_SAMPLE_PER_CLASS = 100
VALIDATION_SAMPLE_PER_CLASS = 100  # None = use all test samples
BATCH_SIZE = 256
EPOCHS = 20
LR = 1e-4
WEIGHT_DECAY = 1e-4
TRAINABLE_LAYERS = 1   # 1 = classifier only, 2 = last block + classifier
EARLY_STOPPING_PATIENCE = 10
SAVE_MODEL = False
USE_TQDM = False

Training mobilenet_v3_small
Epoch 1: Train Loss=2.3177 | Val Loss=2.3455 | Train Acc=11.70% | Val Acc=12.40%
Epoch 2: Train Loss=2.2768 | Val Loss=2.2837 | Train Acc=14.10% | Val Acc=16.10%
Epoch 3: Train Loss=2.2365 | Val Loss=2.2238 | Train Acc=18.60% | Val Acc=20.20%
Epoch 4: Train Loss=2.1967 | Val Loss=2.1654 | Train Acc=22.30% | Val Acc=25.30%
Epoch 5: Train Loss=2.1523 | Val Loss=2.1113 | Train Acc=24.30% | Val Acc=31.70%
Epoch 6: Train Loss=2.1234 | Val Loss=2.0633 | Train Acc=29.80% | Val Acc=35.50%
Epoch 7: Train Loss=2.0863 | Val Loss=2.0215 | Train Acc=32.00% | Val Acc=39.20%
Epoch 8: Train Loss=2.0450 | Val Loss=1.9847 | Train Acc=33.00% | Val Acc=41.70%
Epoch 9: Train Loss=2.0262 | Val Loss=1.9514 | Train Acc=34.40% | Val Acc=43.70%
Epoch 10: Train Loss=1.9843 | Val Loss=1.9185 | Train Acc=35.30% | Val Acc=46.20%
Epoch 11: Train Loss=1.9516 | Val Loss=1.8843 | Train Acc=37.30% | Val Acc=48.10%
Epoch 12: Train Loss=1.9329 | Val Loss=1.8500 | Train Acc=38.80% | Val Acc=49.10%
Epoch 13: Train Loss=1.8963 | Val Loss=1.8137 | Train Acc=40.10% | Val Acc=50.70%
Epoch 14: Train Loss=1.8791 | Val Loss=1.7757 | Train Acc=40.00% | Val Acc=52.40%
Epoch 15: Train Loss=1.8471 | Val Loss=1.7375 | Train Acc=40.40% | Val Acc=53.60%
Epoch 16: Train Loss=1.8110 | Val Loss=1.6985 | Train Acc=42.20% | Val Acc=55.40%
Epoch 17: Train Loss=1.7856 | Val Loss=1.6563 | Train Acc=45.10% | Val Acc=56.10%
Epoch 18: Train Loss=1.7596 | Val Loss=1.6157 | Train Acc=44.50% | Val Acc=57.30%
Epoch 19: Train Loss=1.7511 | Val Loss=1.5755 | Train Acc=45.40% | Val Acc=58.50%
Epoch 20: Train Loss=1.7217 | Val Loss=1.5379 | Train Acc=45.40% | Val Acc=59.90%
✅ Best validation accuracy for mobilenet_v3_small: 59.90%

Training mobilenet_v3_large
Epoch 1: Train Loss=2.3322 | Val Loss=2.3312 | Train Acc=10.20% | Val Acc=11.50%
Epoch 2: Train Loss=2.2975 | Val Loss=2.2714 | Train Acc=12.50% | Val Acc=14.60%
Epoch 3: Train Loss=2.2202 | Val Loss=2.2132 | Train Acc=19.00% | Val Acc=20.20%
Epoch 4: Train Loss=2.1667 | Val Loss=2.1558 | Train Acc=23.00% | Val Acc=23.80%
Epoch 5: Train Loss=2.1245 | Val Loss=2.0989 | Train Acc=25.60% | Val Acc=28.50%
Epoch 6: Train Loss=2.1060 | Val Loss=2.0423 | Train Acc=26.40% | Val Acc=33.10%
Epoch 7: Train Loss=2.0218 | Val Loss=1.9869 | Train Acc=33.70% | Val Acc=36.70%
Epoch 8: Train Loss=1.9899 | Val Loss=1.9310 | Train Acc=35.40% | Val Acc=40.00%
Epoch 9: Train Loss=1.9608 | Val Loss=1.8751 | Train Acc=37.20% | Val Acc=43.30%
Epoch 10: Train Loss=1.9172 | Val Loss=1.8214 | Train Acc=39.60% | Val Acc=46.30%
Epoch 11: Train Loss=1.8726 | Val Loss=1.7687 | Train Acc=41.30% | Val Acc=48.80%
Epoch 12: Train Loss=1.8210 | Val Loss=1.7177 | Train Acc=43.00% | Val Acc=50.00%
Epoch 13: Train Loss=1.7896 | Val Loss=1.6679 | Train Acc=43.60% | Val Acc=52.60%
Epoch 14: Train Loss=1.7819 | Val Loss=1.6190 | Train Acc=43.60% | Val Acc=54.80%
Epoch 15: Train Loss=1.7252 | Val Loss=1.5743 | Train Acc=46.90% | Val Acc=56.00%
Epoch 16: Train Loss=1.6957 | Val Loss=1.5286 | Train Acc=46.10% | Val Acc=57.10%
Epoch 17: Train Loss=1.6737 | Val Loss=1.4864 | Train Acc=48.30% | Val Acc=58.00%
Epoch 18: Train Loss=1.6600 | Val Loss=1.4458 | Train Acc=47.80% | Val Acc=58.70%
Epoch 19: Train Loss=1.6081 | Val Loss=1.4070 | Train Acc=50.10% | Val Acc=59.90%
Epoch 20: Train Loss=1.5894 | Val Loss=1.3731 | Train Acc=51.50% | Val Acc=60.60%
✅ Best validation accuracy for mobilenet_v3_large: 60.60%

Training resnet18
Epoch 1: Train Loss=2.5938 | Val Loss=2.6632 | Train Acc=9.40% | Val Acc=6.70%
Epoch 2: Train Loss=2.5043 | Val Loss=2.6326 | Train Acc=8.60% | Val Acc=6.30%
Epoch 3: Train Loss=2.4533 | Val Loss=2.5928 | Train Acc=9.60% | Val Acc=5.80%
Epoch 4: Train Loss=2.4286 | Val Loss=2.5558 | Train Acc=7.90% | Val Acc=6.70%
Epoch 5: Train Loss=2.3868 | Val Loss=2.5270 | Train Acc=8.80% | Val Acc=6.90%
Epoch 6: Train Loss=2.3919 | Val Loss=2.5020 | Train Acc=10.60% | Val Acc=8.00%
Epoch 7: Train Loss=2.3635 | Val Loss=2.4802 | Train Acc=9.00% | Val Acc=8.40%
Epoch 8: Train Loss=2.3750 | Val Loss=2.4584 | Train Acc=10.20% | Val Acc=8.40%
Epoch 9: Train Loss=2.3513 | Val Loss=2.4373 | Train Acc=11.90% | Val Acc=9.10%
Epoch 10: Train Loss=2.3520 | Val Loss=2.4176 | Train Acc=11.40% | Val Acc=10.00%
Epoch 11: Train Loss=2.3230 | Val Loss=2.4032 | Train Acc=11.40% | Val Acc=10.30%
Epoch 12: Train Loss=2.3147 | Val Loss=2.3892 | Train Acc=11.90% | Val Acc=10.20%
Epoch 13: Train Loss=2.3248 | Val Loss=2.3747 | Train Acc=12.10% | Val Acc=11.20%
Epoch 14: Train Loss=2.2925 | Val Loss=2.3654 | Train Acc=13.50% | Val Acc=11.70%
Epoch 15: Train Loss=2.2954 | Val Loss=2.3534 | Train Acc=12.10% | Val Acc=11.90%
Epoch 16: Train Loss=2.2970 | Val Loss=2.3411 | Train Acc=14.00% | Val Acc=12.10%
Epoch 17: Train Loss=2.2646 | Val Loss=2.3298 | Train Acc=14.90% | Val Acc=12.60%
Epoch 18: Train Loss=2.2509 | Val Loss=2.3168 | Train Acc=16.80% | Val Acc=13.50%
Epoch 19: Train Loss=2.2593 | Val Loss=2.3033 | Train Acc=16.00% | Val Acc=14.00%
Epoch 20: Train Loss=2.2471 | Val Loss=2.2936 | Train Acc=15.60% | Val Acc=14.50%
✅ Best validation accuracy for resnet18: 14.50%

Training resnet34
Epoch 1: Train Loss=2.4739 | Val Loss=2.6085 | Train Acc=8.60% | Val Acc=10.20%
Epoch 2: Train Loss=2.4427 | Val Loss=2.5449 | Train Acc=9.60% | Val Acc=9.50%
Epoch 3: Train Loss=2.4130 | Val Loss=2.4887 | Train Acc=8.70% | Val Acc=10.10%
Epoch 4: Train Loss=2.3812 | Val Loss=2.4540 | Train Acc=9.50% | Val Acc=10.50%
Epoch 5: Train Loss=2.3773 | Val Loss=2.4306 | Train Acc=9.60% | Val Acc=9.80%
Epoch 6: Train Loss=2.3660 | Val Loss=2.4094 | Train Acc=10.40% | Val Acc=10.90%
Epoch 7: Train Loss=2.3541 | Val Loss=2.3940 | Train Acc=10.90% | Val Acc=10.70%
Epoch 8: Train Loss=2.3292 | Val Loss=2.3753 | Train Acc=11.70% | Val Acc=11.20%
Epoch 9: Train Loss=2.3401 | Val Loss=2.3608 | Train Acc=12.60% | Val Acc=11.70%
Epoch 10: Train Loss=2.3171 | Val Loss=2.3462 | Train Acc=12.90% | Val Acc=12.40%
Epoch 11: Train Loss=2.3137 | Val Loss=2.3342 | Train Acc=11.20% | Val Acc=12.30%
Epoch 12: Train Loss=2.3121 | Val Loss=2.3198 | Train Acc=12.00% | Val Acc=12.90%
Epoch 13: Train Loss=2.2840 | Val Loss=2.3081 | Train Acc=14.70% | Val Acc=13.90%
Epoch 14: Train Loss=2.2589 | Val Loss=2.2929 | Train Acc=15.00% | Val Acc=14.20%
Epoch 15: Train Loss=2.2977 | Val Loss=2.2802 | Train Acc=13.00% | Val Acc=14.50%
Epoch 16: Train Loss=2.2577 | Val Loss=2.2687 | Train Acc=14.80% | Val Acc=15.70%
Epoch 17: Train Loss=2.2689 | Val Loss=2.2584 | Train Acc=15.20% | Val Acc=16.50%
Epoch 18: Train Loss=2.2601 | Val Loss=2.2445 | Train Acc=17.20% | Val Acc=16.90%
Epoch 19: Train Loss=2.2375 | Val Loss=2.2317 | Train Acc=17.10% | Val Acc=17.70%
Epoch 20: Train Loss=2.2133 | Val Loss=2.2165 | Train Acc=19.20% | Val Acc=18.50%
✅ Best validation accuracy for resnet34: 18.50%

Initial Hypothesis: Possible Code Issues

My first reaction was that something must be wrong with the code. I suspected issues such as:
– Incorrect freezing or unfreezing logic
– Misconfigured parameters not behaving as intended
– A potential bug in the training pipeline

I spent some time digging into the implementation but could not pinpoint any obvious bugs. To further validate this, I experimented with TRAINABLE_LAYERS = 2, but the results showed no meaningful improvement.

Root Cause: Understanding ResNet Architecture

Eventually, I realized the issue was not the code itself, but rather how TRAINABLE_LAYERS interacted with the ResNet architecture.

A typical ResNet architecture is structured as follows:
[conv1, bn1, relu, maxpool, layer1, layer2, layer3, layer4, avgpool, fc]

When TRAINABLE_LAYERS = 2, the last two modules are unfrozen:
[avgpool, fc]

However, avgpool contains no trainable parameters. As a result, only the fc layer is actually being trained. This means that TRAINABLE_LAYERS = 1 and TRAINABLE_LAYERS = 2 are effectively equivalent for ResNet.

Adjusting the Configuration

Based on this insight, I increased the setting to TRAINABLE_LAYERS = 3, which unfreezes layer4 in addition to the classifier. After rerunning the experiments, the results improved significantly:

– ResNet-18 / ResNet-34: ~77%–79% validation accuracy
– MobileNet: ~59%–67% validation accuracy

Below is the corresponding configuration and training log:

# ======================
# Configuration
# ======================
NUM_CLASSES = 10
TRAINING_SAMPLE_PER_CLASS = 100
VALIDATION_SAMPLE_PER_CLASS = 100  # None = use all test samples
BATCH_SIZE = 256
EPOCHS = 20
LR = 1e-4
WEIGHT_DECAY = 1e-4
TRAINABLE_LAYERS = 3   # 1 = classifier only, 2 = last block + classifier
EARLY_STOPPING_PATIENCE = 10
SAVE_MODEL = False
USE_TQDM = False

Training resnet34
Epoch 1: Train Loss=2.3162 | Val Loss=2.1400 | Train Acc=17.20% | Val Acc=25.60%
Epoch 2: Train Loss=1.8828 | Val Loss=1.6864 | Train Acc=36.20% | Val Acc=45.00%
Epoch 3: Train Loss=1.5692 | Val Loss=1.3554 | Train Acc=47.70% | Val Acc=55.00%
Epoch 4: Train Loss=1.4391 | Val Loss=1.1146 | Train Acc=50.40% | Val Acc=62.60%
Epoch 5: Train Loss=1.3089 | Val Loss=0.9411 | Train Acc=53.70% | Val Acc=69.00%
Epoch 6: Train Loss=1.2026 | Val Loss=0.8654 | Train Acc=58.30% | Val Acc=70.00%
Epoch 7: Train Loss=1.1422 | Val Loss=0.8201 | Train Acc=60.00% | Val Acc=71.70%
Epoch 8: Train Loss=1.1133 | Val Loss=0.7696 | Train Acc=60.00% | Val Acc=73.80%
Epoch 9: Train Loss=1.0301 | Val Loss=0.7183 | Train Acc=62.90% | Val Acc=75.10%
Epoch 10: Train Loss=0.9645 | Val Loss=0.7070 | Train Acc=65.70% | Val Acc=75.90%
Epoch 11: Train Loss=0.9190 | Val Loss=0.7042 | Train Acc=69.50% | Val Acc=75.50%
Epoch 12: Train Loss=0.9551 | Val Loss=0.6816 | Train Acc=66.80% | Val Acc=76.90%
Epoch 13: Train Loss=0.8687 | Val Loss=0.6409 | Train Acc=69.60% | Val Acc=77.80%
Epoch 14: Train Loss=0.8569 | Val Loss=0.6261 | Train Acc=70.60% | Val Acc=78.40%
Epoch 15: Train Loss=0.8373 | Val Loss=0.6420 | Train Acc=71.90% | Val Acc=77.70%
Epoch 16: Train Loss=0.8425 | Val Loss=0.6431 | Train Acc=69.40% | Val Acc=78.50%
Epoch 17: Train Loss=0.8111 | Val Loss=0.6212 | Train Acc=72.40% | Val Acc=78.70%
Epoch 18: Train Loss=0.8147 | Val Loss=0.6209 | Train Acc=72.60% | Val Acc=78.10%
Epoch 19: Train Loss=0.7565 | Val Loss=0.6064 | Train Acc=74.20% | Val Acc=79.10%
Epoch 20: Train Loss=0.7688 | Val Loss=0.6040 | Train Acc=73.20% | Val Acc=78.40%
✅ Best validation accuracy for resnet34: 79.10%

Training resnet18
Epoch 1: Train Loss=2.3781 | Val Loss=2.1589 | Train Acc=13.40% | Val Acc=21.10%
Epoch 2: Train Loss=2.0091 | Val Loss=1.8404 | Train Acc=29.50% | Val Acc=37.80%
Epoch 3: Train Loss=1.7592 | Val Loss=1.5395 | Train Acc=41.50% | Val Acc=50.40%
Epoch 4: Train Loss=1.5662 | Val Loss=1.3322 | Train Acc=48.10% | Val Acc=54.40%
Epoch 5: Train Loss=1.4427 | Val Loss=1.2093 | Train Acc=52.20% | Val Acc=56.10%
Epoch 6: Train Loss=1.3420 | Val Loss=1.1261 | Train Acc=54.00% | Val Acc=59.50%
Epoch 7: Train Loss=1.2779 | Val Loss=1.0259 | Train Acc=57.30% | Val Acc=62.20%
Epoch 8: Train Loss=1.2202 | Val Loss=0.9340 | Train Acc=57.50% | Val Acc=66.00%
Epoch 9: Train Loss=1.1343 | Val Loss=0.8823 | Train Acc=61.50% | Val Acc=68.50%
Epoch 10: Train Loss=1.1146 | Val Loss=0.8320 | Train Acc=62.00% | Val Acc=70.70%
Epoch 11: Train Loss=1.0479 | Val Loss=0.7855 | Train Acc=65.00% | Val Acc=71.70%
Epoch 12: Train Loss=1.0577 | Val Loss=0.7632 | Train Acc=63.30% | Val Acc=72.80%
Epoch 13: Train Loss=1.0376 | Val Loss=0.7399 | Train Acc=64.80% | Val Acc=74.20%
Epoch 14: Train Loss=0.9604 | Val Loss=0.7245 | Train Acc=67.50% | Val Acc=74.20%
Epoch 15: Train Loss=0.9722 | Val Loss=0.7163 | Train Acc=65.00% | Val Acc=73.90%
Epoch 16: Train Loss=0.9594 | Val Loss=0.7047 | Train Acc=65.70% | Val Acc=74.60%
Epoch 17: Train Loss=0.9024 | Val Loss=0.6948 | Train Acc=69.20% | Val Acc=74.70%
Epoch 18: Train Loss=0.8701 | Val Loss=0.6814 | Train Acc=71.10% | Val Acc=75.80%
Epoch 19: Train Loss=0.8892 | Val Loss=0.6533 | Train Acc=70.30% | Val Acc=77.10%
Epoch 20: Train Loss=0.8503 | Val Loss=0.6390 | Train Acc=71.70% | Val Acc=77.80%
✅ Best validation accuracy for resnet18: 77.80%

Training mobilenet_v3_small
Epoch 1: Train Loss=2.3416 | Val Loss=2.3322 | Train Acc=10.10% | Val Acc=9.20%
Epoch 2: Train Loss=2.2764 | Val Loss=2.2613 | Train Acc=14.20% | Val Acc=14.10%
Epoch 3: Train Loss=2.2238 | Val Loss=2.1926 | Train Acc=21.20% | Val Acc=17.60%
Epoch 4: Train Loss=2.1706 | Val Loss=2.1260 | Train Acc=24.00% | Val Acc=22.70%
Epoch 5: Train Loss=2.1219 | Val Loss=2.0639 | Train Acc=28.90% | Val Acc=28.60%
Epoch 6: Train Loss=2.0882 | Val Loss=2.0061 | Train Acc=28.90% | Val Acc=33.40%
Epoch 7: Train Loss=2.0264 | Val Loss=1.9526 | Train Acc=34.50% | Val Acc=39.30%
Epoch 8: Train Loss=1.9684 | Val Loss=1.9010 | Train Acc=40.80% | Val Acc=43.40%
Epoch 9: Train Loss=1.9319 | Val Loss=1.8502 | Train Acc=39.50% | Val Acc=46.20%
Epoch 10: Train Loss=1.8834 | Val Loss=1.7974 | Train Acc=40.50% | Val Acc=47.80%
Epoch 11: Train Loss=1.8380 | Val Loss=1.7407 | Train Acc=43.60% | Val Acc=49.80%
Epoch 12: Train Loss=1.7840 | Val Loss=1.6812 | Train Acc=45.30% | Val Acc=52.80%
Epoch 13: Train Loss=1.7225 | Val Loss=1.6224 | Train Acc=46.50% | Val Acc=54.10%
Epoch 14: Train Loss=1.6673 | Val Loss=1.5620 | Train Acc=49.10% | Val Acc=54.90%
Epoch 15: Train Loss=1.6407 | Val Loss=1.5045 | Train Acc=48.20% | Val Acc=55.70%
Epoch 16: Train Loss=1.5895 | Val Loss=1.4499 | Train Acc=51.30% | Val Acc=56.40%
Epoch 17: Train Loss=1.5427 | Val Loss=1.3994 | Train Acc=49.90% | Val Acc=57.60%
Epoch 18: Train Loss=1.5114 | Val Loss=1.3523 | Train Acc=52.20% | Val Acc=58.40%
Epoch 19: Train Loss=1.5042 | Val Loss=1.3115 | Train Acc=49.30% | Val Acc=58.80%
Epoch 20: Train Loss=1.4911 | Val Loss=1.2781 | Train Acc=48.80% | Val Acc=59.70%
✅ Best validation accuracy for mobilenet_v3_small: 59.70%

Training mobilenet_v3_large
Epoch 1: Train Loss=2.3189 | Val Loss=2.2779 | Train Acc=11.50% | Val Acc=14.90%
Epoch 2: Train Loss=2.2293 | Val Loss=2.1997 | Train Acc=17.20% | Val Acc=18.80%
Epoch 3: Train Loss=2.1549 | Val Loss=2.1214 | Train Acc=25.10% | Val Acc=23.90%
Epoch 4: Train Loss=2.0796 | Val Loss=2.0415 | Train Acc=29.80% | Val Acc=29.30%
Epoch 5: Train Loss=2.0196 | Val Loss=1.9608 | Train Acc=31.70% | Val Acc=36.20%
Epoch 6: Train Loss=1.9393 | Val Loss=1.8787 | Train Acc=39.20% | Val Acc=41.30%
Epoch 7: Train Loss=1.8813 | Val Loss=1.7961 | Train Acc=39.70% | Val Acc=45.80%
Epoch 8: Train Loss=1.8066 | Val Loss=1.7128 | Train Acc=44.10% | Val Acc=48.20%
Epoch 9: Train Loss=1.7320 | Val Loss=1.6270 | Train Acc=45.80% | Val Acc=50.90%
Epoch 10: Train Loss=1.6753 | Val Loss=1.5442 | Train Acc=46.60% | Val Acc=53.00%
Epoch 11: Train Loss=1.6035 | Val Loss=1.4612 | Train Acc=48.80% | Val Acc=55.90%
Epoch 12: Train Loss=1.5834 | Val Loss=1.3842 | Train Acc=48.60% | Val Acc=58.70%
Epoch 13: Train Loss=1.5157 | Val Loss=1.3150 | Train Acc=50.60% | Val Acc=60.60%
Epoch 14: Train Loss=1.4451 | Val Loss=1.2494 | Train Acc=51.90% | Val Acc=61.80%
Epoch 15: Train Loss=1.4311 | Val Loss=1.1944 | Train Acc=51.70% | Val Acc=63.40%
Epoch 16: Train Loss=1.3916 | Val Loss=1.1478 | Train Acc=53.60% | Val Acc=64.30%
Epoch 17: Train Loss=1.3497 | Val Loss=1.1040 | Train Acc=54.10% | Val Acc=65.90%
Epoch 18: Train Loss=1.3411 | Val Loss=1.0652 | Train Acc=54.00% | Val Acc=66.10%
Epoch 19: Train Loss=1.2860 | Val Loss=1.0295 | Train Acc=56.30% | Val Acc=66.90%
Epoch 20: Train Loss=1.2773 | Val Loss=1.0009 | Train Acc=55.00% | Val Acc=67.20%
✅ Best validation accuracy for mobilenet_v3_large: 67.20%

Key Takeaway

The main takeaway from this experiment is that poor performance does not always indicate a bug. Sometimes, the issue lies in how configurable parameters interact with a specific model architecture. A deeper understanding of the model’s internal structure is often necessary to make effective tuning decisions.

Any comments? Feel free to participate below in the Facebook comment section.