Freezing layers, adding custom layers
Description
When using pre-trained models for transfer learning, a common technique is to freeze the existing layers of the model and add new custom layers on top. This allows the model to retain previously learned features and adapt to a new task with minimal training.
Freezing layers means setting the layers' weights as non-trainable, so they are not updated during backpropagation. This is particularly useful when the dataset is small and we want to avoid overfitting.
By freezing layers and only training new layers, we leverage the power of a deep pre-trained model while saving computation and avoiding catastrophic forgetting.
Custom layers are added to adapt the model to a new classification or regression task. These layers are trainable and specifically designed to learn task-specific representations.
- Freeze convolutional layers to retain learned image features
- Add Dense, Dropout, or BatchNorm layers for adaptation
- Train only the new layers or gradually unfreeze others if needed
Examples
Here’s an example of freezing a pre-trained model (like VGG16) and adding custom layers for a new classification task using TensorFlow/Keras:
from tensorflow.keras.applications import VGG16
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Dropout
from tensorflow.keras.optimizers import Adam
# Load pre-trained VGG16 without the top classifier layers
base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))
# Freeze all base model layers
for layer in base_model.layers:
layer.trainable = False
# Add custom layers on top
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(128, activation='relu')(x)
x = Dropout(0.5)(x)
predictions = Dense(10, activation='softmax')(x)
# Create final model
model = Model(inputs=base_model.input, outputs=predictions)
# Compile the model
model.compile(optimizer=Adam(0.0001), loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()This example shows:
- Loading VGG16 without its top classifier layers
- Freezing all layers of the base model
- Adding custom layers: GAP, Dense, Dropout, and Output
After training the custom layers, you can optionally unfreeze some top layers of the base model for fine-tuning.
Real-World Applications
Medical Imaging
Adapting pre-trained CNNs to classify X-rays, MRIs, and CT scans for specific diseases by adding custom layers for each task.
Facial Recognition
Using a pre-trained face recognition model and customizing the final layers for personal identification or emotion detection.
Industrial Defect Detection
Fine-tuning image classifiers for detecting defects in manufacturing lines with limited custom data.
Style Transfer and Art
Freezing convolutional layers and training new output layers to generate stylized or domain-transferred artworks.
Resources
Recommended Books
- Deep Learning by Ian Goodfellow et al.
- Hands-On Machine Learning by Aurélien Géron
Interview Questions
What does it mean to freeze layers in a neural network?
Freezing a layer means setting its weights to non-trainable. During training, these weights remain constant and are not updated via backpropagation.
Why would you freeze layers when using a pre-trained model?
To retain the knowledge the model has learned from the original large dataset and avoid overfitting when adapting it to a smaller dataset.
What are custom layers, and how are they added?
Custom layers are new, trainable layers added to the base of a pre-trained model. These layers are typically added to learn task-specific features for the new dataset.
Can you fine-tune a pre-trained model after freezing it?
Yes. After training the new custom layers, you can unfreeze some top layers of the pre-trained model to fine-tune the network and improve performance on the new task.