Phase 3: Deep Learning

ResNet, EfficientNet & Vision Transformers

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’re building an amazing, super-tall LEGO tower that’s supposed to look at pictures and tell you what’s in them, like whether it’s a dog or a cat. At first, people thought the taller you made your tower (meaning, the more "thinking" layers it had), the smarter it would get. But something weird happened: when the towers got too tall, they became wobbly and confused, like the instructions from the bottom bricks couldn't reach the top properly. It was like trying to send a message through too many friends, and by the time it got to the last person, it was all mixed up!

Then, someone brilliant came up with an idea called "ResNet," which stands for Residual Networks. They said, "What if we add special 'support beams' to our LEGO tower?" These aren't regular stacking bricks. Instead, a support beam would go around a few layers, directly connecting a lower part to a higher part, bypassing some of the middle layers. This simple trick made the tower incredibly stable! Now, you could build truly massive LEGO towers, hundreds of layers tall, and they wouldn't get confused. They became much better at recognizing images because the information could flow smoothly through all the layers.

Once we could build these super-stable, tall towers, the next challenge was to make them even better and more efficient. That’s where "EfficientNet" comes in. Think about building the best possible LEGO castle – not just tall, but also wide, and with lots of fine details. EfficientNet discovered a clever "magic recipe" for scaling up these towers. Instead of just making them taller or wider or adding more detail randomly, it figured out the perfect way to make them all three at the same time: a little taller, a little wider, and seeing a little more detail in the "picture" it was looking at, all in one go. This meant we could build incredibly powerful castles that were super accurate but didn't use up all our LEGO bricks or take forever to build.

And then, a totally new way of looking at LEGO creations appeared, called "Vision Transformers." Instead of looking at a LEGO creation like an old-fashioned system that scans it with a tiny magnifying glass, brick by brick, Vision Transformers take a step back and look at the entire creation all at once. It’s like seeing the whole picture – how the castle walls connect to the towers, where the minifigures are standing, and how everything fits together to tell a story. This lets the computer understand the overall scene and the relationships between different parts in a way that’s sometimes even more powerful.

So, when you build computer programs to understand the world, whether it's recognizing faces, sorting vegetables, or helping self-driving cars see the road, these clever ideas mean you can create super-smart "eyes" that learn from pictures with incredible accuracy and efficiency.

ResNet (Residual Networks) fundamentally changed deep learning by introducing "skip connections" or "residual blocks." Prior to ResNet, stacking many convolutional layers led to vanishing or exploding gradients, making very deep networks difficult to train and often resulting in worse performance. Residual blocks mitigate this by allowing the network to learn a residual mapping H(x) - x, where H(x) is the desired underlying mapping, instead of directly learning H(x). This simple yet powerful innovation enabled the training of extraordinarily deep networks (e.g., ResNet-50, ResNet-101, ResNet-152) and significantly boosted performance on image recognition tasks, establishing ResNet as a cornerstone architecture for many computer vision backbones.

Building upon CNN foundations, EfficientNet focuses on the critical balance between model accuracy and computational efficiency. Instead of arbitrary scaling, EfficientNet proposes a principled compound scaling method that uniformly scales network depth, width, and image resolution using a fixed set of scaling coefficients derived through a neural architecture search. This allows EfficientNet models to achieve state-of-the-art accuracy with significantly fewer parameters and FLOPs compared to other CNNs. For an ML engineer, EfficientNet offers a powerful toolkit for deploying high-performing models, particularly in resource-constrained environments or when latency and inference costs are paramount, by providing a family of models (EfficientNet-B0 to B7) optimized for different computational budgets.

Vision Transformers (ViT) represent a paradigm shift, adapting the highly successful Transformer architecture, originally dominant in NLP, directly to computer vision tasks, moving away from the convolutional inductive biases. ViT processes images by splitting them into fixed-size patches, linearly embedding these patches, and treating them as a sequence of tokens. These tokens are then fed into a standard Transformer encoder, relying on self-attention mechanisms to capture global dependencies across the image. While ViTs often require massive datasets for pre-training to achieve superior performance due to their lack of inherent inductive biases (like locality and translation equivariance), they have demonstrated state-of-the-art results, especially on large-scale vision benchmarks, and offer compelling advantages in modeling long-range interactions within images that traditional CNNs might struggle with.

Key Takeaways

  • ResNet's skip connections resolved vanishing gradients, enabling ultra-deep, high-performing CNNs.
  • EfficientNet introduced principled compound scaling to efficiently balance CNN accuracy and computational cost.
  • Vision Transformers adapt NLP's self-attention to image patches, challenging CNNs and excelling with large datasets.
  • Each architecture addresses different challenges: depth (ResNet), efficiency (EfficientNet), or global context (ViT).
  • Choice depends on performance needs, available data, and computational constraints for the target application.

Code Example

python
import tensorflow as tf
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.models import Model

# Load ResNet50 pre-trained on ImageNet, exclude top classification layer
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(224, 224, 3))

# Add custom classification layers for a new task
x = base_model.output
x = Flatten()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(10, activation='softmax')(x) # Example: 10 classes

model = Model(inputs=base_model.input, outputs=predictions)

# Freeze base model layers for initial training to use it as a feature extractor
for layer in base_model.layers:
    layer.trainable = False

model.summary()

How this code works

This code demonstrates a powerful technique called transfer learning, where a pre-trained deep learning model is adapted for a new task. Its job is to efficiently set up a neural network for image classification by leveraging the features learned by ResNet50 on a vast dataset, then adding specific layers to classify images into a custom set of categories.

The process begins by loading ResNet50 from tensorflow.keras.applications. The weights='imagenet' argument tells it to load weights from the ImageNet competition, meaning the model already understands many general image features. Crucially, include_top=False removes ResNet50's original classification head, allowing us to attach our own layers. The input_shape specifies the expected image size. On top of the base_model.output, new layers like Flatten() and Dense layers are added to create predictions for, in this example, 10 specific classes using a softmax activation. Finally, model = Model(...) constructs the complete network by connecting the base_model.input to these new prediction layers.

A subtle but vital step is the for layer in base_model.layers: layer.trainable = False loop. This "freezes" the weights of the pre-trained ResNet50 layers. Freezing them means they won't be updated during initial training, making the base_model act purely as a feature extractor. This is often done to prevent messing up the well-learned features and to speed up training of the new top layers, especially when the custom dataset is smaller. The model.summary() then provides an overview of the combined architecture and confirms which layers are trainable.

Related Articles

Hand-picked reads to go deeper on convolutional neural networks.

Back to ML Engineer Roadmap