Phase 3: Deep Learning

Positional Encoding & Multi-Head Attention

Advanced ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you have a huge pile of library books, but all the page numbers have been ripped out, and the pages themselves are all mixed up! If you try to read a story, how would you know which sentence comes first or what paragraph follows what? You couldn't tell the difference between "The dog barked loudly at the mailman" and "The mailman barked loudly at the dog" if the words were just floating around. Computers, especially a super-smart system we call a "Transformer," have a similar problem. They're amazing at reading words all at once, super fast, but they don't naturally know the order. They need help to understand which word comes before another.

This is where we add special "magic page numbers" to our story. Instead of simple numbers like 1, 2, 3, these are like unique, shimmery colored patterns printed directly onto each word, blending in with its meaning. So, the first word gets one wavy pattern, the second word gets a slightly different wavy pattern, and so on. These patterns are cleverly designed so that even if the computer just sees a big pile of words, it can immediately tell, "Ah, this word with that shimmery pattern goes in the third spot!" We don't teach the computer to learn these positions; we just give it this special hint. It's like having a secret code on each page that tells you its exact place in the story, no matter where it is in the pile. This way, the computer knows the sequence and can tell the difference between the dog biting the man and the man biting the dog.

Once the computer knows the order of all the words, it then needs to truly understand the story, not just list the words. A single human reader might focus on one main plot point. But imagine you have several super-smart readers, each with a different job, all reading the same story at the same time. One reader might focus only on who the characters are and what they're doing. Another reader might only look for descriptions of where the story takes place. A third might pay attention to how different events cause other events. Each reader highlights different important connections and relationships between words in the story.

These super-readers aren't just guessing; they're all carefully analyzing the words with their "magic page numbers." After each of them has found their own specific types of connections – like the main character's feelings, or the location details, or the sequence of actions – all their insights are gathered and put together. This combined understanding is much richer and more complete than what any single reader could have found alone. This means when you build a program that translates languages, writes stories, or answers questions, it can grasp all the subtle meanings and relationships in a sentence, making its answers super clever and accurate because it truly understands the flow and connections of the words.

Transformers, at their core, process input sequences by parallelizing computations without inherent sequence order. This presents a critical challenge: how does the model differentiate between "dog bites man" and "man bites dog"? Positional Encoding (PE) addresses this by injecting information about the relative or absolute position of each token into its embedding. Rather than learning positions, which can be inefficient for long or varying sequences, fixed sinusoidal functions are typically used. These functions generate a unique, continuous, and scalable signal for each position, which is added to the token's input embedding. This allows the model to leverage both semantic content and sequence order when processing tokens, fundamentally enabling sequence-aware understanding in a parallelized architecture.

Once positional context is established, Multi-Head Attention (MHA) amplifies the model's ability to capture complex dependencies. Instead of a single attention mechanism focusing on one type of relationship, MHA runs multiple 'heads' in parallel. Each head independently projects the input queries, keys, and values into different, lower-dimensional representation subspaces. This allows each head to attend to different parts of the input sequence or different aspects of the same token—for instance, one head might capture syntactic relationships while another focuses on semantic similarities or coreferences. The outputs from these individual attention heads are then concatenated and linearly projected back to the desired output dimension.

Practically, MHA empowers the Transformer to learn a richer, more diverse set of relationships within the input, acting like an ensemble of specialized experts observing the data from different angles. This parallel processing of attention not only enhances the model's capacity to understand intricate patterns but also improves its robustness and generalization capabilities, forming a cornerstone of the Transformer's success across varied tasks in NLP, computer vision, and beyond. Understanding how PE provides essential context and MHA extracts multifaceted insights is key to leveraging and optimizing these powerful models.

Key Takeaways

  • Positional Encoding (PE) provides essential sequence order information to otherwise permutation-invariant Transformers.
  • PE typically uses fixed sinusoidal functions, added to token embeddings, to scale robustly to various sequence lengths.
  • Multi-Head Attention (MHA) allows the model to concurrently focus on multiple distinct relationships within the input sequence.
  • Each attention 'head' learns different projections, enabling the model to capture diverse features like syntax, semantics, and coreference.
  • Together, PE and MHA enable Transformers to understand complex sequences by providing positional context and diverse relational insights.

Code Example

python
import numpy as np

def get_positional_encoding(max_seq_len, d_model):
    """
    Generates sinusoidal positional encodings.
    Args:
        max_seq_len (int): Maximum sequence length to support.
        d_model (int): Dimension of the model's embeddings.
    Returns:
        np.array: (max_seq_len, d_model) array of positional encodings.
    """
    pe = np.zeros((max_seq_len, d_model))
    position = np.arange(0, max_seq_len)[:, np.newaxis]
    # Calculate the divisor term for even/odd dimensions
    div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    
    # Apply sine to even indices in d_model dimension
    pe[:, 0::2] = np.sin(position * div_term)
    # Apply cosine to odd indices in d_model dimension
    pe[:, 1::2] = np.cos(position * div_term)
    
    return pe

# Example usage:
# pe_matrix = get_positional_encoding(max_seq_len=100, d_model=512)
# print(pe_matrix.shape) # Expected: (100, 512)

How this code works

This code generates positional encodings, which are crucial for Transformer models. Since Transformers process all words in a sequence simultaneously, they lack an inherent understanding of word order. These encodings provide a unique pattern added to each word's embedding, conveying its specific position within the sequence. This "positional information" helps the model differentiate between words that are semantically similar but appear at different places in a sentence.

The get_positional_encoding function first initializes a zero-filled pe matrix to store these encodings. It then sets up a position vector, representing each index in the sequence. The key calculation is the div_term, which controls the wavelength of the sinusoidal waves. Notice div_term is generated using np.arange(0, d_model, 2), meaning it only computes values for even indices of the embedding dimension. A subtle point is that this single set of div_term values is then applied to fill both the even dimensions of pe with sine waves (pe[:, 0::2]) and the odd dimensions with cosine waves (pe[:, 1::2]). This pairing of sine and cosine components with the same div_term ensures adjacent dimensions have coupled wavelengths, making it easier for the model to represent relative positions.

Resources

Related Articles

Hand-picked reads to go deeper on transformers & attention.

Back to ML Engineer Roadmap