1. Transformer Interview Questions

The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" (Vaswani et al.) that relies entirely on attention mechanisms instead of recurrence or convolution to process sequential data.

Why it was introduced:

Core idea: Instead of processing tokens one-by-one (RNN) or with fixed-size windows (CNN), the Transformer lets every token directly attend to every other token in a single step.

┌─────────────────────────────────────────────┐
│           TRANSFORMER ARCHITECTURE           │
├─────────────────────────────────────────────┤
│                                             │
│   Input ──► [Token Embedding]               │
│             + [Positional Encoding]          │
│                    │                        │
│          ┌─────────▼──────────┐             │
│          │     ENCODER (Nx)    │             │
│          │  ┌───────────────┐  │             │
│          │  │ Multi-Head    │  │             │
│          │  │ Self-Attention│  │             │
│          │  └──────┬────────┘  │             │
│          │  ┌──────▼────────┐  │             │
│          │  │ Feed-Forward  │  │             │
│          │  │ Network       │  │             │
│          │  └───────────────┘  │             │
│          └─────────┬──────────┘             │
│                    │                        │
│          ┌─────────▼──────────┐             │
│          │     DECODER (Nx)    │             │
│          │  ┌───────────────┐  │             │
│          │  │ Masked Self-  │  │             │
│          │  │ Attention     │  │             │
│          │  ├───────────────┤  │             │
│          │  │ Cross-        │  │             │
│          │  │ Attention     │  │             │
│          │  ├───────────────┤  │             │
│          │  │ Feed-Forward  │  │             │
│          │  │ Network       │  │             │
│          │  └───────────────┘  │             │
│          └─────────┬──────────┘             │
│                    │                        │
│             [Linear + Softmax]              │
│                    │                        │
│                 Output                      │
└─────────────────────────────────────────────┘

The original Transformer used 6 encoder layers and 6 decoder layers, with model dimension \(d_{model} = 512\) and 8 attention heads.

Self-attention (also called intra-attention) allows each position in a sequence to attend to all other positions, computing a weighted sum of values where weights are determined by the compatibility between queries and keys.

Step-by-step computation:

Step 1: Create Q, K, V matrices

For input matrix \(X \in \mathbb{R}^{n \times d_{model}}\):

Where \(W^Q, W^K \in \mathbb{R}^{d_{model} \times d_k}\) and \(W^V \in \mathbb{R}^{d_{model} \times d_v}\)

Step 2: Compute attention scores

Each element \((i, j)\) represents how much token \(i\) should attend to token \(j\).

Step 3: Scale

Step 4: Apply softmax

Step 5: Weighted sum of values

import torch
import torch.nn.functional as F
import math

def scaled_dot_product_attention(Q, K, V, mask=None):
    """
    Q: (batch, seq_len, d_k)
    K: (batch, seq_len, d_k)
    V: (batch, seq_len, d_v)
    """
    d_k = Q.size(-1)

    # Step 1: Compute attention scores
    scores = torch.matmul(Q, K.transpose(-2, -1))  # (batch, seq_len, seq_len)

    # Step 2: Scale
    scores = scores / math.sqrt(d_k)

    # Step 3: Apply mask (optional, for decoder)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))

    # Step 4: Softmax to get attention weights
    attention_weights = F.softmax(scores, dim=-1)

    # Step 5: Weighted sum of values
    output = torch.matmul(attention_weights, V)  # (batch, seq_len, d_v)

    return output, attention_weights

Intuition with analogy: Think of a library search:

• Query (Q): Your search question
• Key (K): Index card labels for each book
• Value (V): The actual book content
• Attention score: How relevant each book is to your question
• Output: A weighted blend of all relevant book contents

We scale by \(\sqrt{d_k}\) to prevent the dot products from growing too large, which would push the softmax into regions with extremely small gradients.

Mathematical justification:

If \(q\) and \(k\) are vectors with components drawn independently from a distribution with mean 0 and variance 1, then their dot product \(q \cdot k = \sum_{i=1}^{d_k} q_i k_i\) has:

• Mean: \(E[q \cdot k] = 0\)
• Variance: \(\text{Var}(q \cdot k) = d_k\)

As \(d_k\) grows, the dot products grow in magnitude proportional to \(\sqrt{d_k}\), pushing softmax outputs toward one-hot vectors (saturation).

What happens without scaling:

import torch
import torch.nn.functional as F

d_k = 512
q = torch.randn(1, d_k)
k = torch.randn(10, d_k)

# Without scaling - softmax saturates
scores_unscaled = q @ k.T
print(f"Unscaled scores std: {scores_unscaled.std():.2f}")      # ~22.6
print(f"Unscaled softmax: {F.softmax(scores_unscaled, dim=-1)}")  # Nearly one-hot

# With scaling - healthy gradient flow
scores_scaled = (q @ k.T) / (d_k ** 0.5)
print(f"Scaled scores std: {scores_scaled.std():.2f}")            # ~1.0
print(f"Scaled softmax: {F.softmax(scores_scaled, dim=-1)}")      # Smooth distribution

After dividing by \(\sqrt{d_k}\), the variance of the dot product becomes 1, keeping the softmax in a region with healthy gradients.

Multi-Head Attention (MHA) runs \(h\) parallel attention functions, each operating on a different learned linear projection of Q, K, and V into lower-dimensional subspaces.

Where each head is:

With \(W_i^Q \in \mathbb{R}^{d_{model} \times d_k}\), \(W_i^K \in \mathbb{R}^{d_{model} \times d_k}\), \(W_i^V \in \mathbb{R}^{d_{model} \times d_v}\), and \(W^O \in \mathbb{R}^{hd_v \times d_{model}}\).

Typically \(d_k = d_v = d_{model} / h\).

Why multiple heads?

1. Different representation subspaces: Each head can learn to attend to different types of relationships:

   • Head 1: Syntactic dependencies (subject-verb agreement)
   • Head 2: Positional proximity (nearby tokens)
   • Head 3: Semantic similarity (synonyms, related concepts)
   • Head 4: Coreference resolution (pronoun-antecedent)

2. Richer representation: A single head computes one set of attention weights — it can only focus on one type of relationship per position. Multiple heads allow attending to multiple positions for different reasons simultaneously.

3. Computational equivalence: MHA with \(h\) heads of dimension \(d_k = d_{model}/h\) has roughly the same computational cost as single-head attention with dimension \(d_{model}\).

import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model=512, n_heads=8):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads

        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

    def forward(self, Q, K, V, mask=None):
        batch_size = Q.size(0)

        # Linear projections and reshape to (batch, n_heads, seq_len, d_k)
        Q = self.W_q(Q).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)
        K = self.W_k(K).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)
        V = self.W_v(V).view(batch_size, -1, self.n_heads, self.d_k).transpose(1, 2)

        # Scaled dot-product attention per head
        scores = torch.matmul(Q, K.transpose(-2, -1)) / (self.d_k ** 0.5)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        attn = torch.softmax(scores, dim=-1)
        context = torch.matmul(attn, V)

        # Concatenate heads and project
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.n_heads * self.d_k)
        return self.W_o(context)

In the original Transformer: \(d_{model} = 512\), \(h = 8\) heads, \(d_k = d_v = 64\).

Positional encoding injects information about the position of each token in the sequence because self-attention is permutation-equivariant — without it, the model cannot distinguish "The cat sat on the mat" from "mat the on sat cat The".

Sinusoidal positional encoding (original Transformer):

Where \(pos\) is the position and \(i\) is the dimension index.

Why sinusoidal functions?

1. Bounded values: Outputs are always in \([-1, 1]\) regardless of sequence length
2. Unique encoding: Each position gets a unique pattern across dimensions
3. Relative position via linear transformation: For any fixed offset \(k\), \(PE_{pos+k}\) can be represented as a linear function of \(PE_{pos}\), enabling the model to learn relative positioning:

1. Generalization to unseen lengths: Can extrapolate to longer sequences than seen during training (in theory)

import torch
import math

def sinusoidal_positional_encoding(max_len, d_model):
    pe = torch.zeros(max_len, d_model)
    position = torch.arange(0, max_len).unsqueeze(1).float()
    div_term = torch.exp(
        torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)
    )

    pe[:, 0::2] = torch.sin(position * div_term)  # Even dimensions
    pe[:, 1::2] = torch.cos(position * div_term)  # Odd dimensions
    return pe

# Each row is a unique positional "fingerprint"
pe = sinusoidal_positional_encoding(100, 512)
print(pe.shape)  # (100, 512)

Alternative approaches: Learned positional embeddings (BERT, GPT-2), Rotary Position Embeddings (RoPE in LLaMA), ALiBi (BLOOM).

The position-wise feed-forward network is applied independently and identically to each position. It is a two-layer MLP with a non-linear activation:

Where \(W_1 \in \mathbb{R}^{d_{model} \times d_{ff}}\), \(W_2 \in \mathbb{R}^{d_{ff} \times d_{model}}\), and \(\sigma\) is the activation function (ReLU in the original paper).

In the original Transformer: \(d_{model} = 512\), \(d_{ff} = 2048\) (4x expansion).

Why is it needed?

Without FFN, the Transformer would only be able to compute weighted averages of value vectors — a linear operation. The FFN adds non-linearity and per-position transformation capacity.

Key properties:

1. Position-wise: The same FFN is applied to each position independently (like a 1x1 convolution)
2. Parameter heavy: FFN accounts for roughly ⅔ of all parameters in a Transformer layer
3. Knowledge storage: Research (Geva et al., 2021) shows FFN layers act as key-value memories, storing factual knowledge learned during pre-training

import torch.nn as nn

class PositionWiseFFN(nn.Module):
    def __init__(self, d_model=512, d_ff=2048, dropout=0.1):
        super().__init__()
        self.w1 = nn.Linear(d_model, d_ff)
        self.w2 = nn.Linear(d_ff, d_model)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        # x: (batch, seq_len, d_model)
        return self.w2(self.dropout(self.relu(self.w1(x))))

Modern variants replace ReLU with SwiGLU (LLaMA, PaLM) which uses a gated linear unit:

Residual (skip) connections and layer normalization are critical for training deep Transformers.

Residual connections add the input of a sublayer to its output:

This creates a "gradient highway" that allows gradients to flow directly through the network, enabling training of very deep models (100+ layers).

Layer normalization normalizes across the feature dimension:

Where \(\mu\) and \(\sigma^2\) are the mean and variance computed across the feature dimension, and \(\gamma\), \(\beta\) are learned scale and shift parameters.

Post-Norm (original Transformer):

x ──┬──► [Sublayer] ──► (+) ──► [LayerNorm] ──► output
    └─────────────────────┘

Pre-Norm (GPT-2, LLaMA, most modern LLMs):

x ──┬──► [LayerNorm] ──► [Sublayer] ──► (+) ──► output
    └──────────────────────────────────────┘

Pre-Norm is preferred for large models because it is much easier to train — the residual stream is the "main trunk" and sublayers are additive updates to it.

Masked (causal) attention prevents the decoder from attending to future positions during training. This is essential for autoregressive generation — the model must predict token \(t\) using only tokens \(1, 2, \ldots, t-1\).

The mask is applied before the softmax:

Where \(M\) is an upper-triangular matrix of \(-\infty\):

After softmax, \(-\infty\) values become 0, so future tokens receive zero attention weight.

import torch

def create_causal_mask(seq_len):
    """Lower triangular mask for autoregressive decoding."""
    mask = torch.tril(torch.ones(seq_len, seq_len))
    return mask  # 1 = attend, 0 = block

# For sequence length 5:
# tensor([[1, 0, 0, 0, 0],
#         [1, 1, 0, 0, 0],
#         [1, 1, 1, 0, 0],
#         [1, 1, 1, 1, 0],
#         [1, 1, 1, 1, 1]])

Why is it necessary?

• During training, we feed the entire target sequence at once (teacher forcing) for efficiency. Without masking, the model could "cheat" by looking at the answer.
• During inference, tokens are generated one at a time, so the mask is naturally satisfied.
• The mask ensures training and inference behavior are consistent.

All decoder-only models (GPT, LLaMA, Claude) use causal masking throughout — every layer is masked self-attention.

Cross-attention (also called encoder-decoder attention) allows the decoder to attend to the encoder's output. Unlike self-attention where Q, K, V all come from the same sequence, in cross-attention:

• Queries (Q): Come from the decoder (previous decoder layer's output)
• Keys (K) and Values (V): Come from the encoder output

Encoder Output ──────────────────┐
     │                           │
     ├──► K (Keys)               │
     └──► V (Values)             │
                                 │
Decoder Hidden ──► Q (Queries)   │
                       │         │
                       ▼         ▼
                [Attention Scores]
                       │
                       ▼
                [Weighted Sum of Encoder Values]

Cross-attention is used in:

• Machine translation: Decoder attends to source language
• Summarization: Decoder attends to input document
• Speech recognition (Whisper): Text decoder attends to audio encoder
• Image captioning: Text decoder attends to image encoder
• Diffusion models: U-Net cross-attends to text embeddings (Stable Diffusion)

┌──────────────────┬──────────────────┬──────────────────┐
│  ENCODER-ONLY    │  DECODER-ONLY    │  ENCODER-DECODER │
│  (Bidirectional) │  (Autoregressive)│  (Seq-to-Seq)    │
├──────────────────┼──────────────────┼──────────────────┤
│                  │                  │                  │
│  ┌──────────┐   │  ┌──────────┐   │  ┌────┐  ┌────┐ │
│  │ Encoder  │   │  │ Decoder  │   │  │Enc │→│Dec │ │
│  │ (bidir.) │   │  │ (causal) │   │  │    │  │    │ │
│  └──────────┘   │  └──────────┘   │  └────┘  └────┘ │
│                  │                  │                  │
│  BERT           │  GPT-2/3/4      │  T5              │
│  RoBERTa        │  LLaMA          │  BART            │
│  DeBERTa        │  Claude         │  Whisper         │
│  ELECTRA        │  Mistral        │  mBART           │
│                  │  DeepSeek       │  FLAN-T5         │
├──────────────────┼──────────────────┼──────────────────┤
│ Attention: Full  │ Attention:      │ Encoder: Full    │
│ (all-to-all)     │ Causal mask     │ Decoder: Causal  │
│                  │ (left-to-right) │ + Cross-attention│
├──────────────────┼──────────────────┼──────────────────┤
│ Pre-training:    │ Pre-training:   │ Pre-training:    │
│ MLM (mask &      │ CLM (next token │ Span corruption  │
│ predict)         │ prediction)     │ (T5) or denoise  │
├──────────────────┼──────────────────┼──────────────────┤
│ Best for:        │ Best for:       │ Best for:        │
│ Classification   │ Text generation │ Translation      │
│ NER, embeddings  │ Code, reasoning │ Summarization    │
│ Retrieval        │ Chat, agents    │ ASR, multimodal  │
└──────────────────┴──────────────────┴──────────────────┘

Why decoder-only dominates today:

• Simpler architecture (one stack of layers)
• Scales better to very large models
• In-context learning and instruction following emerge with scale
• A single model can handle many tasks via prompting (no task-specific head needed)

Autoregressive language modeling trains the model to predict the next token given all previous tokens. The objective is to maximize:

This is equivalent to minimizing the cross-entropy loss between the predicted probability distribution and the one-hot true token at each position.

Teacher forcing is the training strategy where the ground truth sequence is fed as input to the decoder at every timestep, rather than the model's own predictions:

WITHOUT teacher forcing (autoregressive, slow, error propagation):
Input:  <BOS>
Pred:   "The"  →  Input: "The"
Pred:   "cat"  →  Input: "The cat"
Pred:   "sit"  →  Input: "The cat sit"  ← error propagates!

WITH teacher forcing (parallel, fast, ground truth input):
Input:  <BOS>   "The"    "cat"    "sat"    "on"
Target: "The"   "cat"    "sat"    "on"     "the"
Loss:   L1      L2       L3       L4       L5

Advantages of teacher forcing:

1. Parallelization: All positions computed simultaneously (the key speed advantage of Transformers)
2. Stable training: No error accumulation from incorrect predictions
3. Efficient: One forward pass gives loss for all positions

Disadvantage — exposure bias: During training, the model always sees ground truth context, but during inference it sees its own (possibly incorrect) predictions. This mismatch is called exposure bias.

Transformers operate on sub-word tokens, not raw characters or whole words. This balances vocabulary size with representation efficiency.

Byte-Pair Encoding (BPE):

Used in: GPT-⅔/4, LLaMA, Claude, RoBERTa

1. Start with character-level vocabulary
2. Iteratively merge the most frequent adjacent pair
3. Repeat for a fixed number of merges (determines vocab size)

Corpus: "low lower lowest"
Step 1: Characters: l, o, w, e, r, s, t, ...
Step 2: Most frequent pair: (l, o) → merge to "lo"
Step 3: Most frequent pair: (lo, w) → merge to "low"
Step 4: Continue until vocab size reached...

WordPiece:

Used in: BERT, DistilBERT, ELECTRA

Similar to BPE, but selects merges that maximize the language model likelihood of the training data, not just frequency. Uses ## prefix for continuation tokens: "playing" → ["play", "##ing"]

SentencePiece:

Used in: T5, LLaMA, ALBERT, XLNet

Treats the input as a raw byte stream (language-agnostic, no pre-tokenization needed). Supports both BPE and Unigram algorithms. Can handle any language without language-specific preprocessing.

The KV cache stores the Key and Value matrices from all previous tokens during autoregressive generation, avoiding redundant recomputation.

Without KV cache — generating token \(t\) requires computing attention over tokens \(1\) through \(t-1\), recomputing K and V for ALL previous tokens every step:

Step 1: Compute K,V for [tok1]                    → predict tok2
Step 2: Compute K,V for [tok1, tok2]              → predict tok3
Step 3: Compute K,V for [tok1, tok2, tok3]        → predict tok4
Total K,V computations: 1 + 2 + 3 + ... + n = O(n²)

With KV cache — only compute K, V for the new token and append to cached values:

Step 1: Compute K,V for [tok1], cache them        → predict tok2
Step 2: Compute K,V for [tok2], append to cache   → predict tok3
Step 3: Compute K,V for [tok3], append to cache   → predict tok4
Total K,V computations: 1 + 1 + 1 + ... + 1 = O(n)

Memory cost of KV cache:

For LLaMA-2 70B with 4K context:

• 80 layers × 64 heads × 128 \(d_{head}\) × 4096 seq × 2 (K+V) × 2 bytes (FP16)
• = 10.7 GB just for KV cache of one sequence!

This is why KV cache optimization (GQA, MQA, MLA, quantization, PagedAttention) is one of the most active areas in LLM serving.

Rotary Position Embedding (RoPE), introduced by Su et al. (2021), encodes position by rotating the query and key vectors in 2D subspaces. It naturally encodes relative position in the attention dot product.

Core idea: Instead of adding positional encoding, rotate Q and K vectors by an angle proportional to their position:

For a 2D subspace at dimension pair \((2i, 2i+1)\):

Where \(\theta_i = 10000^{-2i/d}\) (similar base frequency as sinusoidal).

The rotated query and key:

The attention score between positions \(m\) and \(n\):

The dot product only depends on the relative position \((n - m)\), not the absolute positions!

Why RoPE is preferred:

Used in: LLaMA ½/3, PaLM, Mistral, DeepSeek, Qwen, Yi, and nearly all modern LLMs.

import torch

def apply_rope(x, freqs_cos, freqs_sin):
    """Apply RoPE to input tensor x of shape (batch, seq_len, n_heads, d_head)."""
    # Split into pairs: (x0, x1), (x2, x3), ...
    x_even = x[..., 0::2]
    x_odd = x[..., 1::2]

    # Apply rotation
    x_rotated_even = x_even * freqs_cos - x_odd * freqs_sin
    x_rotated_odd = x_even * freqs_sin + x_odd * freqs_cos

    # Interleave back
    x_out = torch.stack([x_rotated_even, x_rotated_odd], dim=-1)
    return x_out.flatten(-2)

Grouped Query Attention (GQA) is a compromise between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). It groups query heads to share a single key-value head, reducing KV cache size while retaining most of MHA's quality.

Multi-Head Attention (MHA):          8 Q heads, 8 K heads, 8 V heads
┌──┬──┬──┬──┬──┬──┬──┬──┐  Q
├──┼──┼──┼──┼──┼──┼──┼──┤  K  (1:1 ratio)
├──┼──┼──┼──┼──┼──┼──┼──┤  V
└──┴──┴──┴──┴──┴──┴──┴──┘

Grouped Query Attention (GQA):       8 Q heads, 2 K heads, 2 V heads
┌──┬──┬──┬──┬──┬──┬──┬──┐  Q
├─────┼─────┼─────┼─────┤  K  (4:1 ratio, groups of 4)
├─────┼─────┼─────┼─────┤  V
└─────┴─────┴─────┴─────┘

Multi-Query Attention (MQA):         8 Q heads, 1 K head, 1 V head
┌──┬──┬──┬──┬──┬──┬──┬──┐  Q
├──────────────────────────┤  K  (all share 1)
├──────────────────────────┤  V
└──────────────────────────┘

Example — LLaMA 2 70B:

• 64 query heads, 8 KV heads (GQA with group size 8)
• KV cache reduced by 8x compared to MHA
• Quality nearly matches the MHA baseline

GQA achieves MHA-level quality with MQA-level speed — the best of both worlds. The key insight is that attention heads within a group often learn similar patterns, so sharing KV projections has minimal quality impact.

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses the KV cache into a low-rank latent representation instead of sharing KV heads (as in GQA/MQA).

Key idea: Instead of caching full K and V tensors, compress them into a much smaller latent vector:

Instead of caching \(n_{heads} \times d_{head}\) for both K and V, MLA caches only the latent vector \(c_t^{KV}\) of dimension \(d_c\).

Standard MHA:
Hidden ──► W_K ──► K (cache: n_heads × d_head)
Hidden ──► W_V ──► V (cache: n_heads × d_head)
Total cached per token: 2 × n_heads × d_head

GQA (g groups):
Hidden ──► W_K ──► K (cache: g × d_head)
Hidden ──► W_V ──► V (cache: g × d_head)
Total cached per token: 2 × g × d_head

MLA:
Hidden ──► W_DKV ──► c_KV (cache: d_c, very small)
         ┌──► W_UK ──► K (computed on the fly from c_KV)
c_KV ────┤
         └──► W_UV ──► V (computed on the fly from c_KV)
Total cached per token: d_c (93.3% less than MHA in DeepSeek-V2)

Comparison:

Why MLA is significant:

• DeepSeek-V2 (236B params, 21B active) achieved performance comparable to LLaMA 2 70B
• KV cache compressed by 93.3% compared to standard MHA
• Unlike GQA which restricts which heads share KV, MLA allows all heads to reconstruct rich KV from a shared latent — more expressive
• The up-projection matrices (\(W_h^{UK}\), \(W_h^{UV}\)) are absorbed into the attention computation, adding minimal overhead

Flash Attention (Dao et al., 2022) is an IO-aware exact attention algorithm that is 2-4x faster and uses significantly less memory than standard attention, without any approximation.

The problem: Standard attention materializes the full \(N \times N\) attention matrix in GPU High Bandwidth Memory (HBM), which is:

1. Memory-bound: \(O(N^2)\) memory for the attention matrix
2. IO-bound: Repeatedly reading/writing the large matrix between fast SRAM and slow HBM

Standard Attention (memory bottleneck):
┌─────────┐      ┌─────────────────────────┐
│ GPU SRAM│◄────►│   GPU HBM (slow)        │
│ (fast,  │      │                         │
│  small) │      │  Q: N×d    (read)       │
│         │      │  K: N×d    (read)       │
│         │      │  S: N×N    (write+read) │ ← bottleneck!
│         │      │  P: N×N    (write+read) │ ← bottleneck!
│         │      │  V: N×d    (read)       │
│         │      │  O: N×d    (write)      │
└─────────┘      └─────────────────────────┘

Flash Attention (IO-aware tiling):
┌─────────┐      ┌─────────────────────────┐
│ GPU SRAM│◄────►│   GPU HBM               │
│ (fast)  │      │                         │
│  Q_tile │      │  Q: N×d    (read once)  │
│  K_tile │      │  K: N×d    (read once)  │
│  V_tile │      │  O: N×d    (write once) │
│  [S,P   │      │  (no N×N matrix!)       │
│   tiles]│      │                         │
└─────────┘      └─────────────────────────┘

How it works:

1. Tiling: Split Q, K, V into blocks that fit in SRAM
2. Kernel fusion: Compute attention per block entirely in SRAM (no HBM writes for S, P)
3. Online softmax: Use the online softmax trick (numerically stable running softmax) to compute exact attention without materializing the full matrix
4. Recomputation: In the backward pass, recompute attention from Q, K, V blocks instead of storing the \(N \times N\) matrix — trading compute for memory

Results:

Flash Attention is now the default in PyTorch (torch.nn.functional.scaled_dot_product_attention), Hugging Face, and virtually all modern LLM training/inference frameworks.

Mixture of Experts (MoE) replaces the dense FFN in each Transformer layer with multiple "expert" FFNs, and a router selects a subset of experts per token. This dramatically increases model capacity (total parameters) while keeping compute (active parameters) constant.

Standard Transformer Layer:
Input ──► Self-Attention ──► [One FFN] ──► Output

MoE Transformer Layer:
                              ┌──► Expert 1 (FFN)
Input ──► Self-Attention ──► Router ──► Expert 2 (FFN)  ──► Weighted Sum ──► Output
                              ├──► Expert 3 (FFN)
                              ├──► ...
                              └──► Expert N (FFN)
                           (selects top-k, e.g. k=2)

Router mechanism:

The router produces a probability over all experts, and only the top-\(k\) experts (typically \(k=1\) or \(k=2\)) are activated for each token.

Key components:

1. Expert networks: Identical FFN architectures with independent parameters
2. Router (gating network): Small linear layer that decides which experts to use
3. Load balancing loss: Auxiliary loss to prevent all tokens from routing to the same few experts

Where \(f_i\) is the fraction of tokens routed to expert \(i\) and \(p_i\) is the average routing probability for expert \(i\).

Notable MoE models:

Trade-offs:

• Advantage: 4-8x more parameters at same compute cost
• Disadvantage: Higher memory for storing all expert weights, load balancing challenges, communication overhead in distributed training

RMSNorm (Root Mean Square Normalization) is a simplification of LayerNorm that removes the mean-centering step:

LayerNorm:

RMSNorm:

or equivalently:

Key differences:

Why RMSNorm is preferred:

1. The re-centering (\(x - \mu\)) and shift (\(\beta\)) in LayerNorm are often unnecessary — the re-scaling is what provides the regularization effect
2. Removing mean computation reduces one reduction operation per normalization — significant at scale
3. Empirically matches or exceeds LayerNorm quality

import torch
import torch.nn as nn

class RMSNorm(nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.eps = eps
        self.weight = nn.Parameter(torch.ones(dim))

    def forward(self, x):
        rms = torch.sqrt(torch.mean(x ** 2, dim=-1, keepdim=True) + self.eps)
        return (x / rms) * self.weight

Used in: LLaMA ½/3, PaLM, Mistral, DeepSeek, Gemma — nearly all modern LLMs.

SwiGLU combines the Swish activation with a Gated Linear Unit (GLU) and is used in the FFN of most modern LLMs.

Standard FFN (ReLU):

SwiGLU FFN:

Where:

• \(\text{Swish}(x) = x \cdot \sigma(x)\) (also called SiLU)
• \(\sigma(x)\) is the sigmoid function
• \(\odot\) is element-wise multiplication
• \(W_3\) is an additional "gate" projection

Breakdown:

Standard FFN:
x ──► W1 ──► ReLU ──► W2 ──► output
Parameters: d × 4d + 4d × d = 8d²

SwiGLU FFN:
     ┌──► W1 ──► Swish ──┐
x ───┤                    ├──► ⊙ ──► W2 ──► output
     └──► W3 ─────────────┘
Parameters: d × (8/3)d × 3 ≈ 8d²  (d_ff reduced to 8/3 × d to match param count)

Why SwiGLU is better:

The gating mechanism allows the network to learn which features to pass through, making it more expressive than a simple non-linearity.

Used in: LLaMA ½/3, PaLM, PaLM 2, Mistral, DeepSeek, Gemma.

ALiBi (Press et al., 2022) is a positional encoding method that adds a linear bias to attention scores proportional to the distance between tokens. It uses no positional embeddings at all.

Where the distance matrix is:

And \(m\) is a head-specific slope. For 8 heads: \(m \in \{2^{-1}, 2^{-2}, 2^{-3}, \ldots, 2^{-8}\}\)

For a causal model with 4 positions, the bias matrix looks like:

Head with m=0.5:       Head with m=0.125:
[ 0.0  -∞   -∞   -∞ ]  [ 0.0    -∞     -∞     -∞   ]
[-0.5  0.0  -∞   -∞ ]  [-0.125  0.0    -∞     -∞   ]
[-1.0 -0.5  0.0  -∞ ]  [-0.250 -0.125  0.0    -∞   ]
[-1.5 -1.0 -0.5  0.0]  [-0.375 -0.250 -0.125  0.0  ]

Key insight: Different heads have different slopes — steep slopes create very local attention, gentle slopes allow attending far back. The model gets a "spectrum" of locality.

Advantages over sinusoidal and RoPE:

Used in: BLOOM, MPT.

Research on Transformer interpretability (Clark et al., 2019; Voita et al., 2019) has shown that individual attention heads learn specialized roles:

Common head patterns:

Induction heads (Olsson et al., 2022) are particularly important:

Context: "The cat sat on the mat. The cat"
Induction head detects: "The cat" appeared before → predicts "sat"

Pattern: [A][B] ... [A] → predicts [B]

This is the core mechanism behind in-context learning in LLMs.

Observations:

• Not all heads are equally important — pruning 20-40% of heads often has minimal quality impact
• Earlier layers tend to learn local/syntactic patterns; later layers learn more abstract/semantic patterns
• Some heads are "redundant" and learn overlapping patterns (motivating GQA)

These are the three main paradigms for adapting Transformers to tasks:

Pre-training: Training on a massive corpus (web text, books, code) with a self-supervised objective. This is where the model learns general language understanding.

• Cost: Millions of dollars, weeks on thousands of GPUs
• Data: Trillions of tokens
• Objective: Next token prediction (CLM) or masked language modeling (MLM)

Fine-tuning: Further training the pre-trained model on task-specific data with task-specific supervision.

• Full fine-tuning: Update all parameters
• LoRA / QLoRA: Update only low-rank adapters (0.1-1% of parameters)
• Cost: Hours to days on a few GPUs
• Data: Thousands to millions of labeled examples

In-context learning (ICL): No parameter updates — provide examples in the prompt and the model learns from them on the fly.

Prompt:
"Translate English to French:
 sea otter => loutre de mer
 peppermint => menthe poivrée
 cheese => "

Model output: "fromage"

Key insight: As models scale, in-context learning becomes increasingly powerful, reducing the need for fine-tuning on many tasks.

Self-attention has quadratic complexity in sequence length:

Time complexity: \(O(N^2 \cdot d)\) — computing the \(N \times N\) attention matrix with \(d\)-dimensional vectors.

Memory complexity: \(O(N^2 + Nd)\) — storing the attention matrix plus the Q, K, V, output matrices.

Sequence Length    Attention Matrix     Memory (FP16)
───────────────    ────────────────     ─────────────
512                262K entries         512 KB
2,048              4.2M entries         8 MB
8,192              67M entries          128 MB
32,768             1.07B entries        2 GB
131,072            17.2B entries        32 GB
1,000,000          1T entries           ~2 TB

Why it's a bottleneck:

1. Memory grows quadratically — can't fit in GPU HBM for long sequences
2. Compute grows quadratically — most operations are on the attention matrix
3. For a 4K context model, attention is manageable. For 128K or 1M context, it becomes dominant

Solutions and their trade-offs:

Sliding Window Attention (SWA) restricts each token to only attend to a fixed window of \(w\) preceding tokens, rather than the full context.

Full Causal Attention (seq_len=6):    Sliding Window (w=3):
┌───────────────┐                     ┌───────────────┐
│ 1 0 0 0 0 0   │                     │ 1 0 0 0 0 0   │
│ 1 1 0 0 0 0   │                     │ 1 1 0 0 0 0   │
│ 1 1 1 0 0 0   │                     │ 1 1 1 0 0 0   │
│ 1 1 1 1 0 0   │                     │ 0 1 1 1 0 0   │
│ 1 1 1 1 1 0   │                     │ 0 0 1 1 1 0   │
│ 1 1 1 1 1 1   │                     │ 0 0 0 1 1 1   │
└───────────────┘                     └───────────────┘
Complexity: O(N²)                     Complexity: O(N×w)

How Mistral 7B uses it:

• Window size \(w = 4096\) tokens
• Information propagates beyond the window through stacking layers: after \(L\) layers, the effective receptive field is \(L \times w\) tokens (Mistral: 32 layers × 4096 = 131K effective context)
• Combined with rolling KV cache: only the last \(w\) tokens' KV pairs are stored, enabling constant memory inference regardless of sequence length

Layer 1: Token 10 sees tokens 7-10    (window=4)
Layer 2: Token 10 sees tokens 4-10    (through layer 1 representations)
Layer 3: Token 10 sees tokens 1-10    (full effective context)

Trade-offs:

Learned positional embeddings treat each position as a learnable parameter vector (like a lookup table) rather than computing it with a fixed formula.

import torch.nn as nn

# Learned (BERT, GPT-2)
position_embedding = nn.Embedding(max_seq_len, d_model)
# pe = position_embedding(position_ids)  # Lookup table

# Sinusoidal (original Transformer)
# pe = sin/cos(position / 10000^(2i/d))  # Fixed formula

Why both are largely replaced by RoPE:

• Both are absolute position encodings — they encode \(pos\) directly
• RoPE encodes relative positions in the attention computation, which is more natural for language
• RoPE extrapolates better to longer sequences (with techniques like NTK-aware scaling or YaRN)

The 2017 paper (Vaswani et al.) made several critical contributions:

1. Eliminated recurrence entirely

Previous models (like the Transformer's predecessors) used attention alongside RNNs. The key insight was that attention alone is sufficient — no recurrence needed. This unlocked massive parallelism during training.

2. Established the Transformer block as a universal building block

The pattern [Multi-Head Attention → Add & Norm → FFN → Add & Norm] became the standard building block for nearly all subsequent models in NLP, computer vision (ViT), speech (Whisper), protein folding (AlphaFold), and more.

3. Demonstrated scaling effectiveness

The paper showed that increasing model size (\(d_{model}\), number of layers, number of heads) directly improved translation quality, foreshadowing the scaling laws that would drive the LLM revolution.

4. Practical training recipe

• Label smoothing (0.1)
• Adam optimizer with custom warmup learning rate schedule: \(lr = d_{model}^{-0.5} \cdot \min(\text{step}^{-0.5}, \text{step} \cdot \text{warmup}^{-1.5})\)
• Dropout (0.1) on sublayers and attention weights
• Residual connections + Layer Normalization

5. Benchmark results

Achieved state-of-the-art on WMT 2014 English-to-German and English-to-French translation, training in a fraction of the time of previous best models.

Impact: Every major model since — BERT, GPT, T5, ViT, DALL-E, AlphaFold 2, Whisper, Stable Diffusion — is built on the Transformer.

Transformers process fixed-size tensors, so variable-length sequences in a batch must be padded to the same length and masked to ignore padding tokens.

Batch of 3 sequences (different lengths):
"The cat"       → [The, cat, PAD, PAD, PAD]
"A big dog ran" → [A, big, dog, ran, PAD]
"Hello"         → [Hello, PAD, PAD, PAD, PAD]

Attention mask (1 = real token, 0 = padding):
[1, 1, 0, 0, 0]
[1, 1, 1, 1, 0]
[1, 0, 0, 0, 0]

Two types of masking:

1. Padding mask: Prevents attention to PAD tokens

# Padding mask: (batch, 1, 1, seq_len) for broadcasting
padding_mask = (input_ids != pad_token_id).unsqueeze(1).unsqueeze(2)
# Applied: scores = scores.masked_fill(padding_mask == 0, float('-inf'))

1. Causal mask + Padding mask combined (decoder):

causal_mask = torch.tril(torch.ones(seq_len, seq_len))  # Lower triangular
combined_mask = causal_mask & padding_mask  # Both must be 1 to attend

Efficiency concerns:

• Padding wastes compute on meaningless tokens
• Solutions: Packing (concatenate short sequences to fill the batch dimension), or Flash Attention with variable-length support (no padding needed)

Gradient checkpointing (also called activation checkpointing) trades compute for memory during training by not storing all intermediate activations for the backward pass, instead recomputing them as needed.

Without checkpointing:

• Store activations at every layer during forward pass
• Memory: \(O(L \times N \times d)\) for \(L\) layers
• For a 70B parameter model with long context, this can require hundreds of GB

With checkpointing:

• Only store activations at selected "checkpoint" layers (e.g., every \(\sqrt{L}\) layers)
• During backward pass, recompute activations between checkpoints
• Memory: \(O(\sqrt{L} \times N \times d)\) — typically 60-70% memory reduction
• Cost: ~33% more compute (one extra forward pass per segment)

Without checkpointing (12 layers):
Forward:  L1 → L2 → L3 → L4 → L5 → ... → L12
Stored:   [a1] [a2] [a3] [a4] [a5] ... [a12]  ← stores ALL
Memory:   12 × activation_size

With checkpointing (every 4 layers):
Forward:  L1 → L2 → L3 → L4 → L5 → ... → L12
Stored:   [a1]           [a4]           [a8]  [a12]  ← stores 4 only
Backward: Recompute L9-L12 from a8, then L5-L8 from a4, etc.
Memory:   4 × activation_size + 4 × activation_size (recompute buffer)

# PyTorch gradient checkpointing
from torch.utils.checkpoint import checkpoint

class TransformerBlock(nn.Module):
    def forward(self, x):
        # Wrap sublayers with checkpoint to save memory
        x = x + checkpoint(self.attention, x)
        x = x + checkpoint(self.ffn, x)
        return x

Used in virtually all large-scale Transformer training runs.

Mixed-precision training uses lower-precision formats (FP16 or BF16) for most operations while keeping a master copy of weights in FP32 for numerical stability.

Full precision (FP32):          Mixed precision (BF16/FP16 + FP32):
Weights: FP32 (4 bytes)         Master weights: FP32 (4 bytes)
Activations: FP32               Working weights: BF16 (2 bytes)
Gradients: FP32                 Activations: BF16 (2 bytes)
Optimizer: FP32                 Gradients: BF16 (2 bytes)
                                Optimizer states: FP32 (4 bytes)

Number formats:

BF16 is preferred for LLM training because it has the same exponent range as FP32, avoiding overflow/underflow that can plague FP16.

Benefits:

1. ~2x memory reduction for activations and gradients
2. ~2x throughput on Tensor Cores (NVIDIA GPUs have specialized hardware for FP16/BF16)
3. Enables training larger models that wouldn't fit in memory at FP32

The training loop:

# Simplified mixed-precision training with PyTorch
scaler = torch.amp.GradScaler()  # For FP16 (not needed for BF16)

for data, target in dataloader:
    optimizer.zero_grad()
    with torch.amp.autocast(device_type='cuda', dtype=torch.bfloat16):
        output = model(data)       # Forward in BF16
        loss = criterion(output, target)

    scaler.scale(loss).backward()  # Backward in BF16
    scaler.step(optimizer)         # Update FP32 master weights
    scaler.update()

Transformer training uses a warmup-then-decay learning rate schedule because Adam optimizer with randomly initialized parameters can be unstable with a high initial learning rate.

Original Transformer schedule:

This increases linearly during warmup, then decays proportionally to \(1/\sqrt{\text{step}}\).

Modern LLM schedule (cosine with warmup):

Learning Rate
│
│        ╱╲
│       ╱  ╲
│      ╱    ╲
│     ╱      ╲╲
│    ╱         ╲╲
│   ╱            ╲╲╲
│  ╱                ╲╲╲╲
│ ╱ warmup              ╲╲╲╲╲╲___  min_lr
│╱
└─────────────────────────────── Training Steps
│←warmup→│←──── cosine decay ────→│

Why warmup is necessary:

1. Adam's second moment is not yet estimated: At initialization, Adam has no history of gradient magnitudes. Without warmup, the initial steps can have disproportionately large updates.
2. Random initialization instability: Attention patterns are random at start — large updates can push them into degenerate states.
3. Layer normalization sensitivity: LayerNorm parameters interact with the learning rate; starting too high can cause training divergence.

Typical hyperparameters for modern LLMs:

• Warmup: 1-2% of total training steps (e.g., 2000 steps)
• Peak learning rate: \(1 \times 10^{-4}\) to \(3 \times 10^{-4}\)
• Minimum learning rate: Peak × 0.1
• Decay: Cosine schedule to minimum

PagedAttention (Kwon et al., 2023, used in vLLM) applies virtual memory paging concepts from operating systems to manage the KV cache during LLM inference.

The problem: During serving, each request needs a contiguous block of GPU memory for its KV cache. Due to variable output lengths, this leads to:

• Internal fragmentation: Pre-allocated cache blocks are larger than needed
• External fragmentation: Memory gaps between freed blocks can't be reused
• Typically 60-80% of KV cache memory is wasted

PagedAttention solution:

Traditional KV Cache:
┌──────────────────────────────────────────────────┐
│ Request 1 KV [████████░░░░░]  ← internal frag   │
│ Request 2 KV [█████░░░░░░░░]  ← more waste      │
│ [gap - external fragmentation]                    │
│ Request 3 KV [██████████░░░]                      │
└──────────────────────────────────────────────────┘

PagedAttention (paged KV cache):
┌─────────────────────────────────────────────────────┐
│ Block Table (per request):                          │
│ Req 1: [blk5] → [blk2] → [blk9]                   │
│ Req 2: [blk1] → [blk7]                             │
│ Req 3: [blk3] → [blk8] → [blk4] → [blk6]         │
│                                                     │
│ Physical blocks (non-contiguous, fixed size):       │
│ [blk1][blk2][blk3][blk4][blk5][blk6][blk7][blk8]  │
│  ████   ████  ████  ████  ████  ████  ████  ████   │
│  (all blocks fully utilized, <4% waste)             │
└─────────────────────────────────────────────────────┘

Key ideas:

1. Fixed-size blocks: KV cache is divided into small fixed-size blocks (like memory pages)
2. Block table: Each request has a virtual block table mapping logical → physical blocks
3. Non-contiguous storage: Blocks don't need to be adjacent in GPU memory
4. Copy-on-write: For beam search or parallel sampling, share KV cache blocks and only copy when modified

Results: vLLM achieves 2-4x higher throughput than Hugging Face's text-generation-inference, primarily by eliminating memory waste.

Chinchilla scaling laws (Hoffmann et al., 2022, DeepMind) showed that many LLMs were significantly undertrained — you should scale model size and training data equally.

Key finding: For a given compute budget \(C\), the optimal model size \(N\) and training tokens \(D\) scale as:

The optimal ratio is approximately 20 tokens per parameter.

Impact on the field:

1. Shifted focus from bigger models to more data: Instead of training a 500B model on 300B tokens, train a 70B model on 1.4T tokens — same compute, much better performance
2. LLaMA family: Meta's LLaMA models followed Chinchilla scaling, achieving GPT-3 level performance at 4x fewer parameters
3. Inference efficiency: Smaller, well-trained models are cheaper to serve than larger, undertrained ones
4. Data becomes the bottleneck: High-quality training data is now the scarce resource, not compute

However: In practice, many modern models are intentionally "over-trained" beyond Chinchilla-optimal because a smaller model trained on more data is cheaper to serve (inference cost scales with model size, not training tokens).

RLHF is the technique that transforms a base language model into a helpful, harmless assistant (like ChatGPT or Claude). It has three stages:

Stage 1: Supervised Fine-Tuning (SFT)
┌─────────────────────────────────────┐
│ Base Model ──► Fine-tune on human-  │
│                written examples of  │
│                helpful responses    │
│                                     │
│ Input: "Explain gravity"            │
│ Target: [High-quality explanation]  │
└────────────────┬────────────────────┘
                 │
Stage 2: Reward Model Training
┌────────────────▼────────────────────┐
│ For each prompt, generate multiple  │
│ responses. Humans rank them.        │
│                                     │
│ Prompt → Response A (rank 1) ✓     │
│       → Response B (rank 2)        │
│       → Response C (rank 3)        │
│                                     │
│ Train reward model: R(prompt, resp) │
│ to predict human preferences        │
└────────────────┬────────────────────┘
                 │
Stage 3: RL Optimization (PPO)
┌────────────────▼────────────────────┐
│ Optimize the SFT model to maximize  │
│ reward model score using PPO:       │
│                                     │
│ objective = E[R(x, y)] - β·KL(π||π_ref)│
│                                     │
│ The KL penalty prevents the model   │
│ from diverging too far from the SFT │
│ model (reward hacking prevention)   │
└─────────────────────────────────────┘

The PPO objective:

Alternatives to RLHF:

• DPO (Direct Preference Optimization): Eliminates the reward model and PPO entirely — directly optimizes the policy from preference pairs. Simpler, more stable.
• RLAIF: Uses AI feedback instead of human feedback (constitutional AI approach).

Dense attention computes attention scores between all \(N^2\) token pairs. Sparse attention only computes a subset of these pairs.

Longformer attention pattern (Beltagy et al., 2020):

Combines three patterns:

1. Sliding window (local):     2. Dilated window:       3. Global tokens:
┌─────────────┐               ┌─────────────┐          ┌─────────────┐
│ █ █ ░ ░ ░ ░ │               │ █ ░ █ ░ █ ░ │          │ █ █ █ █ █ █ │ ← [CLS]
│ █ █ █ ░ ░ ░ │               │ ░ █ ░ █ ░ █ │          │ █ █ ░ ░ ░ ░ │
│ ░ █ █ █ ░ ░ │               │ █ ░ █ ░ █ ░ │          │ █ ░ █ ░ ░ ░ │
│ ░ ░ █ █ █ ░ │               │ ░ █ ░ █ ░ █ │          │ █ ░ ░ █ ░ ░ │
│ ░ ░ ░ █ █ █ │               │ █ ░ █ ░ █ ░ │          │ █ ░ ░ ░ █ ░ │
│ ░ ░ ░ ░ █ █ │               │ ░ █ ░ █ ░ █ │          │ █ ░ ░ ░ ░ █ │
└─────────────┘               └─────────────┘          └─────────────┘
O(N×w)                        O(N×w)                   O(N×g)

BigBird (Zaheer et al., 2020, Google) combines:

1. Random attention: Each token attends to \(r\) random tokens
2. Window attention: Local sliding window of size \(w\)
3. Global attention: Selected tokens attend to all others

This achieves \(O(N)\) complexity while theoretically being a universal approximator of sequence functions (proven to be Turing complete).

Modern trend: Rather than using sparse attention patterns, most current LLMs use full attention + Flash Attention for efficiency, combined with RoPE for length extrapolation. Sparse patterns are less common in the latest models.

Emergent abilities (Wei et al., 2022) are capabilities that appear only at a certain model scale — seemingly absent in smaller models and suddenly present in larger ones.

Classic examples:

The debate:

"Emergence is real" view:

• Performance on certain benchmarks shows a sharp phase transition at specific scales
• Qualitatively new capabilities appear (e.g., GPT-4 can pass bar exams, GPT-3 cannot)
• Analogous to phase transitions in physics

"Emergence is a mirage" view (Schaeffer et al., 2023):

• Emergence disappears when using continuous metrics instead of discontinuous ones (e.g., exact-match accuracy shows emergence; token-level accuracy shows smooth improvement)
• The "sudden" appearance is an artifact of the evaluation metric, not the model's capabilities
• Per-token performance improves smoothly with scale

Discontinuous metric (exact match):
Accuracy │
         │                        ●●●●
         │                    ●●●●
         │               ●
         │          ●
         │     ●  ●
         │  ●●●
         └─────────────────── Scale (log)
         Looks like emergence!

Continuous metric (token accuracy):
Accuracy │                         ●●
         │                      ●●
         │                   ●●
         │                ●●
         │             ●●
         │          ●●
         │       ●●
         │    ●●
         │ ●●
         └─────────────────── Scale (log)
         Smooth improvement, no emergence!

Interview takeaway: The truth is likely nuanced — some capabilities do emerge more suddenly than others, but the sharp "phase transition" narrative is partly an artifact of evaluation methodology.

LoRA (Hu et al., 2021) freezes the pre-trained weights and injects trainable low-rank decomposition matrices into attention layers, reducing trainable parameters by 10,000x while matching full fine-tuning quality.

Core idea: Instead of updating a weight matrix \(W \in \mathbb{R}^{d \times d}\), learn a low-rank update \(\Delta W = BA\) where \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times d}\) with rank \(r \ll d\):

Full Fine-tuning:
x ──► [W + ΔW] ──► h       (ΔW: d × d parameters to train)
      d×d trainable

LoRA:
          ┌──► [W] (frozen) ──────┐
x ──►─────┤                        ├──► h = Wx + BAx
          └──► [A]──►[B] (trainable)┘
               d×r    r×d
               (r << d, e.g. r=16)

Parameter savings example (LLaMA 65B):

Why it works:

• Research shows that weight updates during fine-tuning have low intrinsic rank — the update \(\Delta W\) is approximately low-rank even in full fine-tuning
• Rank \(r = 8\) to \(r = 64\) is usually sufficient
• At inference, \(BA\) can be merged into \(W\): \(W' = W + BA\), so there is zero additional latency

QLoRA extends this by quantizing the frozen weights to 4-bit, enabling fine-tuning of 70B+ models on a single 48GB GPU.

These are two fundamental attention mechanisms:

Additive attention (Bahdanau et al., 2015):

Uses a learned feed-forward network to compute compatibility between query and key.

Dot-product (multiplicative) attention (Luong et al., 2015 / Vaswani et al., 2017):

Uses a simple dot product between query and key vectors.

Why Transformers use dot-product attention: It can be computed as a single batched matrix multiplication (\(QK^T\)), which is extremely efficient on GPU hardware. Additive attention requires element-wise operations that are harder to parallelize.

The residual stream view (popularized by Elhage et al., 2021, Anthropic) reframes the Transformer as a series of additive updates to a shared "stream" of information flowing through the network.

Standard view:          Residual stream view:
x → [Attn] → [FFN] →   x ──────────────────────────────────► output
    [Attn] → [FFN] →        +Attn1  +FFN1  +Attn2  +FFN2
    ...                      │       │      │       │
                             ▼       ▼      ▼       ▼
                        x₀ + a₁   + f₁   + a₂    + f₂  = final

Each component READS from the stream and WRITES an additive update.

Mathematically, the final output is:

Why this view is powerful:

1. Linear decomposition: The output is a sum of contributions from each layer and sublayer, making it possible to attribute predictions to specific components

2. Superposition: The residual stream acts as a shared "bandwidth" through which different features communicate across layers

3. Path analysis: Any output logit can be decomposed into contributions from every attention head and FFN layer via path expansion

4. Skip connections matter: Information can "skip" layers entirely — a token embedding at layer 0 directly influences the final output

Key implication: Attention heads and FFN layers don't transform a hidden state in sequence — they each read from and write to a shared communication channel. This is the foundation of mechanistic interpretability research.

Multi-modal Transformers process different input types (text, images, audio) by converting them into a shared token representation space.

Common approaches:

1. Separate encoders + cross-attention (Flamingo, early approaches):

Image ──► [Vision Encoder] ──► Image tokens ──┐
                                                ├──► [Cross-Attention in LLM] ──► Output
Text  ──► [Text Tokenizer] ──► Text tokens  ──┘

2. Early fusion — interleaved tokens (GPT-4V, Gemini):

Image ──► [Vision Encoder] ──► [Projector] ──► Image tokens
                                                 ↓
Input sequence: [text_1] [text_2] [img_1] [img_2] ... [img_N] [text_3] ...
                                                 ↓
                                        [Standard Transformer Decoder]

3. Vision Transformer (ViT) as encoder:

An image is split into patches, each treated as a "token":

Image (224×224) → Split into 16×16 patches → 196 patches
Each patch → Linear projection → Patch embedding (like a word embedding)
+ Positional embedding → Feed into standard Transformer encoder

LLaVA architecture (popular open-source approach):

Image ──► CLIP ViT ──► MLP Projector ──► Visual tokens
                                          ↓
Text  ──► Tokenizer ─────────────────► Text tokens
                                          ↓
Combined: [visual tokens] + [text tokens] ──► LLaMA decoder ──► Response

Key insight: The Transformer's self-attention is modality-agnostic — it operates on sequences of vectors regardless of whether they originated from text, images, audio, or video. The "magic" is in the tokenization/projection step that maps different modalities into a shared vector space.

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) uses a small "draft" model to predict multiple tokens, then verifies them in parallel with the large model — achieving 2-3x speedup with zero quality loss.

The problem: Autoregressive generation is memory-bound, not compute-bound. Each step requires loading the entire model's weights from memory to generate a single token. The GPU compute is vastly underutilized.

How it works:

Standard decoding (1 token per forward pass):
Step 1: Large model → token 1
Step 2: Large model → token 2
Step 3: Large model → token 3
...N steps for N tokens

Speculative decoding:
Step 1: Draft model (fast) → [tok1, tok2, tok3, tok4, tok5] (γ draft tokens)
Step 2: Large model verifies ALL 5 in ONE forward pass
        → tok1 ✓, tok2 ✓, tok3 ✓, tok4 ✗ (reject, sample from large model)
Result: 4 tokens in 2 forward passes instead of 4!

Acceptance criterion: For each draft token, accept it if the large model's probability is at least as high as the draft model's. If rejected, sample from a modified distribution that corrects for the draft model's error. This guarantees the exact same output distribution as the large model alone.

Key requirements:

1. Draft model must be much faster (e.g., 7B drafting for 70B)
2. Draft model should have reasonable agreement with the large model (higher acceptance rate → more speedup)
3. Verification is parallel — verifying \(\gamma\) tokens costs the same as generating 1 token (single forward pass)

Typical speedup: 2-3x with ~70-80% acceptance rate.

Timeline of Sequence Models:

1986        1997        2014          2017           2023+
┌────┐    ┌──────┐    ┌──────────┐  ┌────────────┐  ┌──────┐
│ RNN│───►│ LSTM │───►│ Attention│─►│ Transformer│─►│ SSMs │
└────┘    └──────┘    │ + RNN    │  │            │  │(Mamba)│
                      └──────────┘  └────────────┘  └──────┘

State Space Models (Mamba):

SSMs (Gu & Dao, 2023) are the main challenger to Transformers. They process sequences with a fixed-size recurrent state, like RNNs, but can be parallelized during training via a convolution formulation:

• Training: Computed as a convolution (parallelizable like Transformers)
• Inference: Computed as a recurrence (O(1) per step, like RNNs)
• No attention matrix: Memory doesn't grow with sequence length

Current status: Transformers still dominate for most tasks, but hybrid architectures (Transformer + SSM layers, e.g., Jamba) and pure SSMs are rapidly improving. The key advantage of SSMs is linear scaling with sequence length at inference.

The softmax bottleneck (Yang et al., 2018) is a fundamental expressiveness limitation: the final softmax layer restricts the model to producing log-probability matrices of rank at most \(d_{model}\), even when the true distribution requires higher rank.

The problem:

The output probability for token \(w\) given context \(c\) is:

Where \(h_c \in \mathbb{R}^d\) is the context representation and \(e_w \in \mathbb{R}^d\) is the token embedding. The log-probability matrix \(A\) where \(A_{c,w} = \log P(w|c)\) has rank at most \(d\) — the hidden dimension.

If the true language distribution requires a rank higher than \(d\), no model with this architecture can perfectly represent it.

Practical implications:

• Languages have highly context-dependent word distributions that may require very high rank
• The bottleneck is more severe for smaller \(d_{model}\) and larger vocabularies
• This is one reason why increasing \(d_{model}\) improves perplexity

Solutions:

1. Mixture of Softmaxes (MoS): Use multiple softmax outputs and mix them
2. Larger hidden dimensions: Modern LLMs use \(d_{model}\) = 4096-8192
3. Tied vs untied embeddings: Tying input/output embeddings saves parameters but worsens the bottleneck (used in smaller models; larger models often untie)

Absolute positional encodings assign a fixed vector to each position index. Relative positional encodings encode the distance between token pairs.

Absolute examples:

• Sinusoidal (Vaswani et al.): \(PE(pos, i) = \sin(pos / 10000^{2i/d})\)
• Learned (BERT, GPT-2): Lookup table trained with the model

Relative examples:

• RoPE (LLaMA, Mistral): Rotates Q, K so dot product depends on \((m-n)\) not \(m\) or \(n\) separately
• ALiBi (BLOOM, MPT): Adds \(-m|i-j|\) bias to attention scores
• Relative position bias (T5): Learned scalar bias per relative distance, added to attention logits
• XL-style (Transformer-XL): Modifies attention score to include relative position embedding

Why relative is better: In language, the relationship between tokens typically depends on their distance (the word 3 positions back), not their absolute position (the 47^th word). Relative encodings directly model this.

Router (gating network) decides which expert(s) each token uses:

Expert collapse: A failure mode where the router learns to send most tokens to the same few experts, leaving others undertrained:

Healthy routing:                    Expert collapse:
Expert 1: ████████ (15%)           Expert 1: ████████████████ (60%)
Expert 2: ████████ (14%)           Expert 2: ██ (5%)
Expert 3: ███████ (13%)            Expert 3: █ (2%)
Expert 4: ████████ (12%)           Expert 4: ████████ (25%)
Expert 5: ███████ (12%)            Expert 5: █ (3%)
Expert 6: ████████ (12%)           Expert 6: █ (2%)
Expert 7: ███████ (11%)            Expert 7: █ (2%)
Expert 8: ████████ (11%)           Expert 8: █ (1%)

Load balancing loss (auxiliary loss to prevent collapse):

Where:

• \(f_i\) = fraction of tokens routed to expert \(i\) in the batch
• \(p_i\) = average router probability assigned to expert \(i\)
• \(N\) = number of experts
• \(\alpha\) = balancing coefficient (typically 0.01)

This loss is minimized when \(f_i = p_i = 1/N\) for all experts (uniform routing).

Advanced routing strategies:

DeepSeek-V3 notably uses auxiliary-loss-free load balancing with per-expert bias terms adjusted during training, avoiding the quality trade-off from the auxiliary loss.

Autoregressive generation produces one token at a time, using each predicted token as input for the next step:

Prompt: "The cat"

Step 1: Input: [The, cat]
        Model output: probability distribution over vocab
        Sample/Argmax → "sat"
        Append to sequence

Step 2: Input: [The, cat, sat]  (or just "sat" with KV cache)
        Model output: probability distribution
        → "on"

Step 3: Input: [The, cat, sat, on]
        → "the"

Step 4: Input: [The, cat, sat, on, the]
        → "mat"

...continue until <EOS> or max length

Decoding strategies:

With KV cache (efficient inference):

Step 1: Process full prompt [The, cat] → cache K1,V1 and K2,V2
        Predict → "sat"

Step 2: Only process new token [sat]
        Use cached K1,V1, K2,V2 for attention
        Cache K3,V3
        Predict → "on"

Step 3: Only process [on]
        Use cached K1-K3, V1-V3
        Cache K4,V4
        Predict → "the"

Each step only requires computing Q, K, V for the new token — the KV cache makes this \(O(1)\) per token (instead of reprocessing the entire sequence).

Attention sinks (Xiao et al., 2023) is the phenomenon where Transformer LLMs disproportionately attend to the first token (or BOS token), regardless of its semantic relevance.

Why it happens:

Softmax requires attention weights to sum to 1. When no tokens are particularly relevant to the query, the model needs somewhere to "dump" the excess attention weight. The first token becomes a learned "sink" because:

1. It's always present (every sequence starts with it)
2. Its KV representation becomes optimized during training to absorb "unused" attention
3. This acts like a "no-op" — attending to the sink doesn't corrupt the representation

Typical attention pattern in Layer 15, Head 3:
Token:    [BOS]  The   cat   sat   on   the   mat
Weight:   0.35   0.05  0.15  0.20  0.05 0.10  0.10
          ^^^^
          Attention sink — high weight despite no semantic relevance

Practical impact for streaming/long-context inference:

If you use a sliding window KV cache and evict the first token, quality degrades catastrophically. The solution:

Naive sliding window (breaks!):
Keep tokens: [tok_100, tok_101, ..., tok_200]
← First token evicted → quality crashes

StreamingLLM fix:
Keep tokens: [BOS, tok_100, tok_101, ..., tok_200]
← Always keep the first few "sink" tokens → quality preserved

StreamingLLM (Xiao et al., 2023) shows that keeping just the first 4 attention sink tokens + a sliding window enables infinite length generation with constant memory and stable quality.

Code generation uses the same Transformer architecture but requires special considerations:

1. Tokenization differences:

Natural language: "The cat sat" → ["The", " cat", " sat"] (3 tokens)

Code: "def hello_world():" → ["def", " hello", "_world", "():", ...] (varies)

Problem: Standard BPE can split identifiers badly:
"getUserName" → ["get", "User", "Name"] (ok)
"XMLParser"   → ["XML", "Parser"] (ok)
"x_coord_3d"  → ["x", "_coord", "_", "3", "d"] (fragmented)

Code-specific tokenizers use larger vocabularies and include common code patterns (indentation, brackets, keywords) as single tokens.

2. Indentation sensitivity (Python):

# Indentation is semantically meaningful
if True:
    print("in block")    # ← 4 spaces = part of if-block
print("outside")          # ← no indent = outside if-block

Models must learn that whitespace changes program semantics, unlike natural language.

3. Long-range structural dependencies:

class MyClass:                    # Line 1
    def __init__(self, x):        # Line 2 - must match class indent
        self.x = x                # Line 3 - must reference self
    ...
    # 200 lines later
    def method(self):             # Must know self.x exists from line 3
        return self.x + 1

4. Infilling (Fill-in-the-Middle):

Code models often support FIM — predicting code that goes between a prefix and suffix:

Prefix: "def add(a, b):\n"
Suffix: "\n    return result"
Model fills: "    result = a + b"

This requires special training with <PRE>, <SUF>, <MID> tokens.

Notable code models: Codex (OpenAI), CodeLlama (Meta), StarCoder (BigCode), DeepSeek-Coder, Claude.

After concatenating all head outputs, the output projection \(W^O \in \mathbb{R}^{hd_v \times d_{model}}\) maps the concatenated multi-head result back to \(d_{model}\) dimensions:

Why it exists:

1. Mixes head outputs: Each head operates in its own subspace. \(W^O\) allows the model to learn how to combine information from different heads — which head's output matters most for each feature dimension.

2. Dimension matching: Ensures the attention output has the same dimension (\(d_{model}\)) as the input, enabling the residual connection.

3. Added expressiveness: Without \(W^O\), the output is simply a concatenation of independent subspace projections. \(W^O\) allows cross-head interaction.

Can it be removed?

• If \(d_v = d_{model} / h\) (standard case), the concatenation already has dimension \(d_{model}\), so \(W^O\) could theoretically be an identity matrix
• Experiments show removing \(W^O\) causes a noticeable (but not catastrophic) quality drop
• In practice, \(W^O\) adds relatively few parameters (\(d_{model}^2\)) and is always included

DeepNorm (Wang et al., 2022, Microsoft) is a normalization method that enables training very deep Transformers (up to 1000 layers) by combining a modified residual connection with specific initialization.

The problem: Standard Post-Norm Transformers diverge beyond ~100 layers. Pre-Norm is stable but performs worse. Neither scales well to extreme depths.

DeepNorm formula:

Where \(\alpha > 1\) is a constant that upweights the residual connection, and sublayer weights are initialized with a smaller scale \(\beta < 1\).

For a Transformer with \(N\) encoder and \(M\) decoder layers:

• Encoder: \(\alpha = (2N)^{1/4}\), \(\beta = (8N)^{-1/4}\)
• Decoder: \(\alpha = (2M)^{1/4}\), \(\beta = (8M)^{-1/4}\)

Intuition:

• \(\alpha > 1\) makes the residual connection stronger → gradient flows more easily
• \(\beta < 1\) makes sublayer outputs smaller → prevents destabilizing the residual stream
• As depth increases, \(\alpha\) grows and \(\beta\) shrinks, automatically compensating

Results: Successfully trained a 1000-layer Transformer (2500 attention/FFN sublayers) — previously impossible. The resulting model showed consistent improvements from adding more layers.

Beam search maintains \(B\) (beam width) candidate sequences and expands the most promising ones:

Beam search (B=2):
Step 1: "The" → P=0.3, "A" → P=0.2
Step 2: "The cat" → P=0.12, "The dog" → P=0.09
Step 3: "The cat sat" → P=0.05, "The cat is" → P=0.04
→ Final: highest cumulative probability sequence

Sampling randomly draws from the probability distribution:

Sampling (temperature=0.8, top_p=0.9):
Step 1: Sample from top-90% mass → "The" (drawn)
Step 2: Sample → "happy" (drawn)
Step 3: Sample → "penguin" (drawn)
→ Output: "The happy penguin" (creative, diverse)

When to use each:

• Beam search: Machine translation, summarization, structured output, code completion (need precision)
• Sampling: Creative writing, conversational AI, brainstorming (need diversity)
• Modern LLM chatbots: Almost always use sampling with temperature + top-p. Beam search is rarely used for open-ended generation in modern systems.

Self-attention can be viewed as message passing on a fully connected graph where each token is a node.

Standard Graph Attention Network (GAT):

Where \(\mathcal{N}(i)\) is the neighborhood of node \(i\) and \(\alpha_{ij}\) are attention-based weights.

Transformer self-attention:

Where \(\alpha_{ij} = \text{softmax}_j(q_i^T k_j / \sqrt{d_k})\) — attention over ALL nodes (fully connected graph).

GNN (sparse graph):           Transformer (complete graph):
○─○─○                         ○═══○═══○
│ │                           ║ ╲ ║ ╱ ║
○─○                           ○═══○═══○
Each node attends to           Every node attends to
its neighbors only             every other node

Implications:

• Transformers are a special case of GNNs on complete graphs
• Sparse attention patterns (Longformer, BigBird) make Transformers more like traditional GNNs
• Graph Transformers explicitly combine both: use full attention but incorporate graph structure via edge features or structural encodings

Vocabulary size affects multiple aspects of Transformer models:

Embedding layer parameters: \(V \times d_{model}\)

Trade-offs of vocabulary size:

Key insight — fertility (tokens per word):

English "artificial intelligence" with different vocab sizes:
V=32K:  ["art", "ific", "ial", " intelligence"]     → 4 tokens
V=128K: ["artificial", " intelligence"]              → 2 tokens

2x fewer tokens = 2x faster inference, 2x longer effective context!

Modern trend: Vocabulary sizes are increasing (LLaMA 3: 128K, Gemini: 256K) because the benefits of shorter sequences (faster inference, longer context) outweigh the slightly larger embedding layer.

Standard softmax attention can be viewed through a kernel lens, and this perspective enables linear-complexity approximations.

Standard attention as a kernel:

Where \(\kappa(q, k) = \exp(q^T k / \sqrt{d})\) is the softmax kernel.

Linear attention replaces this kernel with a decomposable feature map \(\phi\):

This allows rewriting attention as:

The key: \(\sum_j \phi(k_j) v_j^T\) can be precomputed once and shared across all queries!

Complexity comparison:

Standard attention:
Q(N×d) × K^T(d×N) = S(N×N) × V(N×d) = O(N²d)
                       ^^^^
                       N×N matrix (bottleneck)

Linear attention:
K^T(d×N) × V(N×d) = KV(d×d)  ← precompute once: O(Nd²)
Q(N×d) × KV(d×d) = O(Nd²)    ← for all queries
Total: O(Nd²) instead of O(N²d)

When \(d \ll N\) (which is true for long sequences), this is a significant speedup.

Examples: Performer (random features), Linear Transformers (ELU+1 feature map), cosFormer.

Limitation: Linear attention approximations generally perform worse than exact attention for language modeling, which is why Flash Attention (exact but IO-efficient) is preferred in practice.

Extending Transformer context windows from 4K to 128K+ tokens requires multiple complementary techniques:

1. RoPE scaling (position interpolation):

Instead of extrapolating to unseen positions, scale existing positions to fit within the training range:

Variants: - Linear interpolation (Meta): Simple division - NTK-aware scaling: Adjusts the base frequency of RoPE - YaRN (Yet Another RoPE extensioN): Combines NTK scaling with attention temperature scaling, achieves best extrapolation

2. Continued pre-training on long data:

Train for a small number of steps on long documents after initial training (e.g., LLaMA 2 Long: additional training on 32K+ documents).

3. Flash Attention: Makes \(O(N^2)\) attention feasible by eliminating memory bottleneck.

4. Ring Attention: Distributes the sequence across multiple GPUs in a ring topology:

GPU 1: tokens 1-32K      ←→  GPU 2: tokens 32K-64K
   ↑                              ↑
   └──── GPU 4: tokens 96K-128K ←→ GPU 3: tokens 64K-96K

Each GPU holds its portion's KV cache.
Attention blocks circulate around the ring.
Total memory: O(N/P) per GPU for P GPUs.

5. Architectural choices:

• Sliding window attention (some layers local, some global)
• Sparse attention patterns for very long sequences
• Efficient KV cache management (GQA, MLA, quantized KV cache)

Current state-of-the-art context lengths:

Weight tying shares the same weight matrix between the input embedding layer and the output (pre-softmax) projection:

Instead of having separate input embedding (\(E\)) and output projection (\(W_{out}\)), we set \(W_{out} = E^T\).

Benefits:

1. Parameter reduction: For LLaMA with \(V = 32000\), \(d = 4096\): saves 131M parameters
2. Regularization: Forces the model to use a consistent representation space for input and output
3. Geometric consistency: Tokens with similar meanings should be close in embedding space — tying ensures input similarity translates to output similarity

When it's used:

Modern trend: Larger models tend not to tie weights because:

• The embedding parameter saving is proportionally small (\(V \times d\) vs billions of parameters)
• Untied embeddings allow the input and output spaces to specialize independently
• Avoids the softmax bottleneck limitation imposed by shared representations

The original Transformer uses the "Noam" schedule (named after the first author Noam Shazeer):

Two phases:

During warmup (\(\text{step} < \text{warmup\_steps}\)):

Linear increase from 0 to peak.

After warmup (\(\text{step} \geq \text{warmup\_steps}\)):

Decay proportional to \(1/\sqrt{\text{step}}\).

Why \(d_{model}^{-0.5}\)? The peak learning rate scales inversely with model size — larger models use smaller learning rates, which is necessary because gradient magnitudes scale with dimension.

def noam_schedule(step, d_model=512, warmup_steps=4000):
    """Original Transformer learning rate schedule."""
    step = max(step, 1)  # Avoid division by zero
    return d_model ** (-0.5) * min(step ** (-0.5), step * warmup_steps ** (-1.5))

# Peak LR at step=warmup_steps:
# For d_model=512, warmup=4000: peak ≈ 0.00070

Modern replacement — cosine with warmup (used by nearly all current LLMs) is simpler and often performs better, with explicit control over peak and minimum learning rates.

Let's trace the forward pass for a decoder-only model (like GPT/LLaMA) on the input "The cat":

INPUT: "The cat"

┌─ Step 1: TOKENIZATION ─────────────────────────────────┐
│ "The cat" → token IDs: [464, 3797]                     │
└────────────────────────────────┬────────────────────────┘
                                 ↓
┌─ Step 2: EMBEDDING ────────────────────────────────────┐
│ Token embedding: E[464] → [0.12, -0.34, ...]  (d=4096)│
│                  E[3797] → [-0.56, 0.78, ...]          │
│ + Positional encoding (RoPE applied later in attention) │
│ → X ∈ ℝ^(2 × 4096)                                    │
└────────────────────────────────┬────────────────────────┘
                                 ↓
┌─ Step 3: TRANSFORMER LAYERS (×32) ─────────────────────┐
│                                                         │
│  For each layer l = 1 to 32:                           │
│                                                         │
│  3a. RMSNorm(X)                                        │
│                                                         │
│  3b. Multi-Head Self-Attention:                         │
│      Q = X·W_Q, K = X·W_K, V = X·W_V                  │
│      Apply RoPE to Q, K                                │
│      Apply causal mask (token 1 can't see token 2)     │
│      Attn = softmax(QK^T/√d_k) · V                    │
│      X = X + Attn·W_O          (residual connection)   │
│                                                         │
│  3c. RMSNorm(X)                                        │
│                                                         │
│  3d. Feed-Forward (SwiGLU):                            │
│      X = X + W2·(Swish(W1·X) ⊙ W3·X)  (residual)     │
│                                                         │
└────────────────────────────────┬────────────────────────┘
                                 ↓
┌─ Step 4: FINAL NORM + OUTPUT ──────────────────────────┐
│ X = RMSNorm(X)                                         │
│ logits = X · E^T    (project to vocabulary, ℝ^(2×32K)) │
│ Take logits for LAST position (token 2 = "cat")        │
│ probs = softmax(logits[-1])                            │
│ next_token = sample(probs)  → e.g., "sat"             │
└────────────────────────────────────────────────────────┘

Key details:

• Only the last position's logits matter for next-token prediction
• The causal mask ensures each position can only attend to itself and earlier positions
• RoPE is applied inside the attention computation, not at the input
• The residual stream carries information from embedding to output, with each layer making additive updates

Quantization reduces the precision of model weights (and optionally activations) to decrease memory usage and increase inference speed.

Original (FP16):    16 bits per weight → 140 GB for 70B model
INT8 quantization:   8 bits per weight →  70 GB
INT4 quantization:   4 bits per weight →  35 GB

Types of quantization:

GPTQ (Frantar et al., 2023):

• Post-training quantization using second-order information (approximate Hessian)
• Quantizes weights one layer at a time, using calibration data to minimize output error
• Compensates for quantization error in remaining weights (Optimal Brain Quantization)

Key insight — not all weights are equal:

Weight distribution:
│     ╱╲
│    ╱  ╲       Most weights are small
│   ╱    ╲      and can be quantized
│  ╱      ╲     aggressively
│ ╱        ╲
─┴──●─●──●──●── A few "outlier" weights are
     ↑↑↑↑       very large and must stay
   outliers     high-precision (mixed precision)

LLM.int8() keeps outlier features (>6σ) in FP16 and quantizes the rest to INT8 — a mixed-precision approach that maintains quality even for very large models.

Differential Transformer (Ye et al., 2024, Microsoft) modifies the attention mechanism to compute the difference between two softmax attention maps, effectively canceling out noise:

Where \(Q_1, Q_2, K_1, K_2\) are two sets of query/key projections and \(\lambda\) is a learnable scalar.

Intuition:

Standard attention:        Differential attention:
┌─────────────────┐       ┌─────────────────┐
│ Signal + Noise  │       │ (Signal + Noise) │
│ ████░░░░░░░░░░  │       │  - λ · Noise     │
│ (noisy weights) │       │ = Clean Signal    │
└─────────────────┘       └─────────────────┘

The second attention map acts as a "noise estimator" — by subtracting it, the model amplifies true signal and suppresses irrelevant attention patterns.

Benefits:

1. Reduced attention noise: Less attention to irrelevant tokens
2. Better long-context performance: Cleaner attention maps degrade less with sequence length
3. Fewer hallucinations: More focused attention leads to more faithful outputs
4. Fewer attention heads needed: Each differential head is more expressive

This is one of the newer architectural innovations gaining attention in the research community.

Vision Transformer (ViT) (Dosovitskiy et al., 2020, Google) applies the standard Transformer encoder to image patches:

Image (224×224×3)
     │
     ▼
Split into 16×16 patches → 196 patches (14×14 grid)
     │
     ▼
Flatten each patch: 16×16×3 = 768 dims
     │
     ▼
Linear projection → Patch embeddings (768 or 1024 dims)
     │
     ▼
Prepend [CLS] token + Add positional embeddings
     │
     ▼
Standard Transformer Encoder (12-24 layers)
     │
     ▼
[CLS] token output → Classification head

Key finding: ViT underperforms CNNs on small datasets (ImageNet alone) because it lacks the spatial inductive bias. But when pre-trained on large datasets (300M+ images), ViT significantly outperforms CNNs, showing that scale can replace inductive bias.

Modern hybrids combine both: early CNN layers for local features, then Transformer layers for global reasoning (e.g., CoAtNet, EfficientFormer).

These three parameters control the randomness and diversity of generated text:

Temperature (\(T\)): Scales logits before softmax.

• \(T \rightarrow 0\): Greedy (deterministic, always picks highest probability)
• \(T = 1\): Standard softmax (original distribution)
• \(T > 1\): Flatter distribution (more random)

Top-k: After temperature scaling, keep only the \(k\) highest-probability tokens, renormalize.

Top-p (nucleus sampling): Keep the smallest set of tokens whose cumulative probability \(\geq p\), renormalize.

Original logits: [cat: 5.0, dog: 3.0, fish: 2.0, xyz: 0.1, ...]

After temperature=0.5:
Softmax → [cat: 0.85, dog: 0.10, fish: 0.04, xyz: 0.00, ...]

After top-k=3:
→ [cat: 0.86, dog: 0.10, fish: 0.04]  (renormalized)

After top-p=0.9:
→ [cat: 0.90, dog: 0.10]  (cumsum: 0.85+0.10=0.95 ≥ 0.9, keep 2)

Typical settings:

Key insight: Temperature controls the shape of the distribution; top-k and top-p control the truncation. They are applied sequentially: temperature first, then top-k or top-p.

LLM inference has two distinct phases with very different computational profiles:

Prefill phase (processing the prompt):

• Processes ALL prompt tokens in parallel (like training)
• Compute-bound: Large matrix multiplications (batch of tokens × model weights)
• Populates the KV cache for all prompt tokens
• Time proportional to prompt length

Decode phase (generating output):

• Generates ONE token at a time
• Memory-bound: Must load entire model weights from memory for each single token
• GPU compute is vastly underutilized (computing with 1 token instead of thousands)
• Time proportional to output length

Prompt: "Explain quantum computing in 3 sentences" (8 tokens)
Output: "Quantum computing uses..." (50 tokens)

┌─ PREFILL ───────────────────────────────┐
│ Process 8 tokens in parallel             │
│ Time: ~50ms                              │
│ GPU utilization: HIGH (compute-bound)    │
│ Arithmetic intensity: HIGH               │
│ Fills KV cache for 8 positions           │
└──────────────────────────────────────────┘

┌─ DECODE ────────────────────────────────┐
│ Generate 50 tokens sequentially          │
│ Time: ~2000ms (40ms per token)           │
│ GPU utilization: LOW (memory-bound)      │
│ Arithmetic intensity: LOW                │
│ Each step: 1 new token, full model load  │
└──────────────────────────────────────────┘

Key metrics:

Time to First Token (TTFT) = prefill time. Time per Output Token (TPOT) = decode time per token. These are the two key latency metrics for LLM serving.