Mastering RNNs, CNNs, and Transformers: The Ultimate 100-Question Exam Preparation Guide

Table of Contents

• Introduction: The Three Pillars of Modern Deep Learning

• Section 1: Recurrent Neural Networks (RNNs) - Questions 1-33

• Section 2: Convolutional Neural Networks (CNNs) - Questions 34-66

• Section 3: Transformers - Questions 67-100

• Section 4: Comparative Analysis & Decision Roadmap

• Section 5: Strategic Exam Preparation Tips

• Conclusion: From Theory to Mastery

Introduction: The Three Pillars of Modern Deep Learning

In the rapidly evolving landscape of artificial intelligence, three architectures stand as the foundational pillars of deep learning: Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers. Whether you are preparing for a university exam, a technical interview at a top-tier AI company, or simply aiming to solidify your understanding, mastering these three paradigms is non-negotiable.

This guide is not just a list of questions - it is a strategic weapon. Each of the 100 questions has been carefully curated to test not only your recall but your ability to reason, compare, and apply these architectures in real-world scenarios. We have structured the content to mirror the depth and rigor expected in advanced machine learning courses and professional certifications.

By the end of this guide, you will be able to:

• Explain the core mathematical principles behind RNNs, CNNs, and Transformers.

• Identify the strengths and weaknesses of each architecture for specific tasks.

• Navigate the trade-offs between computational cost, accuracy, and scalability.

• Confidently answer any exam question thrown your way.

Section 1: Recurrent Neural Networks (RNNs) - Questions 1-33

Foundational Concepts (Questions 1-10)

Q: What is the fundamental difference between a feedforward neural network and a recurrent neural network? A: A feedforward network processes inputs independently, with no memory of previous inputs. An RNN, however, maintains a hidden state that is passed from one time step to the next, enabling it to process sequences of variable length. This recurrence gives RNNs a form of "memory."

Q: Derive the update equation for a simple RNN hidden state. A: The hidden state at time step t is computed as: h_t = tanh(W_h h_{t-1} + W_x x_t + b), where W_h is the recurrent weight matrix, W_x is the input weight matrix, x_t is the input at time t, and b is the bias. The tanh activation introduces non-linearity and keeps values bounded.

Q: Explain the vanishing gradient problem in RNNs. Why is it particularly severe in vanilla RNNs? A: During backpropagation through time (BPTT), gradients are multiplied by the recurrent weight matrix at each time step. If the eigenvalues of this matrix are less than 1, gradients shrink exponentially, making it impossible for the network to learn long-range dependencies. This is the vanishing gradient problem. It is severe because the multiplication chain is as long as the sequence length.

Q: What is the exploding gradient problem, and how is it typically mitigated? A: The opposite of vanishing gradients: when eigenvalues of the recurrent weight matrix are greater than 1, gradients grow exponentially, causing unstable updates. Mitigation strategies include gradient clipping (scaling gradients to a maximum norm) and using architectures like LSTMs or GRUs.

Q: What is Backpropagation Through Time (BPTT)? A: BPTT is the algorithm used to train RNNs. It unfolds the network in time, treating each time step as a layer, and then applies standard backpropagation. The gradients are accumulated across all time steps before updating the weights.

Q: Compare the LSTM and GRU architectures. When would you choose one over the other? A: LSTMs have three gates (input, forget, output) and a cell state, offering more control over memory. GRUs have two gates (reset and update) and are simpler, with fewer parameters. Choose LSTM for tasks requiring very long-term dependencies (e.g., language modeling). Choose GRU for smaller datasets or when computational efficiency is critical.

Q: What is the role of the forget gate in an LSTM? A: The forget gate decides which information from the previous cell state should be discarded. It outputs a value between 0 (completely forget) and 1 (completely retain) based on the previous hidden state and current input, using a sigmoid activation.

Q: Explain bidirectional RNNs. What advantage do they offer over unidirectional RNNs? A: A bidirectional RNN processes the sequence in both forward and backward directions, concatenating or summing the hidden states from both passes. This allows the network to consider future context when making predictions at each time step, which is crucial for tasks like named entity recognition or part-of-speech tagging.

Q: What is teacher forcing in RNN training? A: Teacher forcing is a training technique where, at each time step, the model receives the true previous output (from the training data) as input, rather than its own predicted output. This stabilizes training and speeds up convergence, but can lead to exposure bias during inference.

Q: How do you handle variable-length sequences in an RNN? A: Common strategies include: (1) Padding shorter sequences to the length of the longest sequence, (2) Using masking to ignore padded positions during loss computation, and (3) Bucketing - grouping sequences of similar lengths together to minimize padding.

Advanced RNN Concepts (Questions 11-22)

Q: Explain the concept of "depth" in RNNs. How does stacking RNN layers help? A: Stacked RNNs have multiple layers of recurrent units. Each layer processes the hidden state from the layer below, allowing the network to learn hierarchical representations. Lower layers capture local, short-term patterns, while higher layers capture more abstract, long-term dependencies.

Q: What is the difference between many-to-one, one-to-many, and many-to-many RNN architectures? A: Many-to-one (e.g., sentiment classification): input sequence, single output. One-to-many (e.g., image captioning): single input, output sequence. Many-to-many (e.g., machine translation): input sequence, output sequence, possibly of different lengths (seq2seq).

Q: Describe the encoder-decoder (seq2seq) architecture. What is its primary application? A: The encoder processes the input sequence into a fixed-length context vector (the final hidden state). The decoder then generates the output sequence, conditioned on this context vector. This is the foundation of neural machine translation, text summarization, and conversational AI.

Q: What is the attention mechanism in the context of RNNs? Why was it revolutionary? A: Attention allows the decoder to "look at" different parts of the encoder's hidden states at each decoding step, rather than relying solely on a single context vector. This solved the bottleneck problem in seq2seq models and dramatically improved performance on long sequences.

Q: How does the Peephole LSTM differ from the standard LSTM? A: In a Peephole LSTM, the gate layers also receive the cell state as input, in addition to the hidden state and input. This allows the gates to "peek" at the cell state, potentially improving the model's ability to learn precise timing and counting tasks.

Q: What is the primary limitation of RNNs compared to Transformers for sequence modeling? A: RNNs process sequences sequentially, making them inherently difficult to parallelize. This leads to long training times for long sequences. Additionally, even with LSTMs, capturing very long-range dependencies remains challenging compared to the self-attention mechanism in Transformers.

Q: Explain the concept of "truncated BPTT." When is it used? A: Truncated BPTT processes the sequence in chunks (e.g., 100 time steps at a time), backpropagating gradients only within each chunk. This reduces memory and computational cost, especially for very long sequences. However, it limits the network's ability to learn dependencies longer than the chunk size.

Q: What is the role of the output gate in an LSTM? A: The output gate controls what information from the cell state is exposed to the hidden state (and thus to the next layer or output). It uses a sigmoid to decide which parts of the cell state to output, then multiplies the tanh of the cell state by this gate's output.

Q: How do you initialize the hidden state of an RNN? What are the common practices? A: Typically, the initial hidden state is initialized to a vector of zeros. In some architectures, it can be learned as a trainable parameter, or set from an encoder's final state (as in seq2seq models). Zero initialization is simplest and works well in most practical settings.

Q: What is exposure bias, and why does teacher forcing cause it? A: Exposure bias occurs because during training (with teacher forcing) the model always sees correct previous tokens as input, but at inference it must condition on its own, potentially wrong, previous predictions. This mismatch between training and inference conditions can cause errors to compound during generation.

Q: Explain scheduled sampling and how it addresses exposure bias. A: Scheduled sampling gradually shifts training from using ground-truth tokens (teacher forcing) to using the model's own predictions, following a decaying schedule. This exposes the model to its own error patterns during training, making it more robust at inference time.

Q: What is a Gated Recurrent Unit's reset gate used for, specifically? A: The reset gate controls how much of the previous hidden state to "forget" when computing the candidate hidden state at the current time step. A reset value near 0 lets the unit behave as if it were reading the first item of a new sequence, ignoring past context.

Q: How is an RNN used for sequence labeling tasks like part-of-speech tagging? A: The RNN produces a hidden state at every time step, and a classification layer (softmax) is applied to each hidden state independently to predict a label for the corresponding input token. This is a many-to-many architecture with aligned input and output lengths.

Practical & Applied RNN Concepts (Questions 23-33)

Q: What is Connectionist Temporal Classification (CTC), and why is it used with RNNs in speech recognition? A: CTC is a loss function that allows an RNN to be trained on sequence data without needing frame-by-frame alignment between input audio and output text. It introduces a special "blank" token and sums over all possible alignments, making it ideal for tasks like speech recognition where input and output lengths differ.

Q: How does dropout need to be modified for use in RNNs? A: Standard dropout, applied naively to recurrent connections, disrupts the network's ability to retain long-term memory. Variational dropout solves this by using the same dropout mask at every time step within a sequence, rather than sampling a new mask each step, preserving the recurrent signal.

Q: Compare RNNs to Hidden Markov Models (HMMs) for sequence modeling. A: HMMs assume a fixed number of discrete hidden states and rely on a strict Markov assumption (only the previous state matters). RNNs use a continuous, high-dimensional hidden state and can, in principle, model dependencies over arbitrarily long histories, giving them far greater representational power.

Q: What is the computational complexity of BPTT with respect to sequence length? A: Both time and memory complexity scale linearly, O(T), with sequence length T, since the network must be unrolled for T steps and all intermediate activations stored for the backward pass. This is more efficient per-step than self-attention's O(T²), but the lack of parallelism across steps often makes RNNs slower in wall-clock time.

Q: How can an RNN be applied to time series forecasting? A: The RNN is trained to predict the value at time t+1 given the sequence of values up to time t. For multi-step forecasting, the model can either predict all future steps at once (direct multi-output) or feed its own predictions back in as input for subsequent steps (autoregressive / recursive forecasting).

Q: What is layer normalization, and why is it often preferred over batch normalization in RNNs? A: Layer normalization normalizes activations across the feature dimension for a single example, rather than across a batch. Since RNNs process variable-length sequences and batch statistics can vary across time steps, layer normalization is more stable and works independently of batch size and sequence position.

Q: What is a character-level RNN language model, and how does it differ from a word-level model? A: A character-level model predicts the next character in a sequence, which gives it a small vocabulary and the ability to generate novel words or handle out-of-vocabulary tokens, at the cost of needing to model much longer sequences. A word-level model predicts the next word from a fixed vocabulary, learning higher-level semantics faster but struggling with rare or unseen words.

Q: How are attention alignment scores computed in an RNN-based seq2seq model? A: A common approach (Bahdanau attention) computes a score between the decoder's current hidden state and each encoder hidden state, often via a small feed-forward network. These scores are passed through a softmax to produce attention weights, which are used to compute a weighted sum of encoder states (the context vector) for that decoding step.

Q: What techniques help prevent overfitting when training RNNs? A: Common techniques include variational dropout, weight decay (L2 regularization), gradient clipping (which also stabilizes training), early stopping based on validation loss, and reducing model capacity (fewer layers or hidden units) when data is limited.

Q: Why do LSTMs mitigate the vanishing gradient problem better than vanilla RNNs? A: The LSTM's cell state is updated with an additive interaction (via the forget and input gates) rather than a repeated matrix multiplication. This creates a more direct path for gradients to flow backward through time, largely avoiding the repeated multiplication that causes gradients to vanish in vanilla RNNs.

Q: Write a short PyTorch snippet defining and running a basic LSTM layer. A:

import torch.nn as nn
lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=1, batch_first=True)
x = torch.randn(32, 15, 10)  # (batch, seq_len, input_size)
output, (h_n, c_n) = lstm(x)
output: (32, 15, 20) - hidden state at every time step
h_n, c_n: (1, 32, 20) - final hidden and cell states

Section 2: Convolutional Neural Networks (CNNs) - Questions 34-66

Foundational Concepts (Questions 34-43)

Q: Why are CNNs preferred over fully-connected networks for image data? A: Fully-connected networks treat every pixel independently and ignore spatial structure, leading to an enormous number of parameters for large images. CNNs exploit spatial locality (nearby pixels are related) and use shared filters, drastically reducing parameter count while preserving the ability to detect patterns like edges and textures regardless of their position in the image.

Q: Describe the mathematical operation of convolution in a CNN. A: A filter (kernel) of size k×k slides across the input, and at each position computes an element-wise multiplication with the underlying patch, summing the results into a single output value. This produces a feature map highlighting where the pattern represented by the filter appears in the input.

Q: Derive the formula for output size after a convolution operation. A: Output size = ((N - K + 2P) / S) + 1, where N is input size, K is kernel size, P is padding, and S is stride. This formula is applied independently to height and width for 2D inputs.

Q: What is the purpose of stride and padding in a convolutional layer? A: Stride controls how far the filter moves between applications - a larger stride reduces the output size and computation. Padding adds extra pixels (usually zeros) around the input border, which allows control over output size (e.g., "same" padding keeps output size equal to input size) and prevents excessive shrinkage of feature maps in deep networks.

Q: Compare max pooling and average pooling. A: Max pooling takes the maximum value in each pooling region, preserving the strongest activations and providing some translation invariance; it is more common in classification tasks. Average pooling takes the mean value, producing smoother feature maps; it is often used in the final layers of modern architectures (global average pooling) to reduce parameters before classification.

Q: What is weight sharing in CNNs, and why is it beneficial? A: The same filter (the same set of weights) is applied at every spatial position of the input. This drastically reduces the number of parameters compared to a fully-connected layer, and allows the network to detect a pattern regardless of where it appears in the image.

Q: Explain the difference between translation invariance and translation equivariance in CNNs. A: Equivariance means that if the input shifts, the feature map shifts correspondingly (convolution itself is equivariant). Invariance means the output does not change even if the input shifts - this is introduced by pooling layers, which discard some positional information to make the representation more robust to small translations.

Q: What is the receptive field of a neuron in a CNN, and why does it matter? A: The receptive field is the region of the original input image that influences a given neuron's activation. As layers stack, receptive fields grow, allowing deeper neurons to "see" larger and more abstract patterns. A network needs a sufficiently large receptive field to capture long-range spatial dependencies relevant to the task.

Q: What is the purpose of a 1x1 convolution? A: A 1x1 convolution operates only across the channel dimension at each spatial position, without mixing spatial neighbors. It is commonly used to reduce or expand the number of channels (dimensionality reduction), add non-linearity cheaply, and reduce computational cost before more expensive larger convolutions (as in Inception modules).

Q: How do multiple channels and filters work together in a convolutional layer? A: Each filter in a layer produces one output channel (feature map) by convolving across all input channels simultaneously and summing the results. A layer with F filters produces F output channels, and each filter has its own learned weights, allowing the network to detect F different patterns in parallel at that layer.

Architecture-Specific Concepts (Questions 44-53)

Q: Describe the LeNet-5 architecture and its historical significance. A: LeNet-5 (1998) consists of alternating convolutional and pooling layers followed by fully-connected layers, designed originally for handwritten digit recognition. It was one of the first successful demonstrations that gradient-based learning could be applied directly to raw pixel data, laying the groundwork for modern CNNs.

Q: What innovations did AlexNet introduce that made it a breakthrough architecture? A: AlexNet (2012) used ReLU activations (faster training than tanh/sigmoid), dropout for regularization, data augmentation, and was trained on GPUs, allowing a much deeper and wider network than previously feasible. It won the ImageNet competition by a large margin, sparking the modern deep learning boom.

Q: What is the key design principle behind VGGNet? A: VGGNet uses a simple, uniform design: stacks of small 3x3 convolutional filters (instead of larger filters) followed by max pooling, repeated to build depth. Stacking multiple 3x3 convolutions achieves a larger effective receptive field with fewer parameters than a single larger filter, while adding more non-linearities.

Q: How does the Inception module in GoogLeNet work? A: An Inception module applies multiple filter sizes (1x1, 3x3, 5x5) and pooling operations in parallel on the same input, then concatenates their outputs along the channel dimension. This lets the network capture patterns at multiple scales simultaneously, and 1x1 convolutions are used beforehand to reduce computational cost.

Q: Explain the role of skip (residual) connections in ResNet. A: A residual connection adds a layer's input directly to its output, so the layer only needs to learn a residual function (the difference from the identity mapping) rather than the full transformation. This gives gradients a direct path backward through the network, solving the degradation problem that made very deep networks (100+ layers) hard to train.

Q: Why do very deep CNNs suffer from vanishing gradients, and how does batch normalization help? A: As gradients backpropagate through many layers, repeated multiplication by weight matrices and activation derivatives can shrink them toward zero. Batch normalization normalizes layer activations to have stable mean and variance during training, which keeps gradients well-scaled and allows for higher learning rates and faster, more stable convergence.

Q: How does DenseNet differ from ResNet in its use of skip connections? A: While ResNet adds the input of a layer to its output, DenseNet concatenates the feature maps of every preceding layer as input to each subsequent layer within a dense block. This encourages feature reuse, strengthens gradient flow, and typically achieves strong performance with fewer parameters than comparable ResNets.

Q: What is a depthwise separable convolution, and why is it used in MobileNet? A: It splits a standard convolution into two steps: a depthwise convolution (a single filter per input channel, capturing spatial patterns) followed by a pointwise (1x1) convolution (mixing channels). This factorization drastically reduces the number of parameters and computations compared to a standard convolution, making it well-suited for mobile and edge devices.

Q: What is transfer learning, and why is it common practice with CNNs? A: Transfer learning reuses a CNN pretrained on a large dataset (e.g., ImageNet) as a starting point for a new, often smaller, task - either by fine-tuning all weights or by freezing early layers (which learn generic features like edges and textures) and only training later, task-specific layers. This dramatically reduces the data and compute needed to reach strong performance on the new task.

Q: What problem does batch normalization solve, and where is it typically placed in a CNN block? A: It addresses "internal covariate shift" - the changing distribution of layer inputs during training as earlier layers' weights update. It is typically placed after the convolution operation and before the activation function, normalizing outputs using batch statistics (mean and variance) and learned scale/shift parameters.

Advanced & Applied CNN Concepts (Questions 54-66)

Q: What are dilated (atrous) convolutions, and why are they useful? A: Dilated convolutions insert gaps between kernel elements, expanding the receptive field without increasing the number of parameters or reducing spatial resolution through pooling. They are especially useful in semantic segmentation, where preserving fine spatial detail while capturing broad context is important.

Q: Briefly describe the evolution of the R-CNN family for object detection. A: R-CNN generated region proposals externally and ran a CNN on each region separately (slow). Fast R-CNN shared convolutional computation across the whole image and only processed region-of-interest features per proposal. Faster R-CNN replaced the external proposal step with a learned Region Proposal Network, making the entire pipeline trainable end-to-end and much faster.

Q: What is the U-Net architecture, and what task was it designed for? A: U-Net is an encoder-decoder architecture for semantic segmentation, with a contracting path that captures context and an expanding path that enables precise localization, connected by skip connections between corresponding encoder and decoder layers. These skip connections help recover fine spatial detail lost during downsampling.

Q: What are common data augmentation techniques for training CNNs? A: Common techniques include random cropping, horizontal/vertical flipping, rotation, color jittering, random erasing/cutout, and mixup (blending two images and their labels). These artificially expand the effective training set and improve the model's robustness to variations it will encounter at test time.

Q: How do you compute the number of learnable parameters in a convolutional layer? A: Parameters = (kernel_height × kernel_width × input_channels + 1) × output_channels, where the +1 accounts for the bias term of each filter. Note that this count does not depend on the input's spatial dimensions, unlike a fully-connected layer.

Q: How is the computational cost (FLOPs) of a convolutional layer estimated? A: Approximately: FLOPs ≈ output_height × output_width × output_channels × kernel_height × kernel_width × input_channels. This shows that computational cost scales with both the spatial size of the output feature map and the number of input/output channels, which is why techniques like depthwise separable convolutions target this multiplication.

Q: What is global average pooling, and why has it replaced flatten + fully-connected layers in many modern CNNs? A: Global average pooling reduces each feature map to a single value by averaging over its entire spatial extent, producing a vector with one entry per channel. Compared to flattening and using a large fully-connected layer, this drastically reduces the parameter count and tends to reduce overfitting, while still working well for final classification.

Q: Describe how gradients backpropagate through a convolutional layer. A: Backpropagation through convolution is itself a convolution operation: the gradient with respect to the input is computed by convolving the gradient with respect to the output with a flipped (180-degree rotated) version of the filter. The gradient with respect to the filter weights is computed by convolving the input with the output gradient, summed over all spatial positions where the filter was applied (due to weight sharing).

Q: What are adversarial examples, and why are CNNs particularly vulnerable to them? A: Adversarial examples are inputs deliberately perturbed with small, often imperceptible noise, designed to cause a model to make an incorrect prediction with high confidence. CNNs are vulnerable because their decision boundaries in high-dimensional pixel space can be sensitive to small, carefully-crafted perturbations that exploit the linear behavior of many components in the network.

Q: What is Grad-CAM, and what is it used for? A: Grad-CAM (Gradient-weighted Class Activation Mapping) produces a visual heatmap highlighting which regions of an input image most influenced a CNN's prediction for a given class, using gradients flowing into the final convolutional layer. It is widely used for model interpretability and debugging.

Q: How can CNNs be applied to non-image, sequential data such as text or audio? A: A 1D convolution slides a filter along the sequence dimension (e.g., over word embeddings or audio samples), capturing local patterns like n-grams. Stacking multiple 1D convolutional layers with increasing dilation or larger receptive fields can capture progressively longer-range dependencies, offering a highly parallelizable alternative to RNNs for some sequence tasks.

Q: Compare CNNs and RNNs for sequence modeling tasks. A: CNNs process the entire sequence in parallel and are computationally efficient, but need many stacked layers (or dilation) to capture long-range dependencies, since a single convolution only sees a local window. RNNs process sequentially and can in principle model dependencies over arbitrary distances with a single layer, but suffer from the vanishing gradient problem and cannot be parallelized across time steps.

Q: Write a short PyTorch snippet defining a basic convolutional block. A:

import torch.nn as nn

conv_block = nn.Sequential(
nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1),
nn.BatchNorm2d(16),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2)
)
Input (batch, 3, 32, 32) -> Output (batch, 16, 16, 16)

Section 3: Transformers - Questions 67-100

Foundational Concepts (Questions 67-76)

Q: What is self-attention, and what problem does it solve? A: Self-attention allows every element in a sequence to directly compute a relevance score with every other element, then aggregate information from the whole sequence as a weighted sum based on those scores. It solves the long-range dependency problem faced by RNNs, since any two positions can interact directly in a single computational step, regardless of distance.

Q: Explain the roles of Query, Key, and Value vectors in self-attention. A: Each input token is projected into three vectors: a Query (what this token is looking for), a Key (what this token offers, used to be matched against queries), and a Value (the actual content contributed if a match is found). The Query of one token is compared against the Keys of all tokens to determine how much of each token's Value to include in the output.

Q: Write the scaled dot-product attention formula and explain why scaling is necessary. A: Attention(Q, K, V) = softmax(QK^T / √d_k) V. Scaling by √d_k prevents the dot products from growing too large in magnitude as dimensionality increases, which would otherwise push the softmax into regions with extremely small gradients, slowing or stalling learning.

Q: What is multi-head attention, and why is it used instead of a single attention operation? A: Multi-head attention runs several attention operations in parallel, each with its own learned Q, K, V projections, and concatenates the results before a final linear projection. This allows different heads to specialize in different types of relationships (e.g., syntax, semantics, positional patterns), giving the model a richer representation than a single attention function could.

Q: Why do Transformers need positional encoding, and how is it typically implemented? A: Self-attention is permutation-invariant - it has no inherent sense of token order. Positional encoding injects order information by adding a position-dependent vector to each token's embedding. The original Transformer used fixed sinusoidal functions of different frequencies for each dimension, though many modern models use learned or rotary embeddings instead.

Q: Write the sinusoidal positional encoding formula used in the original Transformer paper. A: PE(pos, 2i) = sin(pos / 10000^(2i/d)), and PE(pos, 2i+1) = cos(pos / 10000^(2i/d)), where pos is the token position and i indexes the embedding dimension. The varying frequencies allow the model to potentially learn to attend to relative positions through linear combinations of these encodings.

Q: Why can Transformers be trained faster (in wall-clock time) than RNNs on the same hardware? A: Self-attention has no sequential dependency between time steps - every position's attention computation can be performed simultaneously as matrix multiplications. This allows full parallelization across the entire sequence and batch on GPUs/TPUs, unlike RNNs, which must process one time step after another.

Q: Describe the high-level structure of the original encoder-decoder Transformer. A: The encoder is a stack of layers, each with self-attention followed by a feed-forward network, processing the input sequence bidirectionally. The decoder is a similar stack, but each layer adds masked self-attention (to prevent seeing future tokens) and a cross-attention sub-layer that attends over the encoder's output, generating the output sequence one token at a time.

Q: What is masked self-attention, and why is it necessary in the decoder? A: Masked self-attention sets the attention scores for all future positions to negative infinity before the softmax, so a token can only attend to itself and earlier tokens. This is necessary because the decoder generates output autoregressively and must not have access to tokens it hasn't generated yet, preserving the causal structure required at inference time.

Q: What is the difference between self-attention and cross-attention? A: In self-attention, the Query, Key, and Value vectors are all derived from the same sequence, letting a sequence contextualize itself. In cross-attention, the Query comes from one sequence (e.g., the decoder) while the Key and Value come from a different sequence (e.g., the encoder's output), allowing one sequence to be conditioned on information from another.

Architectural Details (Questions 77-86)

Q: What is the difference between pre-norm and post-norm Transformer designs? A: Post-norm (used in the original Transformer) applies layer normalization after adding the residual connection to a sub-layer's output. Pre-norm applies layer normalization to the input of each sub-layer before it is processed, which has been shown empirically to produce more stable training in very deep networks and reduces the need for careful learning-rate warmup.

Q: Why are residual (skip) connections used around every Transformer sub-layer? A: They add the sub-layer's input directly to its output, giving gradients a direct path backward through the network during training. This is essential for training very deep Transformer stacks, preventing the vanishing gradient problems that would otherwise occur across dozens or hundreds of layers.

Q: What is the purpose of the feed-forward network within each Transformer block? A: After the attention sub-layer aggregates information across the sequence, the position-wise feed-forward network (typically two linear layers with a non-linearity in between) processes each token's representation independently, adding additional non-linear transformation capacity and increasing the model's representational power.

Q: Why do many modern Transformers use GELU instead of ReLU in the feed-forward network? A: GELU (Gaussian Error Linear Unit) provides a smoother, probabilistic gating of inputs compared to ReLU's hard cutoff at zero, which has been empirically found to improve training stability and final performance in large Transformer-based language models, though the difference is relatively modest.

Q: How does BERT differ architecturally from the original Transformer, and what is it trained to do? A: BERT uses only the encoder stack (no decoder), processing text bidirectionally. It is pretrained using masked language modeling (predicting randomly masked tokens using both left and right context) and next-sentence prediction, making it well-suited for tasks requiring deep understanding of text, such as classification and question answering.

Q: How does GPT differ architecturally from BERT, and why? A: GPT uses only the decoder stack with masked (causal) self-attention, processing text left-to-right and predicting the next token given only previous tokens. This autoregressive design makes it naturally suited for text generation, unlike BERT's bidirectional encoder, which cannot generate text token-by-token in the same way.

Q: What architectural choice does T5 make, and what is its training framework? A: T5 uses the full original encoder-decoder architecture and reframes every NLP task - translation, classification, summarization - as a text-to-text problem, where both input and output are always text strings. It is pretrained using a span-corruption objective, where contiguous spans of text are masked and the model learns to reconstruct them.

Q: How does the Vision Transformer (ViT) adapt the Transformer architecture for images? A: ViT splits an image into fixed-size patches, flattens and linearly embeds each patch (analogous to a token embedding), adds positional encodings, and feeds the resulting sequence through a standard Transformer encoder. This treats an image as a sequence of patches rather than relying on convolutional inductive biases.

Q: What is the computational complexity of self-attention with respect to sequence length, and why does it matter? A: Self-attention has O(n² · d) time complexity and O(n²) memory complexity for a sequence of length n and hidden dimension d, since every token must compute a compatibility score with every other token. This quadratic scaling makes very long sequences expensive, motivating efficient attention variants and hardware-optimized implementations.

Q: What is KV-caching, and why is it important for efficient Transformer inference? A: During autoregressive generation, the Key and Value vectors for already-generated tokens do not change and would otherwise be recomputed at every new generation step. KV-caching stores these vectors after they are first computed, so only the new token's Q, K, V need to be calculated at each step, substantially speeding up generation.

Advanced & Modern Concepts (Questions 87-100)

Q: What is Flash Attention, and how does it improve on standard attention implementations? A: Flash Attention computes exactly the same mathematical result as standard attention, but restructures the computation using tiling and recomputation to avoid materializing the full n×n attention matrix in slow GPU memory (HBM), keeping intermediate values in fast on-chip SRAM instead. This yields significant speed and memory improvements with no loss in accuracy.

Q: What is sparse attention, and how do models like Longformer use it? A: Sparse attention restricts each token to attend to only a subset of other tokens rather than the full sequence, reducing computational cost from quadratic toward linear. Longformer combines a local sliding-window attention pattern with a small number of global attention tokens, balancing efficiency with the ability to model some long-range dependencies.

Q: What are Rotary Position Embeddings (RoPE), and what advantage do they offer? A: RoPE encodes position by rotating the Query and Key vectors by an angle proportional to their position, such that the dot product between two rotated vectors naturally depends on their relative distance. This tends to generalize better to sequence lengths longer than those seen during training, compared to the original fixed sinusoidal encoding.

Q: What is ALiBi, and how does it differ from RoPE? A: ALiBi (Attention with Linear Biases) adds a fixed penalty to attention scores that grows linearly with the distance between two positions, applied directly to the scores rather than to the Q/K vectors themselves. It is simpler to implement than RoPE and has also shown strong extrapolation to longer sequences than seen during training.

Q: What is grouped-query attention (GQA), and what problem does it solve? A: GQA shares Key and Value projections across groups of Query heads (rather than every head having its own K/V), reducing the size of the KV cache needed during inference. This lowers memory bandwidth requirements and speeds up autoregressive generation, with a smaller quality trade-off than full multi-query attention (which shares K/V across all heads).

Q: What is a Mixture-of-Experts (MoE) layer, and why is it used in some large Transformer models? A: An MoE layer replaces a single feed-forward network with multiple "expert" networks and a learned router that selects a small subset of experts (e.g., 2 out of 8) to process each token. This allows the total parameter count of the model to grow substantially while keeping the computation per token roughly constant, since only a fraction of experts are active at once.

Q: What are neural scaling laws, and why are they significant for Transformer-based models? A: Scaling laws describe how model performance (typically loss) improves predictably as a power-law function of model size, dataset size, and compute, when these are scaled together appropriately. They are significant because they let researchers estimate the performance of a much larger model before training it, guiding resource allocation for expensive training runs.

Q: What is the difference between fine-tuning and in-context learning (prompting) for adapting a pretrained language model to a new task? A: Fine-tuning updates the model's weights using gradient descent on task-specific labeled data, permanently changing the model. In-context learning provides examples or instructions directly in the input prompt at inference time, with no weight updates - the model infers the task from context alone, leveraging patterns learned during pretraining.

Q: What role does Reinforcement Learning from Human Feedback (RLHF) play in training modern language models? A: RLHF fine-tunes a pretrained language model using a reward signal derived from human preference comparisons between model outputs, typically via a learned reward model and a reinforcement learning algorithm (such as PPO). It is used to align model behavior with human preferences - for example, making outputs more helpful, honest, or harmless - beyond what pretraining alone achieves.

Q: What is "hallucination" in the context of large language models, and why does it occur? A: Hallucination refers to a model generating text that is fluent and confident but factually incorrect or unsupported by its training data or given context. It occurs partly because language models are trained to produce statistically plausible continuations of text, which does not guarantee factual accuracy, especially for information that is rare, ambiguous, or absent from training data.

Q: What is Byte-Pair Encoding (BPE), and why is it used for tokenization in Transformer models? A: BPE is a subword tokenization algorithm that starts with individual characters and iteratively merges the most frequent adjacent pairs into new tokens, building a vocabulary of common subword units. This lets a model represent rare or unseen words as combinations of known subword pieces, avoiding an unmanageably large vocabulary while still handling arbitrary text.

Q: Explain temperature, top-k, and top-p (nucleus) sampling in text generation. A: Temperature scales the logits before the softmax - lower values make the output distribution sharper (more deterministic), higher values make it flatter (more random). Top-k sampling restricts sampling to the k most likely next tokens. Top-p (nucleus) sampling restricts sampling to the smallest set of tokens whose cumulative probability exceeds p, adapting the candidate pool size based on the model's confidence at each step.

Q: Why have Transformers become the dominant architecture over RNNs and CNNs for most large-scale AI systems? A: Transformers parallelize fully across sequence positions (unlike RNNs), directly model relationships between any two positions regardless of distance (addressing a key weakness of both RNNs and CNNs), and have demonstrated remarkably predictable and favorable scaling behavior as model size, data, and compute increase, making them the default architecture for organizations investing heavily in large-scale training.

Q: Write a short PyTorch snippet using built-in multi-head attention. A:

import torch
import torch.nn as nn

attn = nn.MultiheadAttention(embed_dim=64, num_heads=8, batch_first=True)
x = torch.randn(32, 10, 64)  # (batch, seq_len, embed_dim)
output, attn_weights = attn(x, x, x)  # self-attention: Q=K=V=x
output: (32, 10, 64), attn_weights: (32, 10, 10)

Section 4: Comparative Analysis & Decision Roadmap

Decision roadmap:

• If the task involves images or grid-like spatial data and compute is limited: start with a CNN.

• If the task involves sequences with strong long-range dependencies (language, long time series) and sufficient data/compute is available: use a Transformer.

• If the task involves short sequences with low compute budgets, or an exam question specifically targets recurrence: RNN/LSTM/GRU remain relevant and expected knowledge.

• For very long sequences where full self-attention is too expensive: consider efficient attention variants (sparse, linear, Flash Attention) or hybrid architectures, rather than defaulting to RNNs.

Section 5: Strategic Exam Preparation Tips

• Master the formulas cold. The RNN hidden-state update, the scaled dot-product attention equation, and the convolution output-size formula are the three most commonly tested equations across all three architectures - be able to write each from memory without hesitation.

• Practice deriving, not just reciting. Many exams ask you to derive the vanishing gradient argument or the output size of a conv layer step by step. Recitation without derivation loses partial credit on these questions.

• Always be ready to compare. A large share of exam and interview questions are comparative ("RNN vs Transformer for X," "LSTM vs GRU," "ResNet vs DenseNet"). Prepare short, structured answers - one sentence on the mechanism, one on the trade-off.

• Know the historical narrative. Many questions test whether you understand why each innovation existed - what problem it solved. Framing your answers around "this fixed problem X in the previous approach" demonstrates deeper understanding than a bare definition.

• Don't neglect the coding questions. Even in theory-heavy exams, a short PyTorch snippet question is common. Know the basic layer constructors (nn.RNN, nn.LSTM, nn.Conv2d, nn.MultiheadAttention) and their input/output tensor shapes.

• Time-box your review. With 100 questions across three architectures, allocate review time proportionally: roughly a third of your time to RNNs, a third to CNNs, and a third to Transformers, with extra time reserved for the comparative table, since it tends to appear in some form on almost every exam.

Conclusion: From Theory to Mastery

RNNs, CNNs, and Transformers are not competing curiosities to be memorized in isolation - they are three answers to the same underlying question: how should a model decide what information matters, and from where? RNNs answer this through recurrence and memory, CNNs through locality and shared filters, and Transformers through direct, learned relevance between any two elements of a sequence. Understanding not just what each architecture does, but the specific problem it was built to solve and the trade-offs it accepted in doing so, is what separates rote memorization from real command of the material. Work through these 100 questions until the answers come from understanding rather than recall, and you will be prepared not just for the exam in front of you, but for the technical conversations that follow it.

Author: Shadrackovsky, Antera Group.