Deep Learning

Minimal modern DL training loop with HuggingFace Trainer

from transformers import (AutoModelForSequenceClassification,
                            AutoTokenizer, TrainingArguments, Trainer)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score

# Load pretrained model + tokenizer (transfer learning from BERT)
model_name = "bert-base-uncased"
model     = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Tokenize dataset
dataset = load_dataset("imdb")
def tokenize(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length")
tokenized = dataset.map(tokenize, batched=True)

# Training configuration
args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    learning_rate=2e-5,           # small LR for fine-tuning pretrained model
    weight_decay=0.01,            # AdamW regularization
    evaluation_strategy="epoch",
    fp16=True,                    # mixed precision — 2× faster, half memory
    logging_steps=100,
    load_best_model_at_end=True,
)

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {"accuracy": accuracy_score(labels, preds)}

trainer = Trainer(model=model, args=args,
                  train_dataset=tokenized["train"],
                  eval_dataset=tokenized["test"],
                  compute_metrics=compute_metrics)
trainer.train()

1. What is the universal approximation theorem and why doesn't it guarantee good learning? (Answer: Universal approximation theorem (Cybenko 1989, Hornik 1991): a neural network with one hidden layer and sufficient neurons can approximate any continuous function arbitrarily closely. It guarantees representational capacity exists. It does NOT guarantee: (1) That gradient descent will find the approximating weights. (2) That the network will generalise to unseen data. (3) That the network can be trained efficiently. The theorem says the function space is rich enough; it says nothing about learnability.)
2. What is the difference between a shallow wide network and a deep narrow network for the same parameter count? (Answer: Shallow wide: one hidden layer with many neurons. Can approximate any function (UAT) but may require exponentially more neurons than a deep network for certain functions. Deep narrow: many layers, fewer neurons per layer. Compositionality: learns hierarchical features where each layer builds on the previous. Deep networks are exponentially more efficient at representing compositionally structured functions (images, language, programs). This is the core empirical motivation for depth in modern deep learning.)
3. What is a hyperparameter vs a parameter in deep learning? (Answer: Parameter: learned from data via gradient descent — weights, biases, batch normalisation γ and β. Model does not know their values before training. Hyperparameter: set before training, not learned from data — learning rate, batch size, number of layers, dropout rate, weight decay, architecture choices. Must be tuned by the practitioner via validation performance. Some methods blur this line: learning rate schedules adapt LR during training; neural architecture search learns architectural choices.)
4. What is representation learning and how does deep learning achieve it? (Answer: Representation learning: automatically discovering the features (representations) useful for a task from raw data, rather than hand-crafting features. Deep learning achieves this through hierarchical composition: early layers learn low-level features (edges, frequencies), middle layers learn intermediate combinations (shapes, phonemes), late layers learn semantic features (objects, words). The key insight: the same gradient signal that optimises task performance also shapes what features the network learns — task-relevant features are automatically selected.)
5. What distinguishes deep learning from classical ML in terms of feature engineering? (Answer: Classical ML: practitioner manually designs features from domain knowledge (HOG for images, TF-IDF for text, MFCC for audio). Quality of features determines model ceiling. Deep learning: features are learned end-to-end from raw data — pixels, characters, waveforms. The model discovers what features matter for the task. This shifts human effort from feature engineering to architecture design and data collection. Trade-off: deep learning requires much more data and compute; classical ML with good features can outperform on small datasets.)