Natural Language Processing (NLP)

Classical NLP preprocessing vs modern neural approach

# ── CLASSICAL APPROACH (needed for TF-IDF, n-grams, SVM) ──────────────────
import re, string
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

def classical_preprocess(text: str) -> list[str]:
    text = text.lower()                                # lowercase
    text = re.sub(r'[^a-z0-9\s]', '', text)          # remove punctuation
    tokens = word_tokenize(text)                       # tokenize
    stop_words = set(stopwords.words('english'))
    tokens = [t for t in tokens if t not in stop_words]  # remove stopwords
    lem = WordNetLemmatizer()
    return [lem.lemmatize(t) for t in tokens]          # lemmatize: running→run

# ── MODERN NEURAL APPROACH (BERT/LLM tokenizers) ──────────────────────────
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def neural_preprocess(text: str) -> dict:
    # No manual preprocessing needed — the tokenizer handles everything
    # Uses WordPiece/BPE subword tokenization
    # "running" → ["run", "##ning"] — keeps morphological structure
    return tokenizer(text, truncation=True, max_length=512,
                     return_tensors="pt")
# Neural models learn from raw text — stopwords, casing carry information

1. What is the pipeline difference between NLP in the 2010s (pre-deep learning) and NLP in 2024? (Answer: 2010s NLP: pipeline of specialised components — tokeniser → POS tagger → NER → parser → coreference resolver → task-specific classifier. Each component was separately trained and errors accumulated through the pipeline. 2024 NLP: a single large language model handles tokenisation, understanding, generation, and all downstream tasks through prompting or fine-tuning. One model replaces the entire pipeline, handles task combinations naturally, and achieves better performance on most tasks.)
2. What is the Chomsky hierarchy and how do neural LLMs relate to formal language theory? (Answer: Chomsky hierarchy: Regular < Context-Free < Context-Sensitive < Recursively Enumerable languages. Regular: finite automata (regex). Context-Free: pushdown automata (most programming languages). Natural language: roughly context-sensitive (agreement, cross-serial dependencies). Neural LLMs: empirically capable of generating text that follows complex natural language patterns, including context-sensitive phenomena. However, LLMs are not proven to recognise formal languages beyond their training distribution — they approximate statistical patterns rather than parsing formal grammars.)
3. What is the difference between extractive and abstractive NLP tasks? (Answer: Extractive: the output is a subset of the input — NER (extracting entity spans), extractive QA (finding answer span), extractive summarisation (selecting key sentences). Simpler, more reliable, no hallucination risk. Abstractive: the output requires generating new text not present in the input — abstractive summarisation (paraphrasing), translation, abstractive QA (reasoning across documents). More flexible, more natural-sounding but risk generating incorrect content not supported by the source.)
4. What is the Turing Test and why is passing it no longer considered sufficient evidence of general language understanding? (Answer: Turing Test (Turing 1950): if a human interrogator cannot distinguish a machine from a human in text conversation, the machine exhibits intelligence. Modern LLMs (GPT-4, Claude) consistently fool human judges in short conversations. However, they still: hallucinate facts, fail basic physical reasoning, lack consistent world models, and cannot reliably solve novel logical puzzles. Passing the Turing Test by producing human-like text is a weaker criterion than general language understanding — it tests conversational fluency, not reasoning.)
5. What is the 'bitter lesson' (Rich Sutton) and how has it shaped modern NLP? (Answer: The bitter lesson (Sutton 2019): AI progress consistently comes from leveraging computation rather than incorporating human domain knowledge. Hand-crafted features, linguistic rules, and expert-designed NLP pipelines were repeatedly outperformed by general-purpose methods (neural networks) given enough data and compute. The lesson: invest in methods that scale with compute. Modern NLP took this lesson: transformers + massive data + scale replaced hand-crafted NLP pipelines. The 'bitter' part: human expertise in linguistics ultimately mattered less than raw computation.)