Temperature & Sampling

Softmax with temperature T. At T→0, the highest logit dominates and the distribution collapses to a one-hot. At T→∞, all tokens become equally likely.

Manual nucleus (top-p) sampling — exactly what happens inside every LLM API

import torch
import torch.nn.functional as F

def top_p_sample(logits: torch.Tensor, temperature: float = 0.9, top_p: float = 0.9) -> int:
    """
    Nucleus sampling with temperature.
    logits: raw unnormalized scores, shape (vocab_size,)
    """
    # 1. Apply temperature
    scaled = logits / max(temperature, 1e-8)

    # 2. Softmax → probabilities
    probs = F.softmax(scaled, dim=-1)

    # 3. Sort tokens by probability (highest first)
    sorted_probs, sorted_indices = torch.sort(probs, descending=True)

    # 4. Compute cumulative sum; find the nucleus boundary
    cumulative = torch.cumsum(sorted_probs, dim=-1)

    # 5. Remove tokens once cumulative prob exceeds top_p
    #    (shift by 1 so we always keep at least 1 token)
    remove_mask = cumulative - sorted_probs > top_p
    sorted_probs[remove_mask] = 0.0

    # 6. Renormalize and sample
    sorted_probs /= sorted_probs.sum()
    sampled_pos = torch.multinomial(sorted_probs, num_samples=1)
    return sorted_indices[sampled_pos].item()

1. What happens when you set temperature=0 in an LLM API call? (Answer: Temperature=0 selects the highest probability token at every step — fully deterministic greedy decoding. The same prompt will always produce the same output (with seed fixed). Use for: factual Q&A, code generation where consistency matters, tests. Avoid for: creative writing, brainstorming, where diversity of outputs is valuable.)
2. Top-p=0.9 means the model samples from tokens whose cumulative probability equals 90%. For a distribution where one token has probability 0.95, what tokens are eligible? (Answer: Just that one token — it alone accounts for 95% > 90% of probability mass. With top-p=0.9, the smallest set of tokens totalling ≥90% is just this single dominant token. This is the key advantage over top-k: top-p automatically collapses to near-greedy when the model is highly confident, giving creativity only when the model is genuinely uncertain about what comes next.)
3. What is repetition penalty in LLM sampling and when is it necessary? (Answer: Repetition penalty discounts logits for tokens that have already appeared in the generated text: effective_logit = original_logit / penalty if token appeared, else original_logit. penalty > 1.0 reduces probability of repeating tokens. Default is 1.0 (no penalty). Necessary for models that fall into repetition loops (common without penalty for long generation). Over-penalisation can prevent legitimate word repetition (in lists, technical terms). Typical useful range: 1.1–1.3.)
4. Why might you use min-p sampling instead of top-k or top-p? (Answer: Min-p: filter tokens whose probability is less than min_p × (probability of the top token). Unlike top-k (fixed count regardless of distribution) or top-p (fixed mass), min-p adapts relative to the strongest option. When top token is at 80%, min-p=0.05 keeps tokens with probability ≥4% — very few. When top token is at 10%, min-p=0.05 keeps tokens with probability ≥0.5% — many options. Maintains consistent relative confidence filtering across all probability distributions.)
5. A customer service chatbot uses temperature=0.9 for all responses. What problem might arise? (Answer: High temperature introduces randomness — the bot may give inconsistent answers to the same question, different pricing information in different conversations, varying support procedures. For factual, policy-based responses (return policy, pricing, troubleshooting steps), temperature should be low (0.0–0.3). High temperature is appropriate for creative tasks, not deterministic information retrieval. Many production systems use temperature=0 for factual queries.)