Appendix C. Glossary

This glossary defines every major technical term used in this book. Each entry includes a plain-language definition and a reference to the chapter where the term is first introduced or most thoroughly explained. Terms are listed alphabetically.

A

A2A (Agent2Agent) Protocol: An open protocol introduced by Google on April 9, 2025, that allows AI agents built with different frameworks to discover each other’s capabilities and collaborate on tasks. While MCP (see entry) standardizes how agents connect to tools, A2A standardizes how agents communicate with each other. (Chapter 23)

Absolute position embeddings: A method of encoding word order by adding a learned vector for each position (1st word, 2nd word, etc.) to the token embedding. Used by GPT-2 and BERT. Replaced in modern models by Rotary Position Embeddings (RoPE). (Chapter 6)

Activation function: A mathematical function applied to the output of a neuron or layer that introduces non-linearity, allowing neural networks to learn complex patterns. Common examples include ReLU, GELU, Swish, and SiLU. Without activation functions, stacking multiple layers would be equivalent to a single linear transformation. (Chapters 3, 9)

Activation memory: The GPU memory consumed by intermediate values computed during a forward pass. During training, these values must be stored for the backward pass (backpropagation). Activation memory scales with batch size and sequence length, and is a major component of training memory requirements. (Appendix B)

Active parameters: In a Mixture-of-Experts (MoE) model, the number of parameters actually used to process each token. For example, LLaMA 4 Maverick has 400 billion total parameters but only 17 billion active parameters per token, because only a subset of experts is selected by the router for each token. (Chapters 11, 12)

AdamW: The optimizer used to train virtually all modern LLMs. AdamW maintains two additional values per parameter (momentum and variance estimates) plus a master copy of the weights in FP32, resulting in approximately 16 bytes of memory per parameter during training. The “W” stands for weight decay, a regularization technique that prevents weights from growing too large. Originally proposed by Loshchilov and Hutter (arXiv:1711.05101, ICLR 2019). (Chapter 14)

Agent: In the context of LLMs, a system where a model operates in a loop: it receives a task, reasons about what to do, calls tools (APIs, databases, file systems), observes the results, and repeats until the task is complete. Agents extend LLMs from passive text generators to active problem solvers. (Chapter 23)

Agentic AI Foundation (AAIF): An industry consortium formed on December 9, 2025, under the Linux Foundation, with eight platinum members (AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, OpenAI) each contributing $350,000. AAIF develops governance standards for AI agent interoperability and safety. (Chapter 23)

AgentKit: A visual agent builder released by OpenAI in March 2026 that provides a drag-and-drop interface for creating multi-step agent workflows without writing code. (Chapter 29)

AIME (American Invitational Mathematics Examination): A prestigious math competition used as a benchmark for evaluating LLM mathematical reasoning. AIME problems require multi-step reasoning and creative problem-solving. Key scores: o1 achieved 83.3% on AIME 2024, o3 achieved 96.7%, and DeepSeek-R1 achieved 79.8% (pass@1). AIME scores have become a standard metric for comparing reasoning models. (Chapters 15, 16)

Alignment: The process of making a language model’s behavior match human intentions and values. Alignment techniques include Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and Constitutional AI. The “alignment tax” refers to the tradeoff where safety training can reduce model capabilities. (Chapters 15, 26)

ALiBi (Attention with Linear Biases): A positional encoding method that adds a linear bias to attention scores based on the distance between tokens, rather than modifying the embeddings. Used by the BLOOM model. ALiBi allows some context extension beyond the training length without additional fine-tuning. (Chapter 6)

Annealing: The final phase of a learning rate schedule during pre-training, where the learning rate is gradually reduced to zero (or near zero) over a small number of tokens. LLaMA 3 anneals over 40 million tokens while also upsampling high-quality data, then averages checkpoints from the annealing phase. Annealing helps the model converge to a better minimum. (Chapter 14)

All-gather: A distributed computing operation where each GPU broadcasts its local data shard to all other GPUs, so every GPU ends up with the complete data. Used in Fully Sharded Data Parallelism (FSDP) to reconstruct full parameter tensors when needed for computation. (Chapter 14)

All-to-all communication: A distributed computing operation where each GPU sends a different piece of data to every other GPU. Used by DeepSpeed-Ulysses for sequence parallelism, where the input is partitioned along the sequence dimension and redistributed for attention computation. (Chapter 20)

Attention: The core mechanism of the Transformer architecture. Attention allows each token in a sequence to look at every other token and decide how much to “pay attention” to it. Mathematically, it computes a weighted sum of value vectors, where the weights are determined by the similarity (dot product) between query and key vectors: softmax(QK^T / sqrt(d_k)) * V. (Chapter 7)

Attention head: One parallel instance of the attention mechanism. Each head operates on a subset of the model’s dimensions (e.g., if the model has 8,192 dimensions and 64 heads, each head works with 128 dimensions). Different heads learn to attend to different types of relationships (syntax, semantics, coreference, etc.). (Chapter 8)

Attention sink: A phenomenon where the first few tokens in a sequence consistently receive disproportionately high attention scores, regardless of their semantic content. StreamingLLM (Xiao et al., arXiv:2309.17453, ICLR 2024) exploits this by always retaining these initial “sink” tokens in the KV cache alongside a sliding window of recent tokens, enabling theoretically infinite-length generation with fixed memory. (Chapters 18, 20)

Autoregressive generation: The process by which language models generate text one token at a time, left to right. Each new token is predicted based on all previous tokens. This is why generation is sequential and cannot be fully parallelized, unlike the prompt processing (prefill) phase. (Chapter 17)

AWQ (Activation-Aware Weight Quantization): A quantization method that identifies which weights are most important based on activation patterns and preserves their precision while aggressively quantizing less important weights. Commonly used for INT4 quantization of LLMs for deployment. (Chapters 24, 25)

B

Backpropagation: The algorithm used to train neural networks. It computes how much each weight contributed to the model’s error (the loss) by applying the chain rule of calculus, working backward from the output layer to the input layer. The resulting gradients tell the optimizer which direction to adjust each weight. (Chapter 3)

Base model: A pre-trained language model before any fine-tuning (SFT, RLHF, etc.). Base models are trained only on the next-token prediction objective and produce raw text completions rather than conversational responses. They are the starting point for fine-tuning. Examples: LLaMA 4 Maverick (base), Qwen3-8B (base). Distinct from instruct models, which have been fine-tuned for instruction following. (Chapters 15, 25, 28)

Batch size: The number of input examples processed simultaneously during training or inference. Larger batch sizes improve GPU utilization but require more memory. During inference serving, batch size refers to the number of concurrent requests being processed, each requiring its own KV cache. (Chapters 14, 24, Appendix B)

Beam search: A decoding strategy that maintains multiple candidate sequences (beams) at each generation step, expanding the most promising ones. Unlike greedy decoding (which always picks the single most likely token), beam search explores multiple paths and selects the best overall sequence. Less commonly used in modern LLM inference, where sampling-based methods (temperature, top-p, top-k) are preferred. (Chapter 17)

BERT (Bidirectional Encoder Representations from Transformers): A Transformer model released by Google in October 2018 that processes text bidirectionally (looking at words both before and after a given position). BERT was designed for understanding tasks (classification, question answering) rather than generation. It dominated NLP benchmarks for years but has been largely superseded by decoder-only models like GPT for most tasks. (Chapter 1)

BF16 (BrainFloat16): A 16-bit floating-point format that uses 8 exponent bits and 7 mantissa bits. The wide exponent range (same as FP32) prevents overflow and underflow, making BF16 more numerically stable than FP16 for training. BF16 is the standard precision for LLM pre-training and inference on modern GPUs (H100, B200, B300). Each parameter occupies 2 bytes. (Chapters 14, Appendix B)

BigBird: A sparse attention method (Zaheer et al., arXiv:2007.14062, NeurIPS 2020) that combines sliding window attention, global tokens, and random attention connections. Proven to be a universal approximator and Turing complete, preserving the theoretical properties of full attention while handling sequences up to 8x longer. (Chapter 20)

bitsandbytes: A Python library that provides efficient 8-bit and 4-bit quantization for PyTorch models, enabling QLoRA fine-tuning on consumer GPUs. bitsandbytes implements NF4 (4-bit NormalFloat) quantization and integrates with Hugging Face Transformers and PEFT for seamless quantized training. (Chapter 28)

BPE (Byte Pair Encoding): The dominant tokenization algorithm used by modern LLMs. BPE starts with individual bytes or characters and iteratively merges the most frequent adjacent pairs to build a vocabulary of subword tokens. For example, “understanding” might be split into “under” + “standing.” Originally a data compression algorithm (Gage, 1994), adapted for NLP by Sennrich et al. (2016). (Chapter 4)

Bradley-Terry model: A statistical model used in RLHF to convert pairwise human preference judgments (“response A is better than response B”) into a scalar reward signal. The reward model is trained to predict which response a human would prefer, and its scores are used to guide reinforcement learning. (Chapter 15)

C

Catastrophic forgetting: The phenomenon where fine-tuning a model on new data causes it to lose capabilities learned during pre-training. Mitigated by using low learning rates during fine-tuning, training for few epochs, and techniques like LoRA that modify only a small fraction of parameters. (Chapters 15, 28)

Causal masking: A technique used during training and inference of decoder-only models that prevents each token from attending to future tokens. Implemented by setting attention scores for future positions to negative infinity before the softmax, which zeroes out their attention weights. This ensures the model can only use information from tokens that have already been generated. Also called “autoregressive masking.” (Chapter 7)

Chain-of-thought (CoT) prompting: A technique where the model is prompted to show its reasoning step by step before giving a final answer. First demonstrated by Wei et al. (2022), who showed that adding “Let’s think step by step” to prompts dramatically improved math and reasoning performance. CoT is the foundation of extended thinking and reasoning models. (Chapter 16)

Chat template: A formatting specification that defines how conversation turns (system, user, assistant) are structured as token sequences for a model. Different model families use different templates (e.g., Qwen3 uses <|im_start|> and <|im_end|> special tokens; LLaMA uses <|begin_of_text|> and role headers). Using the wrong chat template causes degraded performance because the model sees token patterns it was not fine-tuned on. (Chapters 23, 28)

Chameleon: A natively multimodal model from Meta (arXiv:2405.09818, May 2024) that uses a unified token space for both text and images. Chameleon encodes images into discrete tokens using a learned codebook (8,192 entries, 1,024 tokens per 512x512 image) and processes them alongside text tokens through the same Transformer. Available in 7B and 34B sizes, trained on 10 trillion tokens of mixed-modal data. (Chapter 22)

ChatGPT: A product released by OpenAI in November 2022, built on GPT-3.5 fine-tuned with RLHF to be conversational and helpful. ChatGPT brought language models to the mainstream, reaching over 100 million users within two months. It is a product, not a model architecture. (Chapter 1)

Chinchilla scaling law: A scaling law published by Hoffmann et al. at DeepMind in 2022 that established the compute-optimal ratio of model parameters to training tokens. The key finding: for a given compute budget, the model size and training data should be scaled roughly equally. A model with N parameters should be trained on approximately 20N tokens. This overturned the previous assumption (from Kaplan et al., 2020) that scaling model size was more important than scaling data. (Chapter 13)

Claude: Anthropic’s family of language models, named after Claude Shannon. Major releases include Claude 3 (March 2024), Claude 3.5 Sonnet (June 2024), Claude 3.7 Sonnet with extended thinking (February 2025), Claude 4 and Opus 4 (May 2025), Claude Opus 4.6 (February 5, 2026), and Claude Sonnet 4.6 (February 17, 2026). Claude models are known for strong instruction following, safety alignment via Constitutional AI, and long context windows (up to 1 million tokens for Opus 4.6). (Chapters 1, 11, 15, 16, 19, 20)

Claude Agent SDK: Anthropic’s framework for building AI agents powered by Claude models. Originally released as the Claude Code SDK on September 29, 2025, then rebranded to Claude Agent SDK. Provides programmatic access to Claude’s agentic capabilities, including tool use, multi-step reasoning, and file system interaction. (Chapters 23, 29)

Claude Code: Anthropic’s agentic coding tool that runs in the terminal, allowing developers to delegate coding tasks to Claude. Released as a beta research preview on February 24, 2025, alongside Claude 3.7 Sonnet, and reached general availability on May 22, 2025. Claude Code can read and edit files, run commands, search codebases, and create pull requests. (Chapter 23)

CLIP (Contrastive Language-Image Pre-training): A model introduced by Radford et al. at OpenAI in January 2021 that trains a vision encoder and a text encoder jointly to map images and text into the same vector space. CLIP enables zero-shot image classification and is the foundation of many vision-language models. Trained on 400 million image-text pairs. (Chapter 21)

Common Crawl: A nonprofit organization that has been crawling the web since 2008, maintaining a freely available archive of hundreds of billions of web pages. Common Crawl data is the largest source of text for LLM pre-training, though it requires extensive filtering and deduplication before use. (Chapter 14)

Computer-Using Agent (CUA): A model or system capable of interacting with a computer’s graphical user interface (GUI) by taking screenshots, interpreting what is on screen, and performing mouse clicks and keyboard actions. OpenAI introduced its CUA model with Operator on January 23, 2025, and integrated CUA capabilities into the Responses API on March 11, 2025. GPT-5.4 (March 5, 2026) achieved 75% on OSWorld-Verified, surpassing the 72.4% human baseline. Anthropic’s Claude Computer Use launched in beta on October 22, 2024. (Chapter 23)

Constitutional AI (CAI): An alignment approach developed by Anthropic where the model is trained to critique and revise its own outputs according to a set of written principles (a “constitution”). Instead of relying solely on human feedback, the model generates its own training signal by evaluating responses against the constitution. This reduces the need for human labelers and makes the alignment process more transparent and scalable. (Chapters 15, 26)

Constitutional Classifiers: A safety system developed by Anthropic that uses trained classifiers to detect and block harmful inputs and outputs in real time. Constitutional Classifiers are trained on synthetic data generated according to the model’s constitution (its set of safety principles). Constitutional Classifiers++ is an enhanced version with improved robustness against adversarial attacks. (Chapter 26)

Context parallelism: A distributed computing technique that splits a long input sequence across multiple GPUs, with each GPU processing a portion of the sequence. Requires communication between GPUs to share attention information. Implementations include Ring Attention and DeepSpeed-Ulysses. NVIDIA’s Dynamic Context Parallelism (January 2026) achieved 1.48x speedup by dynamically adjusting the parallelism strategy based on sequence length. (Chapter 20)

Context rot: The empirically observed phenomenon where LLM performance degrades as input length increases, even on simple tasks. A study by Chroma Research (Hong, Troynikov, and Huber, July 2025) tested 18 state-of-the-art LLMs and found that accuracy drops significantly with longer inputs, challenging the assumption that larger context windows automatically mean better performance. Context rot is distinct from the “lost in the middle” effect; it affects the entire context, not just the middle. (Chapter 20)

Context window: The maximum number of tokens a model can process in a single forward pass. As of March 2026, context windows range from 8,192 tokens (older models) to 10 million tokens (LLaMA 4 Scout). Common sizes include 128K (many open models), 1.05 million (GPT-5.4), 1 million (Gemini 3.1 Pro, Claude Opus 4.6), and 2 million (Grok 4 Fast). Longer context windows require more memory for the KV cache and more compute for attention. (Chapters 6, 20)

Continuous batching: A serving optimization where new requests are added to the processing batch as soon as existing requests finish, rather than waiting for all requests in a batch to complete. This dramatically improves GPU utilization for LLM inference, where different requests have different output lengths. First described in the Orca system (Yu et al., OSDI 2022), which achieved up to 36.9x throughput improvement over FasterTransformer baselines. (Chapter 24)

Cosine schedule: A learning rate schedule that decreases the learning rate following a cosine curve from its peak value to near zero over the course of training. Widely used in LLM pre-training because it provides a smooth, gradual decay. Often combined with a linear warmup phase at the start of training. (Chapter 14)

Cross-attention: An attention mechanism where the queries come from one sequence (e.g., text) and the keys and values come from another sequence (e.g., image tokens). Used in encoder-decoder models and in some vision-language architectures (e.g., Flamingo, LLaMA 3.2 Vision) to connect visual representations to the language model. (Chapters 21, 22)

Cross-entropy loss: The loss function used by virtually all language models. It measures the difference between the model’s predicted probability distribution over the vocabulary and the true next token. Mathematically, for a true token with index y, the loss is -log(p_y), where p_y is the probability the model assigned to the correct token. Lower cross-entropy means better predictions. (Chapter 3)

Cross-Layer Attention (CLA): A KV cache compression technique (Brandon et al., arXiv:2405.12981, NeurIPS 2024) where multiple Transformer layers share the same KV cache instead of each layer maintaining its own. This can reduce KV cache memory by 2x with accuracy comparable to Multi-Query Attention. (Chapter 18)

D

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization): A reinforcement learning algorithm (Yu et al., arXiv:2503.14476, March 2025) that builds on GRPO with modifications including removing the KL divergence penalty, decoupling clipping bounds for positive and negative advantages, and filtering out uninformative prompts. Achieved 50% accuracy on AIME 2024 with Qwen 2.5 32B. (Chapter 15)

Data deduplication: The process of removing duplicate or near-duplicate documents from training data before pre-training. Deduplication prevents the model from memorizing repeated content and improves training efficiency. Common methods include exact hash matching and MinHash-based near-duplicate detection (with a typical Jaccard similarity threshold of 0.8). (Chapter 14)

Data parallelism: A distributed training strategy where each GPU holds a complete copy of the model and processes a different batch of training data. Gradients are averaged across GPUs after each step. The simplest form of distributed training, but limited by the memory required to hold the full model on each GPU. (Chapter 14)

Data wall: The projected point at which the supply of high-quality training data becomes a bottleneck for scaling language models. Estimates suggest the stock of publicly available, high-quality text on the internet is approximately 300 to 510 trillion tokens (Epoch AI, 2024). Since frontier models already train on 15 to 36 trillion tokens, the data wall motivates techniques like synthetic data generation, multi-epoch training, and data-efficient architectures such as MoE. (Chapters 12, 13)

Decoder-only: The Transformer architecture variant used by virtually all modern LLMs (GPT, LLaMA, Claude, Gemini, etc.). Unlike the original encoder-decoder Transformer, decoder-only models process input and output as a single sequence with causal masking, generating tokens autoregressively. (Chapters 1, 7)

DeepSeek-R1: A reasoning model released by DeepSeek on January 20, 2025, that demonstrates strong chain-of-thought reasoning capabilities. DeepSeek-R1 was trained using GRPO (Group Relative Policy Optimization) on the DeepSeek-V3 base model. Notable for making its reasoning process visible in the output and for producing distilled versions (1.5B to 70B) that transfer reasoning capabilities to smaller models. (Chapters 15, 16)

DeepSeek-V3: A 671-billion-parameter Mixture-of-Experts model released by DeepSeek in December 2024, with 37 billion active parameters per token. Notable for its use of Multi-head Latent Attention (MLA), which dramatically reduces KV cache memory, and for pioneering FP8 training. Trained for approximately $5.6 million in GPU compute across 2.788 million H800 GPU-hours. Architecture: 61 layers, kv_lora_rank=512, qk_rope_head_dim=64. (Chapters 8, 11, 12, 18)

DeepSeek-V3.2: An updated version of DeepSeek-V3 released on December 1, 2025, with 685 billion total parameters and 37 billion active parameters per token (the same MLA architecture as V3). Introduces DeepSeek Sparse Attention (DSA), a mechanism that selectively computes attention weights for improved long-context efficiency. Built via continued pre-training from a DeepSeek-V3.1-Terminus checkpoint (the base V3 was pre-trained on 14.8 trillion tokens). (Chapters 11, 12, Appendix B)

DeepSeek Sparse Attention (DSA): A sparse attention mechanism introduced in DeepSeek-V3.2 that selectively computes attention weights rather than processing every token against every other token. DSA achieves fine-grained sparse attention for the first time in production models, delivering substantial improvements in long-context training and inference efficiency while maintaining virtually identical output quality. (Chapter 12)

Dense model: A Transformer model where every parameter is active for every token, as opposed to a Mixture-of-Experts (MoE) model where only a subset of parameters is used per token. Examples of dense models include LLaMA 3 8B, LLaMA 3 70B, and Qwen3-8B. Dense models are simpler to train and serve but less compute-efficient at large scales, which is why most frontier models as of 2026 use MoE architectures. (Chapters 9, 11, 12, 24)

Devin: An autonomous AI software engineering agent developed by Cognition. Devin 2.0 launched on April 3, 2025, with a cloud IDE, interactive task planning, parallel agent usage, and integrated code search. Pricing starts at $20/month (Core plan) with additional usage billed per ACU (Agent Compute Unit). Devin can independently plan, write code, run tests, debug, and submit pull requests. (Chapter 23)

Distillation: The process of training a smaller “student” model to mimic the behavior of a larger “teacher” model. The student learns from the teacher’s output probability distributions (soft labels) rather than from the original training data alone. DeepSeek-R1 distilled models are examples: smaller models trained to replicate the reasoning patterns of the full DeepSeek-R1. (Chapter 15)

DistServe: A prefill-decode disaggregation system (Zhong et al., OSDI 2024, arXiv:2401.09670) that separates the compute-intensive prefill phase from the memory-bound decode phase onto different GPU pools. DistServe achieves up to 4.48x goodput improvement or 10.2x tighter SLO adherence compared to colocated serving. (Chapter 24)

DoRA (Weight-Decomposed Low-Rank Adaptation): An improvement on LoRA (arXiv:2402.09353, ICML 2024) that decomposes weight updates into magnitude and direction components, applying low-rank adaptation only to the direction. This more closely mimics full fine-tuning behavior and can improve quality over standard LoRA. (Chapter 28)

DPO (Direct Preference Optimization): An alignment method (Rafailov et al., NeurIPS 2023) that eliminates the need for a separate reward model by directly optimizing the language model on human preference data. DPO reformulates the RLHF objective as a simple classification loss on pairs of preferred and rejected responses, making it simpler and more stable than PPO-based RLHF. (Chapter 15)

Dropout: A regularization technique that randomly sets a fraction of neuron activations to zero during each training step. This prevents neurons from co-adapting and forces the network to learn more robust features. The original Transformer used dropout rates of 0.1 on attention weights and residual connections. Most modern LLMs use little or no dropout during pre-training, relying instead on the regularizing effect of large datasets and weight decay. (Chapters 3, 14)

Dynamic Context Parallelism: An NVIDIA optimization (announced January 28, 2026) that dynamically adjusts the context parallelism strategy based on sequence length during inference. Instead of using a fixed parallelism configuration, Dynamic CP selects the optimal split across GPUs for each request, achieving 1.48x speedup on long-context workloads. (Chapter 20)

E

Early fusion: A multimodal architecture approach where different modalities (text, images, audio) are combined at the input level and processed together through the same model from the start of training. Unlike pipeline or encoder-attached approaches, early fusion models learn cross-modal relationships from the ground up. Examples include GPT-4o, LLaMA 4 Maverick, Qwen 3.5, and Mistral Small 4. (Chapter 22)

Embedding: A dense vector representation of a token (or image, or other input) in a high-dimensional space. In LLMs, each token in the vocabulary is mapped to a vector (e.g., 4,096 or 8,192 dimensions) via a learned embedding table. Similar tokens have similar embeddings (nearby vectors). The embedding table is the model’s first layer and typically contains hundreds of millions to billions of parameters. (Chapter 5)

Emergent abilities: Capabilities that appear suddenly as models scale up, rather than improving gradually. For example, few-shot arithmetic ability is nearly absent in small models but appears abruptly at a certain scale. The existence and nature of emergent abilities is debated; some researchers argue they are artifacts of evaluation metrics rather than true phase transitions. (Chapter 13)

Encoder-decoder: The original Transformer architecture from Vaswani et al. (2017), which has separate encoder and decoder components. The encoder processes the input sequence bidirectionally, and the decoder generates the output sequence autoregressively while attending to the encoder’s representations via cross-attention. Used by models like T5 and the original Transformer for machine translation. Largely replaced by decoder-only architectures for modern LLMs. (Chapter 7)

EOS token (End of Sequence): A special token that signals the model to stop generating. When the model predicts the EOS token as the most likely next token, generation terminates. Different models use different EOS tokens (e.g., GPT-2 uses token ID 50256, LLaMA 3 uses token ID 128001). (Chapter 17)

Epoch: One complete pass through the entire training dataset. During pre-training, most LLMs train for less than one epoch on their data (they see each document once or fewer times). During fine-tuning, models typically train for 1 to 5 epochs on the smaller fine-tuning dataset. Training for too many epochs risks overfitting, where the model memorizes the training data rather than learning generalizable patterns. (Chapters 14, 15, 28)

Expert (in MoE): One of the parallel feed-forward network (FFN) blocks in a Mixture-of-Experts layer. Each expert has the same architecture but different learned weights, allowing it to specialize in different types of inputs. A router selects which experts process each token. For example, LLaMA 4 Maverick has 128 experts per MoE layer, with the router selecting the top 1 expert per token (plus a shared expert that always runs). (Chapter 12)

Expert parallelism: A distributed training and inference strategy for Mixture-of-Experts models that distributes different experts across different GPUs. When a token is routed to a specific expert, it is sent (via all-to-all communication) to the GPU hosting that expert. Expert parallelism is often combined with tensor parallelism and data parallelism to train large MoE models efficiently. (Chapters 12, 14)

Extended thinking: A capability of reasoning models where the model generates internal “thinking” tokens before producing its final answer. These thinking tokens contain step-by-step reasoning, self-correction, and exploration of different approaches. The thinking process may be visible (as in DeepSeek-R1 and Claude) or hidden (as in OpenAI’s o-series models). Extended thinking improves performance on math, coding, and complex reasoning tasks at the cost of higher latency and token usage. (Chapter 16)

Extrinsic hallucination: A type of hallucination where the model generates claims that cannot be verified from the provided context or any known source. For example, fabricating a citation to a paper that does not exist. Distinct from intrinsic hallucination, which contradicts the provided input. (Chapter 26)

F

Feed-forward network (FFN): The second major component of each Transformer layer (after attention). The FFN processes each token independently through two or three linear transformations with a non-linear activation function in between. In modern models using SwiGLU, the FFN has three weight matrices and an expansion ratio of approximately 2.67x (e.g., hidden size 4,096 expanded to intermediate size 12,288 in Qwen3-8B). The FFN contains approximately two-thirds of each layer’s parameters and is believed to store factual knowledge. (Chapter 9)

Few-shot learning: The ability of a language model to perform a task after seeing only a few examples in the prompt, without any fine-tuning. First demonstrated at scale by GPT-3 (2020). For example, showing the model three examples of English-to-French translation, then asking it to translate a new sentence. (Chapter 1)

FineWeb: A large-scale dataset released by Hugging Face on May 31, 2024, containing over 15 trillion tokens of cleaned and deduplicated English web data sourced from 96 Common Crawl snapshots (arXiv:2406.17557). FineWeb outperforms other open datasets in LLM training benchmarks. Its educational subset, FineWeb-Edu, further improves performance on knowledge and reasoning tasks. (Chapter 14)

Fine-tuning: The process of further training a pre-trained model on a specific dataset to adapt it for a particular task or behavior. Supervised Fine-Tuning (SFT) uses human-written examples of desired responses. Fine-tuning modifies the model’s weights, unlike prompting, which only changes the input. (Chapters 15, 28)

Flamingo: A vision-language model from DeepMind (Alayrac et al., arXiv:2204.14198, NeurIPS 2022) that introduced the Perceiver Resampler architecture for connecting vision encoders to language models. Flamingo uses 64 learned latent queries to compress variable-length visual features into a fixed number of visual tokens, which are then injected into the language model via cross-attention layers. (Chapter 21)

Flash Attention: A memory-efficient algorithm for computing attention that avoids materializing the full n x n attention score matrix. Instead, it computes attention in tiles, loading small blocks into fast on-chip SRAM. This reduces memory usage from O(n^2) to O(n) while producing mathematically identical results. FlashAttention-4 (arXiv:2603.05451, March 2026), implemented entirely in Python-embedded CuTe-DSL, achieves 1,613 TFLOPs/s on NVIDIA B200 GPUs at 71% utilization. (Chapter 20, Appendix A)

FLOPs (Floating-Point Operations): A measure of computational cost. One FLOP is a single arithmetic operation (addition or multiplication) on floating-point numbers. LLM training costs are often measured in FLOPs (e.g., GPT-3 required approximately 3.14 x 10^23 FLOPs). Per-layer FLOPs for a Transformer are approximately 24nd^2 + 4n^2d, where n is sequence length and d is hidden dimension. (Chapters 13, 14, Appendix A)

FP8: An 8-bit floating-point format supported on NVIDIA Hopper (H100) and Blackwell (B200, B300) GPUs. Each parameter occupies 1 byte, half the memory of BF16. A comprehensive study (Kurtic et al., arXiv:2411.02355, ACL 2025) across 500,000+ evaluations found FP8 inference to be “effectively lossless across all model scales.” FP8 is the default precision for production inference on modern data center GPUs as of March 2026. (Chapters 14, 24, Appendix B)

FP32 (Float32): The standard 32-bit floating-point format. Each parameter occupies 4 bytes. Used for optimizer states (master weights, momentum, variance) during training, but too memory-intensive for inference or storing model weights in modern practice. (Appendix B)

Frequency penalty: A generation parameter that reduces the probability of tokens proportional to how many times they have already appeared in the generated text. Higher values discourage repetition. Distinct from presence penalty, which applies a flat reduction regardless of count. (Chapter 17)

FSDP (Fully Sharded Data Parallelism): A distributed training strategy developed by Meta and integrated into PyTorch that shards model parameters, gradients, and optimizer states across all GPUs. Each GPU holds only a fraction of the total state, gathering full parameters on demand via all-gather operations. This trades communication bandwidth for memory savings, enabling training of models too large to fit on a single GPU. (Chapter 14)

Function calling: The mechanism that allows a language model to request the execution of external functions or tools. The model outputs a structured JSON object specifying the function name and arguments, the application executes the function, and the result is fed back to the model. Introduced by OpenAI on June 13, 2023. Also called “tool calling.” (Chapter 23)

G

Gated DeltaNet: A linear attention variant used in Qwen 3.5’s hybrid architecture. Gated DeltaNet replaces standard softmax attention with a linear recurrence that processes tokens in constant time per step, regardless of sequence length. Qwen 3.5 uses a hybrid design where approximately 75% of layers use Gated DeltaNet (linear attention) and 25% use standard sparse MoE attention, achieving 8-19x inference efficiency improvements over pure softmax attention models. (Chapter 12)

GELU (Gaussian Error Linear Unit): An activation function (Hendrycks and Gimpel, 2016) used in BERT and GPT-2. GELU provides a smooth, probabilistic transition around zero, unlike ReLU’s hard cutoff. Largely replaced by SwiGLU in modern LLMs. (Chapter 9)

Gemini: Google DeepMind’s family of multimodal language models. Gemini models are natively multimodal, processing text, images, audio, and video through a unified architecture. As of March 2026, the latest versions include Gemini 3.1 Pro (1 million token context) and Gemini 3.1 Flash. (Chapters 11, 20, 22)

Gemini Diffusion: A text diffusion model from Google DeepMind (announced May 2025) that generates text by iteratively denoising random tokens rather than generating left to right. Gemini Diffusion achieved 1,479 tokens per second on benchmarks (857 tokens per second in practical demos), dramatically faster than autoregressive generation. Represents a research direction that could complement or eventually challenge autoregressive generation. (Chapter 22)

Gemini Embedding 2: A multimodal embedding model released by Google on March 10, 2026, that produces 3,072-dimensional vectors from text, images, video, and audio inputs. Supports Matryoshka Representation Learning (MRL) for flexible dimensionality (128 to 3,072), processes up to 8,192 text tokens (4x the previous limit), and supports 100+ languages. (Chapter 22)

GGUF (GPT-Generated Unified Format): A binary file format for storing quantized LLM weights, designed for efficient loading and inference with llama.cpp and Ollama. GGUF replaced the older GGML format and supports multiple quantization levels (Q2, Q3, Q4, Q5, Q6, Q8, F16, F32) within a single file, along with embedded metadata (tokenizer vocabulary, model architecture, hyperparameters). GGUF is the standard format for running models locally on consumer hardware. (Chapter 25)

GloVe (Global Vectors for Word Representation): A word embedding method published by Pennington, Socher, and Manning at Stanford in 2014. GloVe combines matrix factorization with neural network approaches to produce fixed word vectors. Superseded by contextual embeddings from Transformer models. (Chapter 5)

GPT (Generative Pre-trained Transformer): A family of decoder-only Transformer models developed by OpenAI. GPT-1 (2018, 117M parameters) demonstrated pre-training plus fine-tuning. GPT-2 (2019, 1.5B) showed coherent text generation. GPT-3 (2020, 175B) demonstrated few-shot learning. GPT-4 (2023) introduced multimodal capabilities. GPT-5 (August 2025) unified reasoning and standard generation. GPT-5.4 (March 5, 2026) added native computer use and a 1.05 million token context window. (Chapters 1, 11, 16, 20)

GPT Image: OpenAI’s image generation capability integrated directly into GPT models. GPT Image 1 launched as an API on April 23, 2025, following the viral success of GPT-4o’s native image generation (March 25, 2025), which reached 130 million users and generated over 700 million images in its first week. GPT Image 1.5 (December 16, 2025) delivered 4x faster generation and 20% lower cost. GPT Image represents the shift from separate image generation models (like DALL-E) to native multimodal generation within the language model itself. (Chapter 22)

GPTQ: A post-training quantization method that compresses model weights to lower precision (typically INT4) by minimizing the layer-wise reconstruction error. GPTQ processes weights one layer at a time, using second-order information (the Hessian) to determine the optimal quantization. Widely used for deploying quantized models. (Chapters 24, 25)

Google ADK (Agent Development Kit): An open-source Python framework released by Google at Google Cloud NEXT 2025 on April 9, 2025, for building multi-agent AI applications. ADK is the same framework that powers agents within Google products like Agentspace. It integrates with Vertex AI and Gemini models but supports other LLM providers. (Chapters 23, 29)

GPQA (Graduate-Level Google-Proof Q&A): A benchmark consisting of expert-level science questions (biology, physics, chemistry) designed to be difficult even for domain experts with internet access. The “Diamond” subset contains the hardest questions. Used as a standard evaluation metric for reasoning capabilities. As of March 2026, top scores include Claude Opus 4.6 and GPT-5.4 in the low-to-mid 80s percent range. (Chapters 16, 20, 25)

Gradient: A vector of partial derivatives that indicates how much the loss function would change if each weight were adjusted by a small amount. Gradients are computed during backpropagation and used by the optimizer to update weights in the direction that reduces the loss. (Chapter 3)

Gradient accumulation: A technique that simulates a larger batch size by accumulating gradients over multiple forward-backward passes before performing a single optimizer step. If you accumulate over 4 steps with a micro-batch of 2, the effective batch size is 8. This allows training with large effective batch sizes on GPUs with limited memory. (Chapter 14)

Gradient checkpointing: A memory optimization technique for training that trades compute for memory. Instead of storing all intermediate activations during the forward pass (which are needed for backpropagation), gradient checkpointing discards some activations and recomputes them during the backward pass. This can reduce activation memory by 60-80% at the cost of approximately 30% more compute. Also called “activation checkpointing.” (Chapter 14)

Gradient descent: The optimization algorithm that updates model weights by moving them in the direction opposite to the gradient (the direction that reduces the loss). The step size is controlled by the learning rate. Stochastic gradient descent (SGD) computes gradients on mini-batches rather than the full dataset. Modern LLMs use AdamW, a variant with adaptive learning rates and weight decay. (Chapter 3)

Greedy decoding: The simplest text generation strategy, where the model always selects the single most likely token at each step. Greedy decoding is deterministic (the same input always produces the same output) but can produce repetitive or suboptimal text because it never explores alternative paths. Equivalent to temperature = 0. (Chapter 17)

Grok: xAI’s family of language models. Grok 4 Fast (September 2025) features a 2 million token context window, the largest among frontier models as of March 2026. Grok models are available through the xAI API and the X platform. (Chapters 11, 20)

GQA (Grouped Query Attention): An attention variant (Ainslie et al., 2023) that groups query heads and assigns one key-value head per group. For example, LLaMA 4 Maverick has 40 query heads and 8 KV heads (groups of 5). GQA reduces KV cache memory by a factor of h/h_kv compared to standard Multi-Head Attention while preserving most of the quality. Used by LLaMA 3, LLaMA 4, Qwen3, and Mistral models. (Chapter 8)

GRPO (Group Relative Policy Optimization): A reinforcement learning algorithm introduced in the DeepSeekMath paper (Shao et al., arXiv:2402.03300, February 2024). GRPO eliminates the critic model used in PPO by sampling multiple responses per prompt and computing advantages relative to the group mean. This reduces memory requirements by approximately 50% compared to PPO. Used to train DeepSeek-R1 and widely adopted for open-source reasoning model training. (Chapters 15, 28)

H

Hallucination: When a language model generates text that is factually incorrect, fabricated, or inconsistent with the provided context, while presenting it with the same confidence as accurate information. Intrinsic hallucinations contradict the source material; extrinsic hallucinations introduce unverifiable claims. Hallucinations are a fundamental limitation of current LLMs, arising from the model’s reliance on statistical patterns rather than grounded knowledge. (Chapter 26)

HBM (High Bandwidth Memory): A type of memory used in data center GPUs that provides much higher bandwidth than standard GDDR memory. HBM stacks memory chips vertically and connects them with a wide interface. HBM2e is used in A100 GPUs (2,039 GB/s), HBM3 in H100 (3,350 GB/s), and HBM3e in H200 (4,800 GB/s), B200, and B300 (8,000 GB/s). (Chapters 24, Appendix B)

Head dimension (d_k): The number of dimensions in each attention head’s query, key, and value vectors. Typically 128 in modern models (e.g., LLaMA 4 Maverick, Qwen3-8B). The total model dimension equals the number of heads times the head dimension (e.g., 40 heads x 128 = 5,120 for LLaMA 4 Maverick). (Chapters 7, 8, Appendix A)

Hidden dimension (d_model): The size of the vector representation for each token as it flows through the Transformer layers. Also called the model dimension or hidden size. Typical values range from 4,096 (small models like Qwen3-8B) to 12,288 or larger (frontier models). This is the fundamental “width” of the model. (Chapters 5, 10, 11)

H2O (Heavy-Hitter Oracle): A KV cache eviction method (Zhang et al., arXiv:2306.14048, NeurIPS 2023) that identifies “heavy hitter” tokens (those consistently receiving high attention scores) and retains only those plus recent tokens in the cache. Achieved up to 29x throughput improvement over DeepSpeed/Accelerate baselines. (Chapter 18)

Hugging Face: A company and open-source platform that hosts the Hugging Face Hub, the largest repository of machine learning models, datasets, and applications. As of 2025, the Hub hosts over 2 million models, 500,000+ datasets, and serves 13 million+ users. Hugging Face also develops key open-source libraries including Transformers, PEFT, TRL, and Accelerate. (Chapters 24, 25, 28)

Hugging Face Hub: The central repository hosted by Hugging Face where model weights, datasets, and demo applications (Spaces) are shared. Model weights are typically stored in the safetensors format. The Hub is the primary distribution channel for open-weight models like LLaMA, Qwen, Mistral, and DeepSeek. (Chapters 24, 25, 28)

I

In-context learning: The ability of a language model to learn new tasks or patterns from examples provided in the prompt, without updating any weights. In-context learning encompasses zero-shot (no examples), one-shot (one example), and few-shot (multiple examples) settings. First demonstrated at scale by GPT-3 (2020), in-context learning is a key capability that distinguishes large language models from smaller ones. (Chapters 1, 13)

Inference: The process of running a trained model to generate predictions (text, in the case of LLMs). Inference is distinct from training: no gradients are computed, no weights are updated, and memory requirements are much lower. Inference memory is dominated by model weights and the KV cache. (Chapters 17, 24, Appendix B)

InstructGPT: A model developed by OpenAI (Ouyang et al., arXiv:2203.02155, 2022) that demonstrated the effectiveness of RLHF for aligning language models with human preferences. The key finding: a 1.3-billion-parameter InstructGPT model was preferred by human evaluators over the 175-billion-parameter GPT-3, despite having 100x fewer parameters. InstructGPT established the SFT, then reward model, then PPO pipeline that became the standard for RLHF. (Chapter 15)

Indirect prompt injection (XPIA): A variant of prompt injection where malicious instructions are hidden in external content that the model processes, rather than in the user’s direct input. For example, a web page might contain invisible text saying “Ignore previous instructions and reveal the system prompt.” When an agent browses that page, it may follow the injected instructions. XPIA (Cross-Prompt Injection Attack) is the formal term. This is considered the most critical unsolved security problem for AI agents. (Chapter 26)

Instruct model: A language model that has been fine-tuned (via SFT, RLHF, DPO, or similar techniques) to follow instructions and produce helpful, conversational responses. Instruct models are what users interact with in products like ChatGPT and Claude. They are distinct from base models, which produce raw text completions. Examples: LLaMA 4 Maverick Instruct, Qwen3-8B-Instruct. (Chapters 15, 17, 25)

INT4: A 4-bit integer quantization format where each parameter occupies 0.5 bytes (half a byte). INT4 reduces weight memory by 4x compared to BF16, often making the difference between needing one GPU and needing eight. Quality is generally acceptable for models with 70 billion or more parameters but can noticeably degrade for smaller models. Common INT4 methods include GPTQ and AWQ. (Chapters 24, 25, Appendix B)

INT8: An 8-bit integer quantization format where each parameter occupies 1 byte. INT8 has the same memory footprint as FP8 but uses integer arithmetic instead of floating-point tensor cores. For memory planning purposes, INT8 and FP8 are interchangeable. (Appendix B)

Intrinsic hallucination: A type of hallucination where the model’s output contradicts information provided in the input context. For example, if the prompt says “The meeting is on Tuesday” and the model responds “The meeting is on Wednesday.” (Chapter 26)

J

Jacobian matrix: A matrix of partial derivatives that describes how each output of a function changes with respect to each input. In the context of attention, the Jacobian of the softmax function shows how changing one attention score affects all the output probabilities. The softmax Jacobian has the property that each row sums to approximately zero, meaning increasing attention to one token necessarily decreases attention to others. (Appendix A)

Jailbreaking: Techniques used to bypass a language model’s safety training and elicit responses the model was trained to refuse. Methods include role-playing scenarios, encoding harmful requests in unusual formats, and exploiting the model’s instruction-following tendencies. Jailbreaking is an ongoing adversarial challenge for model developers. (Chapter 26)

K

Key vector (K): In the attention mechanism, the key vector represents what information a token “offers” to other tokens. Each token’s key is compared (via dot product) with every other token’s query to compute attention scores. The key projection matrix W_K maps the input embedding to key vectors. (Chapter 7, Appendix A)

KIVI: A KV cache quantization method (Liu et al., arXiv:2402.02750, ICML 2024) that achieves 2-bit precision with minimal quality loss. KIVI’s key insight is that keys and values have different statistical distributions and should be quantized differently. Achieves 2.6x peak memory reduction and up to 4x batch size increase. (Chapter 18)

KV cache (Key-Value cache): A memory optimization for autoregressive generation that stores the key and value vectors computed for all previous tokens, avoiding redundant recomputation at each generation step. Without the KV cache, generating each new token would require reprocessing the entire sequence. KV cache memory scales linearly with sequence length, batch size, number of layers, and number of KV heads. For long contexts, the KV cache can exceed the model weight memory. (Chapter 18)

L

Layer normalization (LayerNorm): A normalization technique (Ba, Kiros, and Hinton, 2016) that computes the mean and variance across the dimensions of each token’s vector and normalizes them. This prevents values from growing too large or too small as they pass through many layers. Modern LLMs use RMSNorm (Root Mean Square Normalization), a simplified variant that only normalizes by the root mean square, omitting the mean subtraction. (Chapter 10)

Learning rate: A hyperparameter that controls the step size during gradient descent. Too large a learning rate causes training to diverge; too small a learning rate makes training slow. Modern LLM training uses learning rate schedules that warm up from zero, hold at a peak value, and then decay. Fine-tuning uses much lower learning rates (1e-6 to 1e-5) than pre-training (1e-4 to 3e-4) to avoid catastrophic forgetting. (Chapters 3, 14, 15)

LIMA (Less Is More for Alignment): A research paper (Zhou et al., NeurIPS 2023) demonstrating that a pre-trained LLaMA 65B model fine-tuned on only 1,000 carefully curated examples (750 from forums, 250 manually written) could produce high-quality responses competitive with models trained on much larger datasets. LIMA’s key insight, the “Superficial Alignment Hypothesis,” is that almost all knowledge is acquired during pre-training, and alignment is primarily about learning the style and format of interaction. (Chapter 15)

LLaMA (Large Language Model Meta AI): Meta’s family of open-weight language models. LLaMA 1 (February 2023) demonstrated that smaller, well-trained models could match larger ones. LLaMA 2 (July 2023) added chat fine-tuning. LLaMA 3 (April 2024) scaled to 405B parameters. LLaMA 4 (April 2025) introduced MoE architecture with Scout (109B total, 17B active, 16 experts) and Maverick (400B total, 17B active, 128 experts). (Chapters 1, 11, 12)

LLaVA (Large Language and Vision Assistant): A vision-language model (Liu et al., arXiv:2304.08485, NeurIPS 2023 Oral) that connects a CLIP vision encoder to a language model via a simple linear projection layer. LLaVA demonstrated that visual instruction tuning on GPT-4-generated data could produce strong multimodal capabilities. LLaVA-1.5 (arXiv:2310.03744) achieved competitive results with only ~208 GPU-hours of training on 558K pretrain and 665K instruction-tuning examples. (Chapter 21)

llama.cpp: An open-source C/C++ library for running LLM inference on consumer hardware, including CPUs and Apple Silicon. llama.cpp uses the GGUF file format and supports various quantization levels (Q4, Q5, Q8, etc.), enabling models to run on devices without dedicated GPUs. (Chapter 25)

LM Studio: A desktop application for running LLMs locally on personal computers with a graphical user interface. LM Studio supports GGUF model files, provides a chat interface, and includes an OpenAI-compatible local API server. Available for macOS, Windows, and Linux. (Chapter 25)

Load balancing (in MoE): The problem of ensuring all experts in a Mixture-of-Experts model receive a roughly equal share of tokens during training. Without load balancing, the router may learn to send most tokens to a few “popular” experts while others go unused. Solved using auxiliary loss functions that penalize uneven expert utilization. (Chapter 12)

Logit: The raw, unnormalized score that a language model assigns to each token in its vocabulary before applying softmax. Higher logits correspond to higher predicted probabilities. The logit vector has one entry per vocabulary token (e.g., 128,256 entries for LLaMA 3). Temperature scaling divides logits before softmax to control randomness. (Chapters 2, 17)

Longformer: A sparse attention method (Beltagy, Peters, and Cohan, arXiv:2004.05150, 2020) that combines sliding window attention for local context with global attention on designated tokens. This allows long-range information flow while keeping overall complexity linear in sequence length. (Chapter 20)

LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method (Hu et al., 2021) that freezes the pre-trained model weights and adds small trainable low-rank matrices (typically rank 8 to 64) to each layer. Instead of updating a full d x d weight matrix, LoRA updates two smaller matrices of shapes d x r and r x d, where r is the rank. This reduces trainable parameters to typically 0.5-2% of the total, enabling fine-tuning of large models on consumer GPUs. (Chapter 28)

Loss function: A mathematical function that measures how wrong the model’s predictions are. For language models, the standard loss function is cross-entropy loss, which measures the difference between the model’s predicted probability distribution and the true next token. Training minimizes the loss by adjusting weights via gradient descent. (Chapter 3)

Lost in the Middle: A phenomenon documented by Liu et al. (TACL 2024, 12:157-173) where language models perform significantly worse when relevant information is placed in the middle of a long input context, compared to the beginning or end. The effect produces a U-shaped accuracy curve: models attend most strongly to the first and last portions of the input. This is distinct from context rot, which affects the entire context. Lost in the Middle has practical implications for RAG systems and long-document question answering. (Chapter 20)

M

MCP (Model Context Protocol): An open protocol announced by Anthropic in November 2024 that standardizes how LLM applications connect to external tools and data sources. MCP defines a client-server architecture where the LLM application (client) communicates with MCP servers that expose tools, resources, and prompts through a standardized interface. Adopted by OpenAI in March 2025. As of March 2026, the MCP registry lists over 3,000 servers. Donated to the Linux Foundation in early 2026. (Chapter 23)

MetaCLIP: The vision encoder used by LLaMA 4 Maverick, based on Meta’s CLIP variant. MetaCLIP uses a ViT architecture with hidden_size=768, 34 layers, and 16 attention heads, producing vision features that are projected to the language model’s hidden dimension (vision_output_dim=7,680) via a learned projection layer. LLaMA 4 uses MetaCLIP with early fusion training on 40+ trillion tokens of interleaved multimodal data. (Chapter 21)

Micro-batching: A technique used in pipeline parallelism where a training batch is split into smaller micro-batches that flow through the pipeline stages in sequence. This keeps all GPUs busy simultaneously, reducing the “pipeline bubble” (idle time) that would occur if each stage had to wait for the previous one to finish processing the entire batch. (Chapter 14)

Min-p sampling: A dynamic token truncation method (Nguyen et al., arXiv:2407.01082, ICLR 2025 Oral) that sets a minimum probability threshold relative to the top token’s probability. If the top token has probability p_max, min-p filters out all tokens with probability below p_min * p_max. This adapts the candidate set to the model’s confidence: when the model is very confident, few tokens pass the threshold; when uncertain, more tokens are included. Adopted by Hugging Face Transformers, vLLM, and other frameworks. (Chapter 17)

MinHash: A locality-sensitive hashing algorithm used for near-duplicate detection in training data. MinHash computes hash signatures for documents based on their n-grams, and documents with sufficiently similar signatures (typically above a Jaccard similarity threshold of 0.8) are considered near-duplicates. Used extensively in data deduplication pipelines for LLM pre-training. (Chapter 14)

Mistral Small 4: A Mixture-of-Experts model released by Mistral AI on March 16, 2026, under the Apache 2.0 license. Architecture: 119 billion total parameters, 6.5 billion active per token (Mistral’s blog rounds to 6B), 128 experts with top-4 routing, 256K context window. Mistral Small 4 unifies instruct, reasoning, and multimodal (text + image) capabilities in a single model with configurable reasoning effort. Also uses Multi-head Latent Attention (MLA), the same attention variant as DeepSeek-V3. (Chapters 12, 22)

Mixed precision training: A training technique that uses different numerical precisions for different parts of the computation. Typically, model weights and activations are stored in BF16 (2 bytes) for forward and backward passes, while the optimizer maintains a master copy of weights in FP32 (4 bytes) for numerical stability. This halves the memory for weights and activations compared to pure FP32 training while maintaining training quality. (Chapter 14, Appendix B)

MLA (Multi-head Latent Attention): An attention variant used by DeepSeek-V2 and DeepSeek-V3 that compresses key and value information into a single low-rank latent vector. Instead of storing separate K and V vectors per head, MLA stores a compressed vector of dimension d_c (kv_lora_rank) plus a small RoPE vector of dimension d_r. For DeepSeek-V3, this reduces the KV cache to 576 values per token per layer, a 57x reduction compared to standard MHA with 128 heads. (Chapters 8, 18)

MMLU (Massive Multitask Language Understanding): A benchmark consisting of multiple-choice questions across 57 academic subjects (STEM, humanities, social sciences, etc.), ranging from elementary to professional difficulty. MMLU-Pro is a harder variant with 10 answer choices instead of 4 and more reasoning-intensive questions. Widely used to evaluate general knowledge and reasoning. (Chapter 25)

Model collapse: A phenomenon where training a model on synthetic data generated by other models (or by itself) leads to progressive degradation of output quality and diversity over successive generations. Each generation of synthetic data loses information from the tails of the distribution, eventually producing repetitive, low-quality outputs. Model collapse is a key concern as synthetic data becomes a larger fraction of training corpora. (Chapter 13)

MoE (Mixture of Experts): An architecture where each Transformer layer contains multiple parallel feed-forward networks (experts), but only a subset (top-K) is activated for each token. A small router network decides which experts to use. MoE allows models to have enormous total parameter counts (for knowledge capacity) while keeping per-token compute low (for inference speed). As of 2026, MoE is the dominant architecture for frontier models. Examples: DeepSeek-V3 (671B total, 37B active), LLaMA 4 Maverick (400B total, 17B active), Mistral Small 4 (119B total, 6B active). (Chapter 12)

MHA (Multi-Head Attention): The original attention design from Vaswani et al. (2017), where every query head has its own dedicated key and value head. This provides maximum expressiveness but requires the most KV cache memory. Largely replaced by GQA in modern models. (Chapter 8)

MQA (Multi-Query Attention): An attention variant (Shazeer, 2019) where all query heads share a single key head and a single value head. This minimizes KV cache memory but can reduce model quality. Used by PaLM and Falcon-7B. GQA is the more common middle ground in modern models. (Chapter 8)

Multi-token prediction: A training objective where the model predicts multiple future tokens simultaneously instead of just the next one. Proposed by Gloeckle et al. (arXiv:2404.19737, ICML 2024) and adopted by DeepSeek-V3 and Qwen 3.5. Can improve training efficiency and enable speculative decoding at inference time. (Chapter 17)

N

Nano Banana: Google’s viral image generation feature, launched in August 2025 as part of Gemini, that reached 23 million users within two weeks and generated over 500 million images. Named after a popular early prompt. Nano Banana Pro (November 2025) added 4K resolution with 94% text accuracy. Nano Banana 2 (February 2026) uses Gemini 3.1 Flash Image with 40-50% faster generation and 50% lower cost. (Chapter 22)

Needle-in-a-haystack (NIAH): A benchmark for evaluating long-context models by inserting a specific piece of information (the “needle”) at various positions within a long document (the “haystack”) and testing whether the model can retrieve it. Used to measure effective context utilization. LLaMA 4 Scout achieves 95%+ accuracy up to 8 million tokens but drops to 89% at 10 million. (Chapter 20)

NF4 (4-bit NormalFloat): A quantization data type introduced by Dettmers et al. in the QLoRA paper (arXiv:2305.14314, NeurIPS 2023). NF4 is information-theoretically optimal for normally distributed weights, providing better quantization quality than standard INT4 for the same memory footprint (0.5 bytes per parameter). Used specifically in QLoRA to quantize the frozen base model weights during fine-tuning. (Chapter 28)

Neuron: The basic computational unit of a neural network. A neuron takes multiple inputs, multiplies each by a weight, sums the results, adds a bias, and applies an activation function. In the context of LLMs, “neuron” typically refers to a single unit in a feed-forward layer. (Chapter 3)

NVFP4: NVIDIA’s native 4-bit floating-point format for Blackwell GPUs (B200, B300). NVFP4 uses micro-block scaling, where groups of 16 values share an FP8 E4M3 scale factor. It provides approximately 1.8x memory reduction over FP8 and 3.5x over FP16, with typically 1% or less accuracy degradation on large models. NVFP4 is the fastest inference precision available on Blackwell hardware but requires Blackwell GPUs. (Appendix B)

O

Ollama: An open-source tool for running LLMs locally on personal computers. Ollama provides a simple command-line interface for downloading and running models in GGUF format, with support for KV cache quantization via the OLLAMA_KV_CACHE_TYPE environment variable. (Chapters 18, 25)

Open weights: A model distribution approach where the trained model weights are publicly released, allowing anyone to download, run, fine-tune, and deploy the model. “Open weights” is distinct from “open source” because the training code, data, and full reproducibility may not be provided. Examples include LLaMA 4, Mistral, DeepSeek, and Qwen models. Licenses vary: some use Apache 2.0 (fully permissive), while others use custom licenses with restrictions (e.g., Meta’s LLaMA license). (Chapters 1, 24, 25)

OpenAI Agents SDK: A Python framework released by OpenAI on March 11, 2025, for building AI agents. Despite the name, the SDK is provider-agnostic and supports 100+ LLMs. Key features include built-in tool calling, guardrails, handoffs between agents, tracing, human-in-the-loop support, sessions with multiple storage backends, and realtime/voice agents. As of March 2026, the latest version is 0.12.5. (Chapters 23, 29)

OpenAI Codex: A cloud-based software engineering agent from OpenAI, powered by the codex-1 model. Codex CLI was released on April 16, 2025, as an open-source terminal tool under the MIT license. The cloud-based Codex agent launched on May 16, 2025, capable of working on multiple coding tasks in parallel within isolated sandbox environments. Codex can write features, fix bugs, answer codebase questions, and propose pull requests. (Chapter 23)

Operator: An AI agent product released by OpenAI on January 23, 2025, powered by the Computer-Using Agent (CUA) model. Operator can navigate websites and perform tasks autonomously by taking screenshots, interpreting GUI elements, and performing mouse clicks and keyboard actions. Unlike API-based integrations, Operator interacts with standard web interfaces the same way a human would. Initially available as a research preview to ChatGPT Pro subscribers in the US. (Chapter 23)

Optimizer: The algorithm that updates model weights during training based on computed gradients. The optimizer determines the direction and magnitude of weight updates. AdamW is the standard optimizer for LLM training, maintaining per-parameter momentum and variance estimates. (Chapters 3, 14)

Optimizer states: The additional values maintained by the optimizer for each parameter during training. For AdamW, this includes FP32 master weights (4 bytes), momentum (4 bytes), and variance (4 bytes), totaling 12 bytes per parameter on top of the 2-byte BF16 weights and 2-byte gradients. This is why training requires approximately 16 bytes per parameter, 8x more than BF16 inference. (Chapter 14, Appendix B)

OSWorld: A benchmark for evaluating AI agents on real-world computer tasks within actual operating system environments (Ubuntu, Windows, macOS). Tasks include web browsing, file management, and application use. GPT-5.4 achieved 75% on OSWorld-Verified, surpassing the 72.4% human baseline. (Chapters 21, 23, 25)

Output projection (W_O): The weight matrix that combines the outputs of all attention heads back into a single vector of the model’s hidden dimension. Shape: [h * d_v x d]. For LLaMA 4 Maverick, this is a [5120 x 5120] matrix with 26.2 million parameters. (Chapter 8, Appendix A)

Over-refusal: A failure mode of safety-trained models where the model refuses perfectly benign requests because it pattern-matches on surface-level features. For example, refusing to discuss the chemistry of explosives in a mining context because it detects the word “explosives.” (Chapter 26)

Overfitting: A failure mode where a model learns the training data too well, memorizing specific examples rather than learning generalizable patterns. An overfitting model performs well on training data but poorly on new, unseen data. In LLM fine-tuning, overfitting is mitigated by training for few epochs (typically 1 to 3), using low learning rates, and applying regularization techniques like weight decay and LoRA (which limits the number of trainable parameters). (Chapters 3, 15, 28)

P

PagedAttention: A memory management technique for the KV cache (Kwon et al., arXiv:2309.06180, SOSP 2023) that borrows the concept of virtual memory paging from operating systems. Instead of allocating one contiguous memory block per request, PagedAttention divides the KV cache into fixed-size pages, eliminating memory fragmentation and enabling 2-4x throughput improvement. Implemented in the vLLM serving framework. (Chapters 18, 24)

Parameter: A single learnable number in a neural network. Model size is measured in parameters. Each weight in every matrix of every layer is one parameter. A 70-billion-parameter model has 70 billion individual numbers that were adjusted during training. Parameters are stored in a specific precision (BF16, FP8, INT4, etc.), which determines the memory footprint. (Chapters 3, 11)

PEFT (Parameter-Efficient Fine-Tuning): A Hugging Face library that implements parameter-efficient fine-tuning methods including LoRA, QLoRA, DoRA, and others. PEFT integrates with Hugging Face Transformers and TRL, providing a unified interface for adding and managing adapters on pre-trained models. (Chapters 25, 28)

Perceiver Resampler: An architecture component introduced by Flamingo (Alayrac et al., arXiv:2204.14198, NeurIPS 2022) that compresses variable-length visual features into a fixed number of visual tokens using learned latent queries. Flamingo’s Perceiver Resampler uses 64 latent queries to produce 64 visual tokens regardless of the input image resolution. This fixed-length output is then injected into the language model via cross-attention layers. The Perceiver Resampler approach influenced subsequent vision-language architectures. (Chapter 21)

Perplexity: A measure of how “surprised” a language model is by a given text. Mathematically, perplexity is the exponential of the average cross-entropy loss: exp(loss). A perplexity of 1 means the model perfectly predicts every token; higher values indicate worse predictions. Used as a standard evaluation metric for language models. (Chapters 13, 27)

Pipeline parallelism: A distributed training strategy that splits the model’s layers across multiple GPUs, with each GPU responsible for a subset of layers. Data flows through the GPUs in sequence, like an assembly line. Micro-batching is used to keep all GPUs busy simultaneously. (Chapter 14)

Position embedding: See Positional encoding.

Positional encoding: Any method of injecting information about token order into a Transformer model, which otherwise has no built-in sense of sequence position. Methods include absolute position embeddings (GPT-2, BERT), Rotary Position Embeddings (RoPE, used by most modern models), and ALiBi. Without positional encoding, the model would treat “the cat sat on the mat” and “mat the on sat cat the” identically. (Chapter 6)

Power law: A mathematical relationship of the form y = ax^b, where a change in one quantity produces a proportional change in another. Scaling laws for LLMs follow power laws: loss decreases as a power function of model size, training data, and compute. On a log-log plot, power laws appear as straight lines, which is why scaling law papers plot loss vs. compute on logarithmic axes. (Chapter 13)

PPO (Proximal Policy Optimization): A reinforcement learning algorithm (Schulman et al., 2017) widely used in RLHF for language model alignment. PPO uses a clipped objective function to prevent large policy updates, maintaining training stability. It requires a separate critic (value) model, which adds memory overhead. Increasingly being replaced by simpler alternatives like DPO and GRPO. (Chapter 15)

Pre-training: The first and most expensive phase of training a language model, where the model learns to predict the next token on trillions of tokens of text. Pre-training teaches the model language, facts, reasoning patterns, and world knowledge. It requires thousands of GPUs running for weeks to months and costs tens to hundreds of millions of dollars for frontier models. (Chapter 14)

Prefill: The first phase of LLM inference, where the model processes the entire input prompt in parallel to build the KV cache. Prefill is compute-bound (lots of matrix multiplications) and determines the Time to First Token (TTFT). After prefill, the model switches to the decode phase, generating tokens one at a time. (Chapters 17, 18, 24)

Prefill-decode disaggregation: A serving architecture that separates the compute-intensive prefill phase from the memory-bound decode phase onto different GPU pools, each optimized for its workload. Prefill GPUs are configured for maximum compute throughput; decode GPUs are configured for maximum memory bandwidth and batch size. Systems include DistServe (OSDI 2024, up to 4.48x goodput) and Splitwise (ISCA 2024, 1.4x throughput at 20% lower cost). (Chapter 24)

Presence penalty: A generation parameter that applies a flat reduction to the probability of any token that has already appeared in the generated text, regardless of how many times it appeared. Distinct from frequency penalty, which scales with occurrence count. (Chapter 17)

Prompt caching: A server-side optimization where the KV cache computed for a prompt prefix is stored and reused across subsequent requests that share the same prefix. This avoids redundant computation for repeated content like system prompts, tool definitions, and conversation history. OpenAI, Anthropic, and Google all offer prompt caching with discounts of 50-90% on cached input tokens. (Chapter 19)

Prompt engineering: The practice of crafting input prompts to elicit desired behavior from a language model. Techniques include providing clear instructions, using few-shot examples, specifying output format, and chain-of-thought prompting. Prompt engineering is distinct from fine-tuning because it does not modify the model’s weights. (Chapters 16, 23)

Prompt injection: An attack where malicious instructions are embedded in the input to a language model, attempting to override the model’s system prompt or safety training. Direct prompt injection places the attack in the user’s message; indirect prompt injection hides it in external content (web pages, documents) that the model processes. (Chapter 26)

Q

QLoRA (Quantized LoRA): A fine-tuning method (Dettmers et al., arXiv:2305.14314, NeurIPS 2023) that combines LoRA with NF4 (4-bit NormalFloat) quantization of the base model weights. This allows fine-tuning a large model on a single consumer GPU by keeping the frozen base weights in 4-bit precision while training the LoRA adapters in higher precision. (Chapter 28)

Qwen 3.5: Alibaba Cloud’s flagship open-source model, released on February 16, 2026, under the Apache 2.0 license. Architecture: 397 billion total parameters, 17 billion active per token, using a Mixture-of-Experts design with a hybrid attention mechanism that combines Gated DeltaNet (linear attention) with standard softmax attention in a 3:1 ratio. Supports 201 languages, 256K native context (1M via API), and native multimodal capabilities (text, image, video). (Chapters 12, 22)

Qwen3-VL: A family of vision-language models from Alibaba (arXiv:2511.21631, November 2025) available in both dense (2B/4B/8B/32B) and MoE (30B-A3B/235B-A22B) configurations. Features a 256K context window, DeepStack architecture for efficient visual processing, and 99.5% needle-in-a-haystack accuracy on 2-hour video inputs. (Chapter 21)

Quantization: The process of reducing the numerical precision of model weights (and sometimes activations) to use fewer bits per parameter. Common quantization levels include FP8 (1 byte), INT8 (1 byte), INT4 (0.5 bytes), and NVFP4 (0.5 bytes). Quantization reduces memory requirements and can speed up inference, at the cost of some accuracy loss (which varies by method and model size). (Chapters 24, 25, Appendix B)

Query vector (Q): In the attention mechanism, the query vector represents what information a token is “looking for” from other tokens. Each token’s query is compared (via dot product) with every other token’s key to compute attention scores. The query projection matrix W_Q maps the input embedding to query vectors. (Chapter 7, Appendix A)

R

RadixAttention: A tree-based KV cache management system introduced by SGLang (Zheng et al., arXiv:2312.07104, NeurIPS 2024) that organizes cached prefixes in a radix tree data structure. This enables efficient prefix sharing across requests: if multiple requests share the same system prompt or conversation prefix, the KV cache for that prefix is computed once and reused. RadixAttention achieves up to 6.4x throughput improvement over baseline serving. (Chapters 19, 24)

ReAct (Reasoning + Acting): A prompting framework (Yao et al., arXiv:2210.03629, ICLR 2023) that interleaves reasoning steps with action steps (tool calls). The model thinks about what to do, takes an action, observes the result, and repeats. ReAct is the conceptual foundation of modern LLM agents. (Chapter 23)

Reasoning model: A language model specifically trained or configured to perform step-by-step reasoning before producing a final answer. Examples include OpenAI’s o-series (o1, o3), DeepSeek-R1, Claude with extended thinking, and Gemini 2.5 Deep Think. Reasoning models trade latency for accuracy on complex tasks. (Chapter 16)

REINFORCE++: An enhanced variant of the classical REINFORCE algorithm (Hu, arXiv:2501.03262, January 2025) that incorporates key optimization techniques from PPO (clipped surrogate objective, mini-batch updates, advantage normalization) while eliminating the need for a critic network. REINFORCE++ achieves training stability comparable to PPO with the simplicity and reduced memory overhead of REINFORCE. It is one of several critic-free RL methods (alongside GRPO and DPO) used for LLM alignment. (Chapter 15)

Rejection sampling: A technique used in fine-tuning where multiple candidate responses are generated for each prompt, scored by a reward model, and only the best response is kept as training data. Used extensively by Meta for LLaMA 3 SFT data generation. (Chapter 15)

ReLU (Rectified Linear Unit): An activation function that returns the input if positive and zero otherwise: ReLU(x) = max(0, x). Simple and computationally efficient, but has the “dying ReLU” problem where neurons can permanently output zero. Largely replaced by GELU and SwiGLU in modern LLMs. (Chapters 3, 9)

Residual connection: A direct path (skip connection) that bypasses a layer’s computation by adding the layer’s input to its output: output = layer(x) + x. This allows gradients to flow directly through the network during backpropagation, enabling training of very deep networks (80-120+ layers in modern LLMs). Without residual connections, deep Transformers would be untrainable. (Chapter 10)

Responses API: OpenAI’s primary API for interacting with language models, released on March 11, 2025, as the successor to the Chat Completions API for agentic workflows. The Responses API natively supports tool calling, web search, file search, computer use, and multi-turn conversations with built-in state management. It replaces the Assistants API (sunset scheduled for August 26, 2026) and is the recommended interface for building agents with OpenAI models. (Chapters 23, 29)

Reward model: A model trained to predict human preferences, used in RLHF to provide a scalar reward signal for reinforcement learning. The reward model is typically initialized from the same pre-trained model and fine-tuned on human preference data (pairs of responses where annotators indicated which was better). (Chapter 15)

Ring Attention: A distributed attention algorithm (Liu et al., arXiv:2310.01889, 2023) that distributes the attention computation across multiple GPUs arranged in a ring topology. Each GPU holds a portion of the KV cache and passes blocks around the ring, enabling sequences 512x longer than what a single GPU can handle (exceeding 100 million tokens). (Chapter 20)

RLHF (Reinforcement Learning from Human Feedback): A training technique where a language model is fine-tuned using reinforcement learning, with rewards provided by a model trained on human preferences. The standard RLHF pipeline has three stages: (1) supervised fine-tuning, (2) reward model training on human preference data, and (3) RL optimization (typically PPO) to maximize the reward while staying close to the SFT model. Introduced by Christiano et al. (NeurIPS 2017) and popularized by InstructGPT (Ouyang et al., 2022). (Chapter 15)

RMSNorm (Root Mean Square Normalization): A simplified variant of layer normalization used by most modern LLMs (LLaMA, Mistral, Qwen, DeepSeek). RMSNorm normalizes by dividing by the root mean square of the vector elements, omitting the mean subtraction step of standard LayerNorm. This is computationally cheaper and empirically works just as well. (Chapter 10)

RoPE (Rotary Position Embeddings): The dominant positional encoding method in modern LLMs, proposed by Su et al. in 2021. Instead of adding positional information to embeddings, RoPE rotates the query and key vectors in the attention mechanism based on their position. The rotation angle depends on position and dimension, encoding relative position information directly into the attention scores. RoPE naturally supports context extension through techniques like YaRN. Used by LLaMA, Mistral, Qwen, DeepSeek, and most other open models. (Chapter 6)

Router (in MoE): A small neural network in each Mixture-of-Experts layer that decides which experts process each token. The router takes the token’s hidden state as input and produces a probability distribution over experts. The top-K experts (by probability) are selected, and their outputs are combined using the router’s scores as weights. Different models use different gating functions: softmax (Mixtral, LLaMA 4) or sigmoid (DeepSeek-V3). (Chapter 12)

S

Safetensors: A file format developed by Hugging Face for storing model weights safely and efficiently. Unlike pickle-based formats (which can execute arbitrary code on load), safetensors uses a simple binary layout that prevents code injection attacks. Safetensors also supports memory-mapped loading for fast startup. It is the standard format for model weights on the Hugging Face Hub. (Chapters 18, 25)

Scaling law: A mathematical relationship describing how model performance (measured by loss) improves as a function of model size, training data, and compute. The two major scaling laws are from Kaplan et al. (2020), which found power-law relationships between loss and each factor, and Hoffmann et al. (2022, “Chinchilla”), which established compute-optimal training ratios. Scaling laws enable labs to predict the performance of models before training them. (Chapter 13)

Self-attention: The specific form of attention used in Transformers, where the queries, keys, and values all come from the same sequence. Each token attends to every other token in the same input, allowing the model to capture relationships between any pair of positions. This is in contrast to cross-attention, where queries and keys/values come from different sequences. (Chapter 7)

SentencePiece: An open-source tokenization library developed by Google that treats input as a raw stream of Unicode characters (no pre-tokenization). SentencePiece supports both BPE and the Unigram Language Model algorithm and is language-agnostic, making it suitable for multilingual models. Used by LLaMA 1/2, Mistral (older models), and many multilingual models. (Chapter 4)

Sequence length (n): The number of tokens in the input sequence being processed. Sequence length determines the size of the attention score matrix (n x n), the KV cache memory, and the computational cost of attention. Also called “context length” when referring to the maximum supported value. (Chapters 7, 20)

SFT (Supervised Fine-Tuning): The process of fine-tuning a pre-trained model on human-written examples of desired input-output behavior. SFT is typically the first step after pre-training, teaching the model to follow instructions and produce helpful responses. The loss is computed only on the assistant’s response tokens, not the user’s prompt. (Chapter 15)

SGLang: A high-performance LLM serving framework that introduces RadixAttention, a tree-based KV cache management system that enables efficient prefix sharing across requests. SGLang achieves up to 6.4x throughput improvement over baseline serving. Published at NeurIPS 2024 (Zheng et al., arXiv:2312.07104). SGLang v0.4 added a cache-aware load balancer achieving 1.9x throughput and 3.8x cache hit rate improvements. (Chapters 19, 24)

Sigmoid gating: A routing mechanism used in some MoE models (notably DeepSeek-V3) where each expert’s score is passed through a sigmoid function independently, producing values between 0 and 1. Unlike softmax gating, the scores are not normalized to sum to 1, decoupling experts’ contributions from each other. (Chapter 12)

SigLIP (Sigmoid Loss for Language Image Pre-training): An improvement on CLIP (Zhai et al., Google, 2023) that replaces the softmax-based contrastive loss with a simpler sigmoid loss. SigLIP treats each image-text pair independently, allowing training with larger batch sizes and producing better-calibrated similarity scores. SigLIP 2 (arXiv:2502.14786, February 2025) further improved performance with multi-resolution support and enhanced multilingual capabilities. (Chapter 21)

Sliding window attention (SWA): A sparse attention pattern where each token attends only to a fixed window of nearby tokens (e.g., the 4,096 nearest tokens) rather than the entire sequence. This reduces attention complexity from O(n^2) to O(n * w), where w is the window size. Used by Mistral 7B and as a component of Longformer and BigBird. (Chapter 20)

Softmax: A mathematical function that converts a vector of raw scores (logits) into a probability distribution. Each output value is between 0 and 1, and all outputs sum to 1. The formula is: softmax(x_i) = exp(x_i) / sum(exp(x_j)). Used in attention (to convert scores to weights) and in the final layer (to convert logits to token probabilities). (Chapter 2)

Softmax gating: A routing mechanism used in some MoE models (Mixtral, LLaMA 4) where the router scores for the selected top-K experts are passed through softmax, normalizing them to sum to 1. This means the selected experts’ contributions are treated as a weighted average. (Chapter 12)

Speculative decoding: An inference optimization where a small, fast “draft” model generates several candidate tokens, and the large “target” model verifies them in a single forward pass. If the draft tokens match what the target model would have generated, multiple tokens are accepted at once, achieving 2-3x speedup with no quality loss. (Chapter 24)

Splitwise: A prefill-decode disaggregation system (Patel et al., ISCA 2024, arXiv:2311.18677) from Microsoft Research that separates prefill and decode phases onto different GPU pools. Splitwise achieves 1.4x throughput at 20% lower cost, or 2.35x throughput at the same budget, compared to colocated serving. (Chapter 24)

Stanford Alpaca: An early instruction-following model created by Stanford researchers in March 2023 by fine-tuning Meta’s LLaMA 7B on 52,000 instruction-response pairs generated by OpenAI’s text-davinci-003. Alpaca demonstrated that a small, inexpensive fine-tuning run (under $600 in API costs for data generation) could produce a model with behavior qualitatively similar to text-davinci-003. Alpaca was influential in establishing the viability of instruction tuning with synthetic data. (Chapter 28)

StreamingLLM: A KV cache management technique (Xiao et al., arXiv:2309.17453, ICLR 2024) that maintains a small set of “attention sink” tokens (typically the first few tokens) plus a sliding window of recent tokens. This enables theoretically infinite-length generation with fixed memory. Achieved 22.2x speedup over sliding window recomputation. (Chapter 18)

Streamable HTTP: The transport protocol that replaced Server-Sent Events (SSE) in the MCP specification on March 26, 2025. Streamable HTTP uses standard HTTP requests with optional streaming responses, simplifying deployment (no persistent connections required) while maintaining real-time communication capabilities. (Chapters 23, 29)

Structured Outputs: A feature introduced by OpenAI on August 6, 2024, that guarantees the model’s function call arguments exactly match a provided JSON Schema. With strict: true in the tool definition, the output conforms to the schema with 100% reliability, eliminating the need for output validation. (Chapter 23)

System prompt: A special message at the beginning of a conversation that sets the model’s behavior, personality, and constraints. System prompts are processed as part of the input but are typically hidden from the user. They are a primary target for prompt caching because they remain constant across many requests. (Chapters 19, 23)

SWE-bench: A benchmark for evaluating AI coding agents on real-world GitHub issues. Agents must understand bug reports, navigate codebases, and generate patches that pass test suites. SWE-bench Verified is a human-validated subset. As of March 2026, Claude Opus 4.6 (Thinking) leads at 79.2%. OpenAI discontinued reporting SWE-bench Verified scores in March 2026, citing benchmark contamination and flawed test cases. (Chapters 23, 25)

SwiGLU: The dominant feed-forward network activation in modern LLMs, combining the Swish activation with the Gated Linear Unit (GLU) mechanism. Proposed by Noam Shazeer in 2020. SwiGLU uses three weight matrices instead of two, with one matrix producing a gate that modulates the other’s output. Used by LLaMA, Mistral, Qwen, DeepSeek, and most other modern models. (Chapter 9)

Sycophancy: A failure mode of aligned language models where the model agrees with the user’s stated beliefs or preferences even when they are factually incorrect, rather than providing accurate information. Sycophancy arises from RLHF training, where human raters tend to prefer agreeable responses. It is a measurable alignment failure: models will change correct answers to incorrect ones when the user expresses disagreement. (Chapter 26)

Synthetic data: Training data generated by AI models rather than humans. Used to supplement human-generated data, particularly for domains like code and mathematics where correctness can be verified. DeepSeek-V3 uses an internal DeepSeek-R1 model to generate reasoning data. Qwen 3’s training data includes synthetic code and math content. (Chapters 13, 14, 15)

T

Temperature: A generation parameter that controls the randomness of the model’s output by scaling the logits before softmax. Temperature = 0 makes the model deterministic (always picking the most likely token). Temperature = 1 uses the model’s raw probabilities. Temperature > 1 makes the distribution more uniform (more random). Temperature < 1 makes it more peaked (more deterministic). (Chapter 17)

Tensor parallelism: A distributed training and inference strategy that splits individual weight matrices across multiple GPUs. Each GPU computes a portion of each matrix multiplication, and the results are combined. This allows a single layer to span multiple GPUs, enabling models whose individual layers are too large for one GPU’s memory. (Chapters 14, 24)

TensorRT-LLM: NVIDIA’s optimized inference engine for LLMs, achieving up to 10,000 tokens per second on H100 with FP8 precision. TensorRT-LLM applies kernel fusion, quantization, and other optimizations specific to NVIDIA hardware. (Chapter 24)

TGI (Text Generation Inference): A serving toolkit developed by Hugging Face for deploying LLMs. TGI supports tensor parallelism, dynamic batching, and quantization. Placed in maintenance mode on December 11, 2025, with Hugging Face recommending vLLM or SGLang for new deployments. (Chapters 23, 24)

tiktoken: OpenAI’s tokenization library, written in Rust with Python bindings. tiktoken implements byte-level BPE and is used by all OpenAI models (GPT-2 through GPT-5.4), as well as LLaMA 3, LLaMA 4, and Mistral’s Tekken tokenizer. (Chapter 4)

Token: The fundamental unit of text that a language model processes. Tokens are subword units produced by a tokenizer (e.g., BPE). Common English words are typically one token; less common words may be split into multiple tokens. A rough approximation is that one token equals about 0.75 English words, or about 4 characters. Modern vocabularies contain 100,000 to 200,000 tokens. (Chapter 4)

Tokenizer: The component that converts raw text into a sequence of token IDs (integers) that the model can process, and converts token IDs back into text. The tokenizer defines the vocabulary (the set of all possible tokens) and the rules for splitting text. Common tokenizer implementations include BPE (via tiktoken or SentencePiece), Unigram, and WordPiece. (Chapter 4)

Top-k sampling: A decoding strategy that restricts the model’s choices to the k most likely tokens at each step, then samples from only those tokens (after renormalizing probabilities). All other tokens receive zero probability. A fixed k can be suboptimal because the appropriate number of candidates varies by context. (Chapter 17)

Top-nσ sampling: A token sampling method (Tang et al., arXiv:2411.07641, ACL 2025) that operates directly on pre-softmax logits rather than post-softmax probabilities. Top-nσ filters out tokens whose logit scores fall below n standard deviations from the maximum logit. The key insight is that logits naturally separate into a Gaussian-distributed noisy region and a distinct informative region, enabling efficient filtering without complex probability manipulations. Unlike temperature-based methods, top-nσ maintains stable performance across different temperature settings. (Chapter 17)

Top-p sampling (nucleus sampling): A decoding strategy (Holtzman et al., arXiv:1904.09751, ICLR 2020) that dynamically selects the smallest set of tokens whose cumulative probability exceeds a threshold p (e.g., 0.9). This adapts the number of candidates to the shape of the distribution: when the model is confident, fewer tokens are considered; when uncertain, more are included. (Chapter 17)

Transformer: The neural network architecture introduced by Vaswani et al. in “Attention Is All You Need” (NeurIPS 2017) that underlies virtually all modern language models. A Transformer consists of stacked layers, each containing a self-attention mechanism and a feed-forward network, connected by residual connections and layer normalization. The key innovation is the attention mechanism, which allows every token to attend to every other token in parallel, replacing the sequential processing of RNNs. (Chapters 1, 7, 10)

Transfusion: A training approach (arXiv:2408.11039, August 2024) that trains a single model on both text (using next-token prediction) and images (using diffusion loss) simultaneously. This enables a unified model that can both understand and generate images alongside text, without separate vision and language components. (Chapter 22)

TRL (Transformer Reinforcement Learning): A Hugging Face library for training language models with reinforcement learning techniques including SFT, DPO, GRPO, PPO, and reward modeling. TRL integrates with Hugging Face Transformers and PEFT, providing high-level trainer classes (SFTTrainer, DPOTrainer, GRPOTrainer) that handle the complexity of RL-based fine-tuning. (Chapters 17, 25, 28)

t-SNE (t-distributed Stochastic Neighbor Embedding): A dimensionality reduction technique commonly used to visualize high-dimensional data (like word embeddings) in 2D or 3D. t-SNE preserves local neighborhood structure, so tokens that are nearby in the original high-dimensional space remain nearby in the visualization. Used in Chapter 5 to visualize embedding spaces. (Chapter 5)

TPU (Tensor Processing Unit): Google’s custom-designed AI accelerator chips, purpose-built for machine learning workloads. TPUs are used to train and serve Google’s Gemini models and are available to external users via Google Cloud. As of March 2026, the latest generation is TPU v7 (codenamed Ironwood), announced at Google Cloud Next 2025, featuring 4,614 FP8 TFLOPS, 192 GB HBM3e, 7.38 TB/s memory bandwidth, and pods of up to 9,216 chips delivering 42.5 ExaFLOPS. Previous generations include TPU v6e (Trillium, 918 BF16 TFLOPS, 32 GB HBM) and TPU v5e. (Chapter 24)

TTFT (Time to First Token): The latency from when a request is sent until the first response token is generated. TTFT is dominated by the prefill phase, where the model processes the entire input prompt. Longer prompts mean longer TTFT. For short prompts, TTFT is typically 100-300 milliseconds; for very long prompts (100K+ tokens), it can be several seconds. (Chapters 17, 18, 24)

TPS (Tokens Per Second): A measure of generation speed during the decode phase. Also called throughput or inter-token latency. Typical values for frontier models via API range from 30 to 150 tokens per second, depending on model size and hardware. TPS is relatively constant regardless of prompt length but slows slightly as the sequence grows (due to increasing KV cache size). (Chapter 17)

U

Unsloth: An open-source library (Apache 2.0) for efficient LLM fine-tuning and reinforcement learning. Unsloth rewrites performance-critical parts of Hugging Face Transformers with hand-optimized Triton kernels, delivering 2x training speed and 70% VRAM reduction with no accuracy loss. It is a drop-in replacement for standard TRL/PEFT workflows. Supports LoRA, QLoRA, and GRPO training for models including Qwen3, LLaMA 4, DeepSeek, and others. (Chapters 25, 28)

Unified memory: A memory architecture used by Apple Silicon (M-series chips) where the CPU and GPU share the same physical memory pool. This means the entire system memory (up to 512 GB on the Mac Studio M3 Ultra) is available to the GPU for model inference, unlike discrete GPUs where VRAM is separate and limited. The tradeoff is lower memory bandwidth compared to HBM-based data center GPUs. (Appendix B)

Unigram Language Model: A tokenization algorithm (alternative to BPE) supported by SentencePiece. Instead of building the vocabulary bottom-up by merging pairs, Unigram starts with a large vocabulary and iteratively removes tokens that contribute least to the overall likelihood of the training data. (Chapter 4)

V

Value vector (V): In the attention mechanism, the value vector contains the actual information that gets passed forward when a token is attended to. The attention weights (computed from query-key similarity) determine how much of each token’s value vector contributes to the output. The value projection matrix W_V maps the input embedding to value vectors. (Chapter 7, Appendix A)

ViT (Vision Transformer): A Transformer architecture applied to images, introduced by Dosovitskiy et al. (2020). ViT divides an image into fixed-size patches (e.g., 14x14 pixels), treats each patch as a “token,” and processes them through standard Transformer layers. ViT is the basis of most vision encoders in multimodal LLMs. Common variants include ViT-Base (86M parameters), ViT-Large (307M), and ViT-Huge (632M). (Chapter 21)

vLLM: An open-source LLM serving framework that introduced PagedAttention for efficient KV cache management. vLLM achieves 2-4x throughput improvement over naive serving by eliminating memory fragmentation and enabling efficient memory sharing. Published at SOSP 2023 (Kwon et al., arXiv:2309.06180). (Chapters 18, 24, 25)

Vocabulary: The complete set of tokens that a model can recognize and generate. Each token has a unique integer ID. Modern LLM vocabularies range from approximately 32,000 tokens (older models) to over 200,000 tokens (LLaMA 4 Maverick: 202,048). The vocabulary is defined by the tokenizer and fixed after training. (Chapter 4)

VQ-VAE (Vector Quantized Variational Autoencoder): A generative model architecture (van den Oord et al., NeurIPS 2017) that compresses continuous data (such as images) into discrete tokens using a learned codebook of embedding vectors. The encoder maps input patches to the nearest codebook entry, producing a sequence of discrete token IDs. VQ-VAEs are the foundation of visual tokenization in natively multimodal models like Chameleon, which uses a codebook of 8,192 entries to encode images into 1,024 discrete tokens per 512x512 image. (Chapter 22)

VRAM (Video RAM): The dedicated memory on a GPU, used to store model weights, the KV cache, activations, and other data during inference or training. Consumer GPUs have 24-32 GB of VRAM (RTX 4090/5090); data center GPUs have 80-288 GB (H100 through B300). VRAM capacity is often the primary constraint determining which models can run on which hardware. (Chapters 24, Appendix B)

W

Warmup: The initial phase of a learning rate schedule where the learning rate increases linearly from zero (or near zero) to its peak value over a set number of training steps. Warmup prevents training instability at the start, when the model’s weights are randomly initialized and gradients can be large and noisy. Typical warmup durations are 1,000 to 10,000 steps. (Chapter 14)

Weight: A single learnable number in a neural network. Weights are organized into matrices (e.g., the query projection matrix W_Q, the feed-forward up-projection matrix). During training, weights are adjusted by the optimizer to minimize the loss function. The collection of all weights constitutes the model’s parameters. (Chapter 3)

Weight decay: A regularization technique that adds a penalty proportional to the magnitude of the weights to the loss function, preventing weights from growing too large. In AdamW, weight decay is applied directly to the weights rather than through the gradient, which is mathematically different from L2 regularization. (Chapter 14)

Whisper: OpenAI’s speech recognition model, released in September 2022. Trained on 680,000 hours of multilingual audio data. Whisper converts speech to text and is used as the audio input component in pipeline-based multimodal systems. (Chapter 22)

Word2Vec: A word embedding method introduced by Mikolov et al. at Google in 2013. Word2Vec demonstrated that neural networks could learn meaningful word representations from large text corpora, including the famous analogy relationships (King - Man + Woman = Queen). Superseded by contextual embeddings from Transformer models. (Chapter 5)

Y

YaRN (Yet another RoPE extensioN): A context extension technique (Peng et al., 2023) that applies different scaling factors to different frequency components of RoPE. High-frequency dimensions (encoding local positions) are left unscaled, low-frequency dimensions (encoding global positions) are interpolated, and intermediate dimensions get a blend. YaRN also includes an attention temperature correction. Used by Qwen3-8B to extend from 32,768 native context to 131,072 tokens. (Chapter 6)

Z

ZeRO (Zero Redundancy Optimizer): A family of memory optimization techniques from Microsoft’s DeepSpeed library that progressively shard optimizer states (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across GPUs. ZeRO-3 is the most aggressive: each GPU holds only 1/N of the total state (where N is the number of GPUs), gathering full parameters on demand via all-gather operations. This enables training models that would not fit on any single GPU, at the cost of 10-30% communication overhead. Conceptually similar to PyTorch’s FSDP. (Chapter 14, Appendix B)

Zero-shot learning: The ability of a language model to perform a task it has never been explicitly trained on, given only a natural language description of the task in the prompt (no examples). For instance, asking a model to “Translate the following English text to French:” without providing any translation examples. Zero-shot performance improves dramatically with model scale and is a key capability of large language models. (Chapters 1, 13)

Numerical and Symbol Reference

Abbreviation Quick Reference

Appendix D provides a chronological timeline of the key milestones referenced throughout this glossary, from the original Transformer paper in 2017 to the frontier models of March 2026.

Sources: All definitions in this glossary are derived from the corresponding chapters of this book, which contain full source citations for every factual claim. Key primary sources include: Vaswani et al., “Attention Is All You Need,” NeurIPS 2017 (arxiv.org/abs/1706.03762). Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers,” 2018 (arxiv.org/abs/1810.04805). Brown et al., “Language Models are Few-Shot Learners” (GPT-3), NeurIPS 2020 (arxiv.org/abs/2005.14165). Hoffmann et al., “Training Compute-Optimal Large Language Models” (Chinchilla), 2022 (arxiv.org/abs/2203.15556). Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT), 2022 (arxiv.org/abs/2203.02155). Dao et al., “FlashAttention” series, 2022-2026 (arxiv.org/abs/2205.14135, arxiv.org/abs/2307.08691, arxiv.org/abs/2407.08608, arxiv.org/abs/2603.05451). Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023 (arxiv.org/abs/2309.06180). Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” 2021 (arxiv.org/abs/2106.09685). Shao et al., “DeepSeekMath” (GRPO), 2024 (arxiv.org/abs/2402.03300). DeepSeek-V3 Technical Report, 2024 (arxiv.org/abs/2412.19437). DeepSeek-V3.2 model card, 685B total parameters (huggingface.co/deepseek-ai/DeepSeek-V3.2). Kurtic et al., “Give Me BF16 or Give Me Death?” (FP8 study), ACL 2025 (arxiv.org/abs/2411.02355). Liu et al., “Visual Instruction Tuning” (LLaVA), NeurIPS 2023 Oral (arxiv.org/abs/2304.08485). Zhou et al., “LIMA: Less Is More for Alignment,” NeurIPS 2023 (arxiv.org/abs/2305.11206). Nguyen et al., “Min-p Sampling,” ICLR 2025 Oral (arxiv.org/abs/2407.01082). Tang et al., “Top-nσ: Not All Logits Are You Need,” ACL 2025 (arxiv.org/abs/2411.07641). Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs,” NeurIPS 2024 (arxiv.org/abs/2312.07104). Zhong et al., “DistServe,” OSDI 2024 (arxiv.org/abs/2401.09670). Patel et al., “Splitwise,” ISCA 2024 (arxiv.org/abs/2311.18677). Hong, Troynikov, and Huber, “Context Rot,” Chroma Research, July 2025 (research.trychroma.com/context-rot). Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” TACL 2024, 12:157-173 (direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638). Hu, “REINFORCE++: A Simple and Efficient Approach for Aligning Large Language Models,” January 2025 (arxiv.org/abs/2501.03262). van den Oord et al., “Neural Discrete Representation Learning” (VQ-VAE), NeurIPS 2017 (arxiv.org/abs/1711.00937). Stanford Alpaca, 52,000 instruction-response pairs from text-davinci-003 fine-tuned on LLaMA 7B (crfm.stanford.edu/2023/03/13/alpaca.html, huggingface.co/datasets/tatsu-lab/alpaca). GPT Image 1 API April 23, 2025; GPT Image 1.5 December 16, 2025 (wikipedia.org/wiki/GPT_Image, cybernews.com). NVIDIA Dynamic Context Parallelism January 28, 2026, 1.48x speedup (developer.nvidia.com/blog). LLaMA 4 Maverick config from HuggingFace Transformers Llama4TextConfig (huggingface.co/docs/transformers/main/model_doc/llama4). Qwen3-8B config from HuggingFace (huggingface.co/Qwen/Qwen3-8B). Qwen 3.5 released February 16, 2026 (qwen-ai.com, launchberg.com). Mistral Small 4 released March 16, 2026 (mistral.ai/news/mistral-small-4, huggingface.co/mistralai/Mistral-Small-4-119B-2603). GPT-5.4 released March 5, 2026, with 1.05 million token context window (community.openai.com/t/gpt-5-4-deep-dive, buildfastwithai.com, thenextgentechinsider.com). GPT-5.4 75% OSWorld-Verified above 72.4% human baseline (computertech.co, blockchain.news). OpenAI Responses API released March 11, 2025 (openai.com, analyticsvidhya.com). OpenAI Operator with CUA model released January 23, 2025 (openai.com/index/introducing-operator, maginative.com, winbuzzer.com). Grok 4 Fast released September 2025 with 2 million token context window (x.ai/news/grok-4-fast, Wikipedia Grok chatbot versions table). TPU v7 Ironwood 4,614 FP8 TFLOPS 192 GB HBM3e 7.38 TB/s (cloud.google.com/tpu/docs/tpu7x, theregister.co.uk). Claude Opus 4.6 released February 5, 2026 (anthropic.com/research/claude-opus-4-6, felloai.com). Claude Agent SDK released September 29, 2025, rebranded from Claude Code SDK (anthropic.com/engineering/building-agents-with-the-claude-agent-sdk). OpenAI Agents SDK released March 11, 2025, version 0.12.5 as of March 2026 (openai.github.io/openai-agents-python, pypi.org/project/openai-agents). Google ADK released April 9, 2025 (developers.googleblog.com). Devin 2.0 released April 3, 2025 (siliconangle.com, techcrunch.com). TGI maintenance mode December 11, 2025 (huggingface.co/docs/inference-endpoints/engines/tgi). Unsloth 2x faster 70% less VRAM (github.com/unslothai/unsloth). SWE-bench Verified: Claude Opus 4.6 Thinking 79.2% (vals.ai/benchmarks/swebench). OpenAI discontinued SWE-bench Verified reporting March 2026 (openai.com/index/why-we-no-longer-evaluate-swe-bench-verified). MCP registry 3,012 servers as of March 2026 (nimblebrain.ai/blog/state-of-mcp-security-2026). Mistral Small 4 active parameters: 6B per official blog, 6.5B per HuggingFace model card (mistral.ai/news/mistral-small-4). DeepSeek-V3.2 continued pre-training from V3.1-Terminus checkpoint (kili-technology.com/blog/data-story-deepseek-v3-2). Model specifications, GPU hardware details, and pricing verified via web search as of March 2026; see individual chapter source citations for complete references.