update

Author: vukrosic

app/learn/llm-fundamentals/batch-size-vs-sequence-length/page.tsx

@@ -0,0 +1,12 @@
+import { LessonPage } from "@/components/lesson-page";
+
+export default function BatchSizeVsSequenceLengthPage() {
+  return (
+    <LessonPage
+      contentPath="llm-fundamentals/batch-size-vs-sequence-length"
+      prevLink={{ href: "/learn", label: "← Back to Course" }}
+      nextLink={{ href: "/learn", label: "Continue Learning →" }}
+    />
+  );
+}
+

lib/course-structure.tsx

@@ -365,6 +365,22 @@ export const getCourseModules = (): ModuleData[] => [
         href: "/learn/building-a-transformer/training-a-transformer"
       }
     ]
+  },
+  {
+    title: "LLM Training Fundamentals",
+    titleZh: "LLM训练基础",
+    icon: (
+      <svg className="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+        <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M9 3v2m6-2v2M9 19v2m6-2v2M5 9H3m2 6H3m18-6h-2m2 6h-2M7 19h10a2 2 0 002-2V7a2 2 0 00-2-2H7a2 2 0 00-2 2v10a2 2 0 002 2zM9 9h6v6H9V9z" />
+      </svg>
+    ),
+    lessons: [
+      {
+        title: "Batch Size vs Sequence Length",
+        titleZh: "批量大小与序列长度",
+        href: "/learn/llm-fundamentals/batch-size-vs-sequence-length"
+      }
+    ]
   }
 ];
 

public/content/learn/llm-fundamentals/batch-size-vs-sequence-length/batch-size-vs-sequence-length-content.md

@@ -0,0 +1,138 @@
+---
+hero:
+  title: "Batch Size vs Sequence Length"
+  subtitle: "Understanding Two Critical Training Parameters in LLMs"
+  tags:
+    - "🤖 LLM Training"
+    - "⏱️ 12 min read"
+---
+
+# Batch Size vs Sequence Length in LLM Training
+
+When training large language models (LLMs), two of the most important hyperparameters you'll configure are **batch size** and **sequence length**. While they might seem similar at first (both involve "how much data" the model sees), they serve fundamentally different purposes and have distinct impacts on training.
+
+## What is Batch Size?
+
+**Batch size** is the number of independent training examples your model processes in parallel before updating its weights.
+
+### How It Works
+
+Think of batch size as "how many different conversations" your model reads simultaneously:
+
+```python
+# Example with batch_size = 4
+batch = [
+    "The cat sat on the mat",           # Example 1
+    "Machine learning is fascinating",  # Example 2
+    "Python is a great language",       # Example 3
+    "Transformers revolutionized NLP"   # Example 4
+]
+```
+
+Each example in the batch is processed independently, and the gradients from all examples are averaged before updating the model weights.
+
+To understand the trade-offs, imagine trying to find the best path down a mountain (minimizing loss).
+
+**Large Batches (asking a large crowd for directions):** You average the advice from many people. The resulting direction is very reliable and stable, preventing you from overreacting to any single bad piece of advice (this is "low noise"). However, this consensus path might be a slow, winding road that misses clever shortcuts. The updates are less "exploratory."
+
+**Small Batches (asking one or two hikers):** Their advice is less reliable and more random (this is "high noise"). This randomness can be beneficial—it might lead you to discover a hidden, faster trail that the large crowd would have averaged out. This "noise" helps the model explore more diverse solutions and can help it escape from getting stuck in suboptimal valleys (local minima).
+
+Larger batches provide more stable training because they average the gradients over more examples, which reduces noise. However, this also means that each individual update is less "exploratory," potentially making slower progress per step. On the other hand, smaller batches introduce more gradient noise, which can actually help the model make more diverse updates and explore different solutions.
+
+
+## What is Sequence Length?
+
+**Sequence length** is the maximum number of tokens (words/subwords) that the model processes in a single example.
+
+### How It Works
+
+Think of sequence length as "how long of a conversation" your model can read at once:
+
+```python
+# Short sequence (seq_len = 256 tokens)
+"The cat sat on the mat. It was a sunny day..."
+
+# Long sequence (seq_len = 4096 tokens)
+"""The cat sat on the mat. It was a sunny day. The birds were 
+singing in the trees. A gentle breeze rustled the leaves... 
+[continues for 4000+ more tokens, could be an entire chapter]"""
+```
+
+Longer sequences give the model more context to learn fron. It enables learning long-range dependencies and relationships
+BUT, attention mechanism has O(n²) memory complexity with sequence length so memory requirement grows quadratically (n² for seq_len of n).
+
+### Impact on Training
+
+**Longer Sequence Length (e.g., 4096):**
+- ✅ Model can learn long-range dependencies
+- ✅ Better understanding of extended context
+- ✅ More information per training example
+- ❌ Quadratic memory growth (attention is expensive!)
+- ❌ Slower per-step training time
+
+**Shorter Sequence Length (e.g., 256):**
+- ✅ Faster training steps
+- ✅ Less memory required
+- ❌ Limited context window
+- ❌ Cannot learn long-range patterns
+
+## The Key Difference
+
+The fundamental difference between these two parameters:
+
+| Aspect | Batch Size | Sequence Length |
+|--------|-----------|-----------------|
+| **What it controls** | Number of independent examples | Length of each example |
+| **Relationship between data** | Examples are unrelated | Tokens are sequential and dependent |
+| **Memory scaling** | Linear (2x batch = 2x memory) | Quadratic for attention (2x length = 4x memory) |
+| **Learning impact** | Affects gradient stability | Affects context understanding |
+| **Trade-off** | Stability vs exploration | Context vs speed |
+
+### Visual Comparison
+
+```
+Batch Size = 4, Sequence Length = 8:
+┌─────────────────────────────────────┐
+│ Example 1: [A, B, C, D, E, F, G, H] │
+│ Example 2: [I, J, K, L, M, N, O, P] │
+│ Example 3: [Q, R, S, T, U, V, W, X] │
+│ Example 4: [Y, Z, A, B, C, D, E, F] │
+└─────────────────────────────────────┘
+         ↓
+    Averaged Gradients → Weight Update
+
+Batch Size = 2, Sequence Length = 16:
+┌─────────────────────────────────────────────────────┐
+│ Example 1: [A, B, C, D, E, F, G, H, I, J, K, L...] │
+│ Example 2: [M, N, O, P, Q, R, S, T, U, V, W, X...] │
+└─────────────────────────────────────────────────────┘
+         ↓
+    Averaged Gradients → Weight Update
+```
+
+
+## Real-World Experiment Results
+
+In a recent ablation study on MoE transformers, three strategies were tested with equal GPU memory usage:
+
+| Strategy | Batch Size | Seq Length | Final Val Loss | Val Accuracy | Training Time |
+|----------|------------|------------|----------------|--------------|---------------|
+| **Balanced** | 26 | 1024 | 0.0636 | 98.73% | 7.04 min |
+| **Long Seq** | 6 | 4096 | 0.0745 | 98.45% | 6.99 min |
+| **Large Batch** | 104 | 256 | 0.1025 | 98.00% | 6.97 min |
+
+### Key Findings
+
+1. **Balanced approach won** on validation loss metrics
+2. **Large batch trained fastest** per-step but achieved higher final loss
+3. **Long sequence** showed promise but didn't win on short-term metrics
+
+### Important Caveat ⚠️
+
+**Validation loss doesn't tell the whole story!**
+
+While large batch size showed faster convergence in validation loss, longer sequences provide more context and should theoretically enable the model to learn more complex patterns over time. The validation loss metric may favor faster convergence but doesn't necessarily reflect the model's ability to leverage extended context windows.
+
+For applications requiring deep contextual understanding, such as analyzing long documents or multi-turn dialogues, longer sequence lengths are more valuable, even at the cost of a higher validation loss.
+
+**In practice**, sequence length is often between 1024 and 4096, with extension training later.