AI Engineer Exhaustive
EasyNLP

Text Chunking

NLP · Data Processing

Easy

Problem

Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.

Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.

Algorithm

  1. Compute the step size between chunk start positions:

\text{step} = \text{chunk\_size} - \text{overlap}

  1. Starting from position 0, extract a chunk of chunk_size tokens, then advance by step. Stop once a chunk reaches the end of the token list.

Return the chunks as a list of token lists.

Theory

Text chunking divides long documents into smaller, manageable pieces called chunks. This is essential for NLP systems with context limits, retrieval-augmented generation (RAG), and efficient document processing. Good chunking preserves semantic coherence while respecting size constraints.

Why Chunk Text?

Context window limits: LLMs have maximum context lengths (e.g., 4K, 8K, 128K tokens). Documents exceeding this must be split.

Retrieval precision: Smaller chunks allow more precise matching in semantic search. A relevant paragraph is more useful than a vaguely related chapter.

Memory efficiency: Processing entire large documents at once may exceed available memory.

Parallel processing: Chunks can be processed independently across multiple workers.

Chunking Strategies

Fixed-Size Chunking

Split text into chunks of a fixed number of characters or tokens:

\text{num\_chunks} = \lceil \frac{\text{text\_length}}{\text{chunk\_size}} \rceil

Advantages:

Disadvantages:

Overlapping Chunks

Include some overlap between consecutive chunks to preserve context at boundaries:

\text{chunk}_i = \text{text}[i \times \text{stride} : i \times \text{stride} + \text{chunk\_size}]

Where stride = chunk_size - overlap

Example: chunk_size = 100, overlap = 20, stride = 80

Benefit: Information at boundaries is not lost; appears in multiple chunks.

Sentence-Based Chunking

Split at sentence boundaries, grouping sentences until size limit:

Process:

  1. Split text into sentences
  2. Accumulate sentences until adding the next would exceed limit
  3. Start new chunk with next sentence

Advantages:

Challenges:

Paragraph-Based Chunking

Use paragraph breaks (double newlines) as chunk boundaries:

Advantages:

Challenges:

Recursive Chunking

Apply a hierarchy of separators:

  1. Try to split by paragraph (double newline)
  2. If chunks too large, split by single newline
  3. If still too large, split by sentence
  4. If still too large, split by word/character

Benefit: Preserves as much structure as possible while respecting size limits.

Chunk Size Considerations

Too small:

Too large:

Typical sizes:

Worked Example: Fixed-Size with Overlap

Text: "The quick brown fox jumps over the lazy dog. Then it runs away quickly."

Parameters: chunk_size = 30 characters, overlap = 10

Chunks:

Observation: "jumps over" appears in both chunks 0 and 1, providing continuity.

Token-Based vs Character-Based

Character-based chunking:

Token-based chunking:

Relationship: Characters and tokens are not 1:1. Average ratio varies by language and tokenizer.

Metadata Preservation

Each chunk should retain:

Source information: Document ID, filename, URL

Position information: Chunk index, character offset, page number

Structural context: Section heading, chapter title

Why important: After retrieval, need to trace back to original context and combine related chunks.

Handling Special Content

Tables: May need to keep entire table in one chunk or use special formatting

Code blocks: Avoid splitting in the middle of functions or control structures

Lists: Keep list items together when possible

Headers: Include section headers with their content

Images/Figures: Reference them but they may need separate handling

Chunking for Different Use Cases

Semantic search/RAG:

Summarization:

Question answering:

Classification:

Quality Metrics for Chunking

Coherence: Do chunks represent complete thoughts?

Coverage: Is all important content captured (considering overlap)?

Consistency: Are chunks similar in size and scope?

Retrieval performance: Do retrieved chunks contain relevant information?

Where Text Chunking Shows Up

Examples

Example 1

Input
tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output
[["a", "b", "c"], ["d", "e", "f"]]
Explanation
A step of three produces two non-overlapping chunks.

Example 2

Input
tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output
[["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]

Hints

  1. Compute the distance between chunk starts as chunk_size minus overlap.
  2. Slice from each start position and stop after the first chunk that reaches the end.

Requirements

Constraints

Starter Code

def text_chunking(tokens: list, chunk_size: int, overlap: int) -> list:
    """
    Returns fixed-size token chunks with the requested overlap.
    """
    # Write code here
    pass

Test Cases

CaseMatches
No overlapExample 1public
With overlapExample 2public