LangChain Text Splitters

1.1.1 · active · verified Sun Mar 29

LangChain Text Splitters (current version 1.1.1) provides a comprehensive set of utilities for breaking down large text documents into smaller, manageable chunks. This is crucial for applications like Retrieval-Augmented Generation (RAG) and fitting content within Language Model context windows. As an integral part of the LangChain ecosystem, it maintains an active and rapid release cadence, closely aligned with other LangChain libraries.

Warnings

breaking The text splitter modules have been moved from `langchain.text_splitter` to the standalone `langchain-text-splitters` package. Direct imports from `langchain.text_splitter` will no longer work.
Fix: Update all text splitter imports from `from langchain.text_splitter import ...` to `from langchain_text_splitters import ...`.
gotcha When using `create_documents()` method, it expects a *list* of strings (or `Document` objects). Passing a single string will result in each character being treated as a separate document.
Fix: For a single string, use `text_splitter.split_text(your_string)`. If you intend to pass multiple strings to `create_documents`, ensure they are wrapped in a list: `text_splitter.create_documents([your_string])`.
gotcha The `chunk_size` parameter for character-based splitters specifies the *target* maximum chunk size. Due to the splitter's logic (e.g., trying to split on specific separators first), the actual chunk length may not be exactly `chunk_size`.
Fix: Understand that `chunk_size` is a guideline. For more precise length control (e.g., token-based), consider `TokenTextSplitter` or custom `length_function` with an appropriate tokenizer.
gotcha Mixing major versions of LangChain ecosystem packages (e.g., `langchain-text-splitters==1.x.x` with `langchain-core==0.3.x`) can lead to compatibility issues and unexpected behavior.
Fix: Always strive to keep all `langchain-` prefixed packages within the same major version series (e.g., all `1.x.x` or all `0.3.x`) to ensure compatibility.
gotcha Some specialized splitters, like `MarkdownHeaderTextSplitter` and `HTMLHeaderTextSplitter`, do not inherit from the base `TextSplitter` class. This means they might have slightly different method signatures or expectations.
Fix: Always consult the specific documentation or API reference for specialized text splitters to understand their unique behavior and interfaces.

Install

pip install langchain-text-splitters Install latest version

Imports

RecursiveCharacterTextSplitter
```
from langchain_text_splitters import RecursiveCharacterTextSplitter
```
Text splitters were moved to a dedicated package. Old `langchain.text_splitter` imports are deprecated or removed.

CharacterTextSplitter

from langchain_text_splitters import CharacterTextSplitter

MarkdownHeaderTextSplitter

from langchain_text_splitters import MarkdownHeaderTextSplitter

Quickstart

Demonstrates the basic usage of the `RecursiveCharacterTextSplitter`, the most commonly recommended text splitter. It shows how to initialize the splitter with `chunk_size` and `chunk_overlap`, and then split a long string into smaller text chunks.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Example long text
long_text = (
    "LangChain is a framework designed to simplify the creation of applications using large language models. "
    "It provides tools for chaining together different components, making it easier to build complex LLM workflows. "
    "Text splitting is a fundamental step in processing long documents for LLMs, ensuring that chunks fit within context windows and maintain semantic coherence. "
    "The RecursiveCharacterTextSplitter is often the recommended default for general-purpose text."
)

# Initialize the splitter
# chunk_size: maximum size of each chunk (in characters by default)
# chunk_overlap: number of characters to overlap between consecutive chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,
    chunk_overlap=20
)

# Split the text
chunks = text_splitter.split_text(long_text)

# Print the chunks
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:\n{chunk}\n---")

view raw JSON →