Semantic Chunking for RAG: What It Is and How to Implement It
Learn how semantic chunking splits text by meaning for RAG. See the algorithm, Python code, LangChain and LlamaIndex setups, tuning tips, and when it pays off.

A retrieval-augmented generation (RAG) system answers questions from chunks of your documents, so the way you cut those chunks sets a ceiling on answer quality. Cut too early and a fact is separated from its explanation. Cut too late and one chunk carries three topics, and the retriever returns noise along with the signal.
Semantic chunking is one answer to that problem. Instead of cutting every 500 tokens, it looks for the places where the text changes subject. It is popular, easy to build, and also contested, because published tests show mixed results against simpler methods. This guide covers both sides.
Chunking is one stage in a longer pipeline. A related read on staged pipelines is receipt OCR pipeline architecture.
What Semantic Chunking Is (and What It Is Not)
Semantic chunking groups text by meaning. A splitter converts sentences into embeddings, compares neighbors, and starts a new chunk when the topic appears to shift. The chunk length is a result of the text, not an input.
It differs from three related ideas:
- Fixed-size chunking cuts every N tokens or characters, whatever the content says.
- Recursive splitting tries separators in order, such as paragraph breaks, then line breaks, then spaces, until each piece fits a size limit.
- Structure-aware chunking follows headings, sections, and tables that the document already marks.
Semantic chunking changes where you split. A different technique, contextual retrieval, changes what you embed, and the two can be combined. That comparison appears later in this guide.
Mixed piles of files usually get sorted before any splitting happens, and document categorization for public agencies is one example of that step.
File types that are mostly fields instead of prose follow a different path, as in invoice scanning and AP integration.
Why Chunk Boundaries Decide What Your RAG System Retrieves
A retriever scores each chunk as a whole. If a chunk mixes a pricing paragraph with a security paragraph, its embedding sits between the two and matches neither question well. If a chunk is cut in the middle of an argument, the half that reaches the language model may lack the sentence that makes it true.
When answers go wrong, bad boundaries are one place to look before blaming the language model. Another related read is the guide to choosing a document automation tool.
Contracts show the problem clearly, since clauses refer to defined terms several pages away. Readers working with legal text may also want this piece on contract automation.
How Semantic Chunking Works, Step by Step

Most implementations, including the LangChain and LlamaIndex ones, follow the same five steps.
1. Split the Text into Sentences
The splitter breaks the document into sentences with a regular expression or a sentence tokenizer. LangChain's default expression splits after periods, question marks, and exclamation marks. LlamaIndex notes that its regex mainly works for English, so other languages need a different sentence splitter.
2. Embed Each Sentence with Some Context
A single sentence is a weak signal on its own, so most splitters embed a sliding window. A buffer_size of 1 means each sentence is embedded together with one neighbor on each side. A larger buffer smooths the signal and reduces over-splitting.
3. Measure the Distance Between Neighbors
For each pair of adjacent windows, the splitter computes cosine distance, which is one minus cosine similarity. A small distance means the two windows are about the same thing. A large distance means the subject changed.
4. Choose Breakpoints
The splitter needs a rule for what counts as large. The common rule is a percentile: take the distances for the whole document and cut wherever a distance is above, say, the 95th percentile. Other rules use standard deviations or the interquartile range.
5. Assemble the Chunks
Sentences between two breakpoints become one chunk. Good implementations then enforce a minimum and maximum size, because the raw algorithm can produce single-sentence fragments or very long runs.
Implement Semantic Chunking from Scratch in Python
Writing the algorithm yourself takes about thirty lines and makes the tuning knobs concrete. The function below takes any embedding function, so you can plug in a local model or a hosted API.
import re
import numpy as np
ENDINGS = ".?" + chr(33) # period, question mark, exclamation mark
def split_sentences(text):
return [s.strip() for s in re.split(r"(?<=[" + ENDINGS + r"])\s+", text) if s.strip()]
def semantic_chunks(text, embed, buffer_size=1, percentile=90, min_sentences=2):
sents = split_sentences(text)
if len(sents) < 3:
return [text]
# Embed each sentence together with its neighbors
windows = []
for i in range(len(sents)):
lo = max(0, i - buffer_size)
hi = min(len(sents), i + buffer_size + 1)
windows.append(" ".join(sents[lo:hi]))
vecs = np.asarray(embed(windows), dtype=float)
vecs = vecs / np.linalg.norm(vecs, axis=1, keepdims=True).clip(1e-12)
# Cosine distance between each window and the next one
distances = 1 - np.sum(vecs[:-1] * vecs[1:], axis=1)
cutoff = np.percentile(distances, percentile)
chunks, start = [], 0
for i, d in enumerate(distances):
if d > cutoff and (i + 1 - start) >= min_sentences:
chunks.append(" ".join(sents[start:i + 1]))
start = i + 1
chunks.append(" ".join(sents[start:]))
return chunks
To run it with a local embedding model, install sentence-transformers and pass its encode method:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
chunks = semantic_chunks(open("report.txt").read(), model.encode, percentile=90)
for i, c in enumerate(chunks):
print(i, len(c.split()), "words")
Add Size Guards Before You Trust the Output
The min_sentences argument stops the splitter from producing one-sentence chunks. Vectara's authors reported that breakpoint-based chunking sometimes produced single-sentence chunks lacking contextual information, and that lower thresholds led to more of them, so this guard is worth having. LangChain's splitter has an optional min_chunk_size for the same purpose. Add a maximum as well: if a chunk exceeds your embedding model's input limit, split it again with a recursive splitter.
Use LangChain or LlamaIndex Instead
If you already build on one of these frameworks, use its splitter instead of maintaining your own. Both implement the same idea and both default to a 95th percentile cut.
Library Defaults and Breakpoint Rules
LangChain's splitter supports four breakpoint_threshold_type values. The defaults come from a BREAKPOINT_DEFAULTS dictionary in the source.
- Percentile (default 95): cut where the distance is above the Nth percentile of all distances.
- Standard_deviation (default 3): cut where the distance is more than N standard deviations above the mean.
- Interquartile (default 1.5): cut where the distance is above the mean plus N times the interquartile range.
- Gradient (default 95): apply a percentile to the rate of change of the distances, which suits dense or domain-specific text.
LangChain SemanticChunker
- What it automates: Sentence splitting, embedding, distance calculation, and breakpoint selection behind one class, SemanticChunker.
- Pricing: Free and open source. You pay your embedding provider for the API calls it makes.
- Limits: It lives in the separate langchain_experimental package, so expect API changes. Every sentence is embedded, which adds cost on large corpora.
- Best for: Teams whose RAG pipeline already runs on LangChain.
- Pros: Four threshold rules, a number_of_chunks option, add_start_index to keep character offsets, and an optional min_chunk_size that merges small chunks.
- Cons: Experimental status means the API can change between releases.
# pip install langchain-experimental langchain-openai
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
splitter = SemanticChunker(
OpenAIEmbeddings(model="text-embedding-3-small"),
breakpoint_threshold_type="percentile",
breakpoint_threshold_amount=90,
)
docs = splitter.create_documents([open("report.txt").read()])
print(len(docs), "chunks")
LlamaIndex SemanticSplitterNodeParser
- What it automates: The same split, embed, compare, and cut cycle, returning LlamaIndex nodes that plug into an index.
- Pricing: Free and open source. Embedding calls are billed by your provider.
- Limits: The sentence regex mainly works for English, and the percentile threshold usually needs tuning.
- Best for: Teams already building indexes and query engines in LlamaIndex.
- Pros: Only two main knobs, buffer_size and breakpoint_percentile_threshold, both defaulting to 1 and 95.
- Cons: Percentile-only breakpoints, so you cannot switch to standard deviation or interquartile rules.
# pip install llama-index llama-index-embeddings-openai
from llama_index.core import SimpleDirectoryReader
from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding
documents = SimpleDirectoryReader("docs").load_data()
splitter = SemanticSplitterNodeParser(
buffer_size=1,
breakpoint_percentile_threshold=95,
embed_model=OpenAIEmbedding(),
)
nodes = splitter.get_nodes_from_documents(documents)
Tune Thresholds, Buffer Size, and Chunk Size
Three settings decide most of the behavior.
- Percentile or threshold amount. A percentile rule always cuts a fixed share of the gaps, however similar the text is. At the 95th percentile, roughly one gap in twenty becomes a boundary, even in a document about a single topic. A standard deviation rule can instead produce zero cuts when no gap stands out.
- Buffer size. Raising it smooths noisy single-sentence outliers, at the price of blurring real boundaries, so a cut can land a sentence or so away from the true topic change.
- Size limits. Set a floor and a ceiling so chunks stay inside the useful range for your embedding model and your prompt budget.
Tune against questions, not by eye. Write 30 to 50 real queries with known source passages, run each chunking setup, and measure how often the right passage appears in the top results. Unstructured makes the same point: a threshold that works on technical documentation can behave differently on legal or financial content.
Does Semantic Chunking Beat Fixed-Size Chunking? What the Evidence Says
The answer depends on the documents.
A 2024 preprint from Vectara researchers (Qu, Bao, and Tu), "Is Semantic Chunking Worth the Computational Cost?", compared fixed-size, breakpoint-based, and clustering-based chunkers on document retrieval, evidence retrieval, and answer generation. It found that semantic chunking showed slight advantages on stitched datasets, where unrelated passages are glued into one long document. On datasets that reflect real document structure, fixed-size chunking consistently outperformed it in document retrieval, and the authors concluded the extra cost was not justified by consistent gains. The authors also describe their own limits: they chunked at sentence level and had no direct chunk quality labels.
Chroma's 2024 evaluation reached a related conclusion from another angle. It reported that default settings of some popular strategies can perform poorly, and that heuristic splitters such as RecursiveCharacterTextSplitter often perform well when parametrized appropriately. It also introduced a cluster-based semantic chunker and an LLM-based chunker, and found both competitive.
Read together, these results do not say semantic chunking is bad. They say it is not a free upgrade. A tuned recursive splitter is a strong baseline, and the embedding model you choose can matter more than the chunking method.
When Semantic Chunking Is Worth Using
Try it when these conditions hold:
- Documents mix several topics with weak or missing structure, such as transcripts, long emails, support threads, and scraped pages.
- Headings, tables of contents, and paragraph breaks are unreliable or absent.
- Your fixed-size baseline has measurable boundary failures, such as answers split across two chunks.
- Indexing runs offline, so the extra embedding calls do not affect user-facing latency.
Skip it, or run it only as an experiment, when documents already have clean headings, when each file covers one topic, or when your corpus is large and indexing budgets are tight.
Alternatives and Add-Ons

Recursive splitting with overlap. The simplest strong baseline. Start here and keep it as the control in every experiment.
Structure-aware chunking. If the source has headings, sections, or page boundaries, use them as primary boundaries and apply semantic splitting only inside oversized sections.
Contextual retrieval. Anthropic's technique has a model write a short context note for each chunk, usually 50 to 100 tokens, and prepends it before embedding and keyword indexing. Anthropic reported that contextual embeddings cut the top-20 retrieval failure rate by 35 percent (5.7 percent to 3.7 percent), and that adding contextual BM25 brought the cut to 49 percent (to 2.9 percent). With reranking, it reported a 67 percent reduction. These are vendor-reported results on Anthropic's own datasets, so test them on your data.
Hybrid retrieval and reranking. Combining keyword and vector search, then reranking the top results, is a separate lever. Anthropic's numbers above show reranking cutting failures further on top of contextual retrieval.
Common Problems and Fixes
- Tiny chunks. Raise buffer_size, add a minimum size, or merge each short chunk into its neighbor.
- Huge chunks. Lower the percentile or add a maximum size with a recursive fallback.
- Splits in the wrong place for tables and lists. Rows and bullets can look alike to an embedding model, so treat tables and lists as units before running the splitter.
- Non-English text. Replace the default sentence regex with a language-aware tokenizer.
- Slow indexing. Every sentence is embedded, so batch the calls, cache embeddings, and run semantic splitting only on documents that need it.
- Scores that look fine but answers are wrong. Evaluate end-to-end answers as well as retrieval, since a chunk can retrieve well and still lack the context to answer.
Chunking Starts with Clean Text from Documents
Semantic splitters assume flowing prose. Invoices, bank statements, and application forms are mostly fields, where the value you need is a total or a date rather than a paragraph. For those files, a field-level extraction API can take the place of chunking: DeepRead returns named fields, each with the page, bounding box, and source text it came from, and its Free plan covers 2,000 pages a month without a card. Prose-heavy files still go through a chunker, and the quality of the text going in sets what the chunker can do.
Conclusion
Semantic chunking splits where meaning changes, using sentence embeddings, a distance measure, and a breakpoint rule. It is easy to build and worth testing on mixed-topic text with weak structure. The published evidence shows that it does not reliably beat a tuned fixed-size or recursive splitter on documents with a clear structure, so measure it against that baseline on your own questions before you commit to the extra cost. Pair whichever method you choose with size limits, contextual notes, and reranking.
If your corpus is mostly forms and financial records, extract fields rather than chunk prose.
FAQ
What is semantic chunking in RAG?
Semantic chunking splits a document into chunks based on meaning. It embeds sentences, measures how far apart neighboring sentences are in embedding space, and starts a new chunk where the distance is unusually large.
Is semantic chunking better than fixed-size chunking?
Not always. A 2024 Vectara study found only slight advantages on artificially stitched datasets and better results for fixed-size chunking on more realistic documents. Semantic chunking tends to help most on mixed-topic text with weak structure.
What is the best breakpoint threshold?
There is no universal value. LangChain and LlamaIndex both default to the 95th percentile, but you should tune it on your own queries. Lower percentiles give more, smaller chunks, and higher ones give fewer, larger chunks.
How much does semantic chunking cost?
The splitter embeds every sentence, or every sentence window, in each document. With a hosted embedding API that cost shows up on your bill, and with a local model it shows up as indexing time. Fixed-size chunking needs no embedding calls at the chunking stage.
Which embedding model should I use for semantic chunking?
Use the same model you use for retrieval if you can, because the splitter then measures similarity the way your retriever does. Vectara's authors wrote that the effect of chunking strategy was often overshadowed by factors such as embedding quality, so evaluate models before you tune splitters.
Can I combine semantic chunking with other methods?
Yes. A common pattern is to split by headings first, run semantic chunking only inside long sections, add a context note to each chunk, and rerank the retrieved results.
More articles

Unstructured.io Review: Features, Pricing, and Alternatives
Unstructured.io review: what the open-source library and hosted platform do, current pricing, tested Python code, limits, and four alternatives compared.

Resume OCR: Extracting Candidate Data from Resumes Automatically
How resume OCR reads PDFs and scans, fixes two-column order, and returns candidate fields. Tested Python code, a tool comparison, and US hiring rules.