Chunking

Token-aware chunking: the unglamorous part of RAG that decides if it works

Two competing approaches to chunking a fast-changing product catalog — a dedicated chunking library versus embeddings that chunk for you — and the trade-offs between them.

NXT AI · 2026 · 6 min read

Chunking rarely gets the attention embeddings and retrieval get, but it's usually the thing quietly breaking a RAG system. Cut a product description in the wrong place and the embedding for that chunk captures neither the product name nor the spec that mattered — and no embedding model, however good, can retrieve context that was never encoded to begin with.

The basic tension

Smaller chunks — in the 100 to 256 token range — give precise semantic matching, since each chunk represents one specific idea. Larger chunks, 1024 tokens and up, preserve enough surrounding context for the model to actually understand what it retrieved. Every chunking strategy is a negotiation between those two pulls, and a product catalog makes the negotiation harder because a single product page mixes short structured facts (price, size, SKU) with longer free-text description in the same document.

Two approaches, not one

Dedicated chunking library

A tool like Chonkie sits in front of your embedding call. Its token chunker splits text into fixed token-sized pieces with configurable overlap; its semantic chunker groups text by similarity instead of a fixed size. It plugs into whichever embedding provider you're using — Voyage, Cohere, OpenAI, or an open-source model — so the chunking strategy stays decoupled from the embedding choice.

Embeddings that chunk for you

Newer models like Voyage's voyage-context-4 fold chunking into the embedding step itself: send the whole document, and it returns chunk-level embeddings that already carry full-document context, with no separate chunking pipeline to build or tune.

The trade-off in practice: a dedicated chunking library gives you full control and provider flexibility, at the cost of a pipeline stage you have to maintain and re-tune as your content changes. An embedding model with built-in chunking removes that maintenance burden, but ties your chunking strategy to that specific provider's model.

What this means for a product catalog specifically

Document-aware chunking — splitting along the structure the document already has, like headings, spec tables, and description sections, rather than blindly cutting at a token count — tends to hold up best for product data, because product pages already have implicit structure worth preserving. For a catalog that changes as often as inventory does, the maintenance cost of a chunking pipeline is a real, ongoing cost — which is exactly why the built-in-chunking approach is worth serious consideration for teams that don't want chunking logic to become its own subsystem to maintain.

Further reading

Working through a chunking strategy for your catalog?

Happy to walk through the trade-offs for your specific content.

Contact Us — info@nxttechnologies.com