Text Splitter LlamaIndex

This tool provides a programmable way to divide long text into smaller, workflow-ready chunks for LlamaIndex and similar vector stores. It supports both fixed-size chunking and boundary-aware modes that respect sentence or paragraph boundaries. Users supply the source text and a desired chunk size in tokens, plus optional overlap and a boundary strategy. The system then tokenizes the input, builds sequential chunks, and outputs the chunks with per-chunk token counts and overall metadata. Conceptually, it maintains a stable context budget for downstream retrieval, reduces fragmentation, and aids efficient indexing. The tool is designed for data engineers, AI developers, and researchers who preprocess large corpora for AI assistants, chatbots, or search applications. Unique value comes from boundary-aware options, token-budget compliance, and optional language_hint to improve boundary decisions across languages. Core features also include configurable overlap to preserve context between adjacent chunks and optional max chunk controls. Typical use cases include preparing product manuals, research papers, or legal documents for LlamaIndex-based retrieval, enabling consistent chunk sizes and predictable token costs. Outputs include an array of chunk strings, per_chunk_tokens, and a summary of total_chunks, useful for benchmarking and logging in ETL pipelines.