What Segmenter does
Segmenter turns an unstructured document into a set of chunks that you can measure, compare and export. Each capability below answers a question that arises when text is prepared for a language model.
Splits text into chunks
Segmenter sends your text to the Jina Segmenter API with a tokeniser and a max chunk length. The API cuts at structural cues in Markdown, HTML, LaTeX and CJK text rather than at arbitrary character counts, so headings, paragraphs and sentences are kept intact where possible.
Reports exact token counts
Every request asks for the token list, so each chunk carries an exact token count and not an estimate. A local tokeniser also counts tokens in your browser as you type.
Flags weak chunk boundaries
Segmenter analyses every chunk boundary and flags splits that fall mid-sentence, mid-word or inside a code block. Each flag shows the text on either side of the cut.
Shows the tokens themselves
The Tokens view groups tokens by chunk, makes spaces and line breaks visible and shows token IDs on hover. Characters that need several tokens, such as an emoji, are outlined.
Compares tokenisers and chunk lengths
The Compare tab counts the same text with all five tokenisers and runs the Segmenter API at 250, 500, 1,000 and 2,000 characters alongside your own setting. It reports chunk counts, size spread, flagged splits and overruns of your token budget.
Checks context windows
A context window control shows how much of the text fits a chosen model limit and copies exactly that part.
Imports documents locally
You can type, paste, drop or open text, PDF and Word files. Files are read entirely in the browser. Word files are converted to light Markdown so that headings, lists and tables survive. PDFs are read as text only, with repeating headers, footers and page numbers removed.
Exports chunks for a vector database
The Export dialog produces JSONL for retrieval-augmented generation with an ID, the text and metadata for each chunk. It also supports optional overlap, plain text and the raw API response.
How Segmenter works
- Provide text. Type, paste, drop or open a file. A sample can be loaded to try the studio without sending anything.
- Choose a tokeniser and a max chunk length. The tokeniser decides how tokens are counted. The length, in characters, caps the size of each chunk.
- Run Segment. Press the button or use Ctrl+Enter (Cmd+Enter on a Mac). Segmenter requests chunks, chunk positions and tokens from
api.jina.ai/v1/segment. - Handle long documents. Texts above 64,000 characters are sent in parts, cut at paragraph breaks where possible, and merged into one result. The limit for a whole document is 500,000 characters.
- Validate the response. Segmenter checks every chunk position against your text, converts offsets where needed and calculates per-chunk token counts.
- Inspect the result. Review the Chunks, Tokens and JSON views. Each chunk keeps one colour across its highlight in the source, its card, its overview bar and its tokens.
- Compare and export. Test other tokenisers and lengths, then export the chunks as JSONL, plain text or JSON.
Request routing
With a key, the browser calls Jina directly and sends the key only in the Authorization header. Without a key, the browser still tries Jina directly. If the browser blocks that request, a hosted copy of the page falls back to this site’s proxy, which forwards the text to Jina and applies a small free allowance. The key is never sent to the proxy.
Local processing
Token counting, tokeniser comparison, file reading and export all run in your browser. The local tokeniser is a port of the byte pair encoding core of js-tiktoken, checked token for token against the original, and its vocabularies load on demand.
Who Segmenter is for
RAG and search engineers
Engineers who build retrieval pipelines can tune chunk size, check boundaries and produce JSONL that loads into a vector database.
LLM application developers
Developers who work against a context window can measure how many tokens a document needs and how much of it fits a given model.
SEO and content professionals
Specialists who study how AI search systems split pages into passages can paste copy and see whether a heading stays with its paragraph and where each passage ends.
Data and machine learning practitioners
Practitioners who prepare corpora can compare tokenisers and see how vocabulary choice changes token counts.
Technical writers and documentation teams
Teams who publish documentation for retrieval can test whether long pages split cleanly at their headings.
Students and educators
Learners can watch byte pair encoding at work by viewing real tokens and token IDs for any text.
Limits to bear in mind
Segmenter is not a semantic chunker. The Jina Segmenter API uses structural rules and does not score similarity between chunks. Scanned PDFs need OCR, which is not included. Word headers, footers, footnotes and comments are not imported.
Frequently asked questions
What is Segmenter?
Segmenter is a free browser-based studio that splits long text into chunks with the Jina Segmenter API and shows exactly where and how each split was made. It reports exact token counts, lists the tokens in each chunk and exports the result for a vector database.
Is Segmenter free to use?
Yes. The studio needs no account and no payment. Jina documents its Segmenter API as free, and an optional key only raises the rate limit.
Do I need a Jina API key?
No. Without a key, Jina rate-limits requests by IP address. A free key from jina.ai raises that limit. Segmenter stores a key only in your browser and sends it only to api.jina.ai.
Does Segmenter use embeddings to chunk text?
No. The Jina Segmenter API splits text with rules based on structural cues such as Markdown headings, HTML, LaTeX and CJK punctuation. It does not measure semantic similarity between chunks. Segmenter flags structural problems such as mid-sentence splits, but judging meaning across a boundary would need an embeddings call.
Which tokenisers does Segmenter support?
Segmenter supports five byte pair encoding vocabularies: cl100k_base, o200k_base, p50k_base, r50k_base and gpt2. A local tokeniser counts tokens in your browser, so the comparison across all five needs no network request.
Why do token counts differ between tokenisers?
Each tokeniser has its own vocabulary and merge rules, so it splits the same text into different pieces. A larger vocabulary such as o200k_base usually needs fewer tokens than gpt2 for the same passage. Always count with the tokeniser that matches your target model.
What maximum chunk length should I choose?
The length is set in characters, not tokens, and the right value depends on the input limit of your embedding model and on how much context a retrieved passage needs. Use the Compare tab to run 250, 500, 1,000 and 2,000 characters alongside your own setting and review chunk counts, size spread, flagged splits and token budget overruns.
Which files can I open?
You can open plain text formats such as .txt, .md, .json, .csv, .html and .yaml up to 2 MB. You can also open PDF and Word (.docx) files up to 25 MB. Scanned PDFs, password-protected PDFs and legacy formats such as .doc, RTF and PowerPoint are refused with an explanation.
Is my text stored or used for training?
Your text stays in your browser until you run a segmentation. It is then sent to the Jina Segmenter API, or to this site’s proxy if your browser blocks the direct request. Jina states that it does not use API inputs or outputs to train its models. Opening a file or loading a sample sends nothing.
How do I export chunks for a vector database?
Open the Export dialog and choose JSONL for RAG. Each line holds an ID, the chunk text and metadata, with optional token or character overlap and a SHA-256 hash of the source. A Python snippet shows how to load the file.