Segmenter

Token-Boundary Chunking & Tokeniser Inspection
0 chunks 0 tokens cl100k_base

Source Document

0 chars, 0 words
500 chars
Requests go Direct to Jina, no key
Rule-based chunking that follows document structure (Markdown, HTML, LaTeX, CJK)
OpenAI cl100k_base, o200k_base, p50k, r50k & gpt2 vocabularies
Free API: no key needed, and a key only raises the rate limit

Output Chunks

About Segmenter

Documentation

Segmenter is a free, browser-based studio for splitting long text into chunks and inspecting how that text is tokenised. It uses the Jina Segmenter API to cut a document at structural boundaries. It then shows each chunk, its exact token count and the tokens inside it. It is built for people who prepare text for large language models, embeddings and retrieval-augmented generation pipelines.

Segmenter at a glance

Key facts about Segmenter
Application typeBrowser-based web application
Primary functionText chunking with exact token counts per chunk
Chunking engineJina Segmenter API, which applies rules based on structural cues
Tokeniserscl100k_base, o200k_base, p50k_base, r50k_base and gpt2
Max chunk length50 to 8,000 characters
InputTyped or pasted text, text files, PDF and Word (.docx)
OutputChunk cards, token view, JSON view, comparisons and JSONL, plain text or JSON export
CostFree, with no account. An API key is optional
AuthorMike Higgins

What Segmenter does

Segmenter turns an unstructured document into a set of chunks that you can measure, compare and export. Each capability below answers a question that arises when text is prepared for a language model.

Splits text into chunks

Segmenter sends your text to the Jina Segmenter API with a tokeniser and a max chunk length. The API cuts at structural cues in Markdown, HTML, LaTeX and CJK text rather than at arbitrary character counts, so headings, paragraphs and sentences are kept intact where possible.

Reports exact token counts

Every request asks for the token list, so each chunk carries an exact token count and not an estimate. A local tokeniser also counts tokens in your browser as you type.

Flags weak chunk boundaries

Segmenter analyses every chunk boundary and flags splits that fall mid-sentence, mid-word or inside a code block. Each flag shows the text on either side of the cut.

Shows the tokens themselves

The Tokens view groups tokens by chunk, makes spaces and line breaks visible and shows token IDs on hover. Characters that need several tokens, such as an emoji, are outlined.

Compares tokenisers and chunk lengths

The Compare tab counts the same text with all five tokenisers and runs the Segmenter API at 250, 500, 1,000 and 2,000 characters alongside your own setting. It reports chunk counts, size spread, flagged splits and overruns of your token budget.

Checks context windows

A context window control shows how much of the text fits a chosen model limit and copies exactly that part.

Imports documents locally

You can type, paste, drop or open text, PDF and Word files. Files are read entirely in the browser. Word files are converted to light Markdown so that headings, lists and tables survive. PDFs are read as text only, with repeating headers, footers and page numbers removed.

Exports chunks for a vector database

The Export dialog produces JSONL for retrieval-augmented generation with an ID, the text and metadata for each chunk. It also supports optional overlap, plain text and the raw API response.

How Segmenter works

  1. Provide text. Type, paste, drop or open a file. A sample can be loaded to try the studio without sending anything.
  2. Choose a tokeniser and a max chunk length. The tokeniser decides how tokens are counted. The length, in characters, caps the size of each chunk.
  3. Run Segment. Press the button or use Ctrl+Enter (Cmd+Enter on a Mac). Segmenter requests chunks, chunk positions and tokens from api.jina.ai/v1/segment.
  4. Handle long documents. Texts above 64,000 characters are sent in parts, cut at paragraph breaks where possible, and merged into one result. The limit for a whole document is 500,000 characters.
  5. Validate the response. Segmenter checks every chunk position against your text, converts offsets where needed and calculates per-chunk token counts.
  6. Inspect the result. Review the Chunks, Tokens and JSON views. Each chunk keeps one colour across its highlight in the source, its card, its overview bar and its tokens.
  7. Compare and export. Test other tokenisers and lengths, then export the chunks as JSONL, plain text or JSON.

Request routing

With a key, the browser calls Jina directly and sends the key only in the Authorization header. Without a key, the browser still tries Jina directly. If the browser blocks that request, a hosted copy of the page falls back to this site’s proxy, which forwards the text to Jina and applies a small free allowance. The key is never sent to the proxy.

Local processing

Token counting, tokeniser comparison, file reading and export all run in your browser. The local tokeniser is a port of the byte pair encoding core of js-tiktoken, checked token for token against the original, and its vocabularies load on demand.

Who Segmenter is for

RAG and search engineers

Engineers who build retrieval pipelines can tune chunk size, check boundaries and produce JSONL that loads into a vector database.

LLM application developers

Developers who work against a context window can measure how many tokens a document needs and how much of it fits a given model.

SEO and content professionals

Specialists who study how AI search systems split pages into passages can paste copy and see whether a heading stays with its paragraph and where each passage ends.

Data and machine learning practitioners

Practitioners who prepare corpora can compare tokenisers and see how vocabulary choice changes token counts.

Technical writers and documentation teams

Teams who publish documentation for retrieval can test whether long pages split cleanly at their headings.

Students and educators

Learners can watch byte pair encoding at work by viewing real tokens and token IDs for any text.

Limits to bear in mind

Segmenter is not a semantic chunker. The Jina Segmenter API uses structural rules and does not score similarity between chunks. Scanned PDFs need OCR, which is not included. Word headers, footers, footnotes and comments are not imported.

Common workflows

  • Prepare a knowledge base for RAG. Open the source documents, segment them, review flagged splits and export JSONL.
  • Choose a chunk size. Run the length comparison and pick the setting with the fewest flagged splits and no token budget overruns.
  • Estimate token usage. Compare counts across tokenisers to size a prompt or an embedding job.
  • Fit a document to a model. Use the context window control to see how much text a model can accept.
  • Audit page structure for AI retrieval. Segment published copy and check that sections divide at meaningful headings.
  • Debug tokenisation. Inspect the tokens of an unusual string, such as code, emoji or non-Latin text, to see why it costs more than expected.

Supported tokenisers

Segmenter supports five byte pair encoding vocabularies. Choose the one that matches your target model.

Tokenisers supported by Segmenter
TokeniserUsed byNote
cl100k_baseOpenAI GPT-4, GPT-3.5-turbo and text-embedding-ada-002The most common choice for current OpenAI chat and embedding workloads.
o200k_baseOpenAI GPT-4o and the omni reasoning and multimodal modelsA larger vocabulary that usually produces fewer tokens for the same text.
p50k_baseOpenAI Codex and text-davinci-002 edit modelsSuited to code-oriented legacy models.
r50k_baseLegacy text-davinci-001 and text-curie-001Retained for older models and for comparison.
gpt2The classic GPT-2 byte pair encoding vocabularyUseful as a baseline and for Hugging Face GPT-2 workflows.

Technical specifications

Segmenter technical specifications
EndpointPOST https://api.jina.ai/v1/segment
Request fieldscontent, tokenizer, max_chunk_length, return_tokens and return_chunks
Response fieldsnum_tokens, num_chunks, chunks, chunk_positions and tokens
Characters per request64,000, with longer texts sent in parts
Characters per document500,000
Max chunk length50 to 8,000 characters
Request timeout30 seconds, with cancel on the button or the Esc key
Text file limit2 MB
PDF and Word limit25 MB
Live token countUp to 100,000 characters, counted locally
AccessibilityWCAG 2.2 AA colour contrast, visible focus, keyboard operation and landmarks

Privacy, data handling and API keys

  • Opened files are read in your browser and are never uploaded. Their text goes into the editor like typed text.
  • Text is sent to Jina only when you run a segmentation or a comparison that needs the API.
  • Jina states that it does not use API inputs or outputs to train its models.
  • An API key is optional. By default it is kept for the current tab and forgotten when you close it. If you choose to remember it, it is saved in local storage on your device until you remove it.
  • Your key is sent only to api.jina.ai in the request header and never to this site’s proxy.

Glossary of chunking and tokenisation terms

Chunk
A contiguous span of source text produced by segmentation. A chunk stays under a maximum character length and is cut at a structural boundary where one exists.
Chunking
The process of dividing a long document into chunks so that each can be embedded, indexed or passed to a language model on its own. Also called text segmentation.
Token
The unit of text a language model reads. A token can be a whole word, part of a word, a punctuation mark or a sequence of bytes.
Tokeniser
The vocabulary and rules that convert text into tokens and token IDs. Model families use different tokenisers, so the same text produces different token counts.
Byte pair encoding
A subword method that builds a vocabulary by repeatedly merging the most frequent adjacent pairs of bytes or characters. All five tokenisers in Segmenter are byte pair encoding vocabularies.
Max chunk length
The upper limit, in characters, for a single chunk. Segmenter accepts values from 50 to 8,000 characters.
Chunk boundary
The point where one chunk ends and the next begins. A good boundary respects sentences, paragraphs, headings and code blocks. A poor boundary falls mid-sentence or mid-word.
Context window
The maximum number of tokens a language model can process in one request, covering both the input and the output.
Token budget
A ceiling on the tokens in a single chunk, usually set by the input limit of the embedding model that will process it.
Overlap
Repeating the end of one chunk at the start of the next so that context survives the boundary. Segmenter applies overlap when exporting chunks.
Embedding
A numeric vector that represents the meaning of a chunk and supports similarity search.
Vector database
A data store that holds embeddings alongside their source text and metadata and returns the nearest matches to a query.
Retrieval-augmented generation
A pattern in which a system retrieves relevant chunks from an index and supplies them to a language model as context for its answer. Often shortened to RAG.
JSONL
JSON Lines, a text format with one JSON object per line. It is the usual interchange format for loading chunks into a vector database.

Frequently asked questions

What is Segmenter?

Segmenter is a free browser-based studio that splits long text into chunks with the Jina Segmenter API and shows exactly where and how each split was made. It reports exact token counts, lists the tokens in each chunk and exports the result for a vector database.

Is Segmenter free to use?

Yes. The studio needs no account and no payment. Jina documents its Segmenter API as free, and an optional key only raises the rate limit.

Do I need a Jina API key?

No. Without a key, Jina rate-limits requests by IP address. A free key from jina.ai raises that limit. Segmenter stores a key only in your browser and sends it only to api.jina.ai.

Does Segmenter use embeddings to chunk text?

No. The Jina Segmenter API splits text with rules based on structural cues such as Markdown headings, HTML, LaTeX and CJK punctuation. It does not measure semantic similarity between chunks. Segmenter flags structural problems such as mid-sentence splits, but judging meaning across a boundary would need an embeddings call.

Which tokenisers does Segmenter support?

Segmenter supports five byte pair encoding vocabularies: cl100k_base, o200k_base, p50k_base, r50k_base and gpt2. A local tokeniser counts tokens in your browser, so the comparison across all five needs no network request.

Why do token counts differ between tokenisers?

Each tokeniser has its own vocabulary and merge rules, so it splits the same text into different pieces. A larger vocabulary such as o200k_base usually needs fewer tokens than gpt2 for the same passage. Always count with the tokeniser that matches your target model.

What maximum chunk length should I choose?

The length is set in characters, not tokens, and the right value depends on the input limit of your embedding model and on how much context a retrieved passage needs. Use the Compare tab to run 250, 500, 1,000 and 2,000 characters alongside your own setting and review chunk counts, size spread, flagged splits and token budget overruns.

Which files can I open?

You can open plain text formats such as .txt, .md, .json, .csv, .html and .yaml up to 2 MB. You can also open PDF and Word (.docx) files up to 25 MB. Scanned PDFs, password-protected PDFs and legacy formats such as .doc, RTF and PowerPoint are refused with an explanation.

Is my text stored or used for training?

Your text stays in your browser until you run a segmentation. It is then sent to the Jina Segmenter API, or to this site’s proxy if your browser blocks the direct request. Jina states that it does not use API inputs or outputs to train its models. Opening a file or loading a sample sends nothing.

How do I export chunks for a vector database?

Open the Export dialog and choose JSONL for RAG. Each line holds an ID, the chunk text and metadata, with optional token or character overlap and a SHA-256 hash of the source. A Python snippet shows how to load the file.

API Key Transparency & Privacy

1. Do I need an API key?

No. Jina’s Segmenter API at api.jina.ai/v1/segment is free. Without a key, Jina rate-limits requests per IP address. A free key from jina.ai raises that limit and isn’t charged for segmenting.

2. Where does my text go?

When you press Segment, the text in the left pane is sent to Jina’s API. On the hosted site, if your browser blocks that direct request, the text goes through this site’s proxy instead, which forwards it to Jina. Jina states it doesn’t use API inputs or outputs to train its models. Loading a sample sends nothing.

3. Where is my key kept?

Only in this browser. By default it’s kept for this tab and forgotten when you close it. If you tick “Remember my key on this device”, it’s saved in local storage under segmenter:apiKey until you remove it. Either way it’s sent only to api.jina.ai in the request header, never to this site’s proxy.

Export chunks

Format

 
Load the JSONL in Python