Tokens & RAG

Tool Metadata Token Auditor

Measure supplied tool definitions with reconciled array and per-tool token accounting.

Processed on our CPU server.Submitted input is processed for this run and is not stored.

256 KiB limit

Input

Paste your input or load an example,
then run the tool.

Supported formats & limitations

Pinned cl100k_base ordinary-text encoding using tokenizer v0.8.1; vocabulary is compiled locally. Special-token literals count as ordinary text, not reserved token IDs. No universal model/chat-wrapper/billing claim.
32 KiB UTF-8 text maximum. Consecutive whitespace/non-whitespace runs capped at 2048 bytes to bound BPE work. Empty text counts as zero.
Byte offsets are zero-based, half-open. Tokens can split UTF-8 characters; visual spans group tokens until complete code points and retain individual IDs and hex bytes.
1–100 tool objects; full canonical serialization must fit 32 KiB. Names from MCP/flat tools or Chat Completions function wrappers; missing/duplicate names are reported but both values are counted.
Serialization is compact sorted-key JSON, UTF-8 Unicode and unescaped HTML; numeric spellings survive. Total array count is measured directly. Signed wrapper/boundary delta reconciles independent per-tool counts because BPE can merge across boundaries.
Reports include exact counted JSON, per-tool counts and the ten largest definitions. No schema-validity or billed/client-overhead claims.
Server CPU processing with no input history or outbound requests. MCP access and public deployment remain separate launch tasks.

Execution budget: 5s; output limit: 4096 KiB. Browser worker startup has a separate 3s allowance.