Tokens & RAG
Context Budget Planner
Plan a supplied context capacity with measured texts, manual counts, output reserve and unknown components.
Processed on our CPU server.Submitted input is processed for this run and is not stored.
256 KiB limitInput
Result
A useful result starts here.
Paste your input or load an example,
then run the tool.
Supported formats & limitations
- Pinned cl100k_base ordinary-text encoding using tokenizer v0.8.1; vocabulary is compiled locally. Special-token literals count as ordinary text, not reserved token IDs. No universal model/chat-wrapper/billing claim.
- 32 KiB UTF-8 text maximum. Consecutive whitespace/non-whitespace runs capped at 2048 bytes to bound BPE work. Empty text counts as zero.
- Byte offsets are zero-based, half-open. Tokens can split UTF-8 characters; visual spans group tokens until complete code points and retain individual IDs and hex bytes.
- Capacity 1–1,000,000,000; reserve/overhead/counts 0–1,000,000,000. 1–100 named components with 32 KiB aggregate measured text. Text and count together are rejected.
- Missing/null text and count mean unknown, leaving total remaining budget null. Known subtotal includes reserve and overhead; over-budget status can be established even with unknowns.
- Measured texts tokenize separately. Repeated names or identical nonempty texts produce double-count warnings and are still included. Manual counts are estimates. No hidden prompts, live account context or automatic model capacity.
- Server CPU processing with no input history or outbound requests. MCP access and public deployment remain separate launch tasks.
Execution budget: 5s; output limit: 4096 KiB. Browser worker startup has a separate 3s allowance.