Skip to main content

Token Counting

Token counting helps you decide whether the selected context is small enough for the LLM workflow you plan to use. PastePrompt shows estimates before copy or export so you can adjust scope early.

What it does​

PastePrompt estimates token counts for files, folders, selected files, prompt instructions, and generated bundles. The app can show totals at several levels:

  • Individual file token estimates.
  • Folder totals based on eligible child files.
  • Selected total for the current bundle.
  • Format overhead for XML-like or Markdown output.
  • A context budget selector for comparing the selection against a target model window.

Token counts are estimates. They are meant to guide selection, not exactly match every LLM provider interface.

Why it matters​

Audit work often fails when the bundle is too large, too unfocused, or split at the wrong boundary. Token visibility lets you make practical decisions:

  • Include the highest-value files first.
  • Avoid wasting budget on generated code or unrelated tests.
  • See when a folder selection is too broad.
  • Split a large review into multiple reproducible bundles.
  • Leave room for instructions and model output.

How to use it​

  1. Select a repository.
  2. Choose files and folders for the bundle.
  3. Review token estimates in the tree and selected-files panel.
  4. Choose the closest context budget for the LLM you plan to use.
  5. Watch for near-limit or over-limit warnings.
  6. Remove low-value files until the selection fits the target budget.
  7. Add prompt instructions and recheck the selected total.
  8. Generate the final XML-like or Markdown bundle.

File and folder totals​

File totals estimate the source text for that file. Folder totals estimate eligible files below that folder, respecting ignore settings and skipped-file behavior. Folder totals are useful for quickly spotting expensive modules, test suites, generated folders, and vendored dependencies.

Selected total​

The selected total is the number that matters most before copy or export. It should account for selected file contents, file map metadata, Git metadata when included, prompt instructions, and formatting overhead.

Context budget selector​

The budget selector helps you compare the current bundle against a model-sized target such as a smaller chat window, a larger long-context model, or a conservative review budget. Leave margin for follow-up conversation and model output.

Example workflow​

  1. Select src/auth/session.ts, src/billing/strategy.ts, and related interfaces.
  2. Notice the selected total is below the chosen context budget.
  3. Add the relevant test file and Git diff.
  4. The bundle now approaches the budget limit.
  5. Remove unrelated integration tests and deployment scripts.
  6. Keep the final bundle under budget with room for instructions and model output.

Limitations​

  • Token counts can differ from provider UIs because providers use different tokenizers, chat wrappers, hidden system prompts, tool formatting, attachment handling, and message metadata.
  • Counts may change when you switch output format or include Git diff context.
  • Very large files may be estimated differently if preview or scan limits apply.
  • Token estimates do not measure relevance or completeness.
  • Staying under budget does not guarantee high-quality LLM output.