Skip to main content

Token Count Mismatch

PastePrompt token counts are estimates. They can differ from the final count shown by ChatGPT, Claude, Gemini, Codex, Cursor, or provider APIs.

Common causes​

  • Different tokenizer implementations.
  • Chat message wrappers.
  • Attachment handling.
  • System prompts added by the external tool.
  • Formatting changes after copy or export.
  • Model-specific context accounting.

What to do​

  1. Leave margin below the target model limit.
  2. Remove low-value files.
  3. Split the review into smaller bundles.
  4. Keep instructions concise.
  5. Use the external tool's final count as authoritative for that tool.

Practical margins​

For large reviews, do not aim for the exact advertised context limit. Leave room for:

  • the LLM provider's system prompt and message wrappers;
  • follow-up instructions;
  • model-generated reasoning and answer text;
  • provider-side formatting or attachment conversion.

If the provider rejects a bundle, split by module, workflow, or question:

  1. Start with Git metadata, file map, and prompt instructions.
  2. Add only the files needed for the first question.
  3. Export a second bundle for dependencies, tests, or follow-up evidence.
  4. Keep the same output format so comparisons remain easier.

When estimates differ by a lot​

Large differences usually mean the external tool is counting something PastePrompt cannot see, such as chat wrappers, hidden system context, attachments, or provider-specific normalization. Treat PastePrompt counts as planning estimates and the provider count as authoritative for that provider.