How to Strip Hidden Zero-Width Characters & Markdown from LLM Outputs (And Estimate BPE Tokens Locally)

When copying generated text, documentation, or code snippets from ChatGPT, Claude, or Gemini into a CMS, Notion, Gmail, or code editor, two recurring issues come up for developers and technical writers: Formatting & Invisible Unicode Artifacts: Raw **bold** asterisks, ### heading hashes, overused AI em-dashes (—), broken PDF line wraps, and invisible zero-width characters (U+200B, U+FEFF, U+00A0) that silently break string matching, regex validation, and character limits. Context Window & Token Estimation: Knowing how many Byte-Pair Encoding (BPE) tokens a long prompt, JSON payload, or multi-language document consumes before sending it to GPT-4o (128K), Claude 3.5 Sonnet (200K), or Gemini 1.5 Pro (1M)—without uploading confidential text to a third-party server. To solve both problems with zero server round-trips, I built two new 100% client-side utilities as part of TextScope. 1. Stripping Markdown, Em-Dashes, and Hidden Unicode Characters Hidden Unicode characters often sneak into copied text from rich-text editors, PDFs, and AI chat interfaces: Unicode Point Character Name Common Issue in Production U+200B Zero-Width Space Breaks exact string comparisons, SQL queries, and JSON keys U+FEFF Byte Order Mark (BOM) Causes unexpected syntax errors at the start of scripts or CSVs U+00A0 Non-Breaking Space Prevents standard CSS word wrapping and breaks whitespace regex U+2014 Em-Dash (—) Overused by LLMs; makes technical and editorial copy look robotic With the AI Text Cleaner & Markdown Stripper, you can paste any LLM or PDF output and toggle exactly what to clean locally in your browser: Strip Markdown syntax (**bold**, *italic*, ### headings, code fences, and inline links) while keeping clean paragraph breaks. Detect and remove invisible zero-width characters (U+200B, U+200C, U+200D, U+FEFF, U+00A0) with an exact count of removed artifacts. Convert AI em-dashes (—) and curly quotes (“”) into standard ASCII punctuation. Unwrap hard single line breaks copied from multi-column PDFs without merging double-newline paragraphs. 2. Estimating LLM Prompt Tokens & Context Window Headroom In standard English prose, 1 token ≈ 0.75 words (or roughly 4 characters), meaning 1,000 words is ~1,330 tokens. However, source code, JSON keys, Markdown tables, and CJK scripts (Japanese, Korean, Chinese) have a higher token-to-character ratio (1.2–1.8 tokens per character). The LLM Prompt Token Counter & Words-to-Tokens Calculator runs entirely in vanilla JavaScript and calculates: Estimated BPE token count, total words, character count, and token-to-word ratio. Real-time context window utilization across 8K, 32K, 128K (GPT-4o / Llama 3.1), 200K (Claude 3.5 Sonnet), and 1M (Gemini 1.5 Pro) limits. Single-spaced page equivalents for long-form technical docs and books. Full Suite of Client-Side Writing & SEO Utilities All tools in the suite run 100% locally in the browser (no sign-up, no server uploads) and support 6 languages (English, Spanish, German, Portuguese, Japanese, and Korean): Word Counter & Flesch Readability Analyzer – Real-time word/character count, Flesch Reading Ease score, Flesch-Kincaid Grade Level, and 1–3 word n-gram phrase density. AI Text Cleaner & Format Stripper – One-click removal of ChatGPT/Claude Markdown, em-dashes, zero-width spaces, and broken PDF line breaks. LLM Token Counter & Context Calculator – Instant words-to-tokens conversion and context window percentages. SEO Headline Analyzer & Google SERP Preview – Title tag pixel width (600px) and character length (50–60 chars) checker with live search snippet preview. Reading & Speaking Time Calculator – Converts word counts into silent reading (200 WPM) and speech/presentation durations (130 WPM). Case Converter & URL Slug Generator – Converts text between APA Title Case, Sentence case, camelCase, snake_case, and clean SEO URL slugs. You can also check out the full multilingual documentation and reference benchmarks in the public GitHub repository: GS-Seven/textscopes on GitHub. Feedback and feature suggestions are very welcome!