Dev.to · 3 min read

Why does copied text look identical but break your code?

Why does copied text look identical but break your code?

This is the clean version of the example: total = 10 count = 2 print(total / count) # 👩‍💻 Inserting a zero-width space after count produces an error: File "demo.py", line 2 count​ = 2 ^ SyntaxError: invalid non-printable character U+200B The inserted character is U+200B. It has no visible width, so the two versions look the same. Python rejects it on that line. The first Python block deliberately contains no U+200B. To create the failing version, generate the file with explicit escapes: python3 -c "open('demo.py','w').write('total = 10\ncount\u200b = 2\nprint(total / count) # \U0001F469\u200d\U0001F4BB\n')" python3 demo.py That gives the SyntaxError above (Python 3.13). It doesn't always crash. In a JSON key, the lookup just quietly misses: import json d = json.loads('{"user\u200bId": 42}') print(d.get("userId")) # None print(list(d)) # ['user\u200bId'] Seeing it Print the repr instead of the text: $ python3 -c "print(repr(open('demo.py').read()))" 'total = 10\ncount\u200b = 2\nprint(total / count) # 👩\u200d💻\n' Two hidden characters show up. \u200b after count is the bug. \u200d is a zero-width joiner. It glues 👩 and 💻 into one "woman technologist" emoji. To get line numbers, use GNU grep built with PCRE support. The -P option is not supported by the grep bundled with macOS: $ grep -nP '\x{200B}' demo.py 2:count​ = 2 Widen the class when needed, e.g. '[\x{200B}\x{00A0}\x{FEFF}]' for zero-width space, no-break space and BOM. Don't strip everything A regex that removes every invisible character will fix this file and break other text. Some of these characters are doing real work: U+200D (zero-width joiner): emoji sequences like 👩‍💻, and shaping in some Indic scripts. U+200C (zero-width non-joiner): used in normal Persian writing. U+FE0E / U+FE0F (variation selectors): text vs emoji presentation. U+00A0 (no-break space): sometimes intended in prose. In code it's almost always a bug. So remove only what broke the code. Here that's the U+200B: $ python3 -c " s = open('demo.py').read().replace('\u200b', '') open('fixed.py', 'w').write(s) print(repr(s))" 'total = 10\ncount = 2\nprint(total / count) # 👩\u200d💻\n' $ python3 fixed.py 5.0 It runs, and the joiner in the emoji is still there. In the browser I'm dark, and I built InvisiChar for this. It's free, and the text is processed locally in your browser. Everything above works without it. It's handy when the text isn't in a file and you want to go through findings one at a time. For the browser example, use the contents of the generated demo.py, which includes U+200B. The Inspect screenshot below shows 2 findings: the zero-width space is marked "Suggested removal", and the zero-width joiner is "Protected by default". Nothing changes until you confirm. In Clean you pick a preset (Prose, Code & data, International text) and tick the occurrences to remove. The review below selects only U+200B and leaves the joiner unselected. After applying, it lists the change: one U+200B removed at code-point position 17. The emoji is unchanged. When identical text behaves differently, check the code points first.

This is a summary aggregated from Dev.to. Read the complete article on the original site:

Read full article at Dev.to

More Programming & Dev News