There is a specific kind of bug that wastes an afternoon. Two strings look identical on screen. The equality check returns false. You copy both into a text editor and stare at them. They are the same. They are obviously the same.
They are not the same. There is a character in one of them that has no visible form, and until you look at the underlying bytes you will not find it.
The usual suspects
The list is short, which is the good news. Six characters account for nearly all of these incidents.
The non-breaking space (U+00A0, decimal 160). Looks exactly like a normal space. Is not a normal space. It arrives from Word documents, from web pages where was used for spacing, and from PDF extraction. Most whitespace-trimming functions do not remove it, because it is not classified as whitespace in the ASCII sense.
The zero-width space (U+200B, decimal 8203). Has no width at all, so it is genuinely invisible even when you select the text. It is used legitimately for line-break hints in East Asian typography, and it turns up as a stowaway in text copied from web pages and rich editors. A single one in the middle of an email address makes it fail validation with no visible cause.
The zero-width non-joiner and joiner (U+200C, U+200D). Necessary in Arabic, Persian and Indic scripts, and also the glue that holds composite emoji together. A family emoji is several people joined by these characters.
The byte order mark (U+FEFF). A marker some tools write at the very start of a UTF-8 file. Read that file naively and your first column header is id rather than id — which is why an import sometimes fails on the first field only.
The carriage return (decimal 13). Windows ends lines with carriage return plus line feed; Unix uses line feed alone. Read a Windows file with Unix assumptions and every line ends with an invisible extra character. This is the single most common reason two apparently identical lists fail to match.
Smart quotes and dashes. Not invisible, but easily mistaken for their plain counterparts. A curly apostrophe (U+2019) is not the apostrophe on your keyboard. An em dash is not a hyphen. Word substitutes these automatically as you type, which is why text pasted from Word breaks code so reliably.
How to find them
The reliable method is to stop looking at the text and look at the numbers behind it. Convert the string to character codes and read the list.
Our ASCII Converter does this — paste the problem text, encode to decimal with space separators, and every character becomes a number you can actually see. A 32 is a normal space. A 160 is a non-breaking space. A 9 is a tab, a 13 is a carriage return, an 8203 is a zero-width space. The offender is immediately obvious.
Comparing two supposedly identical strings this way takes about thirty seconds and answers the question definitively, which is considerably better than staring at them.
Where they come from
Knowing the source helps you predict the problem.
Microsoft Word produces smart quotes, em dashes and non-breaking spaces automatically. Anything pasted from Word into a code editor or a form should be treated as suspect.
Web pages carry whatever the HTML contained, including sequences that were being used as a layout hack.
PDF extraction is notorious. Text in a PDF has no real word boundaries — the format positions glyphs — so extraction tools infer spaces, and they infer a lot of unusual ones.
Spreadsheets contribute trailing spaces and non-breaking spaces from formatted cells, and Excel in particular will happily store a number as text with an invisible leading character.
Cross-platform file transfer gives you the carriage return problem.
Defending against them
Normalise on input. Anywhere user-supplied text enters your system, strip the known offenders: replace non-breaking spaces with regular ones, remove zero-width characters entirely, normalise line endings, and trim. Doing this at the boundary is far cheaper than debugging it later.
Use Unicode normalisation. The same accented character can be stored either as one code point or as a base letter plus a combining mark. They look identical and compare unequal. Applying NFC normalisation before comparing collapses both forms into one. Every modern language has a function for this.
Turn on invisible characters in your editor. VS Code, Sublime and most others can render spaces, tabs and line endings visibly. It looks cluttered for a day and then you stop noticing, and you catch these problems as you type rather than in production.
Be suspicious of trimming. Most trim() implementations remove ASCII whitespace and stop there. Non-breaking spaces and zero-width characters survive. If your trim is not fixing an obvious whitespace problem, this is why.
The security angle
Worth a brief mention because it is easy to overlook. Zero-width characters can be used to hide content — to slip a word past a keyword filter, or to fingerprint a document by inserting an invisible pattern unique to each recipient.
The practical implication for most people is modest: if you are filtering or comparing user-supplied text for anything that matters, strip zero-width characters first, or a filter can be bypassed by text that looks entirely ordinary.
A short checklist
- Comparison failing on identical-looking strings? Convert both to character codes
- Import failing on the first column only? Look for a byte order mark
- Every line failing to match? Check for carriage returns
- Text came from Word or a PDF? Assume smart quotes and non-breaking spaces
- Normalise text at the point it enters your system, not after it causes trouble
