What Are Invisible Unicode Characters?

Invisible Unicode characters at a glance

Invisible does not mean useless or malicious. Many invisible characters are required for correct typography, language shaping, emoji and line breaking.

What makes a Unicode character invisible?

Some Unicode code points intentionally have no visible glyph. Others affect neighboring characters, create spacing without an obvious symbol, influence text direction, define line-break behavior or act as controls. They still exist in the string, affect bytes and code-unit counts, change comparison and validation, and survive copying. Font and rendering behavior can change what appears visible, so use Unicode Character Inspector and Unicode Sequence Analyzer when debugging.

For lower-level units, read What Is a Unicode Code Point? and Code Points vs Code Units.

Invisible whitespace characters

Ordinary space is U+0020 SPACE. Special spaces may look similar but behave differently. U+00A0 NO-BREAK SPACE prevents normal line breaking, often appears in copied web or document text, can break exact comparison and may be written in HTML as  . U+202F NARROW NO-BREAK SPACE has legitimate typographic uses. Other Unicode spaces include U+2000–U+200A, U+3000 IDEOGRAPHIC SPACE and U+2007 FIGURE SPACE. Do not replace all Unicode spaces with U+0020 without considering layout and language requirements.

Zero-width space

U+200B ZERO WIDTH SPACE marks an optional break opportunity and can support scripts that do not use visible spaces. It may enter text through copying, content-management systems or text processing. It can create hidden identifier differences, search mismatches, broken email or URL validation, unexpected length and source-code confusion.

username
U+0075 U+0073 U+0065 U+0072 U+006E U+0061 U+006D U+0065

user​name
U+0075 U+0073 U+0065 U+0072 U+200B U+006E U+0061 U+006D U+0065

Detect it with Invisible Character Detector and compare strings with Unicode Text Compare.

Zero-width joiner

U+200D ZERO WIDTH JOINER has legitimate uses in emoji ZWJ sequences, script shaping and joining behavior in supported writing systems. In the family emoji 👨‍👩‍👧‍👦, invisible joiners connect visible emoji code points into one displayed sequence.

MAN
ZWJ
WOMAN
ZWJ
GIRL
ZWJ
BOY

Removing it can change one composite emoji into several separate emoji and can affect text shaping. Read What Is a Grapheme Cluster?.

Zero-width non-joiner

U+200C ZERO WIDTH NON-JOINER prevents joining in scripts where neighboring letters might otherwise connect. It can distinguish correct spellings or morphological boundaries and can be semantically important in some languages and writing systems. Removing it may change appearance, spelling, meaning or shaping. Context determines whether U+200C is required, accidental or malicious.

Word joiner and no-break behavior

U+2060 WORD JOINER prevents a line break without visible width and replaced the deprecated word-joining use of U+FEFF. It is not interchangeable with zero-width space or no-break space.

Soft hyphen

U+00AD SOFT HYPHEN marks an optional hyphenation point and may be invisible until a line breaks. It can cause hidden differences in copied words, search issues, unexpected length, extraction discrepancies and broken identifiers when inserted accidentally.

inter­national
Hidden code point: U+00AD between r and n

Removing soft hyphens is often reasonable in identifiers or copied plain text, but not always in layout-sensitive publishing workflows.

Byte order mark and U+FEFF

U+FEFF can appear at the start of a stream as a byte order mark. UTF-8 BOM bytes are EF BB BF. UTF-8 does not need a BOM for byte order, but some software uses it as a signature. A BOM interpreted as text can appear unexpectedly, and mid-string U+FEFF should not be used as a modern word joiner.

UTF-8 BOM:
EF BB BF

Possible mojibake symptom:


Use UTF-8 Validator and read What Is Mojibake? or How to Fix Broken UTF-8 Text.

Bidirectional formatting controls

Bidirectional controls affect visual ordering of mixed left-to-right and right-to-left text. Relevant characters include U+202A LEFT-TO-RIGHT EMBEDDING, U+202B RIGHT-TO-LEFT EMBEDDING, U+202D LEFT-TO-RIGHT OVERRIDE, U+202E RIGHT-TO-LEFT OVERRIDE, U+202C POP DIRECTIONAL FORMATTING, isolates U+2066–U+2069, and marks U+200E and U+200F. Some are legitimate and necessary, but unexpected controls can confuse users and reviewers. Isolates are generally safer than older embeddings or overrides in many contexts.

Control characters and format characters

Control characters have general category Cc, including tab, line feed, carriage return and C0/C1 controls. Format characters have category Cf, including ZWJ, ZWNJ, bidi controls and word joiner. Some controls are expected, some are prohibited in identifiers, and some should be escaped or visualized in logs. Category alone is not enough to decide whether removal is safe.

Variation selectors

Variation selectors are normally invisible and request a particular presentation. U+FE0E often requests text presentation, while U+FE0F often requests emoji presentation. They may be part of one grapheme cluster, and removing them can change appearance.

Text-style heart:
♥
U+2665

Emoji-style heart:
♥️
U+2665 U+FE0F

Do not label variation selectors as junk characters.

Combining marks that appear invisible alone

Combining marks modify a preceding character. They may look invisible or misplaced without a base, but they are not zero-width controls in the same sense as U+200B. They can create visually similar strings with different code points, and long combining sequences can affect display and validation.

e + ◌́
U+0065 U+0301
U+0301 COMBINING ACUTE ACCENT

Read Unicode Normalization Explained and use Unicode Normalizer.

Interactive invisible-character detector

This local analyzer reveals selected invisible and default-ignorable code points, shows names and byte values, and previews policy-based cleanup without changing text automatically.

Why invisible characters appear in text

Invisible characters can enter text by copying from websites, PDFs, rich-text editors, word processors, messaging applications, social media, international keyboards, emoji composition, language-specific shaping, HTML entities such as  , automated formatting, data imports, OCR, extraction pipelines or malicious insertion. Origin alone does not determine whether the character is harmful.

Invisible characters in usernames and identifiers

Risks include visually identical but logically different usernames, duplicate-account confusion, login mismatches, moderation difficulty, hidden separators, bidi display confusion, confusable-character combinations and copy/paste failures.

1. Define allowed scripts.
2. Normalize according to a documented form.
3. Apply case handling separately.
4. Reject or restrict unexpected default-ignorable characters.
5. Allow language-required joiners only under policy.
6. Detect mixed scripts and confusables.
7. Display escaped diagnostics to administrators.
8. Preserve original values for audit when appropriate.

Use Confusable Character Detector, Mixed Script Detector and Unicode Normalization Checker.

Invisible characters in passwords

Passwords may legally contain invisible Unicode characters. Password managers and copy/paste can introduce hidden characters; trimming passwords can break authentication; normalizing passwords may change user input; and security protocols may define exact preprocessing rules. Password forms should not silently remove characters, and diagnostic functions must protect privacy. Do not log password diagnostics or expose password code points in analytics.

Invisible characters in URLs and email addresses

Zero-width characters can invalidate or alter a URL. A copied domain may contain unexpected code points. Percent encoding and normalization are separate, email local parts and domains have different rules, and internationalized domain names require dedicated processing. Parse with standards-compliant libraries, reject unexpected controls, expose suspicious code points during validation and do not use simple regex-only validation. Use URL Encoder and Decoder.

Invisible characters in source code

Hidden whitespace can affect tokens or formatting, bidi controls can alter visual order, zero-width characters can appear in identifiers or strings, and reviewers may not see logical sequence. Make control characters visible in editors, reject unexpected bidi controls according to language tooling, add linter rules, review raw code points and keep policy language-specific. Legitimate Unicode in source code should not be banned without reason.

Invisible characters in databases

Visually identical strings may be stored as different values, unique constraints may not catch hidden differences, search may miss records, trailing or special spaces can behave differently and cleanup may create collisions. Audit first, group by code-point pattern, test collisions, apply transformations transactionally, store a reversible mapping and review language-sensitive joiners separately.

value_original
value_cleaned
invisible_character_count
cleanup_profile
review_status

Invisible characters in copied PDF and document text

PDFs may include soft hyphens, non-breaking spaces, unexpected ligatures, combining marks, directional controls and zero-width characters around line boundaries. OCR errors are separate. Inspect code points, reveal whitespace, compare with the source, remove only known extraction artifacts, preserve meaningful language behavior and recheck word boundaries. Use Clean Copied PDF Text and Invisible Character Detector.

Invisible characters and string comparison

const first = "username";
const second = "user\u200Bname";

console.log(first === second); // false

Direct equality fails because the second string contains U+200B. Normalization may not remove it, trimming may not remove it and case folding does not remove it. Detection and policy-based cleanup are separate steps, and comparison keys may need controlled filtering.

Does Unicode normalization remove invisible characters?

NFC, NFD, NFKC and NFKD are not general invisible-character removers. Many format controls remain unchanged, some compatibility mappings may alter certain characters, and combining marks are not simply removed through normalization.

Which invisible characters are safe to remove?

Safety depends on context.

There is no universal “remove all invisible Unicode” rule. For cleanup workflows, read How to Remove Zero-Width Characters.

Safe cleanup profiles

Copied plain text

May convert U+00A0 to U+0020, remove U+00AD, remove accidental U+200B after review, preserve line breaks and preserve joiners unless the user opts in.

Identifier validation

May reject unexpected controls and default-ignorables, apply a documented normalization form, restrict scripts, detect confusables and preserve only explicitly allowed joiners.

Source-code audit

May reveal all controls, flag bidi overrides and zero-width characters outside approved contexts, and never modify automatically.

Emoji-preserving cleanup

Must preserve U+200D, variation selectors, emoji modifiers and regional indicators.

Invisible characters in JavaScript

const INVISIBLE_PATTERN = /[\u00A0\u00AD\u200B\u200C\u200D\u2060\uFEFF]/gu;

function findInvisibleCharacters(text) {
    return [...text.matchAll(INVISIBLE_PATTERN)].map(match => ({
        value: match[0],
        index: match.index,
        codePoint: `U+${match[0].codePointAt(0).toString(16).toUpperCase().padStart(4, "0")}`,
    }));
}

function inspectCodePoints(text) {
    const result = [];
    let codeUnitIndex = 0;
    for (const character of text) {
        result.push({ character, codePoint: character.codePointAt(0), codeUnitIndex });
        codeUnitIndex += character.length;
    }
    return result;
}

function removeZeroWidthSpace(text) {
    return text.replaceAll("\u200B", "");
}

This pattern is limited, not a complete security policy. Do not remove ZWJ or ZWNJ globally; trim() does not solve every hidden-character problem.

Invisible characters in Python

import unicodedata

def inspect_text(text: str) -> list[dict[str, object]]:
    result = []
    for index, character in enumerate(text):
        result.append({
            "index": index,
            "character": character,
            "code_point": f"U+{ord(character):04X}",
            "name": unicodedata.name(character, "<unnamed>"),
            "category": unicodedata.category(character),
            "bidirectional": unicodedata.bidirectional(character),
        })
    return result

INVISIBLE_CODE_POINTS = {0x00A0, 0x00AD, 0x200B, 0x200C, 0x200D, 0x2060, 0xFEFF}

def find_selected_invisible_characters(text: str) -> list[tuple[int, str]]:
    return [(index, f"U+{ord(character):04X}") for index, character in enumerate(text) if ord(character) in INVISIBLE_CODE_POINTS]

def remove_zero_width_space(text: str) -> str:
    return text.replace("\u200B", "")

str.strip() does not remove every format character. Unicode categories help, but category-based deletion can be dangerous. Preserve ZWJ and ZWNJ unless policy permits removal.

Invisible characters in PHP

PHP strings are byte sequences. Unicode-aware inspection may require intl and mbstring.

function inspectUnicodeText(string $text): array
{
    $characters = mb_str_split($text, 1, "UTF-8");
    $result = [];

    foreach ($characters as $index => $character) {
        $codePoint = IntlChar::ord($character);
        $result[] = [
            "index" => $index,
            "character" => $character,
            "code_point" => sprintf("U+%04X", $codePoint),
            "name" => IntlChar::charName($codePoint),
            "type" => IntlChar::charType($codePoint),
        ];
    }

    return $result;
}

function removeZeroWidthSpace(string $text): string
{
    return str_replace("\u{200B}", "", $text);
}

Do not use byte indexing for Unicode characters, do not delete all format characters by category and validate UTF-8 first.

Common invisible-character mistakes

Removing every zero-width character

This can break emoji and complex-script shaping.

Treating all invisible characters as malicious

Many are legitimate.

Assuming trim removes them

Most format controls are not ordinary trimming whitespace.

Using normalization as cleanup

Normalization does not remove most hidden controls.

Replacing all Unicode spaces with ordinary spaces

This can break typography and no-break behavior.

Deleting variation selectors

This can change text versus emoji presentation.

Removing ZWJ from emoji

This can split composite emoji.

Removing ZWNJ from language text

This can alter spelling or shaping.

Ignoring bidirectional controls

Logical and visual order may differ.

Cleaning identifiers after they are stored

This may create collisions.

Logging hidden characters invisibly

Logs should escape or label suspicious code points.

Using a small regex as a complete security solution

Unicode policies require context and property data.

Practical detection and cleanup workflow

1. Preserve the original text.
2. Inspect code points.
3. Classify spaces, controls, format characters and combining marks.
4. Determine the text context.
5. Identify legitimate language or emoji usage.
6. Flag unexpected characters.
7. Choose a named cleanup profile.
8. Preview every transformation.
9. Check for collisions.
10. Apply changes to a copy.
11. Recheck grapheme clusters and display.
12. Record the cleanup policy.
13. Add regression tests.

Use Invisible Character Detector, Invisible Character Remover, Unicode Character Inspector, Unicode Sequence Analyzer and Unicode Text Compare.

Try these UnicodeNow tools

These tools reveal hidden code points, compare suspicious strings and clean copied text with review.

Frequently asked questions

References

• The Unicode Standard
• Unicode Character Database
• Unicode glossary
• Unicode Standard Annex #29: Unicode Text Segmentation
• Unicode Standard Annex #9: Unicode Bidirectional Algorithm
• Unicode Standard Annex #14: Unicode Line Breaking Algorithm
• Unicode Standard Annex #15: Unicode Normalization Forms
• Unicode emoji data
• MDN: JavaScript strings
• Python documentation: unicodedata
• PHP manual: IntlChar
• ICU Unicode properties