Unicode Normalization Explained

Unicode normalization at a glance

NFC and NFD preserve canonical meaning. NFKC and NFKD may remove formatting or presentation distinctions.

What is Unicode normalization?

Unicode may allow more than one code-point sequence for equivalent text. Normalization converts text into one of four defined forms. It works on code-point sequences; it does not change encoding, choose a font, translate text, remove every invisible character, or guarantee that all visually identical strings become equal. For the underlying units, see What Is a Unicode Code Point?, Code Points vs Code Units and What Is a Grapheme Cluster?.

Precomposed:
é → U+00E9

Decomposed:
e + ◌́ → U+0065 U+0301

These sequences are canonically equivalent. Try Unicode Normalizer, Unicode Normalization Checker and Unicode Text Compare.

Why can identical-looking text have different code points?

Causes include precomposed characters, base characters plus combining marks, compatibility characters, presentation variants, full-width and half-width forms, ligatures, superscript or circled characters, different combining-mark orders, invisible formatting characters and confusable characters from different scripts. Normalization addresses only defined canonical or compatibility relationships; it does not make unrelated confusable characters equivalent.

é
U+00E9

é
U+0065 U+0301

fi
U+FB01

Compatibility decomposition:
f + i
U+0066 U+0069

fi and fi are compatibility-equivalent, not canonically equivalent.

Canonical equivalence

Canonically equivalent sequences represent the same abstract text. Canonical normalization should preserve intended textual identity. NFC and NFD use canonical decomposition mappings, and canonical combining-class ordering is part of normalization. Canonical equivalence does not mean identical code-point sequences.

Å
U+00C5

A + ◌̊
U+0041 U+030A

Compatibility equivalence

Compatibility equivalence includes characters that may have distinct presentation or formatting roles. NFKC and NFKD apply compatibility decomposition, which can change symbols into simpler base text. This is useful for search or identifiers but can be wrong for archival, legal, mathematical or typographic content.

fi → fi
① → 1
Ａ → A
⁵ → 5

How Unicode decomposition works

Decomposition replaces a character with its defined component sequence. Canonical decomposition uses canonical mappings; compatibility decomposition uses canonical and compatibility mappings. NFD uses canonical decomposition and NFKD uses compatibility decomposition. Combining marks are reordered into canonical order, and some characters do not decompose.

é U+00E9 → e + ◌́ U+0065 U+0301
Å U+00C5 → A + ◌̊ U+0041 U+030A

Canonical combining classes define a stable mark order without requiring developers to reproduce the full algorithm.

How Unicode composition works

Composition starts from a decomposed sequence. Eligible sequences are recomposed into precomposed characters where mappings exist. NFC performs canonical decomposition followed by canonical composition; NFKC performs compatibility decomposition followed by canonical composition. Not every decomposed sequence has a precomposed equivalent, and composition exclusions prevent some recompositions.

U+0065 U+0301 → U+00E9

Normalized does not mean one code point per visible character.

NFC explained

NFC, Normalization Form C, applies canonical decomposition and then recomposition. It often produces precomposed characters and is widely suitable for general text interchange and storage. It does not apply compatibility decomposition, so it preserves compatibility distinctions such as ligatures or full-width forms.

e + ◌́ → é
A + ◌̊ → Å

Open the Unicode Normalizer.

NFD explained

NFD, Normalization Form D, applies canonical decomposition. It commonly separates precomposed characters into base characters and combining marks, preserves compatibility distinctions, may increase code-point and byte counts, and can affect filename interoperability. NFD is not more correct than NFC; both are valid normalized representations.

NFKC explained

NFKC, Normalization Form KC, applies compatibility decomposition and then canonical composition. It can simplify full-width forms, ligatures, superscripts and circled characters, making it useful in controlled search or identifier workflows. It may remove meaningful display distinctions and should not automatically replace original stored text.

NFKD explained

NFKD, Normalization Form KD, applies compatibility decomposition without recomposition. It produces fully decomposed compatibility output, may be useful in specialized text analysis, often increases sequence length, and may remove formatting distinctions while retaining combining marks separately. It is rarely the best format for direct user-facing storage.

NFC vs NFD vs NFKC vs NFKD

Choose NFC for general text consistency. Choose NFD when decomposed representation is required. Choose NFKC for controlled compatibility-insensitive comparison. Choose NFKD for specialized decomposition workflows. These are starting points, not absolutes. For a focused canonical comparison, read NFC vs NFD.

Interactive Unicode normalization comparison

This local component compares the original input with all four normalization forms and shows code points, counts and byte size.

Normalization does not change encoding

Normalization changes code-point sequences. Encoding changes code points into bytes or code units. NFC text may be encoded as UTF-8, UTF-16 or UTF-32, and NFD text may also use any Unicode encoding. Converting UTF-16 to UTF-8 does not normalize text; normalizing text does not necessarily change its encoding.

NFC code points:
U+00E9
UTF-8 bytes:
C3 A9

NFD code points:
U+0065 U+0301
UTF-8 bytes:
65 CC 81

Read Unicode vs UTF-8, UTF-8 vs UTF-16 or use the UTF-8 Encoder and Decoder.

Normalization vs case folding

Normalization handles equivalent Unicode representations. Case folding supports case-insensitive comparison. Lowercasing is not identical to full case folding. Normalization does not make text case-insensitive, and case folding does not normalize canonical equivalents. Some workflows require both operations in a deliberate order.

É
é
é

Case and normalization are separate dimensions. Operation order depends on the specification or application.

Normalization vs removing accents

NFD and NFKD can separate many base characters from combining marks, but removing marks is an additional destructive operation. Normalization alone does not remove accents, and some characters do not decompose into a simple Latin base plus accent.

é → NFD → e + ◌́

Deleting U+0301 would produce e, but that is not normalization. Accent removal can change names, words and meaning.

Normalization and string comparison

Direct comparison can fail when strings use different code-point sequences.

const first = "caf\u00E9";
const second = "cafe\u0301";

console.log(first === second); // false
console.log(
    first.normalize("NFC") === second.normalize("NFC")
); // true

Normalize both values to the same form. Do not normalize only one side. Combine normalization with case or locale rules only when required, and preserve original data if exact representation matters. Security-sensitive identifiers may require stricter profiles than normalization alone. Use Unicode Text Compare and Unicode Normalization Checker.

Normalization and search

Search indexes may miss canonically equivalent text when representations differ. Normalizing indexed and query text to the same form can improve matching. NFKC may help compatibility-insensitive search, but it may collapse distinctions users expect. Accent-insensitive and locale-aware search are separate features, and original display text should be preserved.

Original text → preserve for display
Normalized search value → store or index separately

Normalization and usernames or identifiers

Identifiers need consistent comparison rules. Normalization can reduce canonical-equivalence mismatches. NFKC is often considered in identifier profiles, but it is not sufficient by itself. Confusable characters, mixed scripts, invisible controls and case folding need separate documented rules.

User-facing value: preserve original
Comparison key: normalize according to the identifier policy

Use Confusable Character Detector, Mixed Script Detector and Invisible Character Detector.

Normalization and databases

Databases may store canonically equivalent strings as different values, and unique constraints may not treat them as equal. Collations vary by database and configuration. Normalizing before comparison or generating a comparison key can help, but original text may need separate storage. Migration requires careful analysis, normalization does not replace collation design, and byte or character lengths may change.

display_value
normalized_comparison_value

Normalization and filenames

Filesystems may use or expose different normalization behavior. Filenames that look identical may contain different code-point sequences, and copying between systems can reveal differences. Applications should not assume filenames are in NFC, comparing raw code points may fail, and automatic renaming can cause collisions.

résumé.txt

Accented letters could use precomposed or decomposed forms. Preserve the actual filesystem name when performing file operations.

Normalization in URLs, slugs and APIs

URL paths can contain percent-encoded UTF-8 sequences, and visually equivalent paths may produce different bytes. Slug generation should define a normalization policy. API producers and consumers should agree on normalization-sensitive identifiers. JSON strings can contain canonically equivalent but non-identical sequences. Normalization should happen at deliberate boundaries.

Input text
→ validate
→ normalize for slug or comparison
→ apply product-specific transformations
→ encode for URL

URL encoding and Unicode normalization are separate operations. Use URL Encoder and Decoder and Unicode Normalizer.

Normalization in JavaScript

const text = "e\u0301";

console.log(text.normalize("NFC"));
console.log(text.normalize("NFD"));
console.log(text.normalize("NFKC"));
console.log(text.normalize("NFKD"));

function isNormalized(text, form = "NFC") {
    return text === text.normalize(form);
}

const NORMALIZATION_FORMS = new Set(["NFC", "NFD", "NFKC", "NFKD"]);

function normalizeUnicode(text, form = "NFC") {
    if (!NORMALIZATION_FORMS.has(form)) {
        throw new TypeError("Unsupported Unicode normalization form");
    }
    return text.normalize(form);
}

Invalid form names throw an error. Normalization returns a new string, leaves the original unchanged, and does not change JavaScript’s UTF-16 code-unit model or segment graphemes.

Normalization in Python

import unicodedata

text = "e\u0301"

nfc = unicodedata.normalize("NFC", text)
nfd = unicodedata.normalize("NFD", text)
nfkc = unicodedata.normalize("NFKC", text)
nfkd = unicodedata.normalize("NFKD", text)

def is_normalized(text: str, form: str = "NFC") -> bool:
    normalized = unicodedata.normalize(form, text)
    return text == normalized

Python strings represent Unicode text. unicodedata uses Unicode data shipped with the runtime, so tests should use the application’s actual runtime version.

Normalization in PHP

$text = "e\u{0301}";

$nfc = Normalizer::normalize($text, Normalizer::FORM_C);
$nfd = Normalizer::normalize($text, Normalizer::FORM_D);
$nfkc = Normalizer::normalize($text, Normalizer::FORM_KC);
$nfkd = Normalizer::normalize($text, Normalizer::FORM_KD);

$isNormalized = Normalizer::isNormalized($text, Normalizer::FORM_C);

The Intl extension is required. Normalization can fail and should be checked. PHP strings are byte sequences, input should be valid UTF-8 when expected, and normalization does not change PHP into a character-based string model.

When should you normalize text?

Consider normalization for canonical string comparison, search indexing, slug generation, identifier comparison, deduplication, imported text from mixed sources, database comparison keys and Unicode-aware validation pipelines. Define the form, boundary, preservation policy, case handling order, compatibility policy and migration plan.

When should you avoid or delay normalization?

Use caution for digital signatures, cryptographic hashes, exact archival preservation, legal evidence, source-code analysis, binary protocols, exact user-authored text, mathematical notation, specialized typography, existing identifiers with compatibility requirements and data where changes would break external references.

If exact code-point preservation matters, store the original value before producing a normalized comparison form.

Common Unicode normalization mistakes

Normalizing only one value before comparison

Both values must use the same policy.

Assuming visual equality means canonical equivalence

Confusables and different scripts may look alike without being equivalent.

Treating NFKC as harmless cleanup

Compatibility distinctions can be lost.

Replacing original text with a destructive normalized form

Use separate display and comparison values when needed.

Confusing normalization with UTF-8 conversion

Normalization changes code points, not bytes.

Confusing normalization with lowercasing

Case folding is separate.

Assuming NFC produces one code point per character

Some clusters remain multiple code points.

Assuming NFD removes accents

It separates many marks but does not delete them.

Normalizing after hashing or signing

Changed code points produce changed bytes.

Normalizing filenames without collision checks

Duplicate names can result.

Ignoring Unicode-data version differences

Version and test your runtime behavior.

Using normalization as a security filter

It does not detect confusables, mixed scripts or invisible controls.

Practical normalization workflow

1. Preserve original input when fidelity matters.
2. Validate that input is well-formed Unicode text.
3. Identify the business purpose.
4. Choose NFC, NFD, NFKC or NFKD deliberately.
5. Normalize all compared values consistently.
6. Apply case or locale rules separately when required.
7. Validate invisible and control characters separately.
8. Validate scripts and confusables separately for identifiers.
9. Store normalized comparison keys separately when appropriate.
10. Test accented characters, combining marks, ligatures, full-width forms and emoji.

Use Unicode Normalizer, Unicode Normalization Checker, Unicode Text Compare, Confusable Character Detector and Invisible Character Detector.

Which normalization form should you choose?

NFC is often the safest general-purpose starting point, but the correct form depends on the application’s specification and whether compatibility distinctions must be preserved.

Try these UnicodeNow tools

These tools expose normalization forms, code-point sequences, comparison behavior and related Unicode risks.

Frequently asked questions

References

• The Unicode Standard
• Unicode Standard Annex #15: Unicode Normalization Forms
• Unicode Character Database
• Unicode glossary
• Unicode normalization test data
• Unicode Standard Annex #29: Unicode Text Segmentation
• MDN: String.prototype.normalize()
• Python documentation: unicodedata
• PHP manual: Normalizer
• ICU normalization documentation