UnicodeNow Methodology

Purpose and scope

UnicodeNow provides browser and server-based tools for inspecting, converting, comparing and cleaning Unicode text. This methodology describes the general rules shared across the site. Individual tool pages may document additional behavior for specific transformations, and where a tool page's documentation is more specific than this page, the tool page takes precedence.

This methodology applies to:

• Unicode converters (escape, encoding and format converters)
• Character lookup tools
• Character and sequence analyzers
• Text comparison tools
• Encoders and decoders
• Text cleaners
• Invisible-character tools
• Developer utilities such as byte-length and validation tools

Core operating principles

Preserve original input unless a transformation is requested

Input fields are not trimmed, reordered or silently rewritten before analysis.

Apply only explicit, selected transformations

Options such as replacement modes, presets or normalization forms only take effect when selected.

Distinguish grapheme clusters, code points, code units and bytes

These are different counts and are labeled separately rather than conflated into one "character count."

Avoid silently applying Unicode normalization

Normalization only runs inside tools built for that purpose, or when a user enables it.

Prefer browser-local processing for sensitive text

Most tools run entirely in JavaScript so input does not need to leave the browser.

Mark server-assisted tools accurately

A small number of tools submit text to this site's own server; those pages are labeled "Processed securely on the server," not local.

Warn before destructive cleanup

Removal and replacement tools report what would change and surface warnings for settings that discard data.

Show what changed where practical

Reports and diagnostic tables list affected code points, categories and counts rather than only a final result.

Use documented Unicode data sources

Character properties come from the bundled Unicode Character Database, standard library modules or browser APIs, not invented tables.

Expose limitations and assumptions

Where a tool's coverage is partial or approximate, that is stated rather than implied to be complete.

Treat valid Unicode contextually, not automatically as invalid or malicious

Mixed scripts, bidi controls, private-use characters and unusual code points are reported as diagnostic signals, not flagged as attacks.

Never execute submitted text as HTML

Input and diagnostic output are rendered as text, not interpreted as markup or script.

Local and server-side processing

Browser-local tools

Tools marked "Processed locally in your browser" perform analysis or transformation with JavaScript and do not submit the entered text to UnicodeNow servers. This covers most tools on the site.

Server-assisted tools

A small number of tools submit the entered text to this site's own server for processing, and search or lookup pages send a query to the server to retrieve matching Unicode data. Those pages are not labeled as fully local.

The local-processing badge is set per tool from the same setting that determines whether the page loads only client-side JavaScript or also submits requests to the server; it is not a separate, independently maintained label.

Copy and download actions run entirely in your browser on every tool page, including server-assisted ones, since they operate on the result already displayed on the page rather than making a new request.

UnicodeNow does not add its own logging of tool input text or search queries. Standard web-server access logging, where enabled, may still record the request path, which for a GET-based search page includes the query string; POST-based tool requests send text in the request body, which typical access logs do not record. No analytics platform is currently integrated on tool pages, so no analytics service receives raw tool input.

Unicode standards and data versions

Different parts of the site currently use different Unicode Character Database revisions, because character-property lookups use the Python standard library while script, block and age data are loaded from separately downloaded Unicode data files. Browser APIs such as Intl.Segmenter and String.prototype.normalize use whatever Unicode version is bundled with the visitor's own browser, which can differ from both.

HTML named-entity references use Python's built-in html.entities module rather than a separately versioned Unicode file.

See Data Sources for where these files come from.

Code points, code units and grapheme clusters

Unicode code point

A Unicode value such as U+0041 or U+1F600.

Unicode scalar value

A valid Unicode code point excluding the UTF-16 surrogate code points U+D800–U+DFFF.

UTF-16 code unit

The unit counted by JavaScript's string.length. A supplementary-plane character such as most emoji uses two UTF-16 code units, so string.length is not a character count.

Grapheme cluster

A user-perceived character, which may contain multiple code points, such as a decomposed accented letter or a multi-part emoji sequence.

Byte count

The number of bytes after encoding text, most commonly UTF-8.

Implementation rules used across tools:

• Code points are counted with code-point-safe iteration (JavaScript's for...of or spread over a string), not by index.
• UTF-16 code units use JavaScript's native string.length.
• UTF-8 bytes use the browser's TextEncoder, or Python's UTF-8 encoder on the server.
• Graphemes use Intl.Segmenter with grapheme-cluster granularity where the browser supports it, with a documented code-point-based fallback otherwise.
• Array.from(text) or a spread ([...text]) counts code points; it is not grapheme segmentation and is never described as one on this site.

Grapheme segmentation implementation

Tools that report grapheme clusters prefer the browser's built-in Intl.Segmenter with granularity: "grapheme" and the browser's default locale; segmentation is not customized per language. Where Intl.Segmenter is unavailable, tools fall back to counting individual code points and label the resulting count as approximate rather than claiming full grapheme-cluster support. Emoji ZWJ sequences, combining marks and regional-indicator flag pairs are grouped into a single grapheme cluster when Intl.Segmenter is available; under the code-point fallback they are counted as separate code points instead. This means grapheme counts can differ between browsers that support Intl.Segmenter and older browsers that do not.

Related guides: Code Points vs Code Units, What Is a Grapheme Cluster? and How to Count Unicode Characters Correctly.

Unicode normalization

NFC (Normalization Form C) applies canonical decomposition followed by canonical composition, typically producing precomposed characters. NFD applies canonical decomposition only. NFKC and NFKD additionally apply compatibility decomposition, which can replace characters such as ligatures, circled digits or full-width forms with plainer equivalents.

Canonical composition and decomposition preserve the text's abstract identity. Compatibility composition and decomposition can remove representational distinctions that mattered in the original text, such as the difference between a ligature and its separate letters, so NFKC and NFKD are not safe defaults for archival or exact-comparison use. Because normalization can add, remove or reorder code points, it can change code-point counts and byte counts even when the visible text looks identical. Normalization does not remove most invisible or control characters: zero-width joiners, bidi controls and variation selectors are not deleted by any of the four forms.

UnicodeNow does not apply normalization silently unless the selected tool exists specifically to normalize text or the user enables a normalization operation. Tools such as the Unicode Normalizer apply exactly one selected form to produce a new output while leaving the input field unchanged; diagnostic tools such as the Unicode Sequence Analyzer and Unicode Character Counter compute all four forms on separate copies purely for comparison and never modify the original input.

See Unicode Normalization Explained.

Encodings and byte counts

Where implemented, tools calculate or display UTF-8 bytes, UTF-16 code units, UTF-16 byte counts, UTF-32 values, HTML decimal and hexadecimal references, JavaScript escapes, JSON escapes, CSS escapes and URL encoding.

• UTF-16 byte counts (code units × 2) exclude a byte-order mark unless stated otherwise.
• URL encoding is based on the UTF-8 bytes of the text, then percent-encoded.
• Named HTML entities do not exist for every character; numeric character references are shown instead, and they are valid for any scalar value.
• Supplementary-plane JavaScript escapes may use \u{...} notation or a UTF-16 surrogate-pair notation (two \uXXXX escapes) depending on the context a tool is generating for.
• JSON string escapes always use UTF-16 surrogate-pair notation for supplementary-plane characters, since that is what the JSON specification defines.

This page only describes formats actually implemented on the site; consult the specific tool page (for example the UTF-8 Encoder and Decoder or Unicode Escape Converter) for exact output options.

Scripts, blocks and character properties

Where supported, character-detail tools expose the Unicode name, general category, script, block, plane, canonical combining class, bidirectional class, mirrored property, Unicode age, and assigned, private-use or unassigned status of a code point. Script Extensions and East Asian width are not currently bundled, so tools do not claim to report them.

Script

A writing-system property, such as Latin, Cyrillic or Han. Script describes writing systems, not languages.

Block

A contiguous range of code points reserved for a purpose. A single block can contain characters from several scripts.

Common and Inherited scripts

Common covers code points shared across scripts, such as digits, punctuation and most emoji. Inherited covers mainly combining marks, which take their script context from the preceding base character.

Private-use characters

Have no meaning defined by the Unicode Standard. Fonts and applications may assign their own glyphs and interpretations, so no standardized name is shown for them.

Unassigned code points

Have no character assigned in the specific bundled Unicode version reported above; a code point unassigned today may be assigned in a future Unicode version.

Bidirectional and invisible characters

Diagnostic tools classify several distinct categories rather than one combined "invisible" bucket: zero-width space, zero-width joiner, zero-width non-joiner, word joiner, bidirectional controls, directional isolates, variation selectors, soft hyphen, C0/C1 control characters, non-breaking and other special Unicode spaces, combining marks, tag characters and private-use characters.

Invisible does not mean invalid or unwanted: joiners can be required by language text (such as Zero Width Joiner and Zero Width Non-Joiner in some scripts), ZWJ and variation selectors are frequently required inside emoji sequences, and combining marks are meaningful parts of the text they attach to. Bidi controls can be legitimate (for example in genuinely bidirectional content) but are rendered safely rather than left to reorder the surrounding page. Removal tools preserve these characters by default and only remove a category when the user explicitly selects it. Unicode normalization does not remove most of these characters either.

Bidi-safe rendering

Raw bidirectional control characters are not rendered directly in diagnostic tables or reports; they are shown as a visible bracketed label and isolated (using <bdi>, CSS isolation or an equivalent safe-rendering technique) so they cannot reorder surrounding page content. Where a tool reports bidi classes or directionality runs, that is property analysis, not necessarily a full implementation of the Unicode Bidirectional Algorithm, and the final visual ordering in a real document can still depend on surrounding context and the rendering engine.

Tool-specific transformations

Tools that transform text generally follow this workflow:

1. Preserve the original input.
2. Parse by code point or grapheme cluster as appropriate for the tool.
3. Apply only the rules the user selected.
4. Avoid unrelated transformations.
5. Maintain stable positions where practical.
6. Count and report what changed.
7. Show warnings for settings that are destructive.
8. Preserve meaningful Unicode (tabs, line breaks, combining marks, joiners) by default.
9. Generate derived output (reports, previews, normalized copies) separately from the original input.
10. Never execute submitted content as HTML.

For example, the Invisible Character Remover and Control Character Remover only remove the categories a user selects and offer a marker mode that replaces a character with a visible, labeled placeholder instead of deleting it silently. The Whitespace Normalizer and Unicode Text Cleaner apply named presets rather than one fixed, unexplained transformation. The Unicode Normalizer applies exactly one selected normalization form and reports whether the output changed. Unicode Text Compare never modifies either input text; it only computes and displays a comparison. This methodology describes the shared workflow; it does not claim that every tool implements every step where a step does not apply to that tool's purpose.

Testing and validation

Automated and manual tests for text-processing tools are written to cover representative cases, including: ASCII text, accented characters, precomposed and decomposed text, supplementary-plane characters, emoji ZWJ sequences, regional-indicator flag pairs, emoji modifiers, variation selectors, combining marks, zero-width characters, bidi controls, tabs and special spaces, line-ending variants (CRLF, CR, LF, U+0085, U+2028, U+2029), CJK and right-to-left scripts, mixed-script text, private-use characters, unassigned code points, surrogate and out-of-range value validation, XSS-like input, bidi-safe rendering, long input, copy and download output, and no-network checks for tools marked local.

Test types used on this project include unit tests of individual processing functions, integration tests that render a page and inspect its HTML, SEO-focused rendering tests (metadata, canonical URL, structured data), regression tests added when a bug is fixed, and hand-verified fixtures where an expected value (such as a byte count or grapheme count) is computed independently before being written into a test. Automated cross-browser checks are not currently part of this test suite.

Validation sources

Results are checked against browser standards APIs (Intl.Segmenter, TextEncoder, String.prototype.normalize), the bundled Unicode data described above, and hand-verified fixtures. Some values are additionally cross-checked by hand against Python's unicodedata module. The project does not currently run automated parity tests comparing every server-side implementation against every client-side one, and it does not claim official Unicode conformance testing or certification.

Browser and platform differences

• Glyph appearance depends on the visitor's font and operating system.
• Emoji presentation varies by platform even for the same code point sequence.
• A missing-glyph box does not necessarily mean an invalid code point; it can simply mean no installed font supports that character.
• Intl.Segmenter behavior can depend on the browser and the Unicode version it bundles.
• Word segmentation is locale-sensitive; this site uses the browser's default locale rather than a per-language setting.
• Very recently assigned Unicode characters may not render on older browsers or operating systems.
• Browser APIs and this site's bundled Unicode data can use different Unicode versions, as described above.
• Clipboard and file-download behavior can vary by browser and operating system.
• Line-ending representation (CRLF versus LF) can vary by platform and by how text was copied.

Privacy and data handling

Most tools on this site are browser-local and do not send entered text to UnicodeNow servers. A small number of tools (listed above) submit text to this site's own processing endpoint or, for search/lookup pages, as a URL query parameter; the request is used to produce the response and is not intentionally stored afterward. UnicodeNow does not currently run an analytics platform on tool pages, so no analytics service receives raw tool input. Generated downloads (reports and converted text) are built in your browser from the result already on the page. This site's only use of browser local storage is to remember your light/dark theme preference; tool input is not saved to local storage and does not persist after a page reload.

Full privacy terms are documented on the Privacy page.

Limitations

• Visual-similarity (confusable character) detection is not guaranteed to catch every lookalike.
• Mixed-script text is reported as a diagnostic signal; it is not automatically treated as malicious.
• Normalization does not remove all formatting or invisible characters.
• Lowercasing shown by some tools uses JavaScript's toLowerCase(), which is not always equivalent to full Unicode case folding.
• Full case folding is available in some server-side code paths but is not currently exposed by every browser-based tool.
• Bidi property analysis may not reproduce the exact visual order a browser would render.
• Word counts can vary by locale and by which segmentation method a browser supports.
• Glyph rendering always depends on the visitor's installed fonts.
• Private-use character meanings are application-specific and cannot be resolved by this site.
• Unassigned status depends on the specific bundled Unicode version described above and can change in later Unicode releases.
• A visitor's browser API version may differ from the Unicode data version bundled with this site.

Updates and corrections

We review tool behavior when Unicode data, browser APIs or site implementations change. Suspected errors can be reported through the contact page with the affected tool, input, observed result and expected result.

Related resources

Data Sources · Privacy · About · Contact · What Is Unicode? · Code Points vs Code Units · What Is a Grapheme Cluster? · What Are Invisible Unicode Characters? · Unicode Normalization Explained

Representative tools referenced throughout this page:

Frequently asked questions