How to Fix Broken UTF-8 Text

Broken UTF-8 repair checklist

1. Preserve the original source.
2. Determine whether you have bytes or already-decoded text.
3. Check the declared encoding.
4. Validate the bytes as UTF-8.
5. Inspect suspicious character patterns.
6. Identify the likely wrong decoding.
7. Generate repair candidates.
8. Verify the repaired language.
9. Test on a small sample.
10. Write repaired output separately.
11. Audit every transformation.
12. Add regression tests.

First determine what is actually broken

Broken UTF-8 can mean invalid UTF-8 byte sequences, valid UTF-8 decoded using the wrong encoding, double-encoded text, replacement characters caused by failed decoding, missing glyphs, Unicode normalization differences, OCR or PDF extraction errors, truncated multi-byte sequences, incorrect HTML entities or escaped text displayed literally. Do not apply an encoding repair until the type of corruption is known.

Read What Is Mojibake? when garbled characters such as é or ’ appear.

Preserve the original bytes

Raw bytes reveal whether a sequence is valid UTF-8, whether a byte order mark exists, whether a legacy encoding is plausible, whether bytes were truncated, whether text was decoded more than once and whether exact recovery is possible. Copy files before opening them in editors, export affected database rows before migration, save raw API payloads only in secure test environments, avoid resaving through spreadsheet software, record file hashes and keep original and repaired values separately.

original_value
repaired_value
repair_status
repair_method
repair_confidence
repaired_at

Do not log sensitive content unnecessarily. Log repair metadata and sample hashes when possible.

Check whether the bytes are valid UTF-8

In valid UTF-8, ASCII bytes use one byte, other Unicode scalar values use two to four bytes, continuation bytes appear only in valid positions, overlong sequences are invalid, surrogate values must not appear, values above U+10FFFF are invalid and truncated sequences are invalid. For background, read Unicode vs UTF-8 and UTF-8 vs UTF-16.

Valid UTF-8 for é:
C3 A9

Truncated:
C3

Invalid continuation:
C3 20

Use the UTF-8 Validator and UTF-8 Encoder and Decoder.

Inspect common mojibake signatures

These are diagnostic clues, not guaranteed answers. Use Character Encoding Detector and Unicode Character Inspector to verify.

How to reverse UTF-8 decoded as Windows-1252

For the original text café, the correct UTF-8 bytes are 63 61 66 C3 A9. If those bytes are decoded as Windows-1252, the visible result becomes café. The repair reverses the wrong path.

Current mojibake text
→ encode as Windows-1252
→ recover C3 A9
→ decode as UTF-8
→ correct text

Windows-1252 vs ISO-8859-1

Windows-1252 and ISO-8859-1 are single-byte encodings with significant overlap. Windows-1252 defines printable punctuation where ISO-8859-1 treats bytes as controls. Smart quotes and dashes commonly indicate Windows-1252, and some software labels Windows-1252 data as ISO-8859-1. Browser compatibility behavior can complicate labels, so exact repair should use the actual path where known.

’
U+2019
UTF-8: E2 80 99
Common mojibake: ’

The sequence ’ strongly suggests a Windows-1252-style wrong decode.

How to repair double-encoded UTF-8

Double encoding means the text was corrupted and encoded again.

é
→ é
→ é

1. Detect a likely double-encoding pattern.
2. Reverse one layer.
3. Re-evaluate the result.
4. Stop when the result becomes valid and plausible.
5. Never repeat automatically without a maximum depth.
6. Record the number of repair passes.

Set a strict maximum such as two or three layers. Unlimited recursive repair is unsafe.

What to do when text contains the replacement character

� is U+FFFD REPLACEMENT CHARACTER. A decoder inserted it after encountering invalid input, so the original bytes may already be lost. Reversing mojibake from U+FFFD alone is generally impossible; backups, raw files, source database dumps or network captures may be required.

caf�

Interactive broken UTF-8 repair assistant

This local assistant reuses the Mojibake Repair analyzer logic for garbled text and adds a raw-hex byte mode for UTF-8 validation and decoding candidates. It ranks candidates, limits repair depth and leaves low-confidence candidates unapplied.

How to fix broken UTF-8 in a text file

1. Duplicate the file.
2. Determine whether it contains raw bytes or already-corrupted text.
3. Inspect the file with a hex viewer.
4. Check for a BOM.
5. Validate as UTF-8.
6. Test likely legacy encodings.
7. Decode using the correct source encoding.
8. Re-encode once as UTF-8.
9. Save to a new file.
10. Compare line count, byte count and representative text.
11. Keep the original file.

Wrong:
Open café and save as UTF-8
Result remains café

Correct:
Recover the mistaken bytes
Decode them correctly
Then save the repaired Unicode text as UTF-8

How to fix broken UTF-8 in CSV files

CSV does not reliably carry encoding metadata. Spreadsheet software may guess, a UTF-8 BOM can help certain import workflows but affect other tools, delimiter detection is separate from encoding, opening and resaving can introduce corruption, and imports should be tested with accented, non-Latin and emoji values.

1. Preserve the original CSV.
2. Inspect raw bytes.
3. Detect or confirm the source encoding.
4. Convert once to UTF-8.
5. Validate every row.
6. Write a new CSV.
7. Test import settings explicitly.
8. Compare row and column counts.
9. Verify representative multilingual fields.

How to fix broken UTF-8 in a database

Common causes include connection encoding mismatch, an import tool using the wrong encoding, an application encoding text twice, a column character set changed after corruption, data decoded before insertion, or export/import encodings differing. Changing a column charset usually does not repair already-corrupted values.

1. Back up the database.
2. Identify affected columns and rows.
3. Group rows by corruption pattern.
4. Export a sample.
5. Generate repair candidates.
6. Verify with domain owners or native-language reviewers.
7. Write repaired data to a new column or staging table.
8. Compare original and repaired values.
9. Apply updates transactionally.
10. Record repair method and status.
11. Keep a rollback path.

value_original
value_repaired
repair_method
repair_confidence
repair_reviewed

How to fix broken UTF-8 on a web page

Trace the full request path: template or database, application, response bytes, HTTP headers, browser decoding and rendered text. Check template file encoding, database connection encoding, application string handling, reverse proxy behavior, HTTP Content-Type, HTML charset, frontend conversions, API response encoding and static asset encoding.

Template or database
→ application
→ response bytes
→ HTTP headers
→ browser decoding
→ rendered text

<meta charset="utf-8">

Content-Type: text/html; charset=utf-8

Metadata fixes future decoding but does not automatically repair text already stored as mojibake.

How to fix broken UTF-8 in APIs and JSON

JSON strings represent Unicode text and payload bytes are commonly UTF-8. The payload should be decoded once; unnecessary encode() and decode() calls can corrupt text. Base64 is not a text encoding, JSON escapes such as \u00E9 are syntax rather than mojibake, and logging layers may display correct payloads incorrectly.

Receive bytes
→ validate or decode as UTF-8 once
→ parse JSON
→ process Unicode strings
→ serialize JSON
→ encode as UTF-8 once

Check response charset, proxy transformations, double JSON encoding, literal escapes, wrong database values and invalid byte replacement.

How to fix broken UTF-8 copied from PDFs

PDF text extraction may use incorrect character maps. Embedded fonts can map glyphs to unexpected code points, ligatures may be extracted incorrectly, visual order may differ from logical order, OCR errors are not encoding errors, and generic UTF-8 repair may make extraction worse.

1. Inspect extracted code points.
2. Compare with visible PDF text.
3. Determine whether the problem is encoding, mapping or OCR.
4. Try a different extraction engine.
5. Use PDF-specific text cleanup.
6. Use OCR only when no reliable text layer exists.
7. Preserve the source PDF.

Use Clean Copied PDF Text and Unicode Character Inspector.

Fixing broken UTF-8 in JavaScript

JavaScript strings are already decoded Unicode text. Validate raw bytes with TextDecoder, and avoid deprecated escape() or unescape() hacks such as decodeURIComponent(escape(text)).

function decodeUtf8(bytes) {
    return new TextDecoder("utf-8", {
        fatal: true,
    }).decode(bytes);
}

const original = "café";
const bytes = new TextEncoder().encode(original);
const decoded = new TextDecoder("utf-8", {
    fatal: true,
}).decode(bytes);

console.log(decoded);

const result = repairMojibake(input, {
    assumedWrongEncoding: "windows-1252",
    intendedEncoding: "utf-8",
    strict: true,
});

if (result.confidence < 0.8) {
    showRepairCandidates(result.candidates);
}

In this project, the interactive assistant uses the existing Mojibake analyzer utility rather than a replacement dictionary.

Fixing broken UTF-8 in Python

Keep the text/bytes boundary explicit.

from pathlib import Path

raw_bytes = Path("input.txt").read_bytes()
text = raw_bytes.decode("utf-8", errors="strict")

from pathlib import Path

raw_bytes = Path("legacy.txt").read_bytes()
text = raw_bytes.decode("windows-1252", errors="strict")

Path("converted.txt").write_text(
    text,
    encoding="utf-8",
)

def repair_utf8_decoded_as_windows_1252(
    text: str,
) -> str:
    try:
        raw_bytes = text.encode(
            "windows-1252",
            errors="strict",
        )
        return raw_bytes.decode(
            "utf-8",
            errors="strict",
        )
    except (
        UnicodeEncodeError,
        UnicodeDecodeError,
    ) as error:
        raise ValueError(
            "The text does not match the expected encoding path."
        ) from error

Never use errors="ignore" for forensic repair. errors="replace" may destroy recovery information. Read uncertain files as bytes first and test candidate repairs on a sample.

Fixing broken UTF-8 in PHP

PHP strings are byte sequences, so validation and conversion must be deliberate.

if (!mb_check_encoding($value, "UTF-8")) {
    throw new InvalidArgumentException(
        "Input is not valid UTF-8."
    );
}

$bytes = file_get_contents("legacy.txt");

if ($bytes === false) {
    throw new RuntimeException(
        "Unable to read the file."
    );
}

$text = mb_convert_encoding(
    $bytes,
    "UTF-8",
    "Windows-1252"
);

Use the project’s existing tested encoding utilities for mojibake repair. A helper should accept current text, assumed wrong encoding and intended encoding; use strict validation where possible; return warnings and confidence; and preserve the original. mbstring is required, and PHP does not store an encoding label with each string.

How to detect already-correct text

Repair tools must avoid false positives. Checks include valid UTF-8, plausible language, no strong mojibake signatures, a repair candidate that does not introduce replacement or control characters, no reduction in readable text, successful re-encoding and identical original/repaired text.

Useful score factors include reduction in mojibake sequences, fewer control characters, more valid language characters, a valid UTF-8 round trip, fewer replacement characters, a reversible transformation and known source-system metadata. Language scoring is a clue, not proof.

Bulk-repair safety checklist

• Backup completed.
• Raw source preserved.
• Encoding path documented.
• Representative sample reviewed.
• Candidate confidence threshold defined.
• Low-confidence rows excluded.
• Correct text protected.
• Repair depth limited.
• Original and repaired values stored.
• Transaction boundaries defined.
• Rollback tested.
• Metrics recorded.
• Post-repair validation run.
• Native-language review completed when needed.
• Regression tests added.

rows_scanned
rows_flagged
rows_repaired
rows_skipped
rows_low_confidence
rows_with_replacement_character
rows_failed

Common repair mistakes

Using search and replace

Replacing é with é does not solve the general encoding problem.

Saving mojibake as UTF-8

This preserves the wrong Unicode characters.

Applying repair to every row

Correct data can be corrupted.

Ignoring double encoding

One repair pass may be insufficient.

Repeating repair until text looks right

This is unsafe and non-deterministic.

Using lossy error handlers

Ignoring or replacing invalid bytes can destroy evidence.

Treating encoding detection as certainty

Detectors provide likely candidates.

Discarding raw bytes

Exact recovery may become impossible.

Changing database charset only

Stored mojibake remains mojibake.

Confusing normalization with encoding repair

NFC or NFD does not repair é. Read Unicode Normalization Explained and NFC vs NFD.

Treating every PDF extraction problem as UTF-8 corruption

PDF mapping and OCR problems need different tools.

Repairing signed or hashed values

The bytes will change.

How to prevent broken UTF-8 text

Input boundary

Receive bytes, determine or require encoding, validate strictly and decode once.

Internal application

Work with Unicode strings, avoid unnecessary encode/decode operations and do not store guessed encoding state implicitly.

Storage boundary

Configure full-Unicode database support, configure connections correctly and test imports and exports.

Output boundary

Encode once as UTF-8, send correct HTTP headers, declare HTML charset and document API encoding.

café
François
Русский
日本語
中文
العربية
😀
e + U+0301

Practical repair workflow

1. Preserve the original.
2. Determine whether the source is bytes or text.
3. Validate UTF-8.
4. Inspect suspicious sequences.
5. Identify likely mistaken encoding.
6. Test reversible transformations.
7. Rank candidates.
8. Verify language and meaning.
9. Detect double encoding.
10. Exclude irreversible cases.
11. Write repaired output separately.
12. Review and audit.
13. Deploy with rollback.
14. Prevent recurrence.

Use Mojibake Repair, UTF-8 Validator, Character Encoding Detector, Unicode Character Inspector and Unicode Text Compare.

Try these UnicodeNow tools

Use these tools to validate bytes, test candidate repairs and compare before/after text.

Frequently asked questions

References

• The Unicode Standard
• Unicode glossary
• RFC 3629: UTF-8
• WHATWG Encoding Standard
• Unicode Character Database
• HTML character encoding guidance
• MDN: TextEncoder
• MDN: TextDecoder
• Python Unicode HOWTO
• Python codec documentation
• PHP multibyte string documentation