Character Encoding Explained And Why Text Arrives Broken

Character encoding is the table that turns bytes into letters. A file holds bytes. It does not hold letters. Turning bytes into letters needs a table saying which number means which character. A file rarely says which table it was written with. Every garbled name, every question mark where an accent should be, comes from that one gap.

This turns up wherever text moves between programs. A CSV that opens wrong in Excel, a subject line that arrives as symbols, a name with an accent that survives one system and not the next. The bytes are almost always fine. The agreement about reading them is what broke.

The Character Encoding Table Nobody Sends With the File

Start with what a byte can hold. One byte gives 256 possible values. English needs about a hundred characters, so the early agreement, ASCII, used half the range and left the rest unused.

That spare half became the problem. Every region filled it differently.

Nothing inside a plain text file names its encoding. There is nowhere to put one. A program opening it looks for a declaration somewhere outside the file. Failing that it guesses. Failing that it falls back to a regional default set years ago on a machine nobody remembers.

What Unicode Changed About Character Encoding

The Unicode Consortium broke the problem in two. Holding those halves apart clears up most of the confusion.

UTF-8 won the web on one property. Plain English under UTF-8 is byte for byte identical to ASCII, so decades of existing text became valid UTF-8 without anybody touching it.

the word  cafe, with an accent on the e

ASCII        63 61 66 65 ??     the accent has no byte
Windows-1252 63 61 66 E9        one byte, meaning fixed by region
UTF-8        63 61 66 C3 A9     two bytes, meaning fixed everywhere

read the UTF-8 bytes as Windows-1252
             c  a  f  A-tilde  (c)
             cafA(c)          the classic wrong result

Why Garbled Text Looks the Way It Does

Mojibake is not random. It follows the tables exactly, which is why the same wrong pairs appear over and over.

The last row is not an encoding problem at all. Boxes mean the text decoded correctly and the font has nothing to draw. Changing the encoding will not help. This distinction saves a great deal of wasted effort.

The diamond question mark is the one that cannot be undone. Every unreadable byte becomes that same character, so the original values are gone rather than misread. Everything else on this list is recoverable from the original file, which is the argument for never overwriting it.

Two Different Things Both Called Encoding

This is where the word does real damage, because the same term covers two problems that have nothing to do with each other.

UTF-8
Windows-1252
ISO 8859-1
UTF-16

decides which letter
  a byte stands for

base64
quoted-printable

decides how bytes cross
  a channel that only
  carries plain text

Mail keeps the two visible side by side. The transfer layer is why an attachment arrives a third larger than the file. The character layer is why a subject line in French sometimes arrives as symbols. MIME carries both declarations in the same header block.

Where a File Declares Which Character Encoding It Uses

Some formats declare it. Some cannot. Knowing which is which explains why certain files give trouble and others never do.

CSV is the format that causes the most of this. It has no header, no declaration and no specification worth the name. Every program opening one has to guess. Excel falling back to a regional default is behind most spreadsheet trouble people meet.

The Byte Order Mark And Why It Divides People

Since a file cannot declare its encoding, somebody proposed marking the front of it instead. That is the byte order mark.

It works. It also creates a second problem. Software that expects content at byte zero now finds three unexpected ones. The results range from an invisible character at the start of a heading to a script that refuses to run.

Reinterpreting Versus Converting And Why It Matters

Every editor that handles encodings offers two commands that sound alike. Choosing the wrong one is how a recoverable file becomes an unrecoverable one.

bytes  unchanged
table  changed

read the same bytes
  against a different
  table

bytes  rewritten
table  changed

write new bytes for
  the characters
  currently on screen

This is how files get genuinely damaged. Somebody opens a file that looks wrong and saves it as UTF-8 hoping that helps. The editor faithfully writes out the characters it was showing. The original byte values are gone. No later effort recovers them, which is the real reason for working on a copy.

When the Encoding Has Been Wrong Twice

Sometimes the garbling has been through the mill more than once. That shows.

correct        Federation, with an accent
once wrong     FA(c)deration
twice wrong    FAA(c)A(c)deration

Each pass takes the wrong characters and encodes those, so the damage compounds and the string grows every time. It is still reversible. It needs the same number of passes back that it took going forward.

Long runs of accented capitals are the signature. One or two odd characters where an accent belongs is a single misreading. A cluster of five or six is double encoding. It tells you the file passed through two systems that each guessed differently.

Working Out Which Character Encoding a File Uses

1. Ask the editor what it thinks. Notepad++ shows the encoding in the status bar and VS Code in the bottom right corner. On Mac or Linux the file command reports it. That is a guess rather than a reading. It is still a better starting point than your own.
2. Look at where the text goes wrong. Ordinary English fine and accents broken points at a one byte encoding read as UTF-8 or the reverse. Everything broken points at something more serious.
3. Try UTF-8 first. It is the likeliest answer by a wide margin. Text coming out clean means you are finished.
4. Then try the regional default. Windows-1252 for western European text. The matching code page for anything else. Between those two you have covered most files.
5. Check the first three bytes. EF BB BF means a UTF-8 mark is present, which both tells you the encoding and explains a stray character at the start.
6. Keep the original. Work on a copy every time. Once a program has written the diamond question mark into a file, no amount of later effort brings the characters back.

Automatic detection is a guess and the standard says so. It works by statistics rather than by reading a declaration, so it is right most of the time and confidently wrong the rest. Where a declaration exists, believe it over any detector.

Where the file in question is a mailbox rather than a spreadsheet, the encoding is declared per part and a reader that honours those declarations shows the text as intended. Our file viewers do that in a browser tab. The converters carry the declarations through rather than flattening everything to one guess.