File formats

How to Read a File Somebody Else Made

Removing something properly means rewriting. Marking it as gone costs nothing. Formats built decades apart made that trade for the same reason.

6 min read

Reading a file somebody else made comes down to four questions. A filename is a claim. What a program shows you is an interpretation. Underneath both, the bytes carry a record of what the file is, how it should be read and what has happened to it. Almost none of that appears on screen.

This page collects what the rest of the site has worked out about reading files somebody else made. Each section is a summary with the full argument a click away.

Question One, What Is the File Really

Extensions are the least reliable thing about files and the first thing everything checks.

Most binary formats begin with a short fixed run that names the format. Whichever program produced it put it there. Renaming leaves it untouched, so it counts as evidence where the label counts as an assertion. Those opening bytes settle the question when the name has stopped being reliable.

Two things that surprise people. Shared signatures narrow files to a family rather than naming them, so a Word document, a spreadsheet and an Outlook for Mac export all begin identically because all three are zip archives. And several formats carry no marker whatever, mail formats especially, so there is nothing for a program to match on and the extension gets used by default.

Question Two, How Should the File Be Read

Having established what a file is, three separate decisions remain about how to interpret the bytes. Files usually declare none of them.

  1. Which character does this byte mean Character encodingGet it wrong and accents arrive as symbols. The bytes are intact and the table used to read them was the wrong one
  2. Where does a line end Line endingsOne character or two, invisible either way. Mail requires both, which runs opposite to every other habit
  3. Where does a value end Delimiters and quotingA CSV declares neither. Every program opening one guesses. A value may legally contain a line break
Three unstated conventions in a plain text file. A format that looks like the simplest thing on the list is carrying four separate guesses, which is why CSV causes more trouble than formats ten times its complexity.

Some formats do declare these and it is worth knowing which. Mail carries a character set on every text part. HTML and XML say so near the top. JSON is defined as one encoding and needs no declaration. Plain text and CSV say nothing at all, which is the entire difference in how much trouble they cause.

Question Three, What Is the File's History

Files accumulate a history and most of it is readable.

The fileWhat it records about its own past
A PDFEvery earlier version, since saving appends rather than replaces
An imageVisible compression damage that accumulates and never reverses
A messageThree dates from three clocks, disagreeing by design
A scanned pageWhether the text is real or a machine's guess at the shapes
A signed documentWhat the signature covered, plus what arrived afterwards

Converting to a better format restores nothing. A damaged image saved as a lossless format keeps every artefact it had. A signature does not extend to cover pages added later. Formats make promises about what happens next and none about what already happened.

Question Four, What Nobody Meant to Send

This is the one with consequences. It has the same cause in every format.

Removing something properly means rewriting. Marking it as gone costs nothing. That trade was made independently, decades apart, in formats with nothing else in common, so the behaviour turns up everywhere.

What the interface shows

Current
the messages
  in the folder

the pages
  in the document

the visible
  part of the image

Accurate. It is a list rather than the content itself.

What the file holds

Everything not yet overwritten
messages marked
  deleted

earlier revisions
  and removed text

the whole image
  outside the crop

Also accurate. Nothing failed here and nothing was hidden on purpose.

Three formats, one behaviour. Deleting mail marks space reusable. A PDF appends and supersedes. Cropping an image sets a visible region and keeps the rest.

The practical version fits in one sentence. Before files leave, look at them unformatted and hunt for anything the screen does not show. That one habit catches redaction failures, forgotten metadata and crops that never cropped anything.

Read next Inheriting Files Nobody Can Explain What to do when it is not one file but a drive full of them.

The Four Checks in Order

  1. Read the first bytes. Hex editor for binary, plain text editor for anything that might be text. Treat the name as the last thing to trust rather than the first.
  2. Establish the reading conventions. Encoding, line endings, delimiters. Where the format declares them, believe the declaration over any detector.
  3. Look for accumulated history. Size against visible content is the quickest signal. Several megabytes for a handful of text pages says it holds something other than text.
  4. Copy the whole page out into plain text. Anything that lands there without being visible on screen is still inside the file.

None of this is specialist work. Four checks, a few minutes, software everybody already has. Establishing what happened to a file in any formal sense is a different job entirely. It also starts here.

Why All of This Rhymes

The posts behind this page were written separately about unrelated formats. The same few ideas kept arriving.

A name is a claim and the bytes are evidence. Displaying files means interpreting them. The interpretation can be wrong while the file is fine. Removal is usually marking. Conversion promises the future and not the past.

None of that was designed as a principle. Each is a reasonable engineering decision, made independently, arrived at repeatedly because the alternatives cost more. That is why a mail store from the nineties and a document format from the same decade behave alike when you look underneath. It is also why knowing one of them tells you something about the next.

Each claim here is set out with its sources in the linked article. This page summarises rather than establishes.

Questions People Ask

6 questions, answered in full below.

How do I tell what a file actually is?

Read the first few bytes rather than the name. Most formats start with a fixed run announcing themselves, written by whatever created the file and unaffected by anybody renaming it. Text formats have no such marker, which is itself informative.

Why does the same file look different in two programs?

Because showing a file requires interpreting it. Programs differ in how they do that. A message can carry two versions of itself. A PDF can be read in the wrong order. Text can be decoded against the wrong table. None of them is a fault.

Can a file contain things nobody meant to send?

Routinely. Earlier versions of a document. Metadata naming the author and the software. The part of an image outside a crop. Messages a mail store marked deleted and never removed. All of it travels with the file.

Is a wrong extension a sign of damage?

Almost never. Mislabelling leaves the bytes exactly as they were. Work out what the file really is. Give it a matching name and it opens. Repair applies only where the structure itself has broken.

What is the single most useful check?

Look at the raw form rather than the rendered one. A text editor handles text formats and a hex editor handles the rest. Most surprises sit in the first screen of the unformatted view and nowhere in the tidy one.

Why do these problems keep having the same shape?

Because erasing costs work and marking costs nothing. Independent designers reached the same conclusion, so mail stores, documents and images all behave alike once you look past the interface.

Sources

Where the figures and behaviour described above were checked.

  1. List of file signatures Wikipedia
  2. Character encoding Wikipedia
  3. Data remanence Wikipedia