What is an MSG File And Why So Little Can Read One

An MSG is a small file system packed into a single file. Inside it are folders and files. Every one of those files is named after a number instead of after what it holds. Decoding those numbers needs a dictionary that belongs to Outlook, which is the whole reason the format travels so badly.

Drag a message out of Outlook onto the desktop and you get one. So does File then Save As. Drag a contact, an appointment or a task and you get one of those too, since the specification is about Outlook items rather than about email specifically. Feed any of them to a text editor and what comes back is noise. That noise has a shape. Once the shape makes sense, so does everything else the format does. Formats that announce themselves in readable text work quite differently, which the comparison of every mail format lays out side by side.

An MSG Is a File System Inside a File

The container is a compound file, sometimes called OLE structured storage. Microsoft built it so an application could keep many separate pieces of data in one file on disk, with something like folders and something like files inside.

The specification calls the folders storages and the files streams. An MSG uses both heavily.

MSG is not an email format so much as an Outlook item format. Microsoft describes the file as holding a Message object, which covers appointments, contacts, tasks and notes alongside mail. A calendar invitation saved this way is the same structure with different properties filled in, which is why a converter that assumes every MSG is an email will mishandle a folder of them.

How the Streams Get Their Names

A value that will not fit in the property stream gets a stream of its own. The name of that stream is built by taking the string __substg1.0_ and adding the property tag in hexadecimal.

__substg1.0_0037001F    the subject
__substg1.0_0C1F001F    the sender address
__substg1.0_3707001F    an attachment filename
__substg1.0_37010102    attachment contents

The last four hex digits are the data type. A value of 001F means a Unicode string and 0102 means binary, so the type is carried in the name alongside the identity. The first four identify which property it is.

Not every .msg file is an Outlook item. The MDaemon mail server uses the same extension for something structurally unrelated. Third party software has produced its own variants over the years too. The quick check is the first eight bytes. A compound file begins D0 CF 11 E0 A1 B1 1A E1. An MDaemon message will not.

Nothing in the file says what 0037 means. That mapping lives in a separate document, MS-OXPROPS, which lists every property Outlook and Exchange understand and gives 0x0037 the name PidTagSubject. Without it a reader can extract every stream perfectly and still have no idea which one is the subject line. The format is not encrypted or obfuscated. It simply assumes you have the dictionary.

MS-OXMSG is one of three documents. It describes the file layout. MS-OXPROPS names the properties. MS-OXCMSG defines what a Message object is in the first place. Anyone writing a reader needs all three, which is a fair part of why so few exist.

Where Recipients And Attachments Live

Every value in that property stream is a MAPI property, the same kind a PST holds by the thousand. The difference is scope. A PST is a MAPI store for a whole mailbox while an MSG is one item lifted out of it. Each recipient and each attachment is then a small object in its own right, with its own property stream, sitting in its own storage. That is what makes MSG so good at holding an Outlook message and so awkward at being anything else.

Attachment content sits in a stream whose name reflects what it is. Where the attachment is another email, that stream becomes a whole storage holding a nested message object, complete with its own recipients and its own attachments.

__attach_version1.0_#00000000
  __properties_version1.0
  __substg1.0_3707001F
  __substg1.0_37010102

__attach_version1.0_#00000000
  __properties_version1.0
  __substg1.0_3701000D
    __properties_version1.0
    __recip_version1.0_#00000000

Why an Attachment Can Be Anything at All

Compound files came out of Object Linking and Embedding. One consequence of that still shows up in every MSG. When Outlook stores an embedded object it hands the pointer over and stops writing, letting the other application persist whatever it likes into that storage.

Microsoft puts it plainly in its own account of the format. Outlook does not care what is in that storage. It is a custom data store. The application that wrote it could be Word, Excel, another mail client or something written last week.

This is why extraction is harder than it looks. Pulling an attachment out of an MSG can mean unpacking a compound file that was written by a program you have never heard of, using a layout only that program knows. The MSG specification stops at the boundary and says nothing about what lies past it.

The Two Kinds of MSG

An MSG is either Unicode or it is not. The file itself records which. A property called PidTagStoreSupportMask carries the flag. Where it is set, every string in the file is Unicode. If it is absent or unset, the strings are in the older form.

The older form has no idea which character set it used. It relies on the code page of the machine that wrote it, which is fine while the file stays there and unhelpful the moment it moves.

What Travels And What Does Not

MSG holds far more of an Outlook message than the formats built to carry mail between systems, which is exactly why so little of it survives a move.

Nothing in the lower half of that table is a failure of any converter. Those rows describe how Outlook wants to behave with a message rather than describing the message, so a format built to carry messages has nowhere to put them.

Where Even Outlook Has Stopped Understanding It

The awkward part of naming Outlook as the one program that reads MSG is that it is no longer true of every Outlook.

Outlook for Mac has never handled the format natively, since it was built on a different architecture and stores its own data as OLM. What changed more recently is that the new Outlook for Mac dropped even read only support. Microsoft has been steering the whole product toward EML as the format it shares single messages in.

The blank message loop is the giveaway. Double click an MSG on a Mac and Outlook opens a new empty email with the file attached to it. Nothing is broken. The client has no idea what the format is, so it treats the file as a document to send rather than a message to read.

The format still opens perfectly well. What is falling rather than rising is the number of programs willing to do it. The vendor that invented it is walking away first.

That matters most in eDiscovery work, where a collection of MSG files has to be readable years after the client that produced them stopped shipping. Formats that depend on one vendor staying interested are the ones that age badly.

Working With an MSG You Did Not Create

1. Do not judge the file by a text editor. Noise is the expected result, whether that is Notepad on Windows or TextEdit on a Mac. A compound file viewer shows the structure properly, at which point the stream names start to look like the list further up this page.
2. Check whether it is Unicode before converting. If accented characters matter, the encoding question decides whether they survive. It is answerable from the file itself.
3. Expect nested messages to need attention. A forwarded chain is MSG inside MSG. Confirm the conversion reached the bottom rather than stopping at the first level.
4. Accept that Outlook state does not travel. Categories and flags describe your workflow rather than the mail. If they matter, record them separately before the move.

Reading one without Outlook is a parsing job rather than an import, which is what our browser based viewers handle on your own machine. Where the contents need to end up in a format other software can use, that is a conversion job. The mail converters cover those routes.