Why is "Dear X" considered more formal than "To X" in e-mail/writing?

You wouldn't start off an e-mail with "My Dear X", or "Dearest X", since that would be too personal for a professional email, so "To X" being more impersonal seems like it would make the letter more professional-sounding, compared to "Dear X".