If you are a running batter in baseball (where the term originated), then the ball being thrown from left field seems as if it came out of nowhere.
Damn, all these years and I never realized that's what it literally meant. I always knew what the expression meant figuratively but not its literal origin.