I have not been able to find an authoritative origin for this term. However, it is widely used in the American military, going at least back to the Vietnam War. One possibility is that it refers to the fucking which is perpetrated upon the target of a cluster: a large group of fragmentation or incendiary bombs dropped at once. It may also refer to an alternate definition of "cluster," involving groups of mines deployed in a single area to be effective against both tanks and humans.

Evidently, the phrase was also used by Marines during the Vietnam War to refer to a cluster of people large enough to get severely screwed -- that is, a group of marines large enough to draw enemy fire, or to all be injured by a single explosive.

"Clusterfuck" seems to have had a more literal meaning at one point; in the 1960s a mongolian clusterfuck was a slang term for an orgy.
"cluster" is in common military use. My information on the "mongolian" variant comes from the Jargon File entry Mongolian hordes technique. The Marine variant comes from a glossary at http://www.capmarine.com/cap/glossary/glossary.htm.