A recurring feature in most sci-fi universes is the 'advanced' weapon technology that is actually worse at killing unarmed people than a conventional machine gun. What is often overlooked, however, is the advances in armor technology.
There is an often recurring usenet thread about 'how practical would it be to build a giant robot'. The consensus seems to be 'completely impractical - you'd need to make it out of some magic substance that is simultaneously incredibly light, incredibly elastic, and incredibly rigid, otherwise it would shake itself apart if it tried to walk'. So assume you're faced with a battlemech built out of this exact substance. Today's weapon technology wouldn't even scratch it. Battletech's insanely short ranged weaponry can be (sort of) explained by this too - it's really the maximum range where there's a hope of penetrating the armour.
Warhammer 40000's power armour seems a bit pathetic - it can keep the wearer alive for roughly ten seconds of fire from a boltgun. Until you realise that the boltgun is a fully-automatic Rocket-Propelled-Grenade launcher - Think of The Gun from Terminator 2, but firing caseless ammunition at 72 rounds per minute. Just one of them killed the T1000. Do you seriously expect an AK47 to affect a space marine?
Star trek is a bit of an oddity - they carry these incredibly over-powered, slow weapons, but the people they fire them at are essentially unarmoured. Why not use a lighter, rapid-fire weapon? Think force-field technology. A phaser/disruptor is the only weapon powerful enough to shoot through a force field (in The Ensigns of Command (TNG), Data uses a hand phaser to vapourise a whole entire aqueduct). Projectile weaponry wouldn't stand a chance. You don't see personal force-fields in regular use, because phasers make them useless, however you don't see assault rifles for exactly the same reasons. If you were to equip your squad with projectile weapons, they'd have a considerable advantage. For the first encounter. After that they'd be screwed.
The above arguments hold for energy weapons in Star Wars as well. As for Force powers, bear in mind that the energy imparted to a supersonic bullet (Ek = 1/2 m (v2), remember) is around 3.5 kJ*. This pales in comparison to (for instance) the 196 kJ required to lift a 20 tonne X-wing 1 meter into the air. Don't be so cocky about the Jedi.
* - http://www.mpscompany.com/tests.htm - 7.62mm