The G11 (successful project that it was lol) had square ammo and its performance issues don't seem to have come from square ammo.

Notes on other people in this thread:

• The casing during firing acts as a seal for the chamber, which can very safely contain the detonation of the round (which is more of a slow burn, no worries of material damaging unpredictable shockwaves).
• Alignment might be an issue, but it would be coming out of a magazine in a predictable manner. There aren't really mechanical forces imparting spin on the round until it is fired.

Idle thoughts:

• In order to have the same structural integrity, the case would have to be thicker, though not much more so given a round has plenty of strength for supporting itself in a magazine.
• It might reduce friction in a magazine, allowing magazines to be longer. Stacked round bullets impart quite a lot of the magazine spring's force off into the side of the magazine.
• It might allow a greater volume of the magazine to be propellant, allowing for either a shorter magazine for the same ammo count or a larger magazine.
• It would be more difficult to manufacture. Circles are where its at for manufacturing.
• It would be specialist ammunition for one make of firearm. Not insurmountable, but does make it cost a lot for not much benefit.

I propose hexagonal telescoping casings for double stacking tiny magazines.