Maybe some balloon animal structures inside the balloon to give it a shape?
I started off being a bit silly, but this question is serious. If you had a magic material that was rigid and could maintain that inflated shape for the same weight as the current material they use, could you selectively change its buoyancy by vacuum? Don't submarines and fish with their swim bladders do the same thing? That would make it a materials problem correct?