For instance, would you carry an iPhone the size of a shoebox if the rest of the space was a battery? Would you even be willing to carry one of the 80s phones that had a satchel?
Needs to be convenient to hold up to your face for phone calls
Needs to be small enough to be holdable in one hand (highly individual)
Needs to be big enough that you can use the touchscreen effectively (highly individual)
So really you can just use the size of an average pocket and hand as a guide and go from there. Once it stops fitting in pockets and hands, it has to go in the bag, and typing becomes very difficult as you have to prop up the device, which means people won't use it as much and get annoyed with it.
If it's just battery life, you can get bags and such nowadays with convenient rechargable battery packs in them. You don't need to change the device at all. If you want more screen size as a bigger person, there's XL versions of phones. Other than that there's no reason to get a bigger phone