I guess it's not a huge issue for controllers if you use them regularly, but the energy density on lithium-ion batteries is great, but the self-discharge isn't ideal. AA rechargeables are usually NiMH, and those do better on that front.
https://en.wikipedia.org/wiki/Self-discharge
Lithium-ion: 2–3% per month;[3] ca. 4% p.m.[5]
Low self-discharge NiMH: As low as 0.25% per month
I have lithium-ion 18650s in my high-power flashlight, but for devices that I leave around the house that don't really need a lot of energy density
think TV remotes or something
I think that it's probably sensible to use NiMH batteries.