This is pretty much The Other World's Books Depend on the Bean Counter, where the protagonist is an overworked office worker who accidentally gets summoned to another world. He has no special powers, and actually gets sick from magic because his body isn’t immune to it. The treatment for this is somehow gay sex with his aristocrat lover, which as far as I know is one of the first just straight up gay sex scenes in anime shown on screen. Overtime he accrues political power by basically single handedly auditing the country’s treasury.