Elminster doesn’t end up as a companion. He is in the mountain pass before act 2 and you need to have recruited Gale. You probably took the underdark route in which he doesn’t show up.