If they don't make it worse with whatever class system they develop, 120 won't be fatal (at a population level). There's groups that exist with fewer, like Pitcairn islanders, who descend from a shipwreck.

It all comes down to how bad an individual person's bottleneck is. The Habsburgs managed to get it down to a single digit number of great-great-great-... grandparents before they went extinct, IIRC.