It only needs infinite growth everywhere because everyone wants to retire and retirement is a massive drain on an aging society. If you want people to retire at an age where they're still capable of enjoying life, in a society with increasingly few workers compared to non-workers, no economic system works without continued growth.

You absolutely can have a small business that doesn't grow exponentially under capitalism. It just won't let you retire early on your own little yacht.