Also wasn’t the region under Herod at the time, not formally part of the Roman empire?

Herod was the Rome-installed representative, which was a large part of why he was so resented at the time. You're probably getting a bit confused because Herod was from a Hebrew royal dynasty, but his successful ascent to the throne (after bloody internal power struggles in the region, and even within the same Herodic royal family) was a direct product of Rome aiding him and installing him militarily.

Augustus wouldn’t decree a census in a client kingdom, he would have herod make the decree wouldn’t he

I mean, outside of the bible, Josephus also records this same census happening during the same time frame. "Seems like wouldn't have" doesn't really hold up against historical contemporaries saying "did".

A census is meant to document how many people live where. Everyone going to their ancestral town doesn’t make sense

Doesn't make sense, to you, in a modern context, based on your present-day assumptions of how social structures are organized and what censuses are intended to measure.

Take, for example, taxation: in modern contexts, we'd look at the number of people living in various regions and apply their taxation rules accordingly, because variations in tax rules are primarily segmented by region (i.e. this state has different taxes than that one, this county and city have their own tax rules, etc).

In Hebrew contexts, taxation rules (particularly temple taxes) differed based on which tribe you belonged to (i.e. the Levite priests had different taxation rules than other tribes, for example). Herod, though he was a bit of a puppet installed by Rome, was himself still ethnically Hebrew and understood how those tax rules worked (as a layer on top of Roman tax codes), and so it logically follows for him to do what historical accounts record that he actually did: gather everyone in geographic locations that correspond to their tribal identities, and then count whoever's there, then apply the per-tribe rules and do the math.

To take another example, in the rest of Hebrew history, the most common reason for doing censuses wasn't to determine how many people live in a spot; it's to determine how many fighting men (and chariots, and horses, etc) your military was, and there are numerous accounts throughout the bible of those counts being segmented by tribal identity (like most of the book of Numbers, and even the book of Revelation at least references this segmentation because it was such a baseline expectation, though there's dispute about how literally to interpret its use there).

Basically, every time you have a Hebrew ruler going to do a census, they count the people by tribe. So again, a quick way to do that is to split everyone up geographically by tribal identity, then count the people in each spot. The ancestral-town thing is exactly in service of that repeatedly-demonstrated historical precedent.