While I do respect that viewpoint, there's a lot more independent scrutiny of the hardware modules than there are around the parts that would handle any other authentication mechanism you might use.

Pixel phone example iPhone example

Just because something isn't perfect doesn't mean we should keep using the less good thing that it replaces.

Use the PIN if that's more your cup of tea, just so long as you move away from passwords, since it's the HSM that's the protection, not the biometrics. Those are just to make it easier than passwords.