give the sandboxed website access to one specific USB device
That is exactly how it works. You get a prompt like this you have to chose a device, and only then does a website get access to it via WebUSB.
,
(From the documentation: https://developer.chrome.com/docs/capabilities/usb)
You can also disable WebUSB permission requests globally in Chromium: 