They arrest any skin colour coming from abroad without mountains of cash. A German woman who'd been going to the USA for hiking vacations several years in a row suddenly found herself in ICE detention for a month before being sent back to Germany because they figured she was going to the USA for work.
Basically if you're poor, you're ICE fodder. It just happens that brown people are far more likely to be poor.