Yeah, you're right on that point: you're dividing people into behavioural groups using "English speaking world" as identity tag rather than a race.
So the prejudice you voiced was using "geographical area of birth defined by language spoken" to presume unrelated characteristics of people, rather race.
It was indeed incorrect and unfair of my part to accuse you of voicing prejudice by race when the prejudice you voiced was by "geographical area of birth".