Perhaps you’re expecting all colonies to be plantations? The British plantations such as North America, Australia and NZ are still as you’d expect. But most of the empire was run for profit rather than plantation. These colonies were administered by British (later a mix of British and indigenous) civil servants and garrisons but there was no intention to build a lasting presence. The British Empire even told itself it would hand back the non-plantations after they had been “set right” for the benefit of the natives.