Secondary teacher here.
Trying to remove phones from the classroom entirely is just reinforcing that phones are for entertainment. It is a tool and needs to be handled like a tool: we should be teaching responsible use, and limiting it from those who have proved they can't be responsible on a per-case basis.
When students use their phones responsibly, they can be powerful learning aids. And I have zero issues calling out individual students who refuse to use them responsibly and treating their actions like any other misdemeanor.