I asked Google Bard whether it thought Web Environment Integrity was a good or bad idea. Surprisingly, not only did it respond that it was a bad idea, it even went on to urge Google to drop the proposal.
I believe your "They use attention mechanisms to figure out which parts of the text are important" is just a restatement of my "break it into contextual chunks", no?