Because it's not just going to say yes. It's going to say yes, and then present an unique key that browser made for themselves. Other browsers might be able to spoof the key, but the proposal might have cryptographically expensive to even try.