It sounds like hes respecting consent, just being in your underwear doesn't automatically mean you want him all over you and hes respecting that possibility.
If I was playing a video game that I was super sucked into and focused on, and then a girl I was with still in her underwear sat next to me, I wouldn't assume it was an invitation to immediately begin sucking on her titties or something. I'd figure she just wanted to chill next to me in her undies.
If you want to have sex: communicate that. He probably would have been down.