Seems kinda weird, and potentially a lot of people will find themselves in a similar position to you, and get weirdly turned on when they smell something baking.
My sympathies for your gym session, not sure what the answer would be in that situation!