So disclaimer, I don't drink and never have, so I don't have the same perspective. My mother does, though.
None of the others in my family drink and none of her friends drink like 20 yos. Still, she relatively regularly drinks way too much, even without anybody else with her. Other people not drinking as much does not stop her at all.
At least in her case, other people not drinking as much does not stop her at all, even if she intends to "only drink a little".
Also remember that your tolerance will be way way lower now. Whatever you do, be safe! (And I strongly recommend that you don't, fwiw).