In part it's prestige, which for some might matter for promotion purposes, and at least personally I'm more like to cite journals for which I know I trust their judgement in peer review and submission acceptance. There are predatory publishers which abuse the open access concept to make money, and if I'm reviewing literature I don't want to have to also research if a journal can be trusted (unless of course the publication I want to include is novel or especially worthwhile).

Also, in many contexts open access requires payment by the authors; this may be fine if an author is in a large grant-funded lab or at an institution willing to fund the open access fee but for many of us non-research-track folks it's kind of a deal breaker.