I don't think debit cards or their associated bank accounts affect your credit in any way since those transactions don't go through any of the credit agencies.

When I was just starting out in life, I had a credit card but only used it maybe once a year. Just with that I somehow had a credit score in the upper 700s.