Robots.txt has been always ignored by some bots, it's just a guideline originally meant to prevent excessive bandwidth usage by search indexing bots and is entirely voluntary.

Archive.org bot for example has completely ignored it since 2017.