Counter point: literally the next two verses
31 “It was also said, ‘Whoever divorces his wife, let him give her a certificate of divorce.’
32 But I say to you that anyone who divorces his wife, except on the ground of unchastity, causes her to commit adultery; and whoever marries a divorced woman commits adultery.
Not to mention that "adultery in his heart" is essentially thought-crime, which I personally find rather unchill and not based.