The reason behind building around the farm seems to be something quite practical:

“The myth was somewhat debunked last year when a recently unearthed documentary revealed that a geological fault, rather than an awkward farmer, was the real reason for Stott Hall, which lies west of Huddersfield, in West Yorkshire, being left in that peculiar location.”