Does the UK think they have jurisdiction over my server just because some twat from the UK visits my site?
Yes. You are providing services to the UK users and deriving income from these services.
If so, there's a reason why we fought a war nearly 250 years ago
Get used to it. Apple get fined by the EU multiple times, Microsoft and Google also, from memory, lost some cases in the EU despite lack of physical presence.