Edit: Can no longer reproduce with same setup, issue seems fixed at googles side.
TL;DR: See title. How can I tell Google they're probably processing their mail wrong?
After setting up the Matrix Authentication Service (MAS) and exim-relay as mail server, I noticed verification mails sent from the service are often in the spam directory.
When digging deeper, I found out the mails are failing DKIM authentication. This was weird because DKIM is set up correctly, as verified by other mail providers and online DKIM test tools such as DMARC Tester.
Searching online for "gmail fails DKIM authentication, while other providers pass", I found regular reports, posts or similar without resolution, or unrelated resolutions such as DKIM alignment.
Using meld, I compared the original source of mails as received by gmail with those of other providers, and found a difference:
In other providers, the header for "From:" and "Reply-To:" fields are presented with double-quotes:
From: "John Smith" <j.smith@example.com>
Reply-To: "John Smith" <j.smith@example.com>
In gmail, where DKIM fails, there are no double-quotes:
From: John Smith <j.smith@example.com>
Reply-To: John Smith <j.smith@example.com>
As this should be the raw source each, I ruled out presentation issues and digged deeper.
I found out, that specifically the rust crate lettre, as used by the MAS, encodes names with whitespace using double-quotes. Further, from researching a bit more and reading RFC 2822 sections 3.2.4 and 3.2.5, I come to the conclusion that whitespace needs no quoting in mail headers.
I created issues upstream and downstream to report the issue at lettre and MAS, particularly that their mails are failing DKIM checks at gmail:
- https://github.com/lettre/lettre/issues/1125
- https://github.com/element-hq/matrix-authentication-service/issues/5497
If you've read that far, you probably wonder why I post all of that? For one, to provide another data point for people scratching their heads over mail issues.
But other than that: I'm pretty sure the google mail servers should not strip the quotes before doing the DKIM check. I assume they have some kind of decode -> process -> encode workflow, that then simply encodes the headers again, this time without the quotes. But IMHO a correctly signed message should not lead to an authentication error, even if the contents are not perfectly encoded.
I would be curious on getting some feedback from some mail experts on what is happening here. This is not my field of expertise and I'm going by what I've learned over the past 48h.
Hey, if you can figure out how to get Google to listen to you, please let them know they should have updated the DMARC policy for gmail.com to p=reject like 18 months ago, too. Because they're still using p=none.
Speaking of which, landing in spam doesn't mean much as it's post-processing after the mail has been accepted. What are the details of your tests so far - email you're sending from with your MAS, email you're receiving at? If you're using an @gmail.com address to send, it could be the DMARC policy at fault - having a policy of "none" is worse than not having DMARC configured at all, nowadays. Have you received any bounce messages?
And, have you tried a custom domain from somewhere like namecheap? If your mails through your own domain (with fully aligned DNS and a DMARC policy of "reject") still go to spam then the issue is likely in your config for MSA or exim. If not, the issue is likely Gmail's DMARC policy.
Ah interesting, I'm sending from my own domain and IP with DMARC set up to quarantine.
Yes, the mail server is accepting it because gmail accepts the mail if either SPF or DKIM passes (not AND). My observation is that google for some reason sometimes puts the mail into spam, and sometimes into the regular mailbox. In both cases the headers show dkim=fail and spf=pass and I have no idea why it's not deterministic. I've also tested this with same/similar mail contents.
Edit: To be honest, I also don't think that mail from my domain should be "sometimes" in the regular mailbox, if DKIM fails and DMARC has adkim=quarantine.
Ah-ha, I see, it's more clear now. Yeah, if it's an @gmail.com address, then the issue could well be how they're processing the email. Did you notice any other differences in the email received by Gmail vs others, e.g. does the DKIM signature match between them?
My thinking is that if they're dropping the quotes around the name, they might be mangling the DKIM signature too...
Yes, I've seen one other header change: gmail seems to (again sometimes?) enforce Message-ID fields in the header, and may add or change it if it doesn't match it's internal requirements. Interestingly, I've seen both my mailserver getting "rejected because missing message-id" messages, and messages passing to the mailbox, but with a google-added Message-ID in the raw source.
For my specific case of DKIM failures, I've not noticed other differences.
If I take the specific raw source from gmail, i.e. after processing by google, re-add the quotes, and manually check the DKIM signature, the signature passes. With other words, the quotes are literally the only relevant change in my case.
Yeah, sounds like you cracked it, frankly. This does make sense - the signature covers the entire message, including the original headers (which would include those quotes, in your case). If Google's processing is removing quotes that were originally there, the message changed, so DKIM would fail.
I think Google must only leave the quotes in if there are special characters like a comma in the name. Are you able to update your exim to only include quotes when the from name includes a special character? Feels bad having to engineer around Google's incompetence, I know, but that should solve the issue.
Edit: Or you could change your "from" name to "no @ reply" or "daemon @ box" or something, which would force Google to leave the quotes in.
I think it's technically an encoding bug in lettre, which is used in matrix authentication services: https://github.com/lettre/lettre . As far as I can tell, exim is relaying the messages "correctly" or at least without altering them.
I.e. lettre should not add quotes for whitespace. But also google shouldn't alter messages before authenticating. In an ideal world, both sides are fixed ^^
Lettre is following the spec, where if the display name contains a space, the modern way to encode it is to put quotes around it.
See here: https://gehirneimer.de/m/sysadmin@lemmy.world/t/918726/-/comment/8449841
Sorry, was editing while you were replying - I'll reply here in case you don't see my edits, sorry if you read them already. In light of the following:
I'm pretty sure that your setup is fine, including your exim config, and this is an issue specifically with Google's processing like you originally thought. Try making your "from" name for the service "daemon @ box", that will force Google to leave the quotes in. If the email passes dkim at Gmail like that, we have definitively proven your original theory correct.
Yes indeed! Changing the sender name to
Matrix Authentication Service @ <my_homeserver>works, with successful dkim: pass.From original source:
So sending mails with quoted names that do not have to be quoted leads to DKIM failures at gmail.
Nice idea!
I was doing some more reading, and some testing with a colleague, and I'm not sure it's strictly on Gmail's side. In our tests, our emails that sent with quotes arrived at Google without quotes, but also without issue - DKIM was passing.
Now, these tests were done with exim, and the commands sent via telnet, which isn't an exact match for MAS -> Lettre -> exim like you have. I'm coming back around to thinking you might be right that there's a DKIM signing bug somewhere in your stack. Maybe try reproducing this by sending the same email through your stack which would normally fail as you observed by failing DKIM, vs sending one through Lettre -> exim only, vs sending one through only Lettre directly, vs sending one through only EXIM directly?
I'm somewhat expecting the Lettre -> EXIM ones to fail, and maybe the ones through Lettre directly. Edit: And don't forget to update your github issue with any findings!
Interesting, thanks for reporting back.
I went to the config and changed my sender to
Matrix Authentication Systemagain, without @.It works, I have a DKIM pass now. As seen, the quotes are still stripped:
I did not touch my setup otherwise; I'm sure it failed the DKIM check before at gmail. Given the DMARC and SPF checks passed, I also don't think it was a DNS issue.
I'm sure it was not my setup, I manually copied the gmail-processed source and verified the DKIM cert manually, with and without re-adding the quotes. I was only able to verify locally with the quotes added. This, and DKIM consistently passing at other mail servers leads me to believe it actually was an issue at google side.
So either they fixed the issue in light-speed, or I was loosing my mind, which of course is possible as well. Will update the original post and github issues.
Welp, back to believing it was a error on their side, just a transient one. They tell us all the time that emails we've been sending to for years suddenly don't exist anymore; I remove the bounce block, and suddenly emails are delivered fine. So, this tracks.
Worst part is, this isn't something we can monitor for automatically to know for a fact whether or not it's just this time, or if this happens frequently. We just have to trust their giant machine doesn't mangle our mail.
Ayyyyy (๐๏พใฎ๏พ)๐ Nice instincts on ya there in the first place, bud - it was, after all, your idea haha!
Sadly, I don't have tips for getting anyone at Google to give a damn... but at least now we know how to compensate, so we've got that going for us, which is nice.
Will play around with that and report back :)