G suite sometimes rejecting email forwarded domains with overly (in my opinion) restrictive DMARC policies

[arnolddeleon622]

I just moved my personal domain into Sonic after over 20 years at Best/Verio.

I currently forward, via procmail, addresses in the domain to different mailboxes hosted in G suite (formerly Google Apps) and gmail. I can share recipe examples if that is helpful.

I've observed that Sonic, by default, sets the envelope sender of the forwarded emails to my sonic account. This seems like the right choice.

I know that domains like yahoo and paypal have pretty restrictive DMARC policies so I would not at all been surprised if forwarding broke for them. However the great mystery is that it appears to sometimes work and sometimes not and I can't figure out the pattern. I've created a test yahoo accounts and I can successfully send email to them to my domain that is forwarded through Sonic. I have examples of email sent by another person have made it through Sonic to my G suite account. I also have examples of email from the same person that were rejected because of the DMARC policy. I know this because the bounce goes to me (because Sonic set the envelope sender as me).

Does Google treat different Sonic servers differently and the behavior I'm seeing depends on which server I land on? Any other explanation for the behavior I'm seeing?

arnold

[virtualmike]

I also have a domain hosted at Sonic, and several family members have email addresses on the domain. A couple of them receive significant amounts of email, so they forward their mailboxes to regular Gmail mailboxes so they won't fill my account's disk quota. Every so often a message will bounce, in some cases by Google back to the original sender.

In each case that we've investigated, the sender was using a domain or a server known for spamming (even if the , or there were inconsistencies in the set of received headers that appeared to make Google suspicious.

They were able to resolve most of this by changing to having Gmail poll their mailbox on my domain, instead of forwarding messages there. That may not work for you, but it did appear to make Gmail happier.

[arnolddeleon622]

It's really bizarre that it sometimes works and sometimes doesn't. In case it is useful to someone else here are the source domains that I've seen this behavior from:

yahoo.com
paypal.com
facebookmail.com
email.classmates.com
e.dx.com

I've seen bounces for email to Google Apps (personal domain), Gmail (e.g @gmail.com) and Outlook/Hotmail (@hotmail.com) so I don't think it's receiver per se.

I inspected the messages that have been successfully delivered and they (obviously) passed the DMARC policies of the domain (notably DKIM). I checked the DMARC policies of those domains and most of them say reject 100% so the ones getting through are not random from the receiver's perspective.

I think if Google rejects it, Hotmail will reject it too. This makes me suspect that different Sonic servers are doing different things to email. Most likely there is a subset of Sonic servers are changing something in the messages that causes them to fail DKIM/DMARC/SPF on the receiving end.

[arnolddeleon622]

More digging, reveals a possible suspect. Spamassassin looks like it might be re-encoding some messages which explains sometimes working but not other times. I will have to investigate more.

[drew.phillips]

Do you have specific SMTP response codes and messages from the occasional emails that bounce back?

Historically (with companies other than Sonic), I've found email forwarding to be something that cannot be relied upon 100%. The problem gets a lot worse when forwarding mail to large providers like Yahoo and Gmail who's spam filters can become more or less aggressive depending on recent events.

Yahoo and Hotmail/Outlook.com have been particularly problematic in my past experiences. A few years ago I worked at a place that resold McAfee SaaS spam (inbound & outbound) filtering and occasionally our clients would report that messages they were trying to send would get bounced back. Turns out, Yahoo would occasionally block *some* of the McAfee SMTP servers but not others. Since Yahoo never worked with McAfee, our answer was often "keep resending until it goes through a mail server that isn't blacklisted" or try sending from another account temporarily. I've seen the same things happen with Hotmail and Gmail.

If a Sonic account gets compromised and used to send spam, this might make Gmail more prone to block a message forwarded from Sonic if the forwarded message happened to go through that server as well and had some content/keywords that raise any flags.

I also used to see tons of abuse tickets from Hotmail complaining about Sonic because people who were forwarding messages from Sonic to Hotmail would then mark those forwarded messages as spam in Hotmail. Hotmail would then erroneously think Sonic was the spam source even though we were just forwarding a message, potentially giving us a poor reputation for a period of time.

Another potential reason is SPF.

Sender domains with SPF active will usually fail once the message is forwarded. Because the account receiving the forwarded message will check SPF against the original sender and the message is now coming from the forwarded server, SPF failures can occur. When sending a message from Gmail, to my Sonic account, which forwards to my personal server, my system triggers the following rule: "1.5 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail)". DKIM will be fine. Couple this with other "gray" mail, the spam score could quickly go over the threshold thanks in part to SPF.

The volume and type of forwarded messages can also be a factor. If you receive some bona fide spam to your Sonic account that doesn't get classified as Graymail and then forward these on to someone like Google who may properly classify it, they'll think you're spamming and be more likely to block future messages.

So my 2 cents is that when forwarding to freemail providers you have to accept that issues like this could arise and prevent some messages from going through. You could try adding whitelist rules, but a lot of these rejections happen before whitelisting so you may have no control over it.

One workaround I can think of would be to create additional mailboxes with Sonic, and modify your procmail rules to forward messages from the primary Sonic account into these mailboxes (rather than forwarding externally). Then set up each external account to poll from these specific mailboxes. Gmail supports remote IMAP fetching so it should be just as quick as forwarding (if not quicker because it doesn't go through spam filters on Gmail's end).

Hope that helps. And feel free to post or PM me the bounce reasons if you'd like some feedback on that.

[kgc]

A much better solution that forwarding mail (for this, and other reasons) is to configure gmail to retrieve the messages from our server using POP3 or IMAP. This completely does away with any of the associated problems with forwarding and things like DMARC or SPF as well as preventing mail loops.

[arnolddeleon622]

Sigh, it's not spamassassin (at least in the obvious way). Turning it off didn't help (and may have triggered another problem).

Here an example of rejected transaction:

>>> DATA
<<< 550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
<<< 550-5.7.1 DMARC policy. Please contact the administrator of yahoo.com domain if
<<< 550-5.7.1 this was a legitimate mail. Please visit
<<< 550-5.7.1 https://support.google.com/mail/answer/2451690 to learn about the
<<< 550 5.7.1 DMARC initiative. f4si5338626plb.295 - gsmtp
554 5.0.0 Service unavailable
... while talking to mx1.hotmail.com.:
>>> DATA
<<< 550 5.7.0 (SNT004-MC7F22) Unfortunately, messages from (69.12.221.231) on behalf of (yahoo.com) could not be delivered due to domain owner policy restrictions.
554 5.0.0 Service unavailable

OTOH I can successfully send email from my old yahoo account and a test account a I created and here is example of what I believe is relevant header from G suite:

Authentication-Results: mx.google.com;
dkim=pass [email protected];
spf=pass (google.com: best guess record for domain of [email protected] designates 69.12.208.71 as permitted sender) smtp.mailfrom=[email protected];
dmarc=pass (p=REJECT dis=NONE) header.from=yahoo.com

Both dkim and spf pass.

I'm aware of workaround of delivering it locally to a mailbox and I would probably have gone straight there if this didn't work at all but the fact it works some of the time has been both annoyed and curious.

There are several downsides to locally mail delivery and using POP/IMAP to transfer to another account. Problem one is I would need to provide additional "tech support" to my parents, children and spouse in configuring their accounts to retrieve POP/IMAP email. Second issue is that I forward to the some email to multiple accounts offsite so that will require a dedicated account for each account I think. Fortunately Sonic is pretty generous with mail accounts so I could it make it work.

But I'm still wondering, is it possible that some messages are getting altered in a way that would break some DKIM policies some of the time?

As I noted in my original post I previously had a similar setup Best/Verio and was working correctly AFAIK.

arnold

[kgc]

Well, the "troubles" caused today are a good example of why you don't want to set things up this way, particularly using procmail. You could use domain aliases in the membertools to target multiple local or remote accounts. If you want to use procmail, I urge you to put loop detection in place. You can do it several different ways but I've always done something like this.

Code: Select all

#I can't remember if B looks at both the headers and body or just the body (you want both)
#shunt all mail that has the header to a special folder and do not continue
:0 B
* X-Loop: SONIC
$DEFAULT/looped-mail

#add header for loop detection
:0 fwh
| $FORMAIL -A "X-Loop: SONIC"

#rest of procmail rules below

[arnolddeleon622]

Loop detection, good idea. Should have had it. Now it's there.

Right now I'm stuck with a procmailrc as the only way I know of implementing my current email handling scheme. Well I could run on my host and mail system, not interested. I've spent too much time with MMDF, IDA-sendmail, qmail and postfix in my early life.

I'll describe how I handle email. I'm open to other implementation, maybe Sonic can do something that supports it.

For a given address in a my domain, let's say "[email protected]" I also have a prefix and in this example let's make "u".

[email protected] forwards to [email protected] and [email protected], this is intended for humans only. Commercial email should never go here. The original version actually went to a mailbox that got was accessed via POP and gmail and hotmail were just backups, before I was sure I could count on "the cloud" being up all the time. At some point the local computer copy went away, not it's just two cloud copies.

My initial prefix separator was "-", long before Google introduced "+". I'm pretty sure I stole this idea from somewhere but I can't recall anymore. Wouldn't surprise me if MMDF had something like this. So the following example would be a valid address:

[email protected]

u-.*@example.com (u- followed by anything) forwards to [email protected] and [email protected]. This means I can generate unique email address for every site. On the gmail account is easy to have filter that tags the email with "+u" in the Delivered-To: header. This makes really to find misclassified commercial email. If a prefixed address goes bad, it is easy to write a filter to drop it into dev null (For example The San Jose Mercury News for a long time couldn't keep its database secure and I kept having to change my address them). It's also amusing to see a mailing list provider get compromised (multiple email addresses all go bad the same time). Over the years I've added "_" as valid because humans sometimes can tell the difference between "-" and "_". I added "+" sometime after Google started using it and was considering switching to Google Apps (as it was called then) exclusively but too many address parsers still reject it. I think I added single tick as separator because of another human error. I added "." when I encountered a system that would not accept "-" as valid character.

Basically I want a virtual user table where I can use some form regular expressions (even a simplified one) on the left hand side and not just have a single wildcard entry. The hack today is create a catch-all and do it procmailrc. Anyway this is slight digression from the mystery of why some emails are accepted and others rejected Google and Hotmail. I still suspect some sort of transformation that is affecting signed portions of message. After looking the rejections more close I see this in some of messages that were rejected.

X-MIME-Autoconverted: from base64 to 8bit by b.spam.sonic.net id uB56hAaB023442

I suspect this is one of the sources of dkim/dmarc failures. I think yahoo, paypal, facebook, etc., have policies that work if the messages remain unaltered as specified by their policies. I would need to do some more research and check specific policies and content to be sure. Outside of Spamassassin does Sonic transform message content in anyway? The header I cite above seems to suggest so. Perhaps the two headed dog?

arnold

[drew.phillips]

Here an example of rejected transaction:

>>> DATA
<<< 550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
<<< 550-5.7.1 DMARC policy. Please contact the administrator of yahoo.com domain if
<<< 550-5.7.1 this was a legitimate mail. Please visit
<<< 550-5.7.1 https://support.google.com/mail/answer/2451690 to learn about the
<<< 550 5.7.1 DMARC initiative. f4si5338626plb.295 - gsmtp
554 5.0.0 Service unavailable
... while talking to mx1.hotmail.com.:
>>> DATA
<<< 550 5.7.0 (SNT004-MC7F22) Unfortunately, messages from (69.12.221.231) on behalf of (yahoo.com) could not be delivered due to domain owner policy restrictions.
554 5.0.0 Service unavailable

OTOH I can successfully send email from my old yahoo account and a test account a I created and here is example of what I believe is relevant header from G suite:

Authentication-Results: mx.google.com;
dkim=pass [email protected];
spf=pass (google.com: best guess record for domain of [email protected] designates 69.12.208.71 as permitted sender) smtp.mailfrom=[email protected];
dmarc=pass (p=REJECT dis=NONE) header.from=yahoo.com

Both dkim and spf pass.

550-5.7.1 is a relaying failure. Google and Yahoo (via DMARC) have agreed that Sonic is not permitted to relay on behalf of Yahoo without SMTP authentication.

The failure reason is essentially the same as SPF but on a more detailed level. The envelope sender (MAIL FROM) ([email protected]) passes SPF on Google because (1. we don't publish SPF records) and 2. this is a Sonic email server sending as a Sonic user.

DMARC is taking it a step deeper and looking at the "From" header and checking that against the sender domain (to prevent spoofing). This test fails because the "From" header in the email shows Yahoo but the relaying server is Sonic. Effectively this is DMARC preventing spoofing/relaying mail from one host (Sonic) when the message appears to come from another (Yahoo).

These are exactly the kind of problems that can arise from forwarding in 2016, and as you can see with DMARC will continue to get progressively more locked down and make forwarding an unviable option. SPF breaks plain message forwarding, DMARC can make it even more prone to failure.