fedora-infrastructure

#9250 Mails from copr-fe-dev.aws.fedoraproject.org being marked as spam

Closed: Fixed 4 years ago by smooge. Opened 4 years ago by frostyx.

Describe what you would like us to do:

We had an issue with sending mails from copr-frontend instances, that was fixed as #9233. Mails are now successfully being sent from

copr-fe-dev.aws.fedoraproject.org
copr-fe.aws.fedoraproject.org

but they are being marked as spam. Is there something else that needs to be done for us being to able to send emails from copr-frontend instances?

I am attaching an email with all its headers, that came to my spam folder.

Received-SPF: neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) client-ip=35.153.70.58;
MIME-Version: 1.0
Subject: Testing mails from copr-fe-dev
From: root@copr-fe-dev.aws.fedoraproject.org
To: jkadlcik@redhat.com
Message-Id: <20200819105332.78DCD40704@copr-fe-dev.aws.fedoraproject.org>
Date: Wed, 19 Aug 2020 10:53:32 +0000 (UTC)
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Just testing mails from copr-fe-dev

When do you need this to be done by? (YYYY/MM/DD)

The sooner the better. We cannot properly notify users about important events.

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: groomed, medium-gain, medium-trouble, ops

4 years ago

praiskup commented 4 years ago

https://toolbox.googleapps.com/apps/checkmx/check?domain=copr-fe.aws.fedoraproject.org&dkim_selector=

smooge commented 4 years ago

@praiskup copr doesn't send mail directly it sends it through bastion and those are what deliver it to the world. As suck I believe the dkim for bastion.fedoraproject.org is what is to be used.

schlupov commented 4 years ago

SPF entry for bastion.fedoraproject.org is also not set because when I run dig bastion.fedoraproject.org txt +multiline +noall +answer it returns no answer. bastion.fedoraproject.org has only A record. Can you create an SPF record and DKIM record please? Should I open a new issue because of it?
Without it, we will constantly send users spam emails and we don't want that.
I also think that an SPF record for fedoraproject.org is incorrect, because when you run dig fedoraproject.org txt +multiline +noall +answer you can see the answer as
fedoraproject.org. 231 IN TXT "v=spf1 a a:mailers.fedoraproject.org ipv4:38.145.60.11 ipv4:38.145.60.12 ?all" but it should be just ip4 not ipv4. Validators also don't recognize the syntax (https://toolbox.googleapps.com/apps/checkmx/check?domain=fedoraproject.org&dkim_selector=)

smooge commented 4 years ago

@schlupov Thanks for the fix on the spf1. I don't know where I got that syntax on ipv4. I have fixed that part.

Bastion lost its spf record when we had 2 bastions in phx2 and iad2 and I didn't cleanly rename things.. that has been fixed.

I see a DKIM domain key in the zone for bastion. but no other records. NBot sure what else is supposed to be there.

schlupov commented 4 years ago

I tried to send an email to myself from the Copr dev instance and the email ended up in spam

Delivered-To: schlupov@gapps.redhat.com
Received: by 2002:a67:b305:0:0:0:0:0 with SMTP id a5csp305355vsm;
    Wed, 2 Sep 2020 05:49:03 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJxaZv/7RPiG2hBuK2yXuT8J7de2NXdbgW6UKmv1tL37xomCM+R1YFRdacDyiVGVrHkHRzpp
X-Received: by 2002:a25:5741:: with SMTP id l62mr9969248ybb.299.1599050943308;
    Wed, 02 Sep 2020 05:49:03 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1599050943; cv=none;
    d=google.com; s=arc-20160816;
    b=tXLqnoU7wXC8cDaK/5KCYDKpLsu0y1VZafZPqXLxTlCSJaPIzOA66ftCLmyib4pzG8
     3Las52F+9flwZtsJEUAW6ZHunp2bVDCVGj2PCJbrf1EhzMkSpCnNt6VgTANF2Hw6M3tf
     2uRWt0EHoGTYNpSsknMUrKWGoi8dztSqsdwFGSJhZErq72b+vd4FtSlvqXQXy3Njw8fk
     nt/CncMu7eLvf3I+QJxHoSxPJok2P8n+Cmz2R3mnbBU77SOYiK9txiG1/XnRFbrZWgrh
     onb2yhHK8UUYZHFXx2cCKqWDApdN1Aqk+v809l7vEn1ZA5JdWEQ098yZNA7CPyYnxJK6
     GFvw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
    h=content-transfer-encoding:date:message-id:to:from:subject
     :mime-version:dkim-filter:delivered-to;
    bh=6h8spjCSGvcuawNpSpJKQ11Rnr2tlchkBf/qBJmMyAc=;
    b=EcA1DtrHgJM0t7tidsOjzwKAvc6Zu9OS8IXX64a5SXy43fIWo6gCndR28hn/OyNjH+
     mN/hMrvZDCMp3bVHPaTk0ULkjll5GaCz+XdXmvmYI2IEm0mS2N69kuZdGaa+V7vvy1rL
     zyawLphqoN6XNc87thEPs/YLPPzaHt/kbrQJ+B4oNt1d1uGQnkfY2bWAYQcYPgaN7ct+
     yxqlec/41XWUQUyGy6U3mOd0x6taUy1VX0QM+iSuIZGGv/CnvZyUX98JU3EpjhCNFivR
     3IKtWtsD9N0mPizTW8MPxGFlOmdAaKg6/IL0AjkwGqdk+kFodoPSJmiavzkflujLPwnC
     mDoA==
ARC-Authentication-Results: i=1; mx.google.com;
   spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
Return-Path: <root@copr-fe-dev.aws.fedoraproject.org>
Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com. [205.139.110.120])
    by mx.google.com with ESMTPS id t186si4013313ybf.203.2020.09.02.05.49.03
    for <schlupov@gapps.redhat.com>
    (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
    Wed, 02 Sep 2020 05:49:03 -0700 (PDT)
Received-SPF: neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) client-ip=35.153.70.58;
Authentication-Results: mx.google.com;
   spf=neutral (google.com: 35.153.70.58 is neither permitted nor denied by best guess record for domain of root@copr-fe-dev.aws.fedoraproject.org) smtp.mailfrom=root@copr-fe-dev.aws.fedoraproject.org
    Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-318-wA6AV2BHMHO6qYKMUSqoYg-1; Wed, 02 Sep 2020 08:49:01 -0400
X-MC-Unique: wA6AV2BHMHO6qYKMUSqoYg-1
Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3651910082E8 for <schlupov@gapps.redhat.com>; Wed,
  2 Sep 2020 12:49:00 +0000 (UTC)
Received: by smtp.corp.redhat.com (Postfix) id 315AE9F63; Wed,
  2 Sep 2020 12:49:00 +0000 (UTC)
Delivered-To: schlupov@redhat.com
Received: from mx1.redhat.com (ext-mx16.extmail.prod.ext.phx2.redhat.com [10.5.110.45]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2B2E86198E for <schlupov@redhat.com>; Wed,
2 Sep 2020 12:49:00 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 853953082B1A for <schlupov@redhat.com>; Wed,
  2 Sep 2020 12:48:51 +0000 (UTC)
Received: from copr-fe-dev.aws.fedoraproject.org (ec2-35-153-70-58.compute-1.amazonaws.com [35.153.70.58]) by bastion01.iad2.fedoraproject.org (Postfix) with ESMTP id CA45630BDEA5 for <schlupov@redhat.com>; Wed,
  2 Sep 2020 12:48:45 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org CA45630BDEA5
Received: from ec2-35-153-70-58.compute-1.amazonaws.com (localhost [IPv6:::1]) by copr-fe-dev.aws.fedoraproject.org (Postfix) with ESMTP id BC9BB40728 for <schlupov@redhat.com>; Wed,
2 Sep 2020 12:48:45 +0000 (UTC)
MIME-Version: 1.0
Subject: Email from Copr
From: root@copr-fe-dev.aws.fedoraproject.org
To: schlupov@redhat.com
Message-Id: <20200902124845.BC9BB40728@copr-fe-dev.aws.fedoraproject.org>
Date: Wed,
  2 Sep 2020 12:48:45 +0000 (UTC)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16  (mx1.redhat.com [10.5.110.45]); Wed, 02 Sep 2020 12:48:51 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Wed, 02 Sep 2020 12:48:51 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'root@copr-fe-dev.aws.fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: 0.763
  (KHOP_HELO_FCRDNS,RDNS_DYNAMIC,SPF_HELO_NONE) 10.3.163.31  bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <root@copr-fe-dev.aws.fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.45
X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12
X-Mimecast-Spam-Score: 0.0
X-Mimecast-Originator: copr-fe-dev.aws.fedoraproject.org
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Testing email from Copr

SPF is neutral, shouldn't be copr-fe-dev.aws.fedoraproject.org permitted sender? You should add include:_spf.google.com (or ip4:_spf.google.com since we need to use ipv4) into SPF record. SPF must allow Google servers to send mail on behalf of your domain, it's the reason why you can see spf=neutral. Also, I would expect in header Authentication-Results
Authentication-Results: mx.google.com;
dkim=pass...
spf=pass ...

It looks like the DKIM signature is not added to emails, I can't see the DKIM-Signature header.

smooge commented 4 years ago

OK I have made some changes to our DNS.

aws.fedoraproject.org now has an SPF1 record
fedoraproject.org has a fixed dkim record.

Reading through the docs I do not see why it would be reasonable to put ina ip4:_spf.google.com because we do not use them as a sender (Red Hat does in its routing but that is outside of fedoraproject.org servers).

At this point I am going on 2+ weeks of PTO. I am removing myself from 'owning' the ticket as I don't want you to think I can fix anything until I get back. Hopefully someone else can help.

schlupov commented 4 years ago

@smooge thank you for your help :)
Unfortunately, this issue has still not been resolved. Emails still end up in the spam box. Any help will be very appreciated.

kevin commented 4 years ago

Is this still happening?

We may not be able to do anything about it... it may need redhat.com postmaster to look?

schlupov commented 4 years ago

Unfortunately not, please see https://pagure.io/fedora-infra/ansible/pull-request/260. This will not solve the SPF problem, just DKIM. I'll try the postmaster later if it doesn't work.

smooge commented 4 years ago

I am merging this as it corrects an operational mistake i made

schlupov commented 4 years ago

I am merging this as it corrects an operational mistake i made

Thank you

That lines:

# Milter configuration
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:127.0.0.1:8891
non_smtpd_milters = $smtpd_milters

have to be in main.cf on bastion not on Copr machines because we don't run opendkim service, it's my fault, I can fix that by another PR but I have no idea which of the configuration files in roles/base/files/postfix bastion uses. I also assumed that opendkim is listening on 127.0.0.1 and port 8891 on bastion.
When I check DKIM using dig:

dig TXT bastion._domainkey.fedoraproject.org
dig TXT bastion-iad._domainkey.fedoraproject.org

it returns the same public keys, but the file roles/opendkim/files/KeyTable shows that bastion and bastion.iad use different private keys, so I would expect different public keys. Opendkim service has not been restarted or it may take some time for this to change in the DNS record.

smooge commented 4 years ago

I merged it. I need a FBR to run the scripts so it is not updated anywhere.

smooge commented 4 years ago

Things have been restarted.

@schlupov They actually do use the same private key, just with different names. The reason for this was that originally I thought I would need to make new keys and then found that the only problem was that when we had 2 different sets of bastions a different name set was needed. Now that we are onto only one sets of bastions again.. changing the names as Kevin mentioned makes the most sense.

I will put that in place after freeze time though.

kevin commented 4 years ago

Whats the status here? Still happening?

I wonder if we couldn't look at masquerading these messages? ie, you send them to bastion and we rewrite them to be as if they are from 'copr@fedoraproject.org' or something?

schlupov commented 4 years ago

No progress, the last change didn't help as expected due to my mistake I mentioned in my last comment. I don't know which of the configuration files in roles/base/files/postfix bastion uses. I need to know that to move my code to the right config file.
I don't think masquerading messages would help in any way, the bastion doesn't add a dkim signature even though it has a dkim signature, but that's not enough. The signed header must be added to the email using the code currently in the copr configuration file.

smooge commented 4 years ago

So the changes you wanted for roles/base/files/postfix are already in the config, just buried in other areas or defaults. I am going to see if the one line that was not is added but need to see if it breaks other sending.

schlupov commented 4 years ago

Hurray! Thank you. It looks like it works. I sent an email to myself to a private email and I see the SPF pass and dkim signature in the email. The email came to the inbox, not to spam. When I send an email to red hat email I don't see dkim signature but SPF is pass.

@praiskup @frostyx please try it too (dev or production).

smooge commented 4 years ago

OK so I think the email to internal is stripping the dkim somewhere because it is supposed to be there. I am going to close this as fixed.

Metadata Update from @smooge:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

4 years ago

Metadata

Assignee

None

Tags

Blocking

None

Depending on

None

Priority

Waiting on Assignee

Boards 1

ops Status: Done

Attachments 1

Screenshot_20200825_085441.png

Attached 4 years ago View Comment

Related Pull Requests

#260 Merged 4 years ago

fedora-infrastructure

Source Code

#9250 Mails from copr-fe-dev.aws.fedoraproject.org being marked as spam Closed: Fixed 4 years ago by smooge. Opened 4 years ago by frostyx.

Describe what you would like us to do:

When do you need this to be done by? (YYYY/MM/DD)

Metadata

medium-gain medium-trouble ops

Boards 1

Attachments 1

Related Pull Requests

#9250 Mails from copr-fe-dev.aws.fedoraproject.org being marked as spam

Closed: Fixed 4 years ago by smooge. Opened 4 years ago by frostyx.