Describe what you would like us to do:

Hi, I am not sure if it is just me, but it appears that Fedora (maybe just FMN) related emails are massively delayed. E.g. this build failed on 06:24:04 UTC, but I have received notification on 10:47 UTC. This is part of the header:

Precedence: Bulk
Auto-Submitted: auto-generated
From: notifications@fedoraproject.org
To: vondruch@redhat.com
Subject: rubygem-activestorage's builds started to fail in Fedora rawhide
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Message-Id: <20210312104747.88C0930E196A@bastion01.iad2.fedoraproject.org>
Date: Fri, 12 Mar 2021 10:47:47 +0000 (GMT)

When do you need this to be done by? (YYYY/MM/DD)

kevin commented 4 years ago

It's just FMN. It got backloged recently by:

having to be restarted because it was leaking memory. It takes it ~1hour to restart.
There's been a lot of messages lately. I am not sure sure from what.

It's slowly catching up... and I am not sure there's much more we can do.

pingou commented 4 years ago

@vondruch do you still see this?

I'm not sure there is much we can do here :(

Metadata Update from @smooge:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: low-gain, low-trouble, ops

4 years ago

vondruch commented 4 years ago

There is still/again ~1h delay

http://koji.fedoraproject.org/koji/buildinfo?buildID=1723968

Return-Path: notifications@fedoraproject.org
Received: from zmta04.collab.prod.int.phx2.redhat.com (LHLO
 zmta04.collab.prod.int.phx2.redhat.com) (10.5.81.11) by
 zmail26.collab.prod.int.phx2.redhat.com with LMTP; Tue, 16 Mar 2021
 11:05:10 -0400 (EDT)
Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23])
    by zmta04.collab.prod.int.phx2.redhat.com (Postfix) with ESMTP id A70C8DA25F
    for <vondruch@mail.corp.redhat.com>; Tue, 16 Mar 2021 11:05:10 -0400 (EDT)
Received: by smtp.corp.redhat.com (Postfix)
    id A13A019D7D; Tue, 16 Mar 2021 15:05:10 +0000 (UTC)
Delivered-To: vondruch@redhat.com
Received: from mx1.redhat.com (ext-mx20.extmail.prod.ext.phx2.redhat.com [10.5.110.49])
    by smtp.corp.redhat.com (Postfix) with ESMTPS id 9871519D61
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 15:05:10 +0000 (UTC)
Received: from bastion.fedoraproject.org (bastion01.iad2.fedoraproject.org [10.3.163.31])
    (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits))
    (No client certificate requested)
    by mx1.redhat.com (Postfix) with ESMTPS id F3F6E30C3B7D
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 15:05:01 +0000 (UTC)
Received: from notifs-backend01.iad2.fedoraproject.org (notifs-backend01.iad2.fedoraproject.org [10.3.163.63])
    (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
    (Client did not present a certificate)
    by bastion01.iad2.fedoraproject.org (Postfix) with ESMTPS id A9029304C5F7
    for <vondruch@redhat.com>; Tue, 16 Mar 2021 14:59:47 +0000 (GMT)
DKIM-Filter: OpenDKIM Filter v2.11.0 bastion01.iad2.fedoraproject.org A9029304C5F7
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fedoraproject.org;
    s=bastion-iad; t=1615906787;
    bh=Q4x+tokMqG870GUnWlfqOjdFKqY+9IpPtBUyupzRyD4=;
    h=From:To:Subject:Date:From;
    b=K9FWkCqBuVQBK7tlHH1HXgWpUWoNzjh0U5jXD0LQ05S7i6bo35mhURKD5hiXwbvFd
     1jUBYD1WgpyZN8sAfCIgd19tw8WaFSL4EaQ9fwJK9HpgFrJ9/FkdxCqm4N2HPvY8wz
     x5ykQ0ufLtZCMDKE6L/u/bWH0mKOR9HwHm1BNvnI=
Precedence: Bulk
Auto-Submitted: auto-generated
From: notifications@fedoraproject.org
To: vondruch@redhat.com
X-Fedmsg-Topic: org.fedoraproject.prod.buildsys.build.state.change
X-Fedmsg-Category: buildsys
X-Fedmsg-Username: vondruch
X-Fedmsg-Package: rubygem-builder
X-Fedmsg-Num-Packages: 1
Subject: vondruch's rubygem-builder-3.2.4-4.fc34 completed
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Message-Id: <20210316145947.A9029304C5F7@bastion01.iad2.fedoraproject.org>
Date: Tue, 16 Mar 2021 14:59:47 +0000 (GMT)
X-Greylist: Sender DNS name whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 16 Mar 2021 15:05:02 +0000 (UTC)
X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.49]); Tue, 16 Mar 2021 15:05:02 +0000 (UTC) for IP:'10.3.163.31' DOMAIN:'bastion01.iad2.fedoraproject.org' HELO:'bastion.fedoraproject.org' FROM:'notifications@fedoraproject.org' RCPT:''
X-RedHat-Spam-Score: -1.35  (ALL_TRUSTED,DKIMWL_WL_HIGH,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU) 10.3.163.31 bastion01.iad2.fedoraproject.org 10.3.163.31 bastion01.iad2.fedoraproject.org <notifications@fedoraproject.org>
X-Scanned-By: MIMEDefang 2.84 on 10.5.110.49
X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23

kevin commented 3 years ago

So, sadly there's nothing we can do here until we re-write notifs.

At times when there's a flood of messages (like when we tag beta packages or the like) it just gets behind because the way it's implemented. It has to look at each message, then check each list of users to see if they want to be notified on it and this is really slow. ;(

This is definitely something we want to fix in the new version... but not much we can do right now. ;(

Metadata Update from @kevin:
- Issue close_status updated to: Will Not/Can Not fix
- Issue status updated to: Closed (was: Open)

3 years ago

Metadata

Assignee

None

Tags

Blocking

None

Depending on

None

Priority

Waiting on Assignee

Boards 1

ops Status: Done

fedora-infrastructure

Source Code

#9726 Long email delivery times Closed: Will Not/Can Not fix 3 years ago by kevin. Opened 4 years ago by vondruch.

Describe what you would like us to do:

When do you need this to be done by? (YYYY/MM/DD)

Metadata

ops low-trouble low-gain

Boards 1

#9726 Long email delivery times

Closed: Will Not/Can Not fix 3 years ago by kevin. Opened 4 years ago by vondruch.