fedora-infrastructure

#12651 OOM on mailman3
Closed: Fixed with Explanation by zlopez. Opened by zlopez.

Closed: Fixed with Explanation
Reopen Issue

Describe what you would like us to do:

After the DC move the mailman is being killed after some time because it eats all the memory.
The systemd service in question is mailman3.

When do you need this to be done by? (YYYY/MM/DD)

Not urgent, as the service is automatically restarted

Looks like the task runner is consuming a lot of memory.

At some point I remember that the database was full of useless stuff like bounces, maybe that's the case again?

mailman  2150003  0.0  0.2  80516 74380 ?        Ss   07:44   0:00 /usr/bin/python3 /usr/libexec/mailman3/master --force -C /etc/mailman.cfg
mailman  2150005  0.2  0.3 474000 103044 ?       Sl   07:44   0:05  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=archive:0:1
mailman  2150006  0.6  0.2  92396 85896 ?        S    07:44   0:15  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=bounces:0:1
mailman  2150007  0.0  0.2  83392 76656 ?        S    07:44   0:00  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=command:0:1
mailman  2150008  0.1  0.2  90728 84804 ?        S    07:44   0:03  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=in:0:1
mailman  2150009  0.1  0.2 156976 81664 ?        Sl   07:44   0:02  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=lmtp:0:1
mailman  2150010  0.0  0.2  80516 74480 ?        S    07:44   0:00  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=nntp:0:1
mailman  2150011  0.0  0.3 106300 98636 ?        S    07:44   0:01  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=out:0:1
mailman  2150012  0.0  0.3 108296 100756 ?       S    07:44   0:01  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=pipeline:0:1
mailman  2150013  0.0  0.2  89076 82500 ?        S    07:44   0:00  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1
mailman  2150031  0.0  0.3 113408 100876 ?       S    07:44   0:00  |   \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1
mailman  2150032  0.0  0.3 114696 102156 ?       S    07:44   0:01  |   \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1
mailman  2150014  0.0  0.2  80516 74344 ?        S    07:44   0:00  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=retry:0:1
mailman  2150015 51.0 63.0 20568616 20558348 ?   R    07:44  20:46  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=task:0:1
mailman  2150016  0.0  0.3 105444 98404 ?        S    07:44   0:01  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=virgin:0:1
mailman  2150017  0.0  0.2  84156 77804 ?        S    07:44   0:00  \_ /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=digest:0:1

There's no reason why the mailman DB would be 62GB...

postgres=# \l+
                                                                                       List of databases 
         Name         |          Owner          | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |          Access privileges          |   Size   | Tablespace 
----------------------+-------------------------+----------+-----------------+-------------+-------------+------------+-----------+-------------------------------------+----------+------------
[...]
 mailman              | mailmanadmin            | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           |                                     | 62 GB    | pg_default 
[...]
(39 rows)

And indeed, it's the bounces:

mailman=# select
  table_name,
  pg_size_pretty(pg_total_relation_size(quote_ident(table_name))),
  pg_total_relation_size(quote_ident(table_name))
from information_schema.tables
where table_schema = 'public'
order by 3 desc;
     table_name     | pg_size_pretty | pg_total_relation_size 
--------------------+----------------+------------------------
 bounceevent        | 55 GB          |            58562666496
 pendedkeyvalue     | 3476 MB        |             3645120512
[...]
Edited by abompard

I will clean it, thanks for pointing it out.

So I removed all the processed bounceevent from database

mailman=*# delete from bounceevent where processed=true;                                                               
DELETE 160394065

And the size seems to be more manageable now

mailman=# select                           
  table_name,
  pg_size_pretty(pg_total_relation_size(quote_ident(table_name))),
  pg_total_relation_size(quote_ident(table_name))
from information_schema.tables
where table_schema = 'public'
order by 3 desc;
     table_name     | pg_size_pretty | pg_total_relation_size 
--------------------+----------------+------------------------
 bounceevent        | 4557 MB        |             4778762240
 pendedkeyvalue     | 3476 MB        |             3645251584
[...]

Will watch if there will be another OOM incident. But I think we are good for now.

It seems that the pendedkeyvalue table has a lot of entries as well.

mailman=# select count(*) from pendedkeyvalue;
  count   
----------
 21499474
(1 row)

And there are a lot that already expired, so let me clean them as well.

Done.

delete from pendedkeyvalue where pended_id in (select id from pended where expiration_date < '2025-07-10 00:00:00');
delete from pended where expiration_date < '2025-07-10 00:00:00';

Hopefully now it would be stable.

It seems that we finally resolved the OOM. The current memory usage is around 5.0 GB dropped from 16 GB.

I will be watching it for some time to see if we really resolved it, but I will close the ticket as fixed for now.

Metadata Update from @zlopez:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)

And now it looks much better :-)

[root@mailman01 ~][PROD-RDU3]# ps aux | awk 'NR>1 {$5=int($5/1024)"M";}{ print;}' | grep runner
mailman 2173608 0.2 0.3 463M 101108 ? Sl 12:10 0:15 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=archive:0:1
mailman 2173609 0.4 0.2 90M 85868 ? S 12:10 0:33 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=bounces:0:1
mailman 2173610 0.0 0.2 82M 78188 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=command:0:1
mailman 2173611 0.0 0.2 87M 82944 ? S 12:10 0:07 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=in:0:1
mailman 2173612 0.0 0.2 153M 86716 ? Sl 12:10 0:06 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=lmtp:0:1
mailman 2173613 0.0 0.2 78M 74464 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=nntp:0:1
mailman 2173614 0.0 0.3 104M 99572 ? S 12:10 0:04 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=out:0:1
mailman 2173615 0.1 0.3 127M 123828 ? S 12:10 0:08 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=pipeline:0:1
mailman 2173616 0.0 0.2 86M 82240 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1
mailman 2173617 0.0 0.2 78M 73968 ? S 12:10 0:00 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=retry:0:1
mailman 2173619 0.0 0.3 103M 98272 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=virgin:0:1
mailman 2173620 0.0 0.2 84M 79976 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=digest:0:1
mailman 2173642 0.0 0.3 113M 103564 ? S 12:10 0:01 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1
mailman 2173643 0.0 0.3 115M 106380 ? S 12:10 0:02 /usr/bin/python3 /usr/libexec/mailman3/runner -C /etc/mailman.cfg --runner=rest:0:1

Cool. I wasn't sure if those old bounces were useful for something, so I just made an index on it... but I guess it got too big for that too.

They are just marked as processed, but there isn't any task to clean them after that AFAIK. So the db just gets bigger and bigger.

I saw one more OOM after the cleanup, but it's not the same, it happened to gunicorn. Right now the memory consumption is stable. Will try to still monitor it and see if anything else happens.

Should we add a cronjob to remove old bounces and pendedkeyvalues from the DB?

That would be probably helpful, it could even be another systemd timer in mailman role. We already have few of them for hyperkitty jobs.

I also today added auto restart for mailmanweb service as it's still hitting OOM. I assume in this case it's probably AI scrapers consuming all the resources.

We should try and get upstream to add these things instead of adding them ourselves downstream. ;(

This has been fixed upstream since March 2023: https://gitlab.com/mailman/mailman/-/merge_requests/1105

It should be in version 3.3.9, we're currently running 3.3.8.

In that case we need to ask for update of EPEL package https://packages.fedoraproject.org/pkgs/mailman3/mailman3/
Metadata
None

Needs investigation dc-move lists low-gain ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.