fedora-infrastructure

#12669 "Releases Overview" not updated on packages.fedoraproject.org
Closed: Fixed with Explanation by kevin. Opened by pg-tips.

Closed: Fixed with Explanation
Reopen Issue

On https://packages.fedoraproject.org/ I’ve seen several cases where the “Releases Overview” table hasn’t been updated to reflect the current status of the repos, even though the "Recent Activity" data on the same page does seem to be up to date.

Here are a couple of representative examples, as of the time of writing.

a) powerdevil : stable version shown incorrectly
The stable Fedora 42 version is now 6.4.3-1, but the table states that the stable version is 6.4.1-1.

b) restic : testing version shown incorrectly
For Fedora 42, version 0.18.0-1 is now available in testing, but the table states there is no testing version.

Maybe an update process isn't running since the datacenter move?

<!!image>

<!!image>

Screenshot_20250717_124915.png

Screenshot_20250717_124858.png

This seems to be due to mdapi not having a f42 branch at all. ;(

I thought we fixed that, but perhaps not?

CC: @t0xic0der

Metadata Update from @kevin:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

It actually seems to be an issue for versions other than f42 as well.

For example, looking at restic (https://packages.fedoraproject.org/pkgs/restic/restic/index.html), the Bodhi updates in the "Recent Activity" section show that 0.18.0-1 was pushed to testing 5 days ago for both f42 and f41. But neither of those shows in the "Releases Overview" table.

Uhm ... the landing page says "Last refreshed on 2025-07-04 (73777 packages)."

So that seems to be unrelated to F42 specific issues?

As an update ... it's been confirmed that mdapi is failing because there's no sqlite metadata for those repos.

There's an upstream change to let it work with xml metadata, so now we get to decide if we want to try deploying that or generate sqlite for a few more months.

tl;dr It'll probably be broken for at least a few more days.

I think that @t0xic0der deployed new version of mdapi recently, but I'm not sure if it contained this fix.

This was apparently my misunderstanding. packages.fedoraproject.org pushed a workaround (basically it just did a new local createrepo with sqlite data), but mdapi didn't... it was expecting us to re-enable sqlite data until the replacement application for it is deployed. :(

Filed https://pagure.io/fedora-infra/ansible/pull-request/2765 to re-enable sqlite for f41/42 for now.

packages.fedoraproject.org pushed a workaround

Then why is it broken right now? This seems to be unrelated to mdapi issues then?

ok, so... looking more closely/carefully:

fedora-packages-static had some commits to work around this sqlite missing issue:
https://pagure.io/fedora-packages-static/c/dd5202943b194caa2c43b66fa63b0509b5205e80?branch=production
and
https://pagure.io/fedora-packages-static/c/37de99333c249d8ca099157146ee3d02abd6ff28?branch=production

but it's not working, perhaps due to epel being added and epel not having sqlite db either. (also, eln doesn't either).

current traceback:

Cannot remove /tmp/fedorapkgs-0u8paxdg/repodata/94a3167ad1593d3e63d2c8fee95d8bffc1a1bc6d410d67d01c1d1ae3d5ca543d-primary.sqlite.zst: No such file or directory
type group added to list from path: /tmp/fedorapkgs-0u8paxdg/repodata/d148f1d28c05b3ba81f9cb75eeeb5402a99cb7945d5da6ed5da9e5a54fbf3b0b-comps-Everything.x86_64.xml.zst 
type updateinfo added to list from path: /tmp/fedorapkgs-0u8paxdg/repodata/966dc4eabf167d2d36eb2ec9005615c056befe73c4f5e0f085d968dbd3765a8b-updateinfo.xml.zst 
Preparing sqlite DBs
epel-10.0              Extracting /tmp/fedorapkgs-0u8paxdg/repodata/723606f8a6a0aadefc15c67af250ab11133647135246f9b6431b7630ba78cc96-primary.sqlite.bz2 to /tmp/fedorapkgs-0u8paxdg/repodata/epel-10.0_primary.sqlite
epel-10.0              Indexing file: /tmp/fedorapkgs-0u8paxdg/repodata/epel-10.0_primary.sqlite
epel-10.0              Creating diff for file: /etc/packages/repositories/epel-10.0_primary.sqlite
Traceback (most recent call last):
  File "/usr/local/src/packages/bin/fetch-repository-dbs.py", line 437, in <module>
    main()
    ~~~~^^
  File "/usr/local/src/packages/bin/fetch-repository-dbs.py", line 433, in main
    handle(repo, args.target_dir, db_removed)
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/packages/bin/fetch-repository-dbs.py", line 297, in handle
    gen_db_diff(name, tmp_db_path, dst_db_path, False)
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/src/packages/bin/fetch-repository-dbs.py", line 131, in gen_db_diff
    conn.execute(f"ATTACH DATABASE '{old}' as old")
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.DatabaseError: database disk image is malformed
make: *** [Makefile:29: sync-repositories] Error 1
bin/update-solr.py
Loading maintainer mapping...
Loading release name mapping...
> Processing database files for epel-10.0.
Traceback (most recent call last):
  File "/usr/local/src/packages/bin/update-solr.py", line 249, in <module>
    main()
    ~~~~^^
  File "/usr/local/src/packages/bin/update-solr.py", line 123, in main
    for raw in primary.execute("SELECT * FROM packages"):
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.DatabaseError: database disk image is malformed
make: *** [Makefile:25: update-solr] Error 1

I've enabled sqlite on epel10 updates, will see if it helps.

Looking at the https://packages.fedoraproject.org/pkgs/restic/restic/ and comparing to https://src.fedoraproject.org/rpms/restic and it seems that the information are still not updated :/

Yes, it's still not working. The same issues with epel10...

someone needs to figure out why it's failing there and fix it. ;(

@mymindstorm may not be around... so if anyone else sees the problem and can provide a pr that would be great.

Some sqlite files were actually corrupted and were blocking the sync, I've deleted them from the storage and ran the script again, hopefully it'll work now.

Ah, corrupt databases again but this time there's a hint as to why: OSError: [Errno 28] No space left on device

We need to increase the space for the repo files:

$ df -h | grep 100
ntap-rdu3-c02-fedora01-nfs01a.rdu3.fedoraproject.org:/fedora-packages-static-db-storage   10G   10G  1.0M 100% /etc/packages

Does anybody here know how to do that on on the NetApp fileserver?

hopefully it'll work now.

packages.fp.o now says Last refreshed on 2025-08-11 (74694 packages). So it appears to have worked at least once yesterday, but not today (likely again out of storage space?)

 
ntap-rdu3-c02::> df -h fedora_packages_static_db_storage 
Filesystem               total       used      avail capacity  Mounted on                 Vserver
/vol/fedora_packages_static_db_storage/ 
                          10GB    10201MB       38MB      99%  /fedora-packages-static-db-storage  
                                                                                          ntap-rdu3-c02-fedora01
/vol/fedora_packages_static_db_storage/.snapshot 
                            0B     6469MB         0B       0%  /fedora-packages-static-db-storage/.snapshot  
                                                                                          ntap-rdu3-c02-fedora01
2 entries were displayed.
ntap-rdu3-c02::> volume modify fedora_packages_static_db_storage -size +90G
Volume modify successful on volume fedora_packages_static_db_storage of Vserver ntap-rdu3-c02-fedora01.
ntap-rdu3-c02::> df -h fedora_packages_static_db_storage                   
Filesystem               total       used      avail capacity  Mounted on                 Vserver
/vol/fedora_packages_static_db_storage/ 
                         100GB    10201MB       90GB       9%  /fedora-packages-static-db-storage  
                                                                                          ntap-rdu3-c02-fedora01
/vol/fedora_packages_static_db_storage/.snapshot 
                            0B     6469MB         0B       0%  /fedora-packages-static-db-storage/.snapshot  
                                                                                          ntap-rdu3-c02-fedora01
2 entries were displayed.

ok. I cleaned up some more corrupt sqlite db's and got it to get past that.

There's no errors now. The last thing in logs is:

...
>>> 74717 packages have been extracted.
Sending data to Solr index...
DONE.
> 74717 packages submitted to solr.

There was a cpu spike on the solr pod, but then it went back to normal.

However, I still see the old data. ;(

It looks like the restic versions are in sync now, the data looks current to me. Do you have an example of old data?

I looked at a few packages and it seems to be up-to-date now. It also says that the data is from 2025-08-18, so from today, which is a good sign. :)

Metadata Update from @kevin:
- Issue assigned to kevin

Huh. Wonder why it wasn't showing updated right after I looked?

ok then, I guess we can close this now.

Thanks for the help everyone and if anyone sees anything still out of date, please let us know.

Metadata Update from @kevin:
- Issue close_status updated to: Fixed with Explanation
- Issue status updated to: Closed (was: Open)
Metadata

medium-gain medium-trouble ops

None
None
Normal
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.