pagure

#5463 fix unit tests, bump to F39, unpin python packages and replace deprecated code if necessary
Merged by ngompa. Opened by wombelix.
wombelix/pagure tests_5462_fix_issues_with_tox_and_py39  into  master

Download 5463.patch

Early draft and work in progress, started by addressing the issues in https://pagure.io/pagure/issue/5462.
I'm fighting with the unit tests again to bring them to a more recent Fedora and Python version.
This PR will most likely supersede https://pagure.io/pagure/pull-request/5442 and https://pagure.io/pagure/pull-request/5405.

F39 pip tox py39 looks already promising with unpinned pygit2:

============================================================ 1721 passed, 3 skipped, 8723 warnings in 884.93s (0:14:44) ============================================================
py39: exit 0 (885.32 seconds) /pagure> pytest -n auto tests/ pid=33
  py39: OK (885.36=setup[0.04]+cmd[885.32] seconds)
  congratulations :) (885.40 seconds)

Changes only applied to the test container for local testing, I didn't touched the CI container yet, so they will horrible fail at this point.

Requires more to make pip Py310, 311, 312 and F39 RPM tests pass.
I'm also playing around with CentOS Stream 9, but that's not the main focus.

26 new commits added

  • tests: Monkey Patch as temporary workaround for celery task_id = None issues. Needs a better solution later
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • fixup! tests: Debug celery/redis task_id None issue
  • tests: Debug celery/redis task_id None issue
  • build: Version pinning for redis-py package removed
  • build: Pin sqlalchemy to any version below major release 2.0.0, latst 1.4x releases provide Py311 and Py312 support
  • build: Remove version pinning from Celery to support Py311 and Py312
  • fix: Mock 'imp' module to satisfy 'straight.plugin' till https://github.com/ironfroggy/straight.plugin/pull/30 is published.
  • fixup! tests: Add argument --toxenv to run-tests-container.py. Run tests with tox against all environments defined in tox.ini by default. Customizable by the new argument. Passes the value to the '-e' argument of 'tox'.
  • fixup! tests: Add argument --toxenv to run-tests-container.py. Run tests with tox against all environments defined in tox.ini by default. Customizable by the new argument. Passes the value to the '-e' argument of 'tox'.
  • fix: Replace 'imp' with 'importlib' for Python 3.12 compatibility.

Couple of hacky workarounds later down hundreds of failed tests to:

pip py39

============================================================================= short test summary info ==============================================================================
FAILED tests/test_dev_data.py::TestDevData::test_dev_data_all - AssertionError: 42 != 14
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace_changed - AssertionError: 'Are [71 chars]ne!\nProject deleted\n' != 'Are [71 char...
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace - AssertionError: 'Are you sure you want to delete: somenam[53 chars]ed\n' != None
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project - AssertionError: 'Are [57 chars]ne!\nProject deleted\n' != 'Are [57 chars]ne!\nNone\nd6e73...
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_empty_git - AssertionError: False is not true
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_trend_empty_git - AssertionError: False is not true
FAILED tests/test_style.py::TestStyle::test_code_with_black - AssertionError: 1 != 0
FAILED tests/test_style.py::TestStyle::test_code_with_flake8 - AssertionError: 1 != 0
======================================================= 8 failed, 1713 passed, 3 skipped, 8714 warnings in 862.60s (0:14:22) =======================================================

pip py310

============================================================================= short test summary info ==============================================================================
FAILED tests/test_dev_data.py::TestDevData::test_dev_data_all - AssertionError: 42 != 14
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace_changed - AssertionError: 'Are [71 chars]ne!\nProject deleted\n' != 'Are [71 char...
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace - AssertionError: 'Are you sure you want to delete: somenam[53 chars]ed\n' != None
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project - AssertionError: 'Are [57 chars]ne!\nProject deleted\n' != 'Are [57 chars]ne!\nNone\n67a73...
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_empty_git - AssertionError: False is not true
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_trend_empty_git - AssertionError: False is not true
FAILED tests/test_style.py::TestStyle::test_code_with_black - AssertionError: 1 != 0
FAILED tests/test_style.py::TestStyle::test_code_with_flake8 - AssertionError: 1 != 0
======================================================= 8 failed, 1713 passed, 3 skipped, 8737 warnings in 875.14s (0:14:35) =======================================================

pip py311

============================================================================= short test summary info ==============================================================================
FAILED tests/test_dev_data.py::TestDevData::test_dev_data_all - AssertionError: 42 != 14
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace - AssertionError: 'Are you sure you want to delete: somenam[53 chars]ed\n' != None
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace_changed - AssertionError: 'Are [71 chars]ne!\nProject deleted\n' != 'Are [71 char...
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project - AssertionError: 'Are [57 chars]ne!\nProject deleted\n' != 'Are [57 chars]ne!\nNone\n379af...
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_empty_git - AssertionError: False is not true
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_trend_empty_git - AssertionError: False is not true
FAILED tests/test_style.py::TestStyle::test_code_with_flake8 - AssertionError: 1 != 0
FAILED tests/test_style.py::TestStyle::test_code_with_black - AssertionError: 1 != 0
======================================================= 8 failed, 1713 passed, 3 skipped, 8737 warnings in 887.17s (0:14:47) =======================================================

pip py312

============================================================================= short test summary info ==============================================================================
FAILED tests/test_dev_data.py::TestDevData::test_dev_data_all - AssertionError: 42 != 14
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace - AssertionError: 'Are you sure you want to delete: somenam[53 chars]ed\n' != None
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project_namespace_changed - AssertionError: 'Are [71 chars]ne!\nProject deleted\n' != 'Are [71 char...
FAILED tests/test_pagure_admin.py::PagureAdminDeleteProjectTests::test_delete_project - AssertionError: 'Are [57 chars]ne!\nProject deleted\n' != 'Are [57 chars]ne!\nNone\n10355...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_add_remove_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests'...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object has...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_invalid_acl - AttributeError: 'PagureFlaskApiProjectModifyAclTests' obje...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_neither_user_nor_group - AttributeError: 'PagureFlaskApiProjectModifyAcl...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_acl - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object ha...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_permission - AttributeError: 'PagureFlaskApiProjectModifyAclTests' ob...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_project - AttributeError: 'PagureFlaskApiProjectModifyAclTests' objec...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_user - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object h...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_group_not_in_project - AttributeError: 'PagureFlaskApiProjectModi...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_own_acl_ - AttributeError: 'PagureFlaskApiProjectModifyAclTests' ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_own_acl_no_access - AttributeError: 'PagureFlaskApiProjectModifyA...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_someone_else_acl - AttributeError: 'PagureFlaskApiProjectModifyAc...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_user - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object has ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_w_header - AttributeError: 'PagureFlaskApiProjectOptionsTests' obj...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_wo_header - AttributeError: 'PagureFlaskApiProjectOptionsTests' ob...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_wrong_project - AttributeError: 'PagureFlaskApiProjectOptionsTests...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options - AttributeError: 'PagureFlaskApiProjectOptionsTests' object ha...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options2 - AttributeError: 'PagureFlaskApiProjectOptionsTests' object h...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_json - AttributeError: 'PagureFlaskApiProjectOptionsTests' obje...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_no_data - AttributeError: 'PagureFlaskApiProjectOptionsTests' o...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_wo_header - AttributeError: 'PagureFlaskApiProjectOptionsTests'...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_wrong_project - AttributeError: 'PagureFlaskApiProjectOptionsTe...
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_empty_git - AssertionError: False is not true
FAILED tests/test_pagure_flask_internal.py::PagureFlaskInternaltests::test_get_stats_commits_trend_empty_git - AssertionError: False is not true
FAILED tests/test_pagure_flask_ui_repo.py::PagureFlaskRepotests::test_edit_file_default_email - AttributeError: 'PagureFlaskRepotests' object has no attribute 'assertEquals'. Di...
FAILED tests/test_pagure_flask_ui_roadmap.py::PagureFlaskRoadmaptests::test_roadmap_ui - AttributeError: 'PagureFlaskRoadmaptests' object has no attribute 'assertEquals'. Did yo...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey - AttributeError: 'PagureUtilsTests' object has no attribute 'assertNotEquals'. Did you mean: 'assertN...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_different_project - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_non_deploykey - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you mea...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_non_existent_key - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you ...
FAILED tests/test_style.py::TestStyle::test_code_with_black - AssertionError: 1 != 0
FAILED tests/test_style.py::TestStyle::test_code_with_flake8 - AssertionError: 1 != 0
===================================================== 37 failed, 1684 passed, 3 skipped, 146785 warnings in 899.67s (0:14:59) ======================================================

10 new commits added

  • fixup! tests: Fix failed test test_get_stats_commits_empty_git
  • fixup! tests: Fix failed test test_get_stats_commits_empty_git
  • fixup! tests: Fix failed test test_get_stats_commits_empty_git
  • fixup! tests: Fix failed test test_get_stats_commits_empty_git
  • fix: Flake8 formatting issues, findings tests/test_style.py::TestStyle::test_code_with_flake8
  • fix: Black formatting issues, findings of tests/test_style.py::TestStyle::test_code_with_black
  • fixup! tests: Fix failed test test_get_stats_commits_empty_git
  • tests: Fix failed test test_get_stats_commits_empty_git
  • fixup! tests: Undo previous code change during debugging
  • tests: Undo previous code change during debugging

1 new commit added

  • fixup! fix: Mock 'imp' module to satisfy 'straight.plugin' till https://github.com/ironfroggy/straight.plugin/pull/30 is published.

F39 pip: p39, py310 and py311 looking good now. A bit work is still required for py312.
F39 rpm pending, comes when pip is fine.
The next big thing - for another PR - is then to work through the thousands of warnings.

pip py39

============================================================ 1721 passed, 3 skipped, 8723 warnings in 865.10s (0:14:25) ============================================================

pip py310

============================================================ 1721 passed, 3 skipped, 8765 warnings in 872.01s (0:14:32) ============================================================

pip py311

============================================================ 1721 passed, 3 skipped, 8746 warnings in 887.93s (0:14:47) ============================================================

pip py312

============================================================================= short test summary info ==============================================================================
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_add_remove_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests'...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object has...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_invalid_acl - AttributeError: 'PagureFlaskApiProjectModifyAclTests' obje...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_neither_user_nor_group - AttributeError: 'PagureFlaskApiProjectModifyAcl...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_acl - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object ha...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_group - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_permission - AttributeError: 'PagureFlaskApiProjectModifyAclTests' ob...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_project - AttributeError: 'PagureFlaskApiProjectModifyAclTests' objec...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_no_user - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object h...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_group_not_in_project - AttributeError: 'PagureFlaskApiProjectModi...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_own_acl_ - AttributeError: 'PagureFlaskApiProjectModifyAclTests' ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_own_acl_no_access - AttributeError: 'PagureFlaskApiProjectModifyA...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_remove_someone_else_acl - AttributeError: 'PagureFlaskApiProjectModifyAc...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectModifyAclTests::test_api_modify_acls_user - AttributeError: 'PagureFlaskApiProjectModifyAclTests' object has ...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_w_header - AttributeError: 'PagureFlaskApiProjectOptionsTests' obj...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_wo_header - AttributeError: 'PagureFlaskApiProjectOptionsTests' ob...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_get_project_options_wrong_project - AttributeError: 'PagureFlaskApiProjectOptionsTests...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options - AttributeError: 'PagureFlaskApiProjectOptionsTests' object ha...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options2 - AttributeError: 'PagureFlaskApiProjectOptionsTests' object h...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_json - AttributeError: 'PagureFlaskApiProjectOptionsTests' obje...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_no_data - AttributeError: 'PagureFlaskApiProjectOptionsTests' o...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_wo_header - AttributeError: 'PagureFlaskApiProjectOptionsTests'...
FAILED tests/test_pagure_flask_api_project.py::PagureFlaskApiProjectOptionsTests::test_api_modify_project_options_wrong_project - AttributeError: 'PagureFlaskApiProjectOptionsTe...
FAILED tests/test_pagure_flask_ui_repo.py::PagureFlaskRepotests::test_edit_file_default_email - AttributeError: 'PagureFlaskRepotests' object has no attribute 'assertEquals'. Di...
FAILED tests/test_pagure_flask_ui_roadmap.py::PagureFlaskRoadmaptests::test_roadmap_ui - AttributeError: 'PagureFlaskRoadmaptests' object has no attribute 'assertEquals'. Did yo...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey - AttributeError: 'PagureUtilsTests' object has no attribute 'assertNotEquals'. Did you mean: 'assertN...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_different_project - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you...
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_non_existent_key - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you ...
FAILED tests/test_style.py::TestStyle::test_code_with_flake8 - AssertionError: 1 != 0
FAILED tests/test_pagure_utils.py::PagureUtilsTests::test_lookup_deploykey_non_deploykey - AttributeError: 'PagureUtilsTests' object has no attribute 'assertEquals'. Did you mea...
===================================================== 30 failed, 1691 passed, 3 skipped, 147020 warnings in 898.36s (0:14:58) ======================================================
Edited by wombelix

rebased onto 74fc99ebddf07c1393aa2bff3ecd79b6aa01ab70

pretty please pagure-ci rebuild

1 new commit added

  • tests: Revert 'CI pipeline - Drop tests on Fedora RPM temporary', bump rpm container Dockerfile to F39

1 new commit added

  • build: python3-cchardet replaced by python3-chardet in F39

pretty please pagure-ci rebuild

1 new commit added

  • tests: Disable F39 RPM test because of pytest+xdist bug

1 new commit added

  • fixup! tests: Disable F39 RPM test because of pytest+xdist bug

rebased onto 685e61e7b6c62ee0f8db563ad9fb53f0003f7ba2

rebased onto 8a9f4fc51958af6655024c5cf1d5ae5903e464cc

2 new commits added

  • tests: Install missing dependencies in pip container to fix 'Failed building wheel for Pillow'
  • tests: Set executable flag for 'runtests_py3.sh' and 'tox_py3.sh' to align with later usage in unit test containers.

pretty please pagure-ci rebuild

2 new commits added

  • tests: Rename 'alembic-3' to 'alembic' in unit tests, sed commands to replace it during container start removed. Not required anymore after dropping Python 2 a while ago.
  • tests: Make container entrypoint and run test scripts more reliable, fail on error, and always merge feature branch against upstream/master and not fork/master.

Unit tests are back to normal (https://jenkins-pagure.apps.ocp.cloud.ci.centos.org/job/pull-requests/285/).
They run on F39 now and pip/tox based tests cover py39, py310, py311 and py312.
RPM container bumped to F39 but tests removed from CI for now, reason below in F39 RPM pytest runs endless, consumes all resources, triggers OOM Killer section.
Also trololio is replaced now and dropped from requirements and spec.
Related PRs are closed in favour of this one.
I cherry picked a couple of patches that were contributed in the past.
All other relevant information are summarizied below.

F39 RPM pytest runs endless, consumes all resources, triggers OOM Killer

F39 pip isn't affected, but on rpm the tests stuck
till the OOM Killer triggers. There are multiple
similar reports upstream but no fix. To unblock
the unit tests, F39 RPM will be skipped for now.
F39 pip covers py39, py310, py311 and py312.
CI jobs that run into the issue:
https://jenkins-pagure.apps.ocp.cloud.ci.centos.org/job/pull-requests/276/
https://jenkins-pagure.apps.ocp.cloud.ci.centos.org/job/pull-requests/277/
GitHub issues that report similar issues:
https://github.com/pytest-dev/pytest-xdist/issues/110
https://github.com/pytest-dev/pytest-xdist/issues/661
https://github.com/pytest-dev/pytest-xdist/issues/872
https://github.com/pytest-dev/pytest-xdist/issues/1005
  • Disabled via https://pagure.io/pagure/pull-request/5464, https://pagure.io/pagure/pull-request/5465

Version pinning in requirements.txt removed for:
- pygit2
- celery
- redis
- pytest
- flake8
- black
- redis
- setuptools
- straight.plugin
- markdown

Various minor changes in the code base and or tests because of deprecations related to version pin removal.

SQLAlchemy pinned to <2.0.0, the jump to the next major release requires a lot of changes and is a separate activity. 1.4.x is fine for now and works with Python 3.9 and higher.

trololio was replaced by asyncio, I'm not convinced that there is an appropriate test coverage so this needs some more love at a later point.

tox creates and runs the tests now against four environments by default: py39, py310, py311, py312

run-tests-containers.py has a new argument --toxenv to set an individual environment/python version for local testing, default is to use all environments defined in tox.ini.

The CentOS Stream 8 container for local testing was renamed and bumped to CentOS 9. Early stage, some packages are not available, tests failing, to be completed in a separate PR.

Other Issues:
- During container build the repo and branch of the fork was not passed as build-arg. pagure.io/pagure:master was always used which caused some testing related files to never taken into account in the CI. In some cases this also caused a code drift, so the testing locally and in the CI had different results. I also adjusted the shell scripts to fail on errors and that feature branches always get merged into a checkout of upstream/master to identify merge conflicts early.
- straight.plugin doesn't seem to be maintained anymore and the current version uses imp which is removed in Python 3.12. PR pending https://github.com/ironfroggy/straight.plugin/pull/30. It doesn't actually use imp but imports it in one file. imp added as module via Mock as workaround till there is time to evaluate how to replace it.
- There were some wird issues related to importing zope interface. That seemed to be caused by some setuptools versions, was fine after removing the pinning. GH Issues https://github.com/zopefoundation/zope.interface/issues/255, https://github.com/jazzband/pip-tools/issues/1576, https://github.com/pypa/setuptools/issues/3157
- The amount of warnings massively increased, expected but something that has to be addressed next to remove more version pinnings.
- To install Pillow in Python 3.12 pip, additional packages to build the wheel were required and added to the CI and local pip container Dockerfiles

It looks good except for two things:

  • you have two commits with broken attribution
  • you have a fixup commit that needs to be squashed into another commit

55 new commits added

  • tests: Rename 'alembic-3' to 'alembic' in unit tests, sed commands to replace it during container start removed. Not required anymore after dropping Python 2 a while ago.
  • tests: Make container entrypoint and run test scripts more reliable, fail on error, and always merge feature branch against upstream/master and not fork/master.
  • tests: Install missing dependencies in pip container to fix 'Failed building wheel for Pillow'
  • tests: Set executable flag for 'runtests_py3.sh' and 'tox_py3.sh' to align with later usage in unit test containers.
  • tests: Disable F39 RPM test because of pytest+xdist bug
  • build: python3-cchardet replaced by python3-chardet in F39
  • tests: Revert 'CI pipeline - Drop tests on Fedora RPM temporary', bump rpm container Dockerfile to F39
  • fix: black/flake8 formatting issue
  • tests: CI pipeline - Drop tests on Fedora RPM temporary because of pytest xdist issues, tasks stuck till OOM Killer hits. Bump pip based tests to F39 and run against all available tox environments (py39, py310, py311, py312)
  • chore: Remove Celery related MonkeyPatch after applying the Patch from Klaus Koder
  • Fix Celery related exception in get_key_for_task
  • Fix warning jinga2.escape DeprecationWarning
  • Fix PagureHooksPagureHooktests failure when system wide config exist
  • Drop the dependency on python-trololio (in favor of asyncio)
  • fix: Address unit test issues and deprecation warnings
  • tests: Revert 80c9020eae7260851266b539d0aff1fee8222458
  • build: Remove version pinning from pytest* packages
  • tests: Run by default on all logical CPUs in rpm tests containers
  • build: Remove version pinning of flake8 python package
  • tests: 'assertNotEquals' deprecated and removed, replaced by 'assertNotEqual'
  • tests: 'assertEquals' deprecated and removed, replaced by 'assertEqual'
  • fix: Flake8 formatting issues, findings tests/test_style.py::TestStyle::test_code_with_flake8
  • fix: Black formatting issues, findings of tests/test_style.py::TestStyle::test_code_with_black
  • tests: Fix failed test test_get_stats_commits_empty_git
  • tests: Undo previous code change during debugging
  • tests: Monkey Patch as temporary workaround for celery task_id = None issues. Needs a better solution later
  • tests: Debug celery/redis task_id None issue
  • build: Version pinning for redis-py package removed
  • build: Pin sqlalchemy to any version below major release 2.0.0, latst 1.4x releases provide Py311 and Py312 support
  • build: Remove version pinning from Celery to support Py311 and Py312
  • fix: Mock 'imp' module to satisfy 'straight.plugin' till https://github.com/ironfroggy/straight.plugin/pull/30 is published.
  • fix: Replace 'imp' with 'importlib' for Python 3.12 compatibility.
  • build: Don't add tox_py3.sh to fedora pip code container, that file comes from the repo during testing.
  • tests: Add python 3.12 back in tox.ini
  • tests: Add argument --toxenv to run-tests-container.py. Run tests with tox against all environments defined in tox.ini by default. Customizable by the new argument. Passes the value to the '-e' argument of 'tox'.
  • fix: Test for empty string in addition to None Type in 'pagure.lib.git.commit_to_patch'
  • tests: fix test_commit_to_patch_empty_commit
  • tests: fix issues related to log in url in 'tests/test_pagure_flask_ui_login.py'
  • tests: fix issues related to log in url in 'tests/test_pagure_flask_ui_issues_open_access.py'
  • tests: fix issues related to log in url in 'tests/test_pagure_flask_ui_issues.py'
  • tests: fix issues related to log in url in 'tests/test_pagure_flask_ui_issues_acl_checks.py'
  • tests: fix test_view_issue_ticket_access - AssertionError
  • tests: bump CentOS Stream from 8 to 9
  • tests: Fix issue 'No module named pygit2.remote'
  • build: Replace python3-cchardet with python3-chardet
  • tests: bump rpm base container to F39
  • fix: pygit2 Repository.create_remote replaced by Repository.remotes.create
  • build: Verbose / Debug output from tox command during base fedora pip container build
  • build: Fix issue that local volume wasn't mounted inside base container during build initiated by run-tests-container.py
  • build: Fix issue that build-args are not passed correctly to podman in run-tests-container.py
  • build: Enable verbose/debug mode for git clone in fedora base pip container
  • build: Remove version pinning from pygit2
  • tests: bump pip base container to F39
  • tests: Additional packages in fedora pip base container to meet requirements of 'Pillow' python package (compile wheel)
  • tests: Drop py312 from tox tests, pinned Pillow version isn't compatible

pretty please pagure-ci rebuild

Pull-Request has been merged by ngompa

This is all done and merged now. :wine_glass:

Metadata
None
No Tags
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.