PR#4275: Enforce C locale for tox environment - koji

koji

#4275 Enforce C locale for tox environment

Opened 3 months ago by tkopecek. Modified 3 months ago

tkopecek/koji tox-lang into master

Superceded by settings in tox.ini

Tomas Kopecek • 3 months ago

ff36be4

Enforce C locale for tox environment

Tomas Kopecek • 3 months ago

59f1474

tests/test_cli/utils.py

file modified

-3

		`@@ -194,9 +194,6 @@`

		`@mock.patch('koji_cli.commands.activate_session')`
		`def assert_help(self, callableObj, message, activate_session_mock):`
		`- # optarse uses gettext directly and it is driven by LANGUAGE`
		`- # we need english to get comparable strings`
		`- os.environ['LANGUAGE'] = 'C'`
		`self.assert_system_exit(`
		`callableObj,`
		`mock.MagicMock(),`

tox.ini

file modified

+3 -2

		`@@ -23,6 +23,7 @@`
		`sitepackages = true`
		`setenv =`
		`COLUMNS=80`
		`+ LANG=C`
		`# Expected that python-rpm is installed on the host`
		`# If rpm's python bindings are missing, don't continue`
		`# Also, because coverage might be installed system-wide and it serves as our`
		`@@ -36,13 +37,13 @@`
		`[testenv:py3]`
		`deps =`
		`-r{toxinidir}/test-requirements.txt`
		`- setenv =`
		`+ setenv =`
		`{[testenv]setenv}`
		`PYTHONPATH=.:plugins/hub/.:plugins/builder/.:plugins/cli/.:cli/.:www/lib`
		`commands_pre =`
		`{[testenv]commands_pre}`
		`python -m coverage erase --rcfile .coveragerc3`
		`- commands =`
		`+ commands =`
		`python -m pytest -n auto --cov --cov-config .coveragerc3 --cov-report=html`

		`[testenv:py2]`

tkopecek commented 3 months ago

I'm often getting weird thread collisions in cs_CZ.UTF-8. It seems to be limited to path handling in test_prune_signed_copies.py while errors are gone when running under C. One of the errors is here (other one is about already deleted path there, I can't get reproduce it now).

tests/test_cli/test_prune_signed_copies.py:616: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.13/difflib.py:1303: in ndiff
    def ndiff(a, b, linejunk=None, charjunk=IS_CHARACTER_JUNK):
.tox/py3/lib64/python3.13/site-packages/coverage/control.py:395: in _should_trace
    disp = self._inorout.should_trace(filename, frame)
.tox/py3/lib64/python3.13/site-packages/coverage/inorout.py:352: in should_trace
    canonical = canonical_filename(filename)
.tox/py3/lib64/python3.13/site-packages/coverage/files.py:86: in canonical_filename
    cf = abs_file(cf)
.tox/py3/lib64/python3.13/site-packages/coverage/files.py:159: in abs_file
    return actual_path(os.path.abspath(os.path.realpath(path)))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

filename = '/usr/lib64/python3.13/difflib.py'

>   ???
E   TypeError: an integer is required

<frozen posixpath>:440: TypeError

mikem commented 3 months ago

Whenever I see a fix like this, I worry that it's masking a deeper problem.

I can't seem to replicate this simply by setting LANG. Any hints?

tkopecek commented 3 months ago

What increased appearance of these errors was increasing number of parallel threads (simple - it manifests more on new laptop).

If I disable coverage (which is where traceback ends), it will fail on the language issue directly in that comparison. That was the reason why I've tried LANG=C

>           self.assertMultiLineEqual(actual, expected)
E           AssertionError: "Cuto[784 chars]o 2] Adresář nebo soubor neexistuje: 'fakebuil[179 chars] 0\n" != "Cuto[784 chars]o 2] No such file or directory: 'fakebuildpath[174 chars] 0\n"
E             Cutoff date: Sat Apr  20 14:08:17 2023
E             Getting builds...
E             ...got 1 builds
E             DEBUG: package-name-1.3-4
E             Tags: ['test-tag1', 'test-tag2']
E             Sat Apr  20 14:08:17 2023: Tagged package-name-1.3-4 with test-tag1 [still active]
E             Sat Apr  20 14:08:17 2023: Untagged package-name-1.3-4 from test-tag1
E             tag test-tag1: package-name-1.3-4 not latest (revoked Sat Apr  20 14:08:17 2023)
E             Sat Apr  20 14:08:17 2023: Tagged package-name-1.3-4 with test-tag2 [still active]
E             Sat Apr  20 14:08:17 2023: Untagged package-name-1.3-4 from test-tag2
E             tag test-tag2: package-name-1.3-4 not latest (revoked Sat Apr  20 14:08:17 2023)
E             Unlinking: fakebuildpath/data/signed/qwertyuiop/test-arch/test-1.3-1.test-arch.rpm
E           - Error removing fakebuildpath/data/signed/qwertyuiop/test-arch/test-1.3-1.test-arch.rpm: [Errno 2] Adresář nebo soubor neexistuje: 'fakebuildpath/data/signed/qwertyuiop/test-arch/test-1.3-1.test-arch.rpm'
E           ?                                                                                                   ^   ^^^^^^^^^^^^^  ^^^^^^^^^^^
E           + Error removing fakebuildpath/data/signed/qwertyuiop/test-arch/test-1.3-1.test-arch.rpm: [Errno 2] No such file or directory: 'fakebuildpath/data/signed/qwertyuiop/test-arch/test-1.3-1.test-arch.rpm'
E           ?                                                                                                   ^^^^^^^^^^^^^^^^ +  ^^  ^
E             This script needs write access to /mnt/koji
E             (build has no signed copies)
E             --- Grand Totals ---
E             Files: 0
E             Bytes: 0

tests/test_cli/test_prune_signed_copies.py:616: AssertionError

tkopecek commented 3 months ago

But why it happens only sometimes is weird to me. Maybe some other test can change LANG?

tkopecek commented 3 months ago

Maybe this could interfere: https://pagure.io/koji/blob/master/f/tests/test_cli/utils.py#_199

tkopecek commented 3 months ago

Yep, seems that if I disable that settings I can get 100% error there. And if coverage is disabled it also output sensible error.

So, maybe:
1) Drop that setting from CLI test and just set it for whole testsuite
2) Maybe not run coverage by default and make it a separate env/test in tox.ini? Disabling it speeds tests a bit and most time people are not looking to coverage output. So, running it on demand could make sense?

1 new commit added