freeipa

#8116 Pylint parallel execution with custom plugin
Closed: fixed by abbra. Opened by slev.

Closed: fixed
Reopen Issue

According to Pylint docs:
http://pylint.pycqa.org/en/latest/user_guide/run.html#parallel-execution

There are some limitations in running checks in parallel in the current implementation. It is not possible to use custom plugins (i.e. --load-plugins option)...

That is the IPA case:
1) jobs count is set in pylintrc as:

jobs=0

2) Pylint is triggered by the tox task or make targets :

--load-plugins pylint_plugins

Since Pylint-2.4.3 I have unstable linting results for the IPA code on system having many (for example, 32/64/128) CPU cores, for example:

************* Module ipalib.request                                             
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'int' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'bool' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'dict' has no 'disconnect' member)

'Unstable' means that one time - PASS, another - FAIL, different errors...
While on systems with 2 cores (for example, Azure Pipelines) errors are rare, but happen.
The previous Pylints somehow 'work' in all clauses.

Next, I've checked Pylint 2.3.1 and 2.4.3 in one process mode.
Here are new R ((R) refactor, for bad code smell) warnings:

************* Module ipalib.install.kinit
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.frontend -> ipalib -> ipaclient.remote_plugins -> ipaclient.remote_plugins.schema))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.plugins.rpcclient -> ipalib.rpc -> ipalib -> ipaclient.remote_plugins))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipaclient.frontend -> ipalib -> ipaclient.remote_plugins -> ipaclient.remote_plugins.compat))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipatests.pytest_ipa.integration.config -> ipatests.pytest_ipa.integration.env_config))
ipalib/install/kinit.py:1: [R0401(cyclic-import), ] Cyclic import (ipatests.pytest_ipa.integration.config -> ipatests.pytest_ipa.integration.host -> ipatests.pytest_ipa.integration.env_config))

Obviously, that we should run Pylint in 1 process mode.
Of course, this leads to more longer linting, for comparison:
1 process - 14:41 min
2 processes - 7:48 min
4 processes - 4:19 min

Upstream ticket:
https://github.com/PyCQA/pylint/issues/3232

Current status.
1) The single process mode's issue which was tracked at https://github.com/PyCQA/pylint/issues/3245 was fixed at https://github.com/PyCQA/astroid/pull/733 (merged)
2) request.context issue exposed as

************* Module ipalib.request                                             
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'int' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'bool' has no 'disconnect' member)
ipalib/request.py:78: [E1101(no-member), destroy_context] Instance of 'dict' has no 'disconnect' member)

Fix proposed in https://github.com/freeipa/freeipa/pull/3783.

3) Last known issue is pylint#3232
Need to investigate deeply, but there are strong arguments:

https://github.com/PyCQA/pylint/blob/master/pylintrc
https://github.com/PyCQA/astroid/blob/master/pylintrc

Both projects use themselves with jobs=1...

master:

  • 292d686c0b7935dd219559b10de3a393b712a990 pytest: Migrate xunit-style setups to Pytest fixtures
  • fec66942d469188fcd1ccfbd6e14f4e8334056b4 pytest: Migrate unittest/nose to Pytest fixtures
  • 8c7447fd42548d28df6200e354afdc03355b94b6 pytest: Warn about unittest/nose/xunit tests
  • 43a97082bbdf5992dcd355fdcb28ff3bfe66ab59 Update Azure Pipelines to use Fedora 31
  • 19462788f162491e0aedaaf42528cae7104b8068 ipatests: Properly kill gpg-agent
  • 92b440a0baf590fcb3082dfe2ca673813e1e5b7c pylint: Teach Pylint how to handle request.context
  • e128e7d691d0102460a54359dc8855fb7fcd7f35 pylint: Synchronize pylint plugin to ipatests code
  • a309de6c08987b1cf9645111414d1c52b4525190 pylint: Clean up comment
  • ba12165eaf76e6fee30b05b0fe939e13854d70dc lint: Make Pylint-2.4 happy again

Metadata Update from @abbra:
- Issue close_status updated to: fixed
- Issue status updated to: Closed (was: Open)

ipa-4-8:

  • 87bc31464b6133af9befd412af54403665c22628 pytest: Migrate xunit-style setups to Pytest fixtures
  • 356f907fc255ab3a9f93ff2808646b92a6652aec pytest: Migrate unittest/nose to Pytest fixtures
  • 3659b46d6aeea06b4875860ec69a9215afcbdd91 pytest: Warn about unittest/nose/xunit tests
  • f4e2acd1333f0f3d88da81d3fda80e85c9c418c2 Update Azure Pipelines to use Fedora 31
  • 294694ad69fa909e2f699cb2dad0f36b966a246f ipatests: Properly kill gpg-agent
  • 5939c90752db9da1adaf8c0bfe6bec3d6c1e2ad6 pylint: Teach Pylint how to handle request.context
  • 3460db4ee7c7ce6c9a639a644a39c4df09ce31ac pylint: Synchronize pylint plugin to ipatests code
  • 6f48848562f4e9ab9584154fd85e6ad1ac331ecd pylint: Clean up comment
  • 44a59ff39a3f481e90043e546c892c9108231d67 lint: Make Pylint-2.4 happy again
Metadata
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.