#505 postprocess: Workaround for Python timestamp mismatch
Merged by siosm. Opened by siosm.
Unknown source main

Download 505.patch

Python mtime timestamp logic does not work well with ostree force 0
mtime.

Use a small script to post-process all compiled '.pyc' and force the
timestamp to 0.

See: https://github.com/ostreedev/ostree/issues/1469

Co-Authored-By: Miro Hrončok miro@hroncok.cz

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/76269db66ce5463683a8c183e59c1b68

rebased onto 5aaeaa435986691e3133b69e5a0e3a432bb3d632

This function is missing return 0 at the end (when invalidation wasn't ZERO). It now implicitly returns None in that case and will blow up if such .pyc file is found because you cannot add None to an integer.

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci
https://fedora.softwarefactory-project.io/zuul/buildset/6599d1a571744ce2be763456440d5450

The exact thing happens on the CI:

Traceback (most recent call last):
  File "<stdin>", line 34, in <module>
TypeError: unsupported operand type(s) for +=: 'int' and 'NoneType'

Style nitpick: ALL_CAPS are usually used for constants, this isn't a constant (but regex is).

rebased onto 0728189946c058c007b53b03b143d47d07e434f3

rebased onto d085e9fecccaf5d0c164b8a67d3b046dc36df39f

Build succeeded.
https://fedora.softwarefactory-project.io/zuul/buildset/78bd612d79104b49ad8957afce6d3d85

Processed 4855 pyc files \o/

On another look, I guess this len check is redundant because if the len is less than 4, it will never equal ZERO (which is len 4).

Let's give this one a try. I'll merge it so that it lands in Rawhide and when can check if it works as intended or fix it. Thanks a lot @churchyard !

Pull-Request has been merged by siosm

It looks "better" on latest Rawhide but not fully fixed:

$ python3 -v -c 'import mailbox' 2>&1 | grep '# bytecode is stale for' -A2
# bytecode is stale for 'urllib'
# code object from /usr/lib64/python3.12/urllib/__init__.py
# could not create '/usr/lib64/python3.12/urllib/__pycache__/__init__.cpython-312.pyc': OSError(30, 'Read-only file system')

On Rawhide.20240413.n.0:

$ sudo find /usr/ -ipath "*/__pycache__/*.cpython-*.pyc" | wc -l
4988

From https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20240413.n.0/logs/x86_64/Silverblue/ostree-7/create-ostree-repo.log:

Processed 4885 pyc files

Weird.

Shouldn't this be != ZERO ?

Shouldn't this be != ZERO ?

Nah. The second 32bit word is a bitfield that encodes the type of invalidation. ZERO (empty bitfield) means this pyc file is timestamp-invalidated.
Then, and only then, we know we can zero-out the third 32bit word, which carries the mtime.

Nothing special about that file on a regular rawhide system:

>>> REGEX.match(path)
<re.Match object; span=(0, 65), match='/usr/lib64/python3.12/urllib/__pycache__/__init__>
>>> f = open(path, "r+b")
>>> w = f.read(4)
>>> w
b'\xcb\r\r\n'
>>> magic = (w[0] + (w[1] << 8) + (w[2] << 16) + (w[3] << 24)) & 0xFFFF
>>> magic
3531
>>> invalidation = f.read(4)
>>> invalidation
b'\x00\x00\x00\x00'
>>> w = f.read(4) 
>>> mtime = (w[0] + (w[1] << 8) + (w[2] << 16) + (w[3] << 24))
>>> mtime
1712707200

:/

OK, this is weird then. Feels like we're missing something.

We can print the paths we are zeroing-mtimes for and see if we skip this one or if it is zeroed and later unzeroed by something else.

https://pagure.io/workstation-ostree-config/pull-request/507

Next step in https://pagure.io/workstation-ostree-config/pull-request/536

Metadata