I'm getting tired of being prompted by GNOME Software to install updates every day after turning on my computer.
Some thoughts:
We are thinking of reworking the update cadence for Silverblue as well. I've talked to Fedora QA here in Brno and they're not opposed, but first we really have to define everything - https://teams.fedoraproject.org/project/silverblue/epic/95
This is how it works right now:
Every hour, gnome-software wakes up. It looks at the system clock and if it's past 6 am, goes and downloads new update metadata. It rate limits itself to doing it once per day to avoid excessive downloads.
Once metadata is downloaded (doesn't matter if there was any change or not), it then proceeds to download any available updates.
Once updates are downloaded, it then goes and installs any flatpaks it downloaded.
Regular rpms are immediately prepared as an offline update and made available. Once that is done, gnome-software then looks at the set of available updates; if there are any security updates, then it pops up a notification to install them. If not, then it waits until a week has passed since the last update and only then pops up a notification to install offline updates.
@kalev Every time I login (including merely logging out and immediately logging back in), PackageKit consumes 100% CPU for a minute, laptop fans kick in, and Software uses about 1/3 of that for a slightly shorter time. If Software could check once per day, rather than once per day per login, that'd be neat.
Could the downloading of packages could be delayed by ~15 minutes; when I do testing, very often they're throw away VMs. I can't change this behavior via gsettings fast enough. And it also blocks reboot/shutdown.
The last paragraph describes downloading RPMs, and possibly waiting a week to install them. On Fedora that likely means downloading RPMs that will become stale in a week's time. I take it the RPMs themselves contain the metadata to distinguish them as security updates vs regular updates, and that this information isn't in repo metadata?
No, that information comes from the repo metadata. gnome-software downloads them just in case so that they are ready if the user should want to install them earlier before a week has passed.
We are talking about a released Fedora though. How many rpms are revving on a more-than-weekly basis post GA? Should not be that common, really
Should not be that common, indeed.
Still, I think it doesn't really matter. We don't need to preemptively download updates so that the user can update quicker if she manually decides to apply updates. For most users, manual checking for updates should be quite rare, and it's OK to wait a bit for downloads when that happens. I suspect we really just want to ensure we download all updates before proposing a prepared update, and that only needs to happen weekly. Downloading the updates early annoys frequent dnf users, and the benefit of doing so is very limited, so might as well wait until we are actually going to prepare and propose an update before doing so. Fair enough?
Thanks for this summary.
We've previously established that we have security updates almost every day, and this is much too fast. It is extraordinarily rare for security updates to fix vulnerabilities that are being actively exploited, and such fixes should be marked urgent.
Proposal 1: Wait until a week has passed to prepare updates (including downloading updates), unless the updates are urgent priority. Do not treat security updates specially. Urgent updates should be marked urgent.
Proposal 1a): As proposal 1, except two weeks instead of one week. (I split this out separately since I expect it will be controversial, but I really do want to slow things down a bit more.)
Proposal 2: No notifications until one day after an update has been prepared. The user should have the opportunity to apply updates at power off or reboot time, when it's less inconvenient, before being pestered with notifications.
Proposal 3: Figure out a technical solution to the bug preventing the power off dialog from displaying updates reliably.
We don't need to preemptively download updates so that the user can update quicker if she manually decides to apply updates. For most users, manual checking for updates should be quite rare, and it's OK to wait a bit for downloads when that happens. I suspect we really just want to ensure we download all updates before proposing a prepared update, and that only needs to happen weekly. Downloading the updates early annoys frequent dnf users, and the benefit of doing so is very limited, so might as well wait until we are actually going to prepare and propose an update before doing so. Fair enough?
No.
How do you suggest doing that ? We propose an offline update on the logout dialog. That is not the time to go off downloading the updates. We may be offline, etc, the user may want to reboot quickly, etc etc.
Not making the user wait, ever, has been one of the core design principles behind g-s, from the very beginning. We haven't been entirely successful at that, but I don't think we should give that principle up now.
I think our current behavior is already basically following what you want to achieve - we prepare an offline update early, so users get a chance to install it at the 'less inconvenient' time every day (the poweroff dialog), and we don't pester them with notifications until a week has passed.
The one proposal that seems worth discussing is recalibrating what we consider security updates that are worth notifying about.
If I install Fedora 31 today, reboot, go through g-i-s, immediately over 1GiB's worth of updates start downloading and I cannot stop it without a forced reboot. A normal reboot is held up with a systemd wait job on PK (downloading metadata or RPMs or both, not sure). And 4 out of 5 times I'm doing throw away installs, I don't need those updates. And also during beta, it's similar because u-t is enabled but we're also in freeze, so u-t is accumulating updates that need to be applied by default.
These are edge cases, and QA folks are used to it, but I also do throw away Windows and macOS installs and don't run into such an aggressive need to immediately download updates. I'd say UX wise, GNOME Software updates are almost as nice as macOS except for the CPU usage and inhibited reboot. It's way better than Windows updates, which I find exceptionally irritating.
Perhaps instead of a delay downloading the RPMs, if PK could self-throttle CPU and bandwidth usage for this task? Making my laptop hot on every login is just not a good UX. And if it could be interruptible so I can reboot now without having to do a force power off that'd be nice also. I don't think updates need to be available so quickly. Windows and macOS, the first update after a clean install isn't available for maybe an hour (not least of which is that it takes that long to download their incredibly massive updates.)
No. How do you suggest doing that ? We propose an offline update on the logout dialog. That is not the time to go off downloading the updates. We may be offline, etc, the user may want to reboot quickly, etc etc.
No no, that's not what I meant. Of course I agree we should never propose an update before all packages have been downloaded.
According to Kalev's explanation, Software currently downloads updates daily even if it's not scheduled to propose an update for another week. There's not much benefit to doing that as it only saves time for the user if the user manually runs a check for updates in GNOME Software at a time when the user has not been prompted to install a prepared update. Correct? That's really just a tweak to our current behavior, not a major change and not even an essential change. The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software....
Then maybe I misunderstood this point: "Regular rpms are immediately prepared as an offline update and made available." Does "made available" mean the update is immediately presented on the power off dialog, without first waiting for a week since the previous update? In that case, users will still wind up with daily proposed updates when powering off, just not daily notifications. My suggestion is to propose updates no more often than weekly (or biweekly). Then wait some additional time (suggestion: one day, could be even more) before nagging the user with a notification.
The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software....
The cache is not shared because dnf doesn't have a supported api for doing so. I don't think we should accept the blame.
Does "made available" mean the update is immediately presented on the power off dialog,
That is my understanding (modulo the current gnome-shell bug). Kalev, correct me if thats wrong.
Yes, that's correct.
Older discussion in #28
Metadata Update from @chrismurphy: - Issue untagged with: meeting
@churchyard has started: RFC: Security policy adjustments to make it easier to implement and more friendly to maintainers
While that's not directly related to this issue, if there's anything policy, infra, or RPM related that we want or need to better handle this, now might be a good time to take it into account.
The benefit would be to hopefully reduce the number of complaints we receive from 'dnf update' users. Fairly or not, it's GNOME Software that gets blamed for performing duplicate downloads and not sharing a package cache with dnf; for whatever reason, nobody seems to blame dnf for not sharing with GNOME Software.... The cache is not shared because dnf doesn't have a supported api for doing so. I don't think we should accept the blame.
Actually, you should accept the blame, as the fault mostly lies with the PackageKit developers and the DNF team not working together specifically to fix this problem. As the GNOME Software developers are also the PackageKit developers, this is partly their fault. There is also blame to be had with the DNF team, as they have not tried to rectify this either.
As for why nobody blames DNF, it's because it's not visibly DNF's fault. DNF is the package manager. Things are supposed to integrate with it, not the other way around. And all other interfaces for DNF use the shared cache in /var/cache/dnf: dnfdragora, simple-dnf, dnf-automatic, and of course dnf itself. If every frontend to libdnf actually used their own caches, then it might be a different story.
/var/cache/dnf
dnfdragora
simple-dnf
dnf-automatic
dnf
libdnf
But technically the PackageKit backend could be configured to use the same cache as every other libdnf frontend. The main problem is that we don't have an API to "lock" the package manager to a single instance, as far as I know. @jmracek or @dmach might know more there.
@mclasen the DNF API for sharing the cache exists - it's the cachedir option. PackageKit never used it. The required change needs to be made in PackageKit, libdnf or both. If you're interested in switching PackageKit to using cachedir, let us know. Please note that we'll need a commitment from your team to test PackageKit properly, debug any new issues and help us fix them in libdnf. As @ngompa suggested, it will also require proper locking and we need to make sure that running dnf clean all doesn't crash PackageKit etc.
cachedir
dnf clean all
To be honest, we are not planning any PackageKit improvements because it's a dead project for about a year.
The DNF team is planning to stop supporting the libdnf API (in upstream, support in Fedora stays for a while) used by PackageKit shortly and we'll work on a new daemon replacing PackageKit. This new daemon will share cache with DNF. I'm going to send an announcement with more details to fedora-devel in about a week.
I made a note for myself to review this issue and design an update policy for the new daemon.
Does GNOME Software have a way of acting on severity=urgent differently from any of the other severity levels? (high, medium, low) https://github.com/fedora-infra/bodhi/blob/develop/bodhi/client/bindings.py#L262
And are package severity levels being reliably set? Are the updates tagged urgent really urgent, or are they rather high severity?
Yes, the data is there, but PackageKit doesn't support different security levels in its current API, so gnome-software doesn't know anything about them.
Metadata Update from @chrismurphy: - Issue tagged with: meeting
@mclasen the DNF API for sharing the cache exists - it's the cachedir option. PackageKit never used it. The required change needs to be made in PackageKit, libdnf or both. If you're interested in switching PackageKit to using cachedir, let us know. Please note that we'll need a commitment from your team to test PackageKit properly, debug any new issues and help us fix them in libdnf. As @ngompa suggested, it will also require proper locking and we need to make sure that running dnf clean all doesn't crash PackageKit etc. To be honest, we are not planning any PackageKit improvements because it's a dead project for about a year.
Yes, too late now for PackageKit. Back when PackageKit was actively developed, there was no api to share the cache, afaik.
The DNF team is planning to stop supporting the libdnf API (in upstream, support in Fedora stays for a while) used by PackageKit shortly and we'll work on a new daemon replacing PackageKit. This new daemon will share cache with DNF. I'm going to send an announcement with more details to fedora-devel in about a week. I made a note for myself to review this issue and design an update policy for the new daemon.
It would be good to make sure that the new daemon supports all the PackageKit use cases. In particular, the offline update dance is complicated and will requires attention to avoid regressions.
General agreement to reduce the frequency of non-urgent updates to not more than once per week (TBD); and there needs to be a reliable mechanism to indicate truly "urgent" severity updates (possibly entailing automation such that Bodhi requires a bugzilla with CVE reference, to consider an update urgent, otherwise it's demoted to "medium" severity).
ack/nack/patch?
@mattdm is on vaca this week, so tentatively push this issue back to 03 March to give him some time to dig out upon return.
My vote: all non-urgent updates get batched every two weeks. (Weekly is still a bit much!) Start with client-level batching in gnome-software, because that's what we can do today. Ideally in a better future, bodhi would learn to batch updates itself once again, so we can have lightly-tested "update packs" that get sanity-checked by OpenQA.
The definition of what makes a security update urgent is a bit hazy. In general, I'd say only critical-severity CVEs are urgent, or important CVEs actively exploited in the wild (very rare), or CVEs that are receiving special media attention (such that the CVE is an urgent PR issue). I'd also say the update is not urgent if mitigated by a sandbox (so e.g. critical CVEs in Firefox are not urgent, unless the CVE is a sandbox escape). I suggest we trust maintainers to set the priority level of their updates appropriately, using their best judgment, and in accordance with considerations that we document in the Fedora updates policy (which might look like what I just wrote). Abuse can be reported to FESCo to be handled on a case-by-case basis.
None of the above applies to flatpaks. Flatpaks can be updated automatically at any time. In the future, we might want to selectively begin shipping some default apps as flatpaks, e.g. Firefox. (But probably not before we have debuginfo for Fedora flatpaks, and ABRT integration.)
Is there a useful way to informally parse repo metadata over the next 3-7 days, for "urgent" severity updates? Would this help estimate whether this designation might be useful or problematic?
That's probably not smart to make that assumption. Flatpaks are just as dangerous given that they can easily destroy user data with the sandbox. That's what people care about, and you're suggesting that doesn't matter, when it probably matters more to people.
General agreement to reduce the frequency of non-urgent updates to not more than once per week (TBD); and there needs to be a reliable mechanism to indicate truly "urgent" severity updates (possibly entailing automation such that Bodhi requires a bugzilla with CVE reference, to consider an update urgent, otherwise it's demoted to "medium" severity). ack/nack/patch?
One point that was raised yesterday: urgent doesn't necessarily mean "security issue of a certain severity". That could potentially invalidate the "requires a bugzilla with a CVE ref" approach.
The main difficulty in all this is the word "reliable". We need to be realistic in our expectations for what can be achieved while individual maintainers are in control of what gets updated when.
If updates that genuinely need to be pushed out urgently are rare, wouldn't it be possible to have some kind of centralised process for that? If the update needs to go out urgently, the maintainer has to ask person foo and they push a special button.
There is a proposal for updates policy on Fedora Silverblue, most of it can be adapted even for "vanilla" Workstation.
https://docs.google.com/document/d/1gz1inA8e_LQP-dO751bkpS_ZVVLFAqYwFiRlqKJ3G9g/edit?usp=sharing
Comments welcome :)
Very rough draft outline of steps for implementing this. Feel free to reorder, restate, or do a rewrite, fill in details - I'm just trying to get this started so we're prepared for what we know and don't know, what we need but don't yet have, gotchas, etc.
https://etherpad.gnome.org/p/ReconsiderUpdatesPolicy
I'm confused. If gnome-software doesn't know about severity levels, why are update notifications happening so often? In particular, this diagram's "Are they critical?"
What is asking and answering that question?
Metadata Update from @chrismurphy: - Issue tagged with: meeting-request
@mattdm is scheduled to join us for the 17 Mar meeting.
This was just a typo. Kalev meant to write that PackageKit doesn't currently support different severity levels.
It does distinguish between security and non-security updates.
Yes, the data is there, but PackageKit doesn't support different security levels in its current API, so gnome-software doesn't know anything about them. This was just a typo. Kalev meant to write that PackageKit doesn't currently support different severity levels. It does distinguish between security and non-security updates.
This was just a typo. Kalev meant to write that PackageKit doesn't currently support different severity levels. It does distinguish between security and non-security updates.
Yes and no: packagekit has different severity levels, and one of them is a "security" severity level. So it does support distinguishing between security and non-security updates, but doesn't distinguish between different security update severity levels.
I see in Bodhi (both web and CLI views) these two fields
But they are separate fields. Near as I can tell from the interface, they relate only by convention or implication. I think severity describes the update, not the type field value. But...I don't know.
Right. But in PackageKit it's all mangled up, so the update type/severity goes like [ unknown, low, normal, important, security ] (or something along those lines, I'm writing from memory).
This is all of course just implementation details and should be easily fixable if we manage to convince hughsie to include new API in PackageKit (it's in deep maintenance mode right now).
I'm pretty sure PK does support the various update states; it's in the API: https://github.com/hughsie/PackageKit/blob/master/lib/packagekit-glib2/pk-enum.h#L622 -- although I don't believe the DNF backend actually sets it. From memory PK uses the same states that were defined by yum all those years ago.
But it seems there are too many security updates
I don't think you can ask people filing bugs to not mark them as security, nor do I think you can ask the user to specify a CVSS score they consider "important enough".
None of this is a problem. We don't want to change how packagers prepare updates, only how GNOME Software prioritizes them. My proposed new desired behavior is:
I like that plan, especially given it's easily implementable without needing PackageKit API changes. +1 from me.
I do think we need some updated notifications though so that people wouldn't be wondering why they are getting update notifications before 2 weeks have passed.
Currently the update notification reads: Software Updates Available Important OS and application updates are ready to be installed [Not Now] [View]
Can we somehow change the notification so that it says when there's an urgent update? And maybe remove "important" from the regular notification when no urgent updates are included?
Some ideas, Urgent Software Updates Available Urgent Firefox, KDE filesystem, and 5 more updates are available [Not Now] [View]
So basically just adding "urgent" for urgent update notifications and spelling out the names of the urgent updates, maybe? Or should we handwave "KDE filesystem" in some way to avoid spelling out system components?
I'll note that when urgent updates are available, we should be always applying all updates, not just cherry picking the urgent ones (this can easily lead to broken systems as we don't QA cherry-picked combinations).
I do think we need some updated notifications though
I'd like to do a review of all the updates notifications - upstream issue for this.
One question about the proposal: what about if we have updates we want to push out immediately after a release?
If you're upgrading from a previous release, you get the updates as part of the distro upgrade.
If you've just installed fresh, you should be notified about updates immediately because there has not been a successful system update during the past two weeks (or ever).
I like the proposal so far. A possible glitch I can't fully estimate:
$ bodhi updates query --severity=urgent --rows 100
Firefox is often urgent. Sometimes the kernel is. I figure a default installation might get an urgent update once every two weeks, variably. That's better than now. But a more "aged" system may see urgent update notifications more than once per week, possibly back to back days.
Metadata Update from @chrismurphy: - Issue untagged with: meeting-request - Issue tagged with: meeting
Agreed: the WG likes the GNOME Software proposal.
We'll continue discussing with Matthew Miller next week.
Clarification requested. Bodhi provides type distinct from severity. Is the idea that the user is notified of severity=urgent updates, regardless of type=security,bugfix,enhancement,newpackage? Or only when type=security and severity=urgent?
A question and a comment...
Do we have data on how often updates are tagged with "urgent"?
And the comment: urgent bugfixes should probably also be considered as exceptions, because they may be needed to prevent data loss. But we should also discourage packagers from tagging things as urgent in lesser cases. If it turns out that that's a problem. I really have no idea right now.
Whenever severity=urgent, because an urgent update is urgent and delaying urgent updates doesn't make sense.
We won't look at type=security at all.
So that's 100 updates over ... 243 days. Not so great for reducing notifications. I guess that should be filtered out by "packages on the workstation default install set". Some of them seem like they're pretty obscure in practice. Too bad we don't have popcon.
We'll certainly need to ask maintainers to use it more sparingly. E.g. a Firefox update is not urgent unless it fixes a Linux sandbox escape, or a truly critical non-security bug like the recent spurious certificate verification failures. There will never be any shortage of vulnerabilities, but having a sandbox allows us the freedom to wait two weeks to update.
We discussed this today with mattdm and everyone seems to still be on board.
The remaining question that we didn't resolve during the meeting is whether we do updates every two weeks (my proposal), or weekly (as we do now). I suggest we go with two weeks unless somebody wants to advocate for the weekly option.
Metadata Update from @catanzaro: - Issue untagged with: meeting
General consensus to proceed with this change to GNOME Software behavior:
Additionally:
@aday, regarding your gnome-software design changes, could we change the end session dialog to prompt to install updates by default? i.e. users would have to manually uncheck the checkbox if they don't want to install updates, instead of having to proactively take action to do so?
<img alt="Screenshot_from_2020-03-17_19-45-12.png" src="/fedora-workstation/issue/raw/files/b2313b8973e1cbde28b1f21d12283d70a19f610c65ef3d872e5ab54f85d31ad2-Screenshot_from_2020-03-17_19-45-12.png" /><img alt="Screenshot_from_2020-03-17_19-45-20.png" src="/fedora-workstation/issue/raw/files/e77db63cf604fc9092ef670d2dc282acf17dac1d96d3701714448a5b887ad17f-Screenshot_from_2020-03-17_19-45-20.png" />
@aday, regarding your gnome-software design changes, could we change the end session dialog to prompt to install updates by default?
Sure, makes sense
We agreed non-urgent updates will occur every two weeks.
Kalev, were you going to implement this in gnome-software?
We agreed non-urgent updates will occur every two weeks. Kalev, were you going to implement this in gnome-software?
Yes.
We're approaching the end of the current release cycle. Are you still targeting 3.38 for this?
My plan was to spend some time next week looking at this. Let's see how far we get with this then.
I guess this will slip to next cycle?
Yes, I think so at this point.
Metadata Update from @aday: - Issue set to the milestone: Fedora 34
Looks like this is still stalled.
@mcrha, is this something you could help with? The desired policy is:
regular update notifications will happen not more than once per week, ideally once every two weeks availability of "urgent" severity updates, of any type, will cause a notification to appear to inform and encourage the user to apply updates when there are urgent updates to apply, all available updates (any type and severity) will also be applied at the same time
Let's start with updates once per week for now. We could always switch to two weeks in the future, but I'm worried that delaying Firefox updates that long might result in unhappy users.
And to clarify the second point: GNOME Software should no longer prioritize security updates for immediate notification. It should only prioritize Urgent updates, regardless of whether it is a security update or not.
We have an ongoing discussion about this with @aday at the moment.
Hi Milan, any updates here?
Yeah, I plan to cover this within https://gitlab.gnome.org/GNOME/gnome-software/-/issues/182 , and I have it in my short term To Do, in fact I wanted to make a proposal the last week, but I've been distracted with other work, thus it has got postponed. I'm about to review a large change upstream and then I plan to look on this.
I guess this got fixed? @mcrha do you have any advice around testing these changes?
Yes, this is now fixed, in theory. Thanks so much Milan!
Testing updates is a little tricky. I think the best approach is going to be to keep a record of how often we are prompted to update in Fedora 34. If it's more than every two weeks, then that should be clearly attributable to a particular update that was marked as Urgent. If not, we have a problem.
I'm going to close this for now. If we think it's not working properly, then we can revisit, of course.
Metadata Update from @catanzaro: - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
I would personally like to have some kind of verification that the changes are working as expected. It could be as simple as documenting the intended behaviour and have a few people look out for updates notifications to see if they are behaving as expected.
Verification sounds good.
You can also count on me to take yell and start taking notes on my proposed updates if I suspect it's not working as intended. ;)
do you have any advice around testing these changes?
The only thing I can think of, apart of playing with the code, is to check the:
$ gsettings get org.gnome.software update-notification-timestamp
The other option is to run gnome-software with --verbose, which will print why it claimed, or why not, the update notification. Grep the output for should_notify_about_pending_updates.
--verbose
should_notify_about_pending_updates
One more thing, you can modify the GSettings key to simulate the day moves.
@mrcha thanks, that's useful! Could you give us a summary of the behaviour as implemented? That would help with testing.
It follows your design from the https://gitlab.gnome.org/GNOME/gnome-software/-/issues/182#note_970125
Reopening since Milan says this was not fully implemented, see https://bugzilla.redhat.com/show_bug.cgi?id=1930401#c22.
Metadata Update from @catanzaro: - Issue status updated to: Open (was: Closed)
To be precise, the upstream change does not change the classification of the updates, thus any security update is considered as a "critical urgency" and any "important" update is considered as a "high urgency". The code considers important both high & important updates.
This needs to be filled upstream [1] and discussed there first.
[1] https://gitlab.gnome.org/GNOME/gnome-software/-/issues
OK, but keep in mind Workstation WG only sets policy for Fedora. That is, even if upstream rejects the change, it still needs to be implemented in Fedora.
Upstream ticket
Thanks, I proposed a change there.
Thanks for looking into it, Milan. It seems your GNOME Software change is stalled pending PackageKit API changes, which nobody is assigned to implement. So this falls back to the Working Group....
Metadata Update from @catanzaro: - Issue tagged with: meeting
It seems your GNOME Software change is stalled pending PackageKit API changes, which nobody is assigned to implement. So this falls back to the Working Group....
Do we have an issue or PR for those changes? I don't see one.
Right, I kind of forgot to update this ticket with the findings from https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/648#note_1055290 and above that comment. Long story short: libdnf mimics bodhi (or vice versa), there's an update severity and an update kind. The problem is that the PackageKit doesn't expose (neither uses) the severity, it only exposes the kind.
Your request here is to recognize update notification importance solely on the severity.
The issue is interconnected, the libappstream doesn't have any such thing like severity, it also has only the kind. That's important, because gnome-software uses enums from the libappstream.
What to do depends how great (and clean) the change should be. I can foresee two options: either add the severity everywhere and use it instead of the kind for the notifications, or extend the kind with "urgent" and check only for the urgent severity in the PackageKit. The later has its limitations, as you can see. There can be other possibilities too, these are just two I can think of right now.
ACTION: Neal to discuss with Milan and determine whether PackageKit changes are really needed here
Metadata Update from @catanzaro: - Issue untagged with: meeting - Issue tagged with: pending-action
This is curious, because Cockpit exposes the severity and it also uses PackageKit, so I wonder how it's doing it if the PackageKit API isn't normally offering this.
<img alt="Screenshot_from_2021-03-16_08-18-12.png" src="/fedora-workstation/issue/raw/files/355ff70e5b8d40ffb00c753c13fd25aea37e47b8a1755022185a6325b08085b2-Screenshot_from_2021-03-16_08-18-12.png" />
Okay, so Cockpit just confuses severity with type. We may need an API extension in PackageKit, which should hopefully not be too bad to do...
As I said in today's meeting, one thing that we're obviously missing is a way to programatically test changes like this. Without that, it is impossible to a) know what users are going to get in terms of experience or b) iteratively improve the design.
That said, I do wonder whether having so much logic in the client is one of the reasons for these testing woes. If we do conclude that the current design approach is impossible to test that might be a reason to look at this whole subject afresh.
Metadata Update from @aday: - Issue untagged with: pending-action
Cockpit's 'severity' looks like it's bodhi's 'type' which is either newpackage, security, bugfix, enhancement. The number after the bugfix or security icon appears to be a reference to a rhbz associated with a particular updateid. I'm not sure how that is exposed via packagekit to cockpit, but it does not correlate with bodhi's 'severity' which is either unspecified, urgent, high, medium, low.
The gnome-software has some internal tests, but I do not see any similar for the PackageKit. The reason might be that providing updates for the PackageKit is not that simple as with the Flatpak plugin (the Flatpak plugin does quite a lot of things, including updates).
I do not know whether I understood your question correctly though.
In any case, libdnf returns only strings. PackageKit transforms the update type into an enum and it doesn't have anything like that for the severity (because PackageKit doesn't use severity at all, I suppose). If there can be assured the severity string will be one of the predefined values, then it'll make things much simpler. I'd not define the enum on the PackageKit side, it can be done on the gnome-software side instead. I do not know how to get the values libdnf can return, is that anyhow dictated by the bodhi itself? I checked the PackageKit code briefly and exposing the severity might not be a toy, depending how hackish the change would be (the change either requires to add a new parameter to pk_backend_job_package (it's used a lot and it's a semi-public API function) or add a new variant of it, with added severity argument).
pk_backend_job_package
That said, I do wonder whether having so much logic in the client is one of the reasons for these testing woes. If we do conclude that the current design approach is impossible to test that might be a reason to look at this whole subject afresh. The gnome-software has some internal tests, but I do not see any similar for the PackageKit. The reason might be that providing updates for the PackageKit is not that simple as with the Flatpak plugin (the Flatpak plugin does quite a lot of things, including updates). I do not know whether I understood your question correctly though.
That said, I do wonder whether having so much logic in the client is one of the reasons for these testing woes. If we do conclude that the current design approach is impossible to test that might be a reason to look at this whole subject afresh. The gnome-software has some internal tests, but I do not see any similar for the PackageKit. The reason might be that providing updates for the PackageKit is not that simple as with the Flatpak plugin (the Flatpak plugin does quite a lot of things, including updates).
One issue is that, since we don't batch updates on the server side, we implement the batching logic on the client side (say by waiting 2 weeks before notifying the user about updates). Practically speaking, that's a difficult thing to test.
I've filed a ticket with the DNF team for an API for severity: https://bugzilla.redhat.com/show_bug.cgi?id=1959066
The libdnf provides the severity, it's the PackageKit not passing it forward.
Just a note, I'm (slowly) working on the PackageKit side of this, adding the required API. Once done, it can be used on the gnome-software side. I'll update the above bug accordingly.
A little status update: * the PackageKit upstream accepted the change, it'll be part of the 1.2.4 release, if I'm not mistaken. What date it's planed for I do not know. * the gnome-software upstream accepted the change, which will be part of the 40.2 release, which will happen this Friday.
That means, once the PackageKit is built with the patch (either by adding it as a custom change to the 1.2.3 package or when the 1.2.4 is released) and gnome-software is rebuilt against it, the things will start working as is requested here. I do not have commit rights for the PackageKit package, but I'd not add the patch there myself anyway, because I do not maintain the package. Such things should be done by the maintainer.
Thanks Milan! I think it's sufficient for this change to land in rawhide after the next PackageKit release.
For Fedora 34, the priority is to fix the two major regressions (a) system updates no longer prepared automatically, (b) flatpak apps no longer updated daily. I see both are already fixed in gnome-software 40.2, which is good.
This ticket has the F34 milestone and there hasn't been an update for a couple of months. What's the status?
I've been testing development versions of Software and I do see daily automatic flatpak updates. It's a while since I've seen system updates prepared for me, though.
I've been testing development versions of Software and I do see daily automatic flatpak updates.
It seems to be updating a couple days behind schedule for me. Currently my Epiphany version is "41.beta-8-g796ef52d3+" which is from Saturday, but there were two new builds on Sunday and today is Tuesday, so this seems like clear proof that it's not working reliably. (There are also new builds today, but it's OK to be one day behind.)
Anyway, being off by a couple days on flatpak updates doesn't seem serious enough to keep this issue open, especially since this issue was originally intended to deal only with OS updates.
It's a while since I've seen system updates prepared for me, though.
This actually seems to be working for me:
$ gsettings get org.gnome.software install-timestamp int64 1628812036
Plugging that into https://www.unixtimestamp.com/index.php I find that I last installed updates on Thursday, which is well within our two week target. Note that PackageKit 1.2.4 is required for this to work reliably, and that just went out quite recently.
What results do you see, Allan? I was inclined to close this issue, but it sounds like you're not confident that GNOME Software is really preparing the updates.
Metadata Update from @aday: - Issue set to the milestone: Fedora 35 (was: Fedora 34)
I think we're probably good here, unless somebody suspects they are not seeing biweekly updates being prepared. Seems to be working for me.
Metadata Update from @aday: - Issue tagged with: testing
Test to see if this is working properly:
$ gsettings get org.gnome.software install-timesamp 1631660863
Convert it to a human-readable timestamp, e.g.: https://duckduckgo.com/?q=1631660863+unix+time&t=epiphany&ia=answer spits out "Tue Sep 14 18:07:43 2021 -0500".
If you power off your computer every day, and agree to install updates the same day you're prompted to do so, then this time should be no more than two weeks ago. (In my case, it's just under two weeks, so I'm expecting to be prompted to update tomorrow.)
If you do not power off every day, then I think there are a few extra days of delay before you receive a nag notification to install updates.
I briefly checked the code and if I read it properly, then the notification is shown up to once per day. To achieve that, a g_timeout_add_seconds is scheduled for 24 hours, which might be the weak point of this logic, because, according to the GLib documentation, this is using the monotonic time, which may or may not continue to tick during times where the machine is suspended.
g_timeout_add_seconds
Yeah that's not ideal, it should use wall clock time rather than monotonic time. What I would do is: schedule a g_timeout_add_seconds() every 10 minutes (this is arbitrary, could alternatively 60 minutes or something), then check the wall clock time in the timeout.
Better: change GNOME Software to quit when it's not doing anything, so it's not constantly running in the background, and schedule the check for updates using a systemd timer unit.
BTW I noticed yesterday that my F34 laptop has not been updated since mid-August, and I always apply updates when prompted, so something is still broken as of mid-August. I tried updating it manually and discovered that the Download button was broken in GNOME Software: it failed trying to download an outdated package version that no longer existed on Fedora mirrors. My solution was to press the Refresh button in the top-left corner of the header bar, after which the update worked fine. Milan, do you known if you've fixed any related bugs recently? I've updated it now and we'll see if it gets more updates two weeks from now.
Update on this: today is September 29, and I have not yet been prompted to install updates. So it's now been more than two weeks. This indicates something is still wrong.
I'm actually using the https://copr.fedorainfracloud.org/coprs/mcrha/gnome-software-devel/ copr, so my version of GNOME Software is theoretically newer and better than what we have in F34. (Of course, this invalidates the main goal of my test, which was to see whether the issue is fixed in F34.)
P.S. To avoid confusion: this comment references my main development machine, which is currently on F34 (I'll upgrade to F35 beta tomorrow probably) and using Milan's copr. This comment is unrelated to my previous comment, which refers to my F34 laptop (which I won't upgrade until F35 is released) that does not use the copr.
@catanzaro I filed https://bugzilla.redhat.com/show_bug.cgi?id=2009063 since I'm seeing the same thing. Flatpaks are automatically updated without prompt, and I'm informed by gnome-shell notification after the update that two apps have been updated. But there's no notification for pending RPM updates which have been downloaded.
Milan, do you known if you've fixed any related bugs recently? I've updated it now and we'll see if it gets more updates two weeks from now.
Those things are managed by the PackageKit, thus Id guess a change on that side is needed. I've nothing really recent, but I also do not know how old the PackageKit and gnome-software were before the manual update. As there had been some change to recognize when the updates are downloaded and when not.
What I would do is: schedule a g_timeout_add_seconds() every 10 minutes (this is arbitrary, could alternatively 60 minutes or something), then check the wall clock time in the timeout.
Yes, I see it the same. I'll open a merge request upstream soon.
Milan, do you known if you've fixed any related bugs recently? I've updated it now and we'll see if it gets more updates two weeks from now. Those things are managed by the PackageKit, thus Id guess a change on that side is needed. I've nothing really recent, but I also do not know how old the PackageKit and gnome-software were before the manual update. As there had been some change to recognize when the updates are downloaded and when not.
PackageKit has a serious problem, actually. I traced it down since this problem affected Plasma Discover during the Fedora Linux 34 timeframe too. Basically, PackageKit defaults to UINT_MAX for cache-age, and doesn't use the DNF backend's cache age settings. This caused RH#1950041 and RH#1903294.
I updated Chris' bug with a link to a merge request for the gnome-software. There had been an issue in the gnome-software code regarding the last notification timestamp reset. The merge request contains a change for the suspend as well.
Presumably we need to wait until GNOME 41.1 (scheduled for 30 October) with those latest changes, and then retest?
The COPR https://copr.fedorainfracloud.org/coprs/mcrha/gnome-software-devel/ contains the changes now, within the version gnome-software-42~alpha-0.20211004git1bf340de. Restart of the gnome-software process is required to take the changes into the effect.
gnome-software-42~alpha-0.20211004git1bf340de
Otherwise yes, it will be included in the 41.1 release.
Honestly, I would ship a git snapshot of the GNOME 41 branch in F35, so users can test this without needing the copr. We hit final freeze tomorrow, so there is no better time than now to batch up the latest fixes and get them into Fedora.
Running gnome-software 41.0 on F35, over 2 weeks have passed and I'm yet to get an updates reminder notification. Waiting on the 41.1 release to retest.
Please run:
$ gsettings get org.gnome.software install-timestamp $ rpm -qi gnome-software | grep -C1 Version
I believe this is expected to be fixed in 41.0-2.
I haven't either...or alternatively I've forgotten if it has notified me recently. It is automatically updating flatpaks, however, and I get a notification each time that happens.
$ gsettings get org.gnome.software install-timestamp int64 1633005263 $ rpm -qi gnome-software | grep -C1 Version Name : gnome-software Version : 41.0 Release : 5.fc35
So you last installed updates on Thursday, September 30. Two weeks after that was Thursday, October 14, so I would have expected an update to have been prepared no later than Friday, October 15.
One last question: have you powered off or rebooted your computer since then? If not, you might not notice the prepared update. It's supposed to wait some number of days (I forget how many) before creating a desktop notification to nag you to install the prepared update, in order to avoid unnecessary nagging. If you go to power off your computer now, does it have the checkbox allowing you to install updates? If not, that seems like concrete proof that something is still not working correctly.
$ gsettings get org.gnome.software install-timestamp int64 1633098127 $ rpm -qi gnome-software | grep -C1 Version Name : gnome-software Version : 41.0 Release : 1.fc35
I don't see the install updates check box in the power off dialog. I do reboot my machine from time to time.
You're going to have to update manually with dnf once, because the two-week timeout check was not working properly in 41.0-1.fc35. Milan fixed known issues in 41.0-2.fc35.
Now on 41.0-5.fc35. Will check back in 2 weeks. :smile:
One last question: have you powered off or rebooted your computer since then?
Many times.
If you go to power off your computer now, does it have the checkbox allowing you to install updates? I
Yes. Both restart and poweroff dialogs have a "install pending" option which is ticked by default. So I guess it's working.
Yes, that indicates it is working.
One last question: have you powered off or rebooted your computer since then? Many times.
Er, but this kinda suggests otherwise. You didn't notice this checkbox anytime between Friday and today? It should have been there since Friday.
If I'm not mistaken, the GNOME Shell's "Install pending updates" checkbox is tight to the /var/lib/PackageKit/prepared-update existence. That file sometimes disappears, maybe when the packagekitd is closed or something, I did not find when it happens.
/var/lib/PackageKit/prepared-update
If I'm not mistaken, the GNOME Shell's "Install pending updates" checkbox is tight to the /var/lib/PackageKit/prepared-update existence.
Correct.
That file sometimes disappears, maybe when the packagekitd is closed or something, I did not find when it happens.
I fixed that in https://github.com/PackageKit/PackageKit/pull/397. PackageKit needs to stay constantly running until it learns not to remove its prepared update. Looks like there were other problems with the update not working reliably if PackageKit is not running.
PackageKit needs to stay constantly running until it learns not to remove its prepared update. Looks like there were other problems with the update not working reliably if PackageKit is not running.
That can trick gnome-software. I noticed that the PackageKit claims "all packages for update are downloaded", which the gnome-software interprets as "nothing to download, can just install it", thus the Updates page shows "Restart & Update", even though the prepared-update file is missing, thus the PackageKit rejects the update. The "Download" button on the Updates page in the Software does make sure the prepared-update file is created. I did not try the latest version, I recall some semi-recent upstream gnome-software bug report about the similar thing, but I lost track of it.
It's possible that I might have got the update notification around the correct time, two days ago - that would be exactly 2 weeks since my last update.
I say it's possible because I was clicking around and the Software updates view showed up... I suspect that the notification popped up at the wrong moment under the cursor.
$ gsettings get org.gnome.software install-timestamp int64 1634746916 $ rpm -qi gnome-software | grep -C1 Version Name : gnome-software Version : 41.0 Release : 5.fc35
Do you have a prepared update?
There's no check box in the shell's power off / restart dialog.
Then that's QA fail. The requirement is that updates are prepared after two weeks, and that clearly did not happen for you. I'm removing the testing label from this issue, as Allan's experience is sufficient to show there are still problems here. I think we need an upstream bug report now to figure out why your update was not properly prepared.
testing
Metadata Update from @catanzaro: - Issue untagged with: testing
See: https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1114#note_1296453 for a possible reason. The change from there is part of the 41.1. It helps in realizing whether the update is prepared on the gnome-software side.
OK, so... I've added the testing label once again. We'll just have to wait another two weeks to see if it works properly with Software 41.1.
Metadata Update from @catanzaro: - Issue tagged with: testing
For what it's worth, I'm notified about the pending updates regularly, sometimes to download sometimes to install them. It's a development machine, thus might not count. I do not know.
I have gnome-software-41.1-1.fc35 installed. Today I received that "updates are waiting and ready to be installed." However, when I went to reboot, I discovered that no update was prepared. GNOME Software showed updates waiting to be downloaded, but this notification should be displayed only when an update is actually prepared, never when updates are merely waiting to be downloaded. So I think this is QA fail again....
Secondary problem: I received this notification first thing in the morning after I turned on my computer. Our notification policy is designed to make this unlikely by ensuring users are notified only a few days after the update has been prepared if automatic downloads are enabled.
I'm confused, you implemented the notification policy yourself, following the rules presented by Allan, but you don't know whether GNOME Software is following the policy or not? Allan, maybe it would help if you could dig up a link to your mockup that shows the rules for when to display notifications?
I'm confused, you implemented the notification policy yourself, following the rules presented by Allan, but you don't know whether GNOME Software is following the policy or not?
No, I did not mean it that way. I meant: "works for me", but as happened too many times in the past, things which work on my side not always work somewhere else.
Okay, after some more investigation in the https://bugzilla.redhat.com/show_bug.cgi?id=2026108 and re-reading the new design ( https://gitlab.gnome.org/Teams/Design/software-mockups/-/blob/master/updates-logic.png ), there was another bug regarding critical updates, fixed by https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1103 for 41.2+. As the security updates were not recognized, no early download was made. That means the code behaved like the design says for the top line of the offline update, which means the first download should happen only after 14 days of waiting after the last install, and then the user will be notified about pending updates every 3 days.
I have gnome-software-41.2-1.fc35, but my gsettings get org.gnome.software install-timestamp reports 1639354440, which is December 12. I generally apply updates the same day that they are prepared, so something is definitely still wrong. :(
gsettings get org.gnome.software install-timestamp
Perhaps for testing purposes we should temporarily switch to checking for updates every hour instead of every two weeks? Then we would know within an hour of an updates push if there is a bug, and we wouldn't need to wait weeks to report back that it's not working. We've had quite a few problems here and it seems like faster iteration is going to be required to get this working reliably.
Could you run in a terminal:
$ gnome-software --quit $ gnome-software --verbose &>log.txt
then wait say for 5 minutes, not doing anything in the gnome-software, and then:
$ gnome-software --quit
and provide the log.txt to me, please?
Sure, here it is:
<img alt="log.txt" src="/fedora-workstation/issue/raw/files/8f830d15e8df50bb58e57c83cfa3d036b1b86efdd9507d252ce5181ad14fa4ac-log.txt" />
Thanks. I'm a bit late here, I'm sorry. I see in the log there's running First hourly updates check with running get-langpacks after which should follow Getting updates, but it is not, there is only the Getting upgrades. I believe it's because the daily check already happened that day, as saved in the "check-timestamp" GSetting key. When you gsettings reset org.gnome.software check-timestamp, then the next start of the gnome-software process the "Getting updates" will be included in the log as well.
First hourly updates check
running get-langpacks
Getting updates
Getting upgrades
gsettings reset org.gnome.software check-timestamp
When I try it here, using gnome-software-41.2-1.fc35.x86_64, then the log contains the Getting updates, which takes some time to finish and it ends with:
13:13:47:0876 Gs running download on plugin=flatpak with dedupe-flags=7 with propagate-error=True with timeout=60 on apps system/package/fedora/org.gnome..... , elapsed time since creation 22669ms 13:13:47:0876 Gs should_notify_about_pending_updates: last_test_days:1 n-apps:12 should_download:1 has_important:0 any_important_downloaded:0 all_downloaded:1 any_downloaded:1 res:0 13:13:47:0876 Gs No update notification needed
The last_test_days:1 looks suspicious. Checking the install-timestamp it says Dec 9, the last year, which is much longer than two weeks. I do have both automatic updates and automatic updates notification enabled. I guess I've a setup, which reproduces at least one part of the problem.
last_test_days:1
install-timestamp
The last_test_days:1 looks suspicious.
Aha, no, it's okay. It means I've been notified yesterday, thus nothing is shown to me. See https://wiki.gnome.org/Design/Apps/Software/Updates#Tentative_Design , it's the three days period between notifications for the offline updates without a critical update.
Affected GSettings keys in org.gnome.software are: * install-timestamp the date/time, when the last install of the udpates was done * check-timestamp the last check for new updates - used also to not try more than once per day * update-notification-timestamp when the user had been notified the last time
check-timestamp
update-notification-timestamp
When I "correct" the changed keys to some previous day(s), then I get:
13:48:57:0869 Gs should_notify_about_pending_updates: last_test_days:4 n-apps:14 should_download:1 has_important:0 any_important_downloaded:0 all_downloaded:1 any_downloaded:1 res:1 reason:Software Updates Ready to Install|Software updates are waiting and ready to be installed. 13:48:57:0869 Gs Notify about update: 'Software Updates Ready to Install'
which is reflected in the Shell with that notification and I also see /var/lib/PackageKit/prepared-update. When I ask GNOME Shell to turn off the machine I see [x] Install pending software updates in the "Power off" dialog. I did let it update.
[x] Install pending software updates
One of the important values of the debug log is the all_downloaded:1 - that should reflect existing PackageKit's prepared-update file as well.
all_downloaded:1
prepared-update
My gsettings get org.gnome.software check-timestamp pointed to earlier today, so it is at least checking. I reset it and then obtained this new log. This new log does say "Getting updates" twice, so at least it's doing something this time.
gsettings get org.gnome.software check-timestamp
<img alt="log.txt" src="/fedora-workstation/issue/raw/files/c6e301ff6a914354876329e6a1adcd6fd0490656faa34bbe9acd51938517a587-log.txt" />
If I read it correctly, then you've got 19 updates, which had been (tried to) downloaded, but I do not see any running download in the log.
got 19 updates
running download
I made some more testing here, including clean up my PackageKit cache, thus everything needed to be downloaded. The gnome-software logs here:
08:58:21:0470 Gs got 3 updates 08:58:21:0470 Gs Received 3 apps to update, 0 are online and 3 offline updates; will download online updates 08:58:21:0470 Gs should_notify_about_pending_updates: last_test_days:0 n-apps:3 should_download:1 has_important:0 any_important_downloaded:0 all_downloaded:0 any_downloaded:1 res:0 08:58:21:0470 Gs No update notification needed
The log says will download online updates with 0 are online, thus there's is no download of the online updates.
will download online updates
0 are online
The offline updated are somewhat different and it can be my misunderstanding of the design. My branch is "non-critical updates". It says to wait for 14 days, then download what's new and then notify every 3 days about pending updates. The tricky part here is what to do when the user does not update on the 14th day. My understanding was that the updates should not be downloaded immediately after the 14 days waiting period, but when I think of it now, and what it seems you suggest, it should always download on the 15th day, on the 16th day, ... until the user installs, but still notify only every 3 days about pending updates. This will also ensure the "Install updates" check in the Reboot/Shutdown dialog of the Shell.
Could you confirm that, please? If agreed, I'll propose a patch upstream.
it should always download on the 15th day, on the 16th day, ... until the user installs, but still notify only every 3 days about pending updates.
Err, I had install-timestamp on yesterday, which is not 14 days. When I made it lot longer, it did call the download, and the code said all is downloaded, but the PackageKit did not prepare the offline update. I'm investigating this further.
Okay, if I read the code properly, then to download the packages the PackageKit plugin in the gnome-software calls pk_task_update_packages_sync with the task having set "download only". I can see it being used with the pkmon. This does not create the /var/lib/PackageKit/prepared-update file for some reason [*]. When I click the "Download" button in the Updates section of the GNOME Software, then I see in the pkmon the same two functions being called (get-updates followed by update-packages), but this time the prepared_update is properly created. When I cleaned up the PackageKit directories under the /var/ it created the prepared-update file as expected, after it downloaded all the packages.
pk_task_update_packages_sync
pkmon
get-updates
update-packages
prepared_update
/var/
[*] of course, the more I debug it I fail to reproduce it. There plays its role also the /var/lib/PackageKit/offline-update-competed which I had there, with the today date (which was then propagated into the install-timestamp GSettings key).
/var/lib/PackageKit/offline-update-competed
That being said, I cannot reproduce it right now. The conditions are quite complex, many things influence the behavior. I'd need to know all of that to see what could fail.
The offline updated are somewhat different and it can be my misunderstanding of the design. My branch is "non-critical updates". It says to wait for 14 days, then download what's new and then notify every 3 days about pending updates. The tricky part here is what to do when the user does not update on the 14th day. My understanding was that the updates should not be downloaded immediately after the 14 days waiting period, but when I think of it now, and what it seems you suggest, it should always download on the 15th day, on the 16th day, ... until the user installs, but still notify only every 3 days about pending updates. This will also ensure the "Install updates" check in the Reboot/Shutdown dialog of the Shell. Could you confirm that, please? If agreed, I'll propose a patch upstream.
Yes, after 14 days the prepared update should be available until the user installs it. The only reason an update should ever not be prepared after the 14-day period would be if GNOME Software is actively working on preparing it, or if an error has occurred.
What more, exactly, do you need from me? All of /var/lib/PackageKit?
Yes, after 14 days the prepared update should be available until the user installs it.
Thanks. Reading the code it does that, granted the prerequisites are satisfied.
No no, something else. Just run these three lengthy commands:
for key in install-timestamp check-timestamp update-notification-timestamp ; do val=`gsettings get org.gnome.software $key | sed 's/int64 //'`; dt=`date --date="@$val"`; now=`date +%s`; echo "$key: $dt; days:$((($now - $val)/(60*60*24)))"; done; echo "should download: `gsettings get org.gnome.software download-updates`"; ls -l /var/lib/PackageKit/
when the check-timestamp says days:0, then change it to at least a day back, then restart gnome-software, thus it tries again a minute after it is started.
days:0
I do this in this case:
gsettings reset org.gnome.software check-timestamp; gnome-software --quit; gnome-software --verbose
and after the log showed running get-updates on plugin=packagekit with .... followed by setting state from action-get-updates to idle (has-update:1, has-upgrade:0), I see a new /var/lib/PackageKit/prepared-update, which was not there when the days:0 (the machine was restarted meanwhile).
running get-updates on plugin=packagekit with ....
setting state from action-get-updates to idle (has-update:1, has-upgrade:0)
Here's what I have:
$ for key in install-timestamp check-timestamp update-notification-timestamp ; do val=`gsettings get org.gnome.software $key | sed 's/int64 //'`; dt=`date --date="@$val"`; now=`date +%s`; echo "$key: $dt; days:$((($now - $val)/(60*60*24)))"; done; install-timestamp: Sun Dec 12 06:14:00 PM CST 2021; days:37 check-timestamp: Wed Jan 19 07:36:08 AM CST 2022; days:0 update-notification-timestamp: Wed Jan 12 07:23:33 AM CST 2022; days:7 $ echo "should download: `gsettings get org.gnome.software download-updates`"; should download: true
Hm, I already did this before providing that second log yesterday. (I used gsettings reset org.gnome.software check-timestamp.)
Could you install this scratch build and then run from a terminal:
$ gnome-software --quit; \ gsettings reset org.gnome.software check-timestamp; \ gnome-software --verbose &>log.txt
and let it running say for 5 minutes, thus it has enough time to download the updates, please? Use gnome-software --quit again to gracefully stop it. The build adds some new debug prints (denoted with *** for easier searching in the log), which, I hope, will help to show why it behaves differently for you.
gnome-software --quit
***
Will do. Note: this upgraded from 40.2 to 40.3, so it's not the same version that I was testing before.
This time, it actually did prepare an update. Is it possible that you fixed something between 41.2 and 41.3? Otherwise, perhaps this is just bad luck.
Is it possible that you fixed something between 41.2 and 41.3? Otherwise, perhaps this is just bad luck.
Nope, the 41.3 contains only a single semi-cosmetic change related to an application details page.
Hm, a little while later, somehow my update has un-prepared itself. O_O I was just about to reboot to install updates, but the update is gone. I didn't touch dnf or do anything. That shouldn't happen.
I ran this a second time:
And it immediately prepared an update once again. So update preparation now seems to be working, but something unknown causes the update to get invalidated? I am going to refrain from installing this update for diagnostic purposes, because I want it to still be available when you look at this issue again.
Now my second prepared update has also disappeared. So even when the update does get prepared, it breaks somehow.
Looking in my journal for anything potentially-suspicious... could it be dnf makecache?
dnf makecache
The prepared-update file is fully under the PackageKit's control. If you connect gdb to the packagekitd process and place a breakpoint into pk_offline_auth_invalidate you might see from where it is called (some are idle callbacks).
packagekitd
pk_offline_auth_invalidate
There are few tasks, which can invalidate the update also from the Software (it's rather a side effect of those tasks), to name some: when the get-updates does not find anything; when calling refresh-cache, repo-set-data or repo-enable; on install/update/remove of a package, which is scheduled for an update. There are couple more places, I only briefly checked the PackageKit code.
refresh-cache
repo-set-data
repo-enable
when calling refresh-cache, repo-set-data or repo-enable; on install/update/remove of a package, which is scheduled for an update.
All of these should result in the update being prepared again, though, right?
Obviously installing some app should not cause an update to be delayed by two more weeks.
This is called even before any update is prepared:
(gdb) bt #0 0x00007fc261734cb0 in pk_offline_auth_invalidate () at /lib64/libpackagekit-glib2.so.18 #1 0x00005600af9e047d in pk_transaction_offline_finished (transaction=0x5600b1522770) at ../src/pk-transaction.c:996 #2 pk_transaction_finished_cb (job=<optimized out>, exit_enum=PK_EXIT_ENUM_SUCCESS, transaction=0x5600b1522770) at ../src/pk-transaction.c:1044 #3 0x00005600af9e7f56 in pk_backend_job_call_vfunc_idle_cb (user_data=<optimized out>) at ../src/pk-backend-job.c:588 #4 0x00007fc26161747b in g_idle_dispatch () at /lib64/libglib-2.0.so.0 #5 0x00007fc26161b130 in g_main_context_dispatch () at /lib64/libglib-2.0.so.0 #6 0x00007fc261670208 in g_main_context_iterate.constprop () at /lib64/libglib-2.0.so.0 #7 0x00007fc26161a853 in g_main_loop_run () at /lib64/libglib-2.0.so.0 #8 0x00005600af9db3df in main (argc=<optimized out>, argv=<optimized out>) at ../src/pk-main.c:245
It seems to be called very frequently when GNOME Software is running. Sadly, GNOME Software is presenting the "software is up to date" screen even though dnf says I have 1.1 GB of updates to install. It's been a seriously long time since my last update, due to this bug....
Screenshot demonstrating failure to prepare update (after running gnome-software --quit then gsettings reset org.gnome.software check-timestamp):
<img alt="Screenshot_from_2022-02-24_15-47-10.png" src="/fedora-workstation/issue/raw/files/390613a4566cd53a6bb7849a30c1c72161079f94aab4b20cbff813eb1618b9f4-Screenshot_from_2022-02-24_15-47-10.png" />
I do not know. I do not know PackageKit. If it's due to the way gnome-software calls it, then it should be fixed.
Sadly, GNOME Software is presenting the "software is up to date" screen even though dnf says I have 1.1 GB of updates to install.
I'd really like to know why it behaves differently for you than for me. It can be just a small thing. I saw similar issue recently on Silverblue, but it could be due to the rpm-ostree daemon doing other things in the background, which prevented gnome-software to show the update state. This is PackageKit, not rpm-ostree, I know.
I see this on my rawhide machine. Would you mind to file an upstream gnome-software bug, please? I do not know where the problem currently is, but I'd start there.
Hrm, I rebuild gnome-software and simply restart it and it shows me the updates. Wait with the upstream report, please.
Here is a test build for f35: https://koji.fedoraproject.org/koji/taskinfo?taskID=83455312
It prints on the gnome-software terminal some debug prints. You might know there are two calls to get updates. One is done after start to populate the Updates page, then, a minute after start, there is done a daily check, which can be postponed for several reasons - all are printed on the terminal, if they were used. The daily check calls also a cache refresh, with an acceptance of a day old cache. The cache age can be changed to 0 days, when the gnome-software is ran with GS_UPD=1 exported (the value does not matter).
GS_UPD=1
I'd probably rename /usr/bin/gnome-software from a text terminal before logging in as a user, to avoid a problem, which happened to me (the updates shown up after restart of the gnome-software process).
/usr/bin/gnome-software
Here is a test build for f35: https://koji.fedoraproject.org/koji/taskinfo?taskID=83455312 It prints on the gnome-software terminal some debug prints. You might know there are two calls to get updates. One is done after start to populate the Updates page, then, a minute after start, there is done a daily check, which can be postponed for several reasons - all are printed on the terminal, if they were used. The daily check calls also a cache refresh, with an acceptance of a day old cache. The cache age can be changed to 0 days, when the gnome-software is ran with GS_UPD=1 exported (the value does not matter). I'd probably rename /usr/bin/gnome-software from a text terminal before logging in as a user, to avoid a problem, which happened to me (the updates shown up after restart of the gnome-software process).
I didn't need to do this. I just ran gnome-software --quit, gsettings reset org.gnome.software check-timestamp, and then gnome-software as usual.
gnome-software
Here is the output:
$ gnome-software 14:16:46:0695 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.openh264/x86_64/2.0beta is already installed: No such ref 'runtime/org.freedesktop.Platform.openh264/x86_64/2.0beta' in remote gnome-nightly 14:16:46:0695 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.openh264/x86_64/2.0 is already installed: No such ref 'runtime/org.freedesktop.Platform.openh264/x86_64/2.0' in remote gnome-nightly 14:16:46:0696 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.GL.default/x86_64/21.08beta is already installed: No such ref 'runtime/org.freedesktop.Platform.GL.default/x86_64/21.08beta' in remote gnome-nightly 14:16:46:0696 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.GL.default/x86_64/20.08 is already installed: No such ref 'runtime/org.freedesktop.Platform.GL.default/x86_64/20.08' in remote gnome-nightly gs_plugin_add_updates: pk_client_get_updates returned 0 packages check_updates: daily update check, with cache age 1 days refresh_cache_finished_cb: going to get updates with refreshed cache 14:18:11:0385 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.openh264/x86_64/2.0beta is already installed: No such ref 'runtime/org.freedesktop.Platform.openh264/x86_64/2.0beta' in remote gnome-nightly 14:18:11:0386 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.openh264/x86_64/2.0 is already installed: No such ref 'runtime/org.freedesktop.Platform.openh264/x86_64/2.0' in remote gnome-nightly 14:18:11:0386 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.GL.default/x86_64/21.08beta is already installed: No such ref 'runtime/org.freedesktop.Platform.GL.default/x86_64/21.08beta' in remote gnome-nightly 14:18:11:0386 flatpak Warning: Treating remote fetch error as non-fatal since runtime/org.freedesktop.Platform.GL.default/x86_64/20.08 is already installed: No such ref 'runtime/org.freedesktop.Platform.GL.default/x86_64/20.08' in remote gnome-nightly gs_plugin_add_updates: pk_client_get_updates returned 527 packages get_updates_finished_cb: got 17 updates
Note that it did NOT prepare an update!
And even though it says "got 17 updates" the GUI says there are no updates available:
<img alt="Screenshot_from_2022-02-28_08-21-29.png" src="/fedora-workstation/issue/raw/files/288b9cc18edbb25690bbbedf73cfafdb69deffa06467cef72b81b72d82881f46-Screenshot_from_2022-02-28_08-21-29.png" />
This honestly seems like a different bug to me. Previously I was complaining that prepared updates were mysteriously invalidated. But now it looks like the update is not prepared at all.
I tried again with gnome-software --verbose in the hopes that more logging might help. This time it detected that updates were available to download, but it did not properly download them. I've attached this log: <img alt="log.txt" src="/fedora-workstation/issue/raw/files/b73ad7f0851e0174c8dbc94ba98dedc0f91511fb92279f3abda45903a53dcecb-log.txt" />
gnome-software --verbose
Thanks. The first gs_plugin_add_updates: pk_client_get_updates returned 0 packages means the local cache did not know about any update. The second call did refresh the cache and found some updates. The second call is not done within the Updates page - a bug can be the Updates page is not refreshed when the update monitor finishes its work. I opened https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1251 for it.
gs_plugin_add_updates: pk_client_get_updates returned 0 packages
The verbose log shows:
14:24:20:0929 Gs should_notify_about_pending_updates: last_test_days:0 n-apps:17 should_download:1 has_important:0 any_important_downloaded:0 all_downloaded:1 any_downloaded:1 res:0
After this there does exist /var/lib/PackageKit/prepared-update (the above merge request helps to propagate the change into the Updates page). These downloads are done only once per day, but they can be invalidated with some operations, as we've seen above (like in https://pagure.io/fedora-workstation/issue/107#comment-783282 ). The update is prepared again, but only the next day. The Software does not delete the prepared-update file on its own, as far as I know. You might want to consult PackageKit why it does that and what can be done to avoid its early deletion.
Or the Software can call "download/prepare updates" more than once per day, but I do not know when it would be.
The update is prepared again, but only the next day.
OK, this is probably part of the problem. When a prepared update is invalidated, I'd like a new update to be prepared right away, not the next day. Otherwise it could be invalidated every single day and we'll never present updates at all, right? I suspect that is a big part of the problem here, although I don't know what exactly is causing my updates to be invalidated.
I also suspect updates are not being prepared in the first place, though. (Maybe your MR 1251 will help with that?)
Does Software know when PackageKit invalidates an update? If so, then that would be the right time to prepare a new update. If not, maybe PackageKit should be responsible for preparing a new update when one is invalidated? I do not know.
PackageKit has a bug where it never invalidates anything by default right now (rhbz#1903294, boo#1196522), so any invalidation is triggered by GNOME Software.
I suspect that is a big part of the problem here,...
I agree, the prepared updates invalidation is bad. The file is private for the PackageKit, the gnome-software does not invalidate it on purpose. We/I can investigate it further, though rather on the PackageKit level, than on the gnome-software level.
I need to debug this more. If I recall correctly, the prepared update file is removed shortly after gnome-software is started. There is no install involved or any such thing, of course. The upstream merge request only ensures refresh of the Updates page in the Software, when needed, thus it reflects the current state.
Does Software know when PackageKit invalidates an update?
I do not think so. As I mentioned, it's a private file of the PackageKit.
PackageKit has a bug where it never invalidates anything by default right now
This is a different kind of invalidation ;) The gnome-software always set the cache age when calling refresh, thus it can be a different thing. Specifically the daily refresh from the update monitor in the Software sets the cache age to one day.
the prepared updates invalidation is bad. The file is private for the PackageKit, the gnome-software does not invalidate it on purpose. We/I can investigate it further, though rather on the PackageKit level, than on the gnome-software level.
Okay, so, after gnome-software start, when automatic updates are enabled, the gnome-software calls "refresh" on its plugins, which means it calls "refresh-cache" on the PackageKit. This refresh-cache operation invalidates the previously prepared offline updates, regardless it found anything new or not.
Any later refresh call can cause the invalidation of the prepared updates. There are more/other reasons to invalidate the files, this one is probably the most common. I suggest to open a PackageKit bug/enhancement request.
While debugging the above, I noticed pk_offline_get_prepared_monitor() exists, which let's the caller notify when that file changes/deletes/... - it returns a GFileMonitor. The PackageKit plugin in the Software can use it. How precisely, with respect of the design diagram, I'm not sure. https://wiki.gnome.org/Design/Apps/Software/Updates#Tentative_Design
OK good news, Software has prepared an update for me and it hasn't been invalidated yet. I've attached with gdb and set the requested breakpoint. We'll see if it gets hit.
I always apply updates when presented, but it's been over a month since my last system update, so I assume the prepared update gets invalidated every single day. I assume I'd have to be pretty unlucky to not catch it today. The only other possibility is that Software doesn't properly prepare the updates in the first place, and while there seems to be some weirdness indicated by the logs I took above, I don't think that's the main problem here.
Er, could you (or Neal) handle this please? You have a much better understanding than I do.
I've attached with gdb and set the requested breakpoint. We'll see if it gets hit.
I tried it here too. The only place, where the file was deleted for me, was a result of a finished refresh call. I could install an RPM application, then remove it, and the prepared-update file was left there.
I do not have any understanding why the refresh call deletes the prepared-update file. Maybe it makes sense, maybe not. Or it makes sense sometimes. I do not know.
The state diagram does not understand the concept of an invalidated prepared update. That's fine. The way for the code to respect that is to always prepare a new update whenever a prepared update is invalidated, if possible. E.g. if the prepared update was invalidated because the user applied all available updates via dnf, and now no updates are available, then of course it's not possible.
It might make sense to use a timeout, e.g. "try again five minutes later."
Does it really matter why the prepared update was invalidated, though? What really matters is that we expect invalidation to occur and handle it, regardless of why it happens. Right?
From my point of view, if the reason to invalidate the update is wrong, then the problem is on the PackageKit side. Adding "prepare update five minutes after it had been invalidated" would be just a workaround for that broken reason. It's still with a bold if at the beginning of this paragraph.
Let me think about something sane for the gnome-software for this.
Of course, we all know that there are technical reasons that might cause a prepared update to be invalidated, e.g. if the user runs dnf manually. But please remember that according to our design, there is never a reason to invalidate an update. We always want the update to be prepared until it is finally applied.
How you accomplish that is up to you: either by changing PackageKit to never invalidate updates (seems impossible?), by changing PackageKit to prepare a new update when it invalidates one (seems inconsistent with its current API design?), or by having GNOME Software prepare the update itself. We don't care how it's solved, so long as it operates by design.
Amazingly, the update is still prepared now at the end of the day when I'm powering off my computer. I'm going to apply it.
I opened https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1260 . From the commit message:
The auto-prepare of the update is triggered only if the file was deleted and the user has set to download updates. It's not triggered when the Software is started and there is no prepared update. It's because the file name is not exposed in the public PackageKit API.
I know it's a weak point, but the reason would make sense too, I hope.
Thanks Milan. I wonder if this might be all that we need to have updates working reliably again!
I've separately reported https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1669 for the separate issue regarding improper display of "up to date" on the updates page when we are not up to date. This seems to be only a cosmetic bug, so it shouldn't block resolution of this design ticket.
I understand this should be fixed in F36 with Software 42.rc. Once beta freeze ends, we should update to F36, then run:
$ gnome-software --quit $ gsettings reset org.gnome.software check-timestamp
and then verify that we see updates prepared reliably. I'm hopeful that we've fixed the problem where prepared updates are invalidated and then not prepared again, but I still wonder whether something else is preventing updates from being prepared in the first place.
Hi Milan, today I received a desktop notification "Critical software update available" but when I clicked the notification and GNOME Software opened, I saw the update was available to download, but not actually prepared. This was with gnome-software-41.4-1.1.fc35. Do you want another bug report for this? The notification must not be displayed until after the update is prepared.
This was with gnome-software-41.4-1.1.fc35.
It does not contain the change to re-prepare the update when the prepared-update file vanishes. The 41.4-1.1 contains only debug prints about "how many updates had been received from the PackageKit and why the update monitor failed.
This would need change for PackageKit to notify about changed updates and a change to auto-re-prepare the update.
To be clear: I clicked immediately when the notification appeared, and there was no update prepared, so there would have been almost no time for the update to be unexpectedly invalidated. I could be wrong, but I don't think the update was ever prepared in the first place.
This would need change for PackageKit to notify about changed updates
Is there an issue report for this?
and a change to auto-re-prepare the update.
This is what you already fixed in version 42.rc, right?
This would need change for PackageKit to notify about changed updates Is there an issue report for this?
https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1251
and a change to auto-re-prepare the update. This is what you already fixed in version 42.rc, right?
https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1260
Both are part of the 42.rc, yes.
Unfortunately this has failed my QA once again. I have a freshly-installed F36 system, just installed on Tuesday, that prompted me to install updates immediately after I turned it on today. When I clicked the desktop notification, GNOME Software opened and I could see that no update was prepared: I had to click the Download button.
This violated two rules:
There is supposed to be a multi-day waiting period after preparing the update before the user is notified.
The delay is made after the last check. When you install the machine, there is no check made yet, which is treated as "notify now".
The notification should never be displayed if the update is not prepared.
I agree. I think I saw something like this before, the update monitor check does not correspond to the Updates page, or vice versa. The updates are checked every day/hour, while the Updates page does not update that often.
It should also probably withdraw any notifications when it quits
You can open a request upstream. Similar one (for different notifications) is here: https://gitlab.gnome.org/GNOME/gnome-software/-/issues/859 Sadly, GNotification doesn't allow to set timeout for the notification, which would make things much simpler.
Since that is a one-time issue, I suppose that is OK. Hopefully you manage to fix the state of the updates page, then.
It should also probably withdraw any notifications when it quits You can open a request upstream. Similar one (for different notifications) is here: https://gitlab.gnome.org/GNOME/gnome-software/-/issues/859 Sadly, GNotification doesn't allow to set timeout for the notification, which would make things much simpler.
ack. The simple solution is to just only ever use one well-known notification ID, then it's easy to withdraw without having to remember what it previously was.
Michael, could you give a try to this test build, please? It prints some debug information about the update page and the update monitor. To get it, run these commands:
$ gnome-software --quit $ gsettings set org.gnome.software check-timestamp 0; gsettings set org.gnome.software update-notification-timestamp 0 $ gnome-software
The build contains one change, which tries to catch a little race condition about the refreshing the Updates page and it prints things about it, but it works properly here even without that change. What I see:
This cannot fail, unless you face the race condition, which the patch tries to address, but I consider it rather rare.
I tested this build and it failed on the first try: Software displayed the notification that updates are available, but it showed the Download button rather than the Restart & Update button. Here is the debug log:
<img alt="log" src="/fedora-workstation/issue/raw/files/c36cd478dab914443686341978e4b3496296b8134ca607e498ff274303e515fd-log" />
Hmm, the last notification about changed updates emitted by the plugins had been received ~4 seconds before the update monitor finished its "get updates" call. The Updates page had been still updating itself at that time. The monitor's "get updates" call should continue with download and such, but it did not happen here.
I suppose you've only few pending updates, and some gnome-nightly installed apps, which are not part of that remote (according to the log). Maybe that's the reason why it works differently in your environment than here.
I updated the test package, I forced the Updates page refresh on the update-monitor finishing its work. It can be downloaded here.
Sadly it failed again, exactly the same as before. Here is a new log:
<img alt="log" src="/fedora-workstation/issue/raw/files/f369b9bd1f2886668fdc776dad2a1426c446e36ca74d1c1b7a4cac55c87bf4ac-log" />
I also saw this bug just now on the freshly-installed system where I have no flatpaks installed (which is not my normal desktop, which is where I am testing currently).
Thanks for a quick test. I see it did what it was supposed to do, thus the problem is elsewhere. As you can reproduce on a fresh system I'll try to reproduce it there. No need to waste your time.
Okay, I see it too and the above makes sense, because when you look at https://wiki.gnome.org/Design/Apps/Software/Updates#Tentative_Design then you can see for the offline updates: Checks for updates daily -> finds non-critical update -> wait 2 weeks -> download updates -> ... , thus the prepare for offline updates depends on the criticality of the updates. What I've got during debugging:
Checks for updates daily -> finds non-critical update -> wait 2 weeks -> download updates -> ...
should_notify_about_pending_updates: last_test_days:365 n-apps:6 should_download:1 has_important:0 any_important_downloaded:0 all_downloaded:0 any_downloaded:1 res:1 reason:Software Updates Ready to Install|Software updates are waiting and ready to be installed.
after which the Updates page refreshes and shows Restart & Update. This is only if I also reset:
Restart & Update
$ gsettings set org.gnome.software install-timestamp 0
which counts for the 14 day delay between non-critical updates download (and I missed it in the above comment with other two keys for reset. As the PackageKit has downloaded the packages I get the "Restart&Update" and the same notification as above after the update monitor finishes its duty also without resetting the install-timestamp. When I had it set for 11 days (before I reset it), the gnome-software did not ask PackageKit to prepare offline updates, which corresponds to the design. I think this is the reason also for you.
Just to be sure we're on the same page here: with automatic downloading enabled (default), you're never supposed to notify about updates unless there is currently a prepared update. Thus, any time GNOME Software displays a desktop notification when an update is not prepared is a bug. Right?
Thus, any time GNOME Software displays a desktop notification when an update is not prepared is a bug.
I do not think so. There are two types of updates, online and offline. Offline updates are divided into two groups, critical and non-critical. All these three types/groups behave differently. See https://wiki.gnome.org/Design/Apps/Software/Updates#Tentative_Design .
Perhaps the diagram needs to be clarified. The diagram assumes that "downloading" and "preparing" an update are the same thing. It further assumes that an update that has been prepared will stay prepared. So when the diagram says "after download, notifies to install," this assumes the update is currently prepared. If GNOME Software is notifying the user when there is no prepared update -- which I've seen it do repeatedly today -- then GNOME Software failed to properly follow the diagram.
To be clear, the behavior that's not working properly is the "Automatic updates on, offline updates" case. It should be the same behavior regardless of whether the update is "critical" or not: notify only after the update is prepared.
There is supposed to be a multi-day waiting period after preparing the update before the user is notified. The primary mechanism for informing users about the update is supposed to be the gnome-shell power off dialog. The notification is here to support users who do not power off regularly, not to nag users to update ASAP.
I see this multi-day waiting period is not actually included on the diagram. Let's not worry about that for now. We can add it later if need be. This is a minor consideration overall. It's more important to ensure the notifications appear only when an update is prepared.
I'm on Fedora 36 for some time now and just got a notification in the shell saying updates are available. I click on the notification, which brings up Software, and it says "Up to date, last checked 5 months ago". Both of these are incorrect. The one non-standard thing set in my setup is gsettings get org.gnome.software download-updates is false.
gsettings get org.gnome.software download-updates
false
Maybe I should try https://pagure.io/fedora-workstation/issue/107#comment-775923 (do a reset)?
I'm on Fedora 36 for some time now and just got a notification in the shell saying updates are available. I click on the notification, which brings up Software, and it says "Up to date, last checked 5 months ago". Both of these are incorrect.
It might be a disagreement between the update monitor and the Updates tab.
There are three timestamps related to the behaviour, apart of the check-timestamp, it's the install-timestamp and update-notification-timestamp. Before reseting any of those, do backup those values, please. Though I'm not sure what the reset would help with, it should be similar to what you have now, 5 months since the last check.
With automatic updates off you should rather get notification about "Software Updates are Out of Date" (see the design chart).
The only option I see from the brief code reading is that something else prepared updates in the background, then the Software's update monitor received either from the PackageKit or from the Flatpak an least one app for update, which then gives the notification you received. Then the Updates page failed to refresh and has got out of sync with the update monitor.
Michael's https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1669 references https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1251 , whose change is part of the 42.rc, thus you might have it, even the change landed two months ago and you did not check for updates for 5 months. What is your gnome-software version, please? Did you restart it since the last update of it (gnome-software)?
If GNOME Software is notifying the user when there is no prepared update -- which I've seen it do repeatedly today -- then GNOME Software failed to properly follow the diagram.
Note I'm still seeing this on three different computers. It wasn't a one-time error.
Metadata Update from @chrismurphy: - Issue set to the milestone: Fedora 37 (was: Fedora 35)
Perhaps we could add an assert right before the code that displays the notification, to crash GNOME Software if automatic updates are on but an update is not currently prepared? Surely that would be possible?
Yes, it is possible, but what it is good for I'm not sure. Instead of showing "incorrect" notification crash the app?
Prepared updates are tricky with the PackageKit, because anything can talk to the PackageKit daemon and cause prepared-updates file disappear. The Software tries to workaround it, but it's not enough. The PackageKit itself invalidates the prepared-updates file after certain operations and there's nothing one can do about it.
I do not know whether it's the cause here, it can be anything else, just the situation of having prepared update in time of the notification and then that being invalidated and the Updates tab showing "Download" instead of "Restart & Update", can certainly happen.
I was thinking this would help you receive bug reports with backtraces, and hopefully fix the problem. Do you have another plan to solve this?
Thing is, I do not think Software is actually checking whether an update is prepared before it displays its notification that an update is available. If I check for a prepared update immediately after the notification is displayed, it's usually not there. I guess either PackageKit invalidates its update immediately after Software notifies, or Software doesn't check before notifying.
Aren't there several possible solutions for this? (a) PackageKit could inform Software of state changes, e.g. by changing a D-Bus property. (b) Software could continually check for state changes from PackageKit on a predetermined interval to re-prepare an invalidated update, but certainly it should check immediately before displaying a notification. (c) PackageKit could automatically maintain its state, and re-prepare any invalidated update.
Can we pick one approach and make it work? I assume approach (b) could fix this without any changes in PackageKit?
I need to know what precisely is misbehaving here. The last time I tried to reproduce this it worked fine, thus the circumstances matter. You have it running as a user, I do not. It surely plays its role as well. Getting crash reports is useless in this case, apart of making pressure on me (which should be rather done on the gnome-software upstream anyway, because I doubt this is Fedora specific).
I do not think Software is actually checking whether an update is prepared before it displays its notification that an update is available.
The update monitor does determine what message to show according to the state of the update. See it here. Note of those all_downloaded and any_downloaded variables. The PackageKit plugin does verify the update is prepared.
all_downloaded
any_downloaded
While giving you the pointers into the code, I noticed the all_downloaded is determined slightly differently above and in the updates-section, the later checks also app dependencies. Let me check whether it's the difference causing the problem (I'm not sure whether PackageKit apps have any dependencies, maybe apart of the "System Update" app, which contains bare packages).
Software could continually check...
You'd surely find a time window , when the UI state won't match reality. A file monitor could be an option here, supposing the prepared-update file placement can be determined.
Note the prepared-update is invalidated on any repo changes (add/remove/update), which makes sense.
I need to know what precisely is misbehaving here.
I don't know what more to tell you, Milan, other than that GNOME Software is creating desktop notifications to tell me an update is available when the update is not actually prepared. This happens on three different computers, one of which was freshly installed with Fedora 36. It sure looks like the code in GNOME Software that decides to display the notification is just generally broken.
The update monitor does determine what message to show according to the state of the update. See it here. Note of those all_downloaded and any_downloaded variables.
The bad notification that I see is always "Software Updates Ready to Install" so that is where something is going wrong. Looking at this line:
https://gitlab.gnome.org/GNOME/gnome-software/-/blob/8cbce258c3ed1e581477ce56972c9ce9c6225aa5/src/gs-update-monitor.c#L237
I see you display that notification if only some packages are downloaded, but not all, right? But is the update really prepared if only some packages are downloaded? I don't think so? Again, the requirement here is to display that notification only if an update is prepared.
Interesting. Sounds like another good avenue to investigate.
Does PackageKit have an easy way to check if an update is prepared or not? That would seem safer than trying to recompute this yourself?
A file monitor sounds ideal.
I would expect GNOME Software to prepare the update again when this happens.
The comment above that line explains why it does so. It can be changed, but it's not the source of the problem here.
I'm able to reproduce it here and I see several issues with the determination of the update availability and the propagation of the update changes into the Updates page. The good news is, the PackageKit plugin with the updates-monitor do the correct thing, it's the UI part, which is blocked/broken. It;s also about timing, it sometimes works fine.
Does PackageKit have an easy way to check if an update is prepared or not?
Not helpful here, the gnome-software is not about PackageKit only.
A file monitor sounds ideal. ... I would expect GNOME Software to prepare the update again when this happens.
It looks like it's already there, or something close to it, but the GUI is not updated properly.
I opened https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1401
I had the chance to discuss the translation issue with @mcrha yesterday.
The other gnome-software maintainer didn't want the heading string to live upstream. He's currently away so won't be possible to discuss it any more with him for a while.
I think that Milan's idea is to have a new project to contain the configuration for the deployment featured section, and to set that project up to be translated using Fedora's translation infrastructure and translation teams. He's pointed out that the heading is a single, simple string. If it comes to it, we can fill in the translations ourselves as best we can, including relying on machine translation if necessary.
This seems OK to me, and seems like something that we can make work.
One question to resolve is which package the configuration should be shipped in. Milan's suggestion is to use appstream-data. Alternatives to that could be gnome-software or fedora-release.
appstream-data
fedora-release
I guess you wanted to write the previous comment into the https://pagure.io/fedora-workstation/issue/203 instead ;)
We discussed this briefly during today's WG call. We're hoping that the issue will be resolved for F37 - something to test once it's available.
Okay, if there's anything I can help with, then please let me know.
I thought this was finally working properly, but unfortunately with gnome-software-43.1-1.fc37 I just received a notification telling me that critical updates are available, but they are not yet downloaded; accordingly, the new update policy is still not properly implemented.
I think I see bug remaining in should_notify_about_pending_updates(). If has_important and timestamp_days >= 1 are both true, but all_downloaded is false, then the update notification is displayed even if should_download is true. But if should_download is true and all_downloaded is false, no notification should ever be displayed. So I think it's now working properly for non-critical updates, but still improperly notifying for critical updates before the updates are all downloaded.
has_important
timestamp_days >= 1
should_download
I'm also concerned that GNOME Software does not seem to have downloaded these updates yet, even though it should download immediately once it knows there exists a critical update.
As had been mentioned several times on many places, updates are prepared by the PackageKit and it's all driven by a file /var/lib/PackageKit/prepared-update, which the PackageKit deletes on various (and many) occasions and gnome-software tries to re-create it in the background. This all means there can be race conditions involved, as multiple processes are making assumptions on the file existence and how/when to re-create it. It can be that the updates are downloaded in the /var/cache/PackageKit/.... or where it stores it, but the prepared-update file is gone, thus the gnome-software claims "needs to download first". Such download can be lightning fast, with minimum network use, due to the packages being ready in the PackageKit's local cache.
/var/cache/PackageKit/....
Milan, do you agree this code looks wrong?
According to https://wiki.gnome.org/Design/Apps/Software/Updates#Tentative_Design , when there are important updates, they should be downloaded immediately (this part is tricky), then the user should be notified daily to install the updates (the timestamp_days is there to notify just once per day).
timestamp_days
The "faulty" part can be the call of the notify_about_pending_updates from the get_updates_finished_cb. Having there critical updates, they should be downloaded as the "online" updates, and only then the second call to notify_about_pending_updates should apply, if needed. But this is done when the critical updates are recognized (see the security_timestamp use at the get_updates_finished_cb() of the src/gs-update-monitor.c). Unless the critical update has unknown download size, that can cause trouble.
notify_about_pending_updates
get_updates_finished_cb
security_timestamp
get_updates_finished_cb()
src/gs-update-monitor.c
So I just saw this bug on a second computer now, also running F37. I paid special attention to the text of the notification and I believe it said "Critical Software Updates Available to Download," so GNOME Software actually knew the update was not yet downloaded at the time it improperly decided to show the notification. I think this should fix the errant notification:
diff --git a/src/gs-update-monitor.c b/src/gs-update-monitor.c index 292d2bc4b..d46f6d069 100644 --- a/src/gs-update-monitor.c +++ b/src/gs-update-monitor.c @@ -220,7 +220,7 @@ should_notify_about_pending_updates (GsUpdateMonitor *monitor, *out_title = _("Critical Software Update Ready to Install"); *out_body = _("An important software update is ready to be installed."); res = TRUE; - } else { + } else if (!should_download) { *out_title = _("Critical Software Updates Available to Download"); *out_body = _("Important: critical software updates are waiting."); res = TRUE;
Milan, is it OK for me to submit this change as a merge request upstream?
Err, I thought I replied to that long ago, but it seems I lost track of this instead.
From my point of view, the question is: when you see this notification, what is the state the offline update? Is it ready or not ready? Do you just want to mute the notification, regardless what the state under the hood is? It depends which call the notification is provided in. It can be valid notifications when called from one place, but invalid from the other.
In any case, you can open a merge request and it can be discussed upstream. It would end there anyway.
Err, I thought I replied to that long ago, but it seems I lost track of this instead. From my point of view, the question is: when you see this notification, what is the state the offline update? Is it ready or not ready?
From my point of view, the question is: when you see this notification, what is the state the offline update? Is it ready or not ready?
The update is not ready. That's the bug.
Do you just want to mute the notification, regardless what the state under the hood is?
I want to mute the notification if the update is not ready, since GNOME Software is not supposed to ever notify about updates except when they're ready to install (by default, when the automatic download updates setting is enabled).
I'll propose this upstream, then.
OK, proposed merge requests https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1565 and https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/1566. I'm pretty sure that will be enough to finally resolve this longstanding issue. We did it! I think....
Metadata Update from @catanzaro: - Issue untagged with: testing - Issue tagged with: pending-action
I think there are involved a lot of corner cases with the "updates are ready" thing, one can be that you'll miss the notification, due to a bug/behaviour in the related library (PackgeKit) in the future, but let's see what Philip thinks of the change in the upstream.
I really don't understand, Milan. It seems very simple to me. With default settings, we are not allowed to notify unless the update is prepared. i.e. the "updates are available to download" must never be shown unless default settings are changed and automatic downloads are disabled. Software already knows the update is not prepared because it decided to show the "available to download" version of the notification rather than the "ready to install" version, so it's not some edge case where Software is confused about its state. It already knows that the update is not prepared, but it notifies improperly anyway. There is no corner case involved here, it's just a logic error. Yes?
Can be.
OK, my change was merged upstream. I'm reasonably confident that this will be fixed in Software 43.3, so I'm going to optimistically close this. (We can always reopen if we detect unexpected further problems.)
Thanks for working with us for so long on this, Milan!
Metadata Update from @catanzaro: - Issue untagged with: pending-action - Issue close_status updated to: Fixed - Issue status updated to: Closed (was: Open)
Log in to comment on this ticket.