PR#143: Bugzilla: Do a full sync from time to time - fedora-qa/oraculum

fedora-qa / oraculum

#143 Bugzilla: Do a full sync from time to time

Merged 4 years ago by frantisekz. Opened 4 years ago by frantisekz.

fix_bz_sync_cmp_change into master

Bugzilla: Do a full sync from time to time

František Zatloukal • 4 years ago

c43da14

conf/settings.py.example

file modified

		`@@ -93,6 +93,9 @@`
		`SYNC_INTERVALS["packager_dashboard_package_prs"] = 7200`
		`SYNC_INTERVALS["package_versions_generic"] = 43200`

		`+ # Every BZ_FULL_EVERY_N_TIMES sync will be full instead of partial fast path sync`
		`+ BZ_FULL_EVERY_N_TIMES = 12`
		`+`
		`# Specify time (seconds) after which are tasks considered to be in "rotting state"`
		`TASKS_ROT_AFTER = 7200`

oraculum/config.py

file modified

		`@@ -120,6 +120,9 @@`
		`SYNC_INTERVALS["packager_dashboard_package_prs"] = 7200`
		`SYNC_INTERVALS["package_versions_generic"] = 43200`

		`+ # Every BZ_FULL_EVERY_N_TIMES sync will be full instead of partial fast path sync`
		`+ BZ_FULL_EVERY_N_TIMES = 12`
		`+`
		`# Specify time (seconds) after which are tasks considered to be in "rotting state"`
		`TASKS_ROT_AFTER = 7200`

oraculum/utils/bugzilla.py

file modified

+16 -3

		`@@ -23,7 +23,7 @@`

		`import urllib.parse`

		`- from oraculum import CACHE, BUGZILLA`
		`+ from oraculum import app, celery_app, CACHE, BUGZILLA`
		`from oraculum.utils.dashboard_helpers import release_from_number`


		`@@ -78,8 +78,21 @@`
		`Returns all open Bugs for a single package`
		`"""`
		`data = []`
		`- # Try to get the since value from db if we already have something`
		`- since = CACHE.get_refreshed_time("packager-dashboard__all_package_bugs", package)`
		`+`
		`+ # Get BZ sync counter to decide if it's a time to do a full blown sync`
		`+ # For later us or other oraculum maintainers:`
		`+ # We can't pass this as a function parameter as our caching architecture is saving and getting data`
		`+ # from the database according to provider, which includes also the function arguments`
		`+ redis_conn = celery_app.broker_connection().default_channel.client`
		`+ counter = int(redis_conn.get("bz_sync_mode_counter")) or 1`
		`+`
		`+ if counter % app.config["BZ_FULL_EVERY_N_TIMES"] == 0:`
		`+ # # Refresh the whole bugzilla`
		`+ since = False`
		`+ else:`
		`+ # Try to get the since value from db if we already have something`
		`+ since = CACHE.get_refreshed_time("packager-dashboard__all_package_bugs", package)`
		`+`
		`if since:`
		`# Let's have a faster BZ if we have something in db`
		`since = since.strftime("%Y-%m-%d")`

oraculum/utils/celery_utils.py

file modified

		`@@ -150,6 +150,11 @@`
		`@celery_app.task`
		`def celery_sync_package_bugs():`
		`users = get_users_for_sync()`
		`+`
		`+ # Bump counter to decide which sync mode we should use (full/partial)`
		`+ redis_conn = celery_app.broker_connection().default_channel.client`
		`+ counter = redis_conn.incr("bz_sync_mode_counter")`
		`+`
		`tasks_pool = []`

		`for user in users:`

frantisekz commented 4 years ago

Fixes https://pagure.io/fedora-qa/packager_dashboard/issue/93

zuul commented 4 years ago

Build succeeded.

oraculum-pytest : SUCCESS in 1m 47s

rebased onto d8de052

4 years ago

rebased onto ba1099d

4 years ago

zuul commented 4 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

oraculum-pytest : TIMED_OUT in 30m 26s

jskladan commented on line 24 of oraculum/utils/bugzilla.py 4 years ago

If it were to me, I'd write it in just the opposite way, but that is just a personal preference:

if counter % app.config["BZ_FULL_EVERY_N_TIMES"] == 0:
    # Refresh the whole bugzilla
    since = False
else:
    # Try to a sk bugzilla only for changes since our last request
    since = CACHE.get_refreshed_time("packager-dashboard__all_package_bugs", package)

For some reason my brain parses this easier than what you have there, but as I said, just a personal preference.

rebased onto c43da14

4 years ago

jskladan commented 4 years ago

LGTM, thanks

Pull-Request has been merged by frantisekz

4 years ago

zuul commented 4 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci