centos-sig-hyperscale / package-updates

Created 2 years ago
Maintained by oidoming
Tools to create issues for Hyperscale SIG packages
Members 3
Davide Cavalca committed a year ago

Package Updates | Centos package updates notification system

Usage

MQTT

git clone https://pagure.io/centos-sig-hyperscale/package-updates.git
cd package-updates
podman build -t package-updates .
podman run -it \
  --mount type=bind,src=$HOME/.centos-server-ca.cert,dst=/app/.centos-server-ca.cert,ro=true,relabel=private \
  --mount type=bind,src=$HOME/.centos.cert,dst=/app/.centos.cert,ro=true,relabel=private \
  --mount type=bind,source=$(pwd)config.json,dst=/app/package-updates/config.json \
  -e PAGURE_API_KEY=<pagure-api-key> \
  -e CAFILE=/app/.centos-server-ca.cert \
  -e CERT=/app/.centos.cert \
  -e KEY=/app/.centos.cert \
  -e CONFIG_PATH=/app/package-updates/config.json \
  -e FEDORA_MESSAGING_CONF=/app/package-updates/fedora.toml \  
  -e RUN_MODE=MQTT \
  package-updates

AMQP

git clone https://pagure.io/centos-sig-hyperscale/package-updates.git
cd package-updates
podman build -t package-updates .
podman run -it \
  --mount type=bind,src=$HOME/.centos-server-ca.cert,dst=/app/.centos-server-ca.cert,ro=true,relabel=private \
  --mount type=bind,src=$HOME/.centos.cert,dst=/app/.centos.cert,ro=true,relabel=private \
  --mount type=bind,source=$(pwd)/config.json,dst=/app/package-updates/config.json \
  -e PAGURE_API_KEY=<pagure-api-key> \
  -e CAFILE=/app/.centos-server-ca.cert \
  -e CERT=/app/.centos.cert \
  -e KEY=/app/.centos.cert \
  -e CONFIG_PATH=/app/package-updates/config.json \
  -e FEDORA_MESSAGING_CONF=/app/package-updates/fedora.toml \
  -e RUN_MODE=AMQP \
  package-updates

Configuration

Use config.json to specify your own project configuration values. Here you can specify: - MQTT: - The MQTT broker, port, and topic. - Pagure: - repo_url: Your desired Pagure repo URL, here is where the package updates tickets are going to be created. - api_repo_url: Pagure repo API endpoint for repo_url. - Cbs: - tags: Koji CBS tags. Add cbs tag ids if you want to get package updates from a specific list of cbs tags. - tags_keyword. If this is specified, you will get package updates from all cbs tags that contains this keyword.

Config.json example

{
    "mqtt": {
        "broker": "mqtt.git.centos.org",
        "port": 8883,
        "topic": "git.centos.org/git.tag.creation"
    },
    "pagure": {
        "repo_url": "https://pagure.io/centos-sig-hyperscale/package-updates/",
        "api_repo_url": "https://pagure.io/api/0/centos-sig-hyperscale/package-updates/"
    },
    "cbs": {
        "tags" : [],
        "tags_keyword": "hyperscale"
    }
}

The tags field is prioritized over the keyword field, if you provide both you will only get package updates from the list of tags specified.

For AMQP and logging configuration use the fedora.toml file

Enviroment variables

The script needs these enviroment variables:

  • PAGURE_API_KEY: Pagure API key, It is necessary to have an API Key with issue_create, issue_comment, issue_change_status and issue_update ACLs for the repo in order to allow the script to work with pagure issues.
  • CAFILE: .centos-server-ca.cert file
  • CERT: .centos.cert file
  • KEY: .centos.cert file
  • CONFIG_PATH: path to config.json file
  • FEDORA_MESSAGING_CONF: path to fedora messaging configuration file (fedora.toml)

For more information on how to get the centos cert files, see: https://wiki.centos.org/Authentication#TLS_certificate \ Cert files needed for MQTT, see the Message Broker (MQTT) section: https://wiki.centos.org/Sources#Message_Broker_.28MQTT.29

*Note: Inside fedora.toml file it is needed to specify the cacert.pem, fedora-key.pem and fedora-cert.pem files path. Files can be downloaded from here: https://github.com/fedora-infra/fedora-messaging/tree/stable/configs*

Test enviroment

You can use tests/test_mqtt_pub/publish.py to run a mqtt publisher to test the script with a localhost mqtt server.

You can use tests/test_amqp_pub/publish.py to run a amqp publisher to test the script with the fedora messaging server.

Note: git.centos.org notifications don't appear very often so if you want to test the git.centos.org server leave the script for a long time running (hours or days).

MQTT payload

To receive notifications from git.centos.org package updates the script listens to "git.centos.org/git.tag.creation" topic on mqtt, this topic is for git tag release creation. Here is an example of the payload for this topic:

{
  "repo": {
    "custom_keys": [],
    "name": "selinux-policy",
    "parent": null,
    "date_modified": "1553627665",
    "access_users": {
      "owner": [
        "centosrcm"
      ],
      "admin": [],
      "ticket": [],
      "commit": []
    },
    "namespace": "rpms",
    "priorities": {},
    "close_status": [],
    "access_groups": {
      "admin": [],
      "commit": [],
      "ticket": []
    },
    "milestones": {},
    "user": {
      "fullname": "CentOS Sources",
      "name": "centosrcm"
    },
    "date_created": "1553627665",
    "fullname": "rpms/selinux-policy",
    "url_path": "rpms/selinux-policy",
    "id": 6059,
    "tags": [],
    "description": " SELinux policy configuration "
  },
  "tag": "imports/c8s/selinux-policy-3.14.3-93.el8",
  "rev": "56e29e64a64cb48a0889fd502c636b26dc7800e3",
  "agent": "centosrcm",
  "authors": [
    {
      "fullname": "CentOS Sources",
      "name": "centosrcm"
    }
  ]
}

AMQP payload

To receive notifications from cbs.centos.org builds the script listens to "org.centos.prod.cbs.buildsys.build.state.change" topic on amqp, this topic is for cbs/koji builds status change. Here is an example of the payload for this topic:

{
  "body": {
    "attribute": "state",
    "build_id": 41063,
    "epoch": null,
    "instance": "primary",
    "name": "zlib",
    "new": 0,
    "old": null,
    "owner": "aekoroglu",
    "release": "21.hs+intel.el8",
    "request": [
      "git+https://git.centos.org/rpms/zlib.git#b65ea150e9ae1699e1227bee2dec884d4f2f1c0b",
      "hyperscale8s-packages-intel-el8s",
      {
        "custom_user_metadata": {},
        "wait_builds": []
      }
    ],
    "task_id": 3000379,
    "version": "1.2.11"
  },
  "headers": {
    "fedora_messaging_schema": "base.message",
    "fedora_messaging_severity": 20,
    "sent-at": "2022-09-08T15:58:07+00:00"
  },
  "id": "cace298e-72dd-440f-a652-ddf8afcf1ca5",
  "queue": null,
  "topic": "org.centos.prod.cbs.buildsys.build.state.change"
}