PR#1748: Add jenkins job definition for rebuilding test images - pungi

pungi

#1748 Add jenkins job definition for rebuilding test images

Opened a year ago by lsedlar. Modified a year ago

lsedlar/pungi rebuild-images into master

Add jenkins job definition for rebuilding test images

Lubomír Sedlář • a year ago

a26158f

tests/jenkins-image-rebuild-job.yaml

file added

+68

		`@@ -0,0 +1,68 @@`
		`+ # This file defines a Jenkins job that periodically rebuilds the container`
		`+ # images used for running unit tests.`
		`+ #`
		`+ # To use it, install jenkins-job-builder via pip and create a configuration`
		`+ # file with this content:`
		`+ #`
		`+ # [jenkins]`
		`+ # user=<your jenkins ID, displayed on People page in UI>`
		`+ # password=<API token created in the Jenkins UI>`
		`+ # url=https://jenkins-fedora-infra.apps.ocp.cloud.ci.centos.org/`
		`+ #`
		`+ # Then run the tool like this:`
		`+ #`
		`+ # $ jenkins-jobs --conf jenkins-jobs.ini update tests/jenkins-image-rebuild-job.yaml`
		`+ #`
		`+ # The job expects a credential with pungi-quay-io-push-token to exist in`
		`+ # Jenkins. It should be a secret file with login information for the account`
		`+ # pushing the image.`
		`+ - job:`
		`+ name: pungi-test-image-rebuild`
		`+ project-type: freestyle`
		`+ description: Rebuild container images for running Pungi unit tests and push them to quay.io`
		`+ concurrent: false`
		`+ node: cico-workspace`
		`+ logrotate:`
		`+ numToKeep: 20`
		`+ triggers:`
		`+ - timed: '@weekly'`
		`+ wrappers:`
		`+ - credentials-binding:`
		`+ - file:`
		`+ credential-id: pungi-quay-io-push-token`
		`+ variable: QUAY_AUTHFILE`
		`+ builders:`
		`+ - shell: \|`
		`+ # Set up Duffy node`
		`+ set +x`
		`+ SESSION="$(duffy client --url https://duffy.ci.centos.org/api/v1 --auth-name fedora-infra --auth-key "$CICO_API_KEY" request-session pool=virt-ec2-t2-centos-9s-x86_64,quantity=1)"`
		`+ DUFFY_SESSION_ID="$(jq -r <<<"$SESSION" ".session.id")"`
		`+ trap 'set +x && echo "SHUTTING DOWN DUFFY NODE" && duffy client --url https://duffy.ci.centos.org/api/v1 --auth-name fedora-infra --auth-key "$CICO_API_KEY" retire-session "$DUFFY_SESSION_ID" >/dev/null' EXIT`
		`+`
		`+ # Set up environment`
		`+ remote_dir=/tmp/$JENKINS_AGENT_NAME`
		`+ hostname=$(jq <<<"$SESSION" -r '.session.nodes[0].hostname')`
		`+`
		`+ echo "duffy session id: $DUFFY_SESSION_ID hostname: $hostname"`
		`+`
		`+ # Prepare job script`
		`+ cat >job.sh <<EOF`
		`+ set -x`
		`+ dnf install -y git podman skopeo`
		`+ git clone https://pagure.io/pungi.git`
		`+ cd pungi`
		`+ podman build -t pungi-test -f tests/Dockerfile-test`
		`+ podman build -t pungi-test-py2 -f tests/Dockerfile-test-py2`
		`+ skopeo copy --authfile=$remote_dir/authfile docker://quay.io/exd-guild-compose/pungi-test:latest docker://quay.io/exd-guild-compose/pungi-test:previous`
		`+ skopeo copy --authfile=$remote_dir/authfile docker://quay.io/exd-guild-compose/pungi-test-py2:latest docker://quay.io/exd-guild-compose/pungi-test-py2:previous`
		`+ podman push --authfile=$remote_dir/authfile pungi-test:latest docker://quay.io/exd-guild-compose/pungi-test:latest`
		`+ podman push --authfile=$remote_dir/authfile pungi-test-py2:latest docker://quay.io/exd-guild-compose/pungi-test-py2:latest`
		`+ EOF`
		`+`
		`+ set -x`
		`+ ssh -o StrictHostKeyChecking=no root@$hostname mkdir $remote_dir`
		`+ scp job.sh root@$hostname:$remote_dir`
		`+ scp $QUAY_AUTHFILE root@$hostname:$remote_dir/authfile`
		`+`
		`+ # Run the actual workload`
		`+ ssh root@$hostname sh $remote_dir/job.sh`

lsedlar commented a year ago

This is definition for a jenkins job:
https://jenkins-fedora-infra.apps.ocp.cloud.ci.centos.org/job/pungi-test-image-rebuild/21/console

The job will run weekly, pull the latest dockerfiles from git, build the test images and push them to quay.io. The previous image is tagged so that there's a way to roll back.

rebased onto 8064a18e636f599d8103a519353dfcd82c66e907

a year ago

rebased onto a26158f

a year ago

lsedlar commented a year ago

The point of using container for the test is to avoid complexity of getting dependencies like dnf and createrepo into python virtualenv.

The automatic rebuild should solve the problem of the images getting stale, but it again adds a fair bit of complexity.

An alternative is to not store the images in a registry, and instead build it fresh each time. It would be simple and the images would always be up-to-date. On the negative side the pipeline would be slower, and less resource efficient.

--- a/tests/Jenkinsfile
+++ b/tests/Jenkinsfile
@@ -39,8 +39,10 @@ git remote add proposed "$params.REPO"
 git fetch proposed
 git checkout origin/master
 git merge --no-ff "proposed/$params.BRANCH" -m "Merge PR"
-podman run --rm -v .:/src:Z quay.io/exd-guild-compose/pungi-test tox -r -e flake8,black,py3,bandit
-podman run --rm -v .:/src:Z quay.io/exd-guild-compose/pungi-test-py2 tox -r -e py27
+podman build -t pungi-test -f tests/Dockerfile-test
+podman run --rm -v .:/src:Z pungi-test tox -r -e flake8,black,py3,bandit
+podman build -t pungi-test-py2 -f tests/Dockerfile-test-py2
+podman run --rm -v .:/src:Z pungi-test-py2 tox -r -e py27
                         """
                         sh "cat job.sh"
                         sh "ssh -o StrictHostKeyChecking=no root@$hostname mkdir $remote_dir"

Edited a year ago by lsedlar