pungi

#485 Write osbs JSON manifest for scratch builds
Closed: Fixed Opened by ktdreyer.

Closed: Fixed
Reopen Issue

The osbs phase writes an osbs JSON file with metadata about a container. However, if "scratch" is enabled in the config, Pungi will not write this JSON file.

pungi/phases/osbs.py has the following:

# Only real builds get the metadata.
if not config.get('scratch', False):
    self._add_metadata(koji.koji_proxy, variant, task_id)

This JSON data is useful for CI environments where we are generating composes + containers continuously. In these cases, I don't want our CI system to fill up Koji/OSBS/Pulp with "real" containers.

Writing the metadata for scratch containers will allow other tools in the CI pipeline to consume this JSON data and run QE tests on the resulting containers.

Here's a standalone hacky tool that does this: https://github.com/ktdreyer/bucko

This tool will scratch-build containers from a compose, and write an osbs JSON data file like so:

{
  "compose_id": "MYPRODUCT-RHEL-7-20161208.t.1",
  "compose_url": "http://example.com/composes/MYPRODUCT-RHEL-7-20161208.t.1",
  "koji_task": 12300940,
  "repositories": [
    "brew-pulp-docker01.example.com:8888/myprod:myproduct-rhel-7-docker-candidate-20170104182159"
  ]
}

These are the only attributes I need for scratch builds.

Hi @ktdreyer, we can enable writing manifest for scratch builds, however there are some problems with the info of scratch builds we can provide in manifest:

  1. we can't guarantee the compose_url will stay correct as sometime the compose can be promoted to other place after composing. For this, we can just drop compose_url, having compose_id in metadata should be enough to figure out which compose the containers are built from.

  2. scratch tasks of buildContainer doesn't contain any build, there will be limited info we can know for scratch builds, we can only have 'compose_id', 'koji_task', 'repositories', very close to what you want, but we missing arch info, which will result in different data structure of metadata than real builds.

for example, metadata for real builds can be something like:

{
    "Server": {
        "x86_64": [
            {
                "checksum": "4b3f393f6f34725c49b470d68c56222e",
                "compose_id": "DP-1.0-20170302.t.0",
                "creation_time": "2017-02-28 16:42:00.422646",
                "docker": {
                    "config": {
                        "architecture": "amd64",
                        "author": "\"Jason Brooks\" <jbrooks@redhat.com>",
                        "config": {
                            "ArgsEscaped": true,
                            "AttachStderr": false,
                            "AttachStdin": false,
                            "AttachStdout": false,
                            "Cmd": null,
                            "Domainname": "",
                            "Entrypoint": null,
                            "Env": [
                                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                                "DISTTAG=f25docker",
                                "FGC=f25",
                                "FBR=f25",
                                "NAME=kubernetes-master",
                                "VERSION=0.1",
                                "RELEASE=4",
                                "ARCH=x86_64"
                            ],
                            "Hostname": "bdcf92595a02",
                            "Image": "817f5a3ac97d6189134f4b19027fb39b467bb0c10ba0bd49c85fcd917963e6cb",
                            "Labels": {
                                "Architecture": "x86_64",
                                "BZComponent": "kubernetes-master",
                                "Name": "f25/kubernetes-master",
                                "Release": "4.f25docker",
                                "Version": "0.1",
                                "architecture": "x86_64",
                                "authoritative-source-url": "registry.example.com",
                                "build-date": "2017-02-28T16:36:33.652649",
                                "com.redhat.build-host": "osbs-node01.phx2.fedoraproject.org",
                                "com.redhat.component": "kubernetes-master",
                                "distribution-scope": "private",
                                "name": "f25/kubernetes-master",
                                "release": "4.f25docker",
                                "vcs-ref": "bb3a5eb8521fb1226d9e812e957023d0e6632707",
                                "vcs-type": "git",
                                "vendor": "Fedora Project",
                                "version": "0.1"
                            },
                            "OnBuild": [],
                            "OpenStdin": false,
                            "StdinOnce": false,
                            "Tty": false,
                            "User": "",
                            "Volumes": null,
                            "WorkingDir": ""
                        },
                        "created": "2017-02-28T16:39:58.517634Z",
                        "docker_version": "1.10.3",
                        "history": [
                            {
                                "author": "[Adam Miller <maxamillion@fedoraproject.org>] [Patrick Uiterwijk <patrick@puiterwijk.org>]",
                                "created": "2017-02-16T18:36:36.200746321Z",
                                "created_by": "/bin/sh -c #(nop)  MAINTAINER [Adam Miller <maxamillion@fedoraproject.org>] [Patrick Uiterwijk <patrick@puiterwijk.org>]",
                                "empty_layer": true
                            },
                            {
                                "author": "[Adam Miller <maxamillion@fedoraproject.org>] [Patrick Uiterwijk <patrick@puiterwijk.org>]",
                                "created": "2017-02-16T18:36:37.533508569Z",
                                "created_by": "/bin/sh -c #(nop)  ENV DISTTAG=f25docker FGC=f25 FBR=f25",
                                "empty_layer": true
                            },
                            {
                                "author": "[Adam Miller <maxamillion@fedoraproject.org>] [Patrick Uiterwijk <patrick@puiterwijk.org>]",
                                "created": "2017-02-16T18:36:55.213572136Z",
                                "created_by": "/bin/sh -c #(nop) ADD file:0099a3bf2430ffb5bbfdee48fb5b131f9f67427a2c7f78f23aba062f902666b8 in / "
                            },
                            {
                                "comment": "",
                                "created": "2017-02-28T16:39:58.517634Z"
                            }
                        ],
                        "os": "linux",
                        "rootfs": {
                            "diff_ids": [
                                "sha256:bf973ec11f37bce787f28f7ccf15aa42e6c20188c5267caea4973f802edec873",
                                "sha256:3a293b476afc4ff14c82dde39042427e90ba8207e9ccf5d7e74f8e1b6b3af317"
                            ],
                            "type": "layers"
                        }
                    },
                    "id": "856f6d7770170a6a8a2242e614f53ae0c10c730cfab1724cfafca4a52172709c",
                    "parent_id": "sha256:bdb2c3acbdd83741aaf37d8b74871c1e30ad4fcaae2c81762cd0a629653851fe",
                    "repositories": [
                        "candidate-registry.fedoraproject.org/f25/kubernetes-master:0.1-4.f25docker",
                        "candidate-registry.fedoraproject.org/f25/kubernetes-master@sha256:29399c884b53d117993564772c30b687554e0b15cbd303cef342e21129360fbc"
                    ],
                    "tags": [
                        "0.1-4.f25docker",
                        "0.1",
                        "latest"
                    ]
                },
                "filename": "docker-image-856f6d7770170a6a8a2242e614f53ae0c10c730cfab1724cfafca4a52172709c.x86_64.tar.gz",
                "image": {
                    "arch": "x86_64"
                },
                "koji_task": 18109608,
                "name": "kubernetes-master",
                "release": "4.f25docker",
                "size": 153072853,
                "version": "0.1"
            }
        ]
    }
}

and a proposal metadata of scratch build can be:

{
    "Server": {
        "compose_id": "DP-1.0-20170302.t.0",
        "koji_task": 17640131,
        "repositories": [
            "candidate-registry.fedoraproject.org/f25/mirrormanager2-mirrorlist:f25-docker-candidate-20170207003922"
        ]
    }
}

You will see the metadata for scratch build missing a level of arch in it. Will this work for you?

https://pagure.io/pungi/pull-request/554 is created for review.

There is another proposal from @lsedlar which I believe is much better: add a fake arch 'scratch' for scratch builds, for example:

{
    "Server": {
        "scratch": [
            {
                "compose_id": "DP-1.0-20170302.t.0",
                "koji_task": 17640131,
                "repositories": [
                   "candidate-registry.fedoraproject.org/f25/mirrormanager2-mirrorlist:f25-docker-candidate-20170207003922"
                ]
            }
        ]
    }
}

This can avoid to have a very weird metadata especially when we have both real and scratch build in the same compose. I'll updated the PR to adopt the new proposal.

Metadata Update from @lsedlar:
- Issue assigned to qwan
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

Metadata Update from @lsedlar:
- Issue tagged with: 4.1.14

(For the record, another reason I use scratch builds here instead of real builds is because it reduces confusion when it's time to ship. Scratch built container == maybe not fully GPG-signed; Real build == must be GPG-signed with the GA key).

This issue has been migrated to Fedora Forge:
https://forge.fedoraproject.org/pungi/pungi/issues/485

Please continue any further discussion there.
Metadata

4.1.14

None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.