#2322 Add optional argument pkg_name to webhooks API
Merged 2 years ago by praiskup. Opened 2 years ago by nikromen.
copr/ nikromen/copr add-pkgname  into  main

file modified
+70 -8
@@ -173,20 +173,82 @@ 

  "delete after days" option in web UI or on command-line:

  ``copr-cli create your-project ... --delete-after-days 10``

  

+ Webhooks

+ --------

  

- GitHub Webhooks

- ---------------

+ Set up an integration with a Git hosting website and get Copr rebuilds for pull requests, tags and commits.

  

- Webhooks allows you to automatically trigger build.

+ Simple guide:

+   1. Create an SCM package and set its default source by specifying an https:// "Clone URL".

+   2. Make sure the package auto-rebuild option is checked.

+   3. Now you can navigate to **Setting** tab and then **Integrations**

+   4. There is your webhook url in the form of ``https://copr.fedorainfracloud.org/webhooks/<GIT_FORGE>/<ID>/<UUID>/``

+   5. Finish it by following the Git host specific guide below.

  

- First you need to go to your Copr project and tab "Packages" and define some package. The only source type which make sense together with webhooks is "SCM". Check the "Webhook rebuild" option. You may hit "rebuild" and test the build actually works.

+ And next time you push anything to your git, Copr will automatically rebuild your package.

  

- Now you can navigate to "Setting" tab and then "Integrations" There is your webhook url in the form of `https://copr.fedorainfracloud.org/webhooks/github/<ID>/<UUID>/`.

+ Triggerring builds by tag events

+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  

- Then in your GitHub project, go to Settings / Webhooks and services. Click on the Add webhook button.

- Fill in the Payload URL field with the url you noted previously. Set the other fields to the values: `content: application/json; send just push event; no secret`. Click the Add webhook button.

+ One forge may have multiple packages. For this reason, Copr needs to know what package or set of

+ packages should be rebuilt for the tag event. Copr gets this information from the name of the tag, so

+ it is important that the tag contains the name of the package, in a predefined format, that will

+ have to rebuild.

  

- And next time you push anything to your git, Copr will automatically rebuild your package.

+ The tag name should be in this format: ``PKGNAME-VERSION[-RELEASE]`` with possibility of

+ replacing the dash with an underscore.

+ 

+ In case you use different tag name patterns (different Copr package name than tag name), Copr

+ has no idea what package build should be triggered. You have to be explicit and tell Copr your

+ **copr package name** in the webhook URL like this ``https://copr.fedorainfracloud.org/webhooks/<GIT_FORGE>/<ID>/<UUID>/<copr_package_name>/``.

+ 

+ Consider this example:

+ 

+ Your Copr package name is **my-package** and tag name on Github is only a version e.g. **1.22.3**, in that case

+ you have to add an optional argument to your URL containing your **copr package name**.

+ 

+ So if your Copr package name is **my-package** your Github URL would be:

+ ``https://copr.fedorainfracloud.org/webhooks/github/<ID>/<UUID>/my_package/``

+ 

+ GitHub

+ ^^^^^^

+ 

+ How to use it:

+   1. In your GitHub project, go to **Settings** / **Webhooks**

+   2. Click on the **Add webhook** button.

+   3. Fill in the Payload URL field with the url above.

+   4. Select **application/json** as the content type.

+   5. If you want to react to **Tag push events** click **Let me select individual events.** and then select **Branch or tag creation**.

+   6. Click the **Add webhook** button.

+ 

+ Gitlab

+ ^^^^^^

+ 

+ How to use it:

+   1. In your GitLab project, go to **Settings** / **Webhooks**.

+   2. Fill in the URL field with the url above.

+   3. Select **Push events** and **Tag push events** (if you want to react to tags) as event triggers.

+   4. Click the **Add webhook** button.

+ 

+ Bitbucket

+ ^^^^^^^^^

+ 

+ How to use it:

+   1. In your Bitbucket project, go to **Settings** / **Workflow** / **integrations** / **Add webhook**.

+   2. Name the hook, e.g., **Copr**.

+   3. Fill in the URL field with the url above.

+   4. Select to trigger on **Repository Push**.

+   5. Click the **Save** button.

+ 

+ Custom webhook

+ ^^^^^^^^^^^^^^

+ 

+ How to use it:

+ Use the GitLab/GitHub/Bitbucket steps above (when needed), or simply::

+ 

+     $ curl -X POST https://copr.fedorainfracloud.org/webhooks/custom/<ID>/<UUID>/<PACKAGE_NAME>/

+ 

+ Note that the package of name 'PACKAGE_NAME' must exist within this project, and that the 'POST' http method must be specified.

  

  Pagure Integration

  ------------------

@@ -1,5 +1,6 @@ 

  import json

  import re

+ from typing import List, Optional

  

  from sqlalchemy import bindparam, Integer, func

  from sqlalchemy.sql import true, text
@@ -13,6 +14,7 @@ 

  

  from coprs.logic import users_logic

  from coprs.logic import builds_logic

+ from coprs.models import Package

  from copr_common.enums import StatusEnum

  

  log = app.logger
@@ -131,7 +133,9 @@ 

          return package

  

      @classmethod

-     def get_for_webhook_rebuild(cls, copr_id, webhook_secret, clone_url, commits, ref_type, ref):

+     def get_for_webhook_rebuild(

+         cls, copr_id, webhook_secret, clone_url, commits, ref_type, ref, pkg_name: Optional[str]

+     ) -> List[Package]:

          clone_url_stripped = re.sub(r'(\.git)?/*$', '', clone_url)

  

          packages = (models.Package.query.join(models.Copr)
@@ -152,20 +156,60 @@ 

              if not package.copr.active_copr_chroots:

                  continue

  

-             if cls.commits_belong_to_package(package, commits, ref_type, ref):

-                 result += [package]

+             if cls._belongs_to_package(package, commits, ref_type, ref, pkg_name):

+                 result.append(package)

  

          return result

  

      @classmethod

-     def commits_belong_to_package(cls, package, commits, ref_type, ref):

+     def _belongs_to_package(

+         cls, package: Package, commits, ref_type: str, ref: str, pkg_name: Optional[str]

+     ) -> bool:

          if ref_type == "tag":

-             matches = re.search(r'(.*)-[^-]+-[^-]+$', ref)

-             if matches and package.name != matches.group(1):

+             return cls._tag_belongs_to_package(package, ref, pkg_name)

+ 

+         return cls.commits_belong_to_package(package, commits, ref)

+ 

+     @staticmethod

+     def _ref_matches_copr_pkgname(ref: str, copr_pkg_name: str) -> bool:

+         """

+         We accept N-V-R and N-V.  Version and Release needs to contain at least one

+         digit.

+         """

+         separators = ["-", "_"]

+ 

+         def _parts_match(ref_parts):

+             return any(separator.join(ref_parts) == copr_pkg_name for separator in separators)

+ 

+         def _has_a_version_part(ref_parts):

+             if len(ref_parts) < 2:

                  return False

-             else:

-                 return True

+             return any(char.isdigit() for char in ref_parts[-1])

+ 

+         for sep in separators:

+             parts = ref.split(sep)

+             # in case that ref has both version and release we need to do this twice

+             for _ in range(2):

+                 if not _has_a_version_part(parts):

+                     break

+                 # drop the last version component, and compare

+                 parts.pop()

+                 if _parts_match(parts):

+                     return True

+ 

+         return False

  

+     @classmethod

+     def _tag_belongs_to_package(

+         cls, package: Package, ref: str, pkg_name: Optional[str]

+     ) -> bool:

+         if package.name == pkg_name:

+             return True

+ 

+         return cls._ref_matches_copr_pkgname(ref, package.name)

+ 

+     @classmethod

+     def commits_belong_to_package(cls, package: Package, commits, ref: str) -> bool:

          committish = package.source_json_dict.get("committish") or ''

          if committish and not ref.endswith(committish):

              return False

@@ -10,24 +10,8 @@ 

  <div class="row">

      <div class="col-sm-8 col-md-9">

          <h2> Integrations </h2>

-         <p>

-         Set up an integration with a Git hosting website and get Copr build statuses reported

-         in pull requests and commits.

-         </p>

  

-         <h3> Simple guide: </h3>

-         <ol>

-           <li> Create an SCM package and set its default source by specifying an https:// "Clone URL". </li>

-           <li> Make sure the package auto-rebuild option is checked. </li>

-           <li> Finish it by following the Git host specific guide below. </li>

-         </ol>

- 

-         <p> Copr will now rebuild the package every time it receives an update event -

-             usually for every new commit, every new tag, or every opened pull request. </p>

- 

-         <p> If valid credentials for a source repository are entered below, Copr will

-             report back build results for a particular pull request or commit.

-             Only one source repository at time can be set for pull request and commit flagging. </p>

+         For detailed information, please see <a href="https://docs.pagure.org/copr.copr/user_documentation.html#webhooks">the docs</a>.

  

          <h3> Pagure </h3>

          <ol>
@@ -56,53 +40,30 @@ 

          <div class="well well-sm">

              {{ github_url }}

          </div>

-         <h4> How to use it: </h4>

-         <ol>

-           <li> In your GitHub project, go to <code>Settings</code> / <code>Webhooks</code>. </li>

-           <li> Click on the <code>Add webhook</code> button. </li>

-           <li> Fill in the <code>Payload URL</code> field with the url above. </li>

-           <li> Select <code>application/json</code> as the content type. </li>

-           <li> Click the <code>Add webhook</code> button. </li>

-         </ol>

+         <div class="well well-sm">

+             {{ github_url }}&ltPKG_NAME&gt/

+         </div>

  

          <h3> Gitlab </h3>

          <div class="well well-sm">

              {{ gitlab_url }}

          </div>

-         <h4> How to use it: </h4>

-         <ol>

-           <li> In your GitLab project, go to <code>Settings</code> / <code>Webhooks</code>. </li>

-           <li> Fill in the <code>URL</code> field with the url above. </li>

-           <li> Select <code>Push events</code> and <code>Tag push events</code> as event triggers. </li>

-           <li> Click the <code>Add webhook</code> button. </li>

-         </ol>

+         <div class="well well-sm">

+             {{ gitlab_url }}&ltPKG_NAME&gt/

+         </div>

  

          <h3> Bitbucket </h3>

          <div class="well well-sm">

              {{ bitbucket_url }}

          </div>

-         <h4> How to use it: </h4>

-         <ol>

-           <li> In your Bitbucket project, go to <code>Settings</code> / <code>Workflow</code> / <code>integrations</code> / <code>Add webhook</code>. </li>

-           <li> Name the hook, e.g., “Copr”.

-           <li> Fill in the <code>URL</code> field with the url above. </li>

-           <li> Select to trigger on <code>Repository Push</code>. </li>

-           <li> Click the <code>Save</code> button. </li>

-         </ol>

+         <div class="well well-sm">

+             {{ bitbucket_url }}&ltPKG_NAME&gt/

+         </div>

  

          <h3> Custom webhook </h3>

          <div class="well well-sm">

              {{ custom_url }}

          </div>

-         <h4> How to use it: </h4>

-         <p> Use the GitLab/GitHub/Bitbucket steps above (when needed), or simply </p>

-         <p>

-           <div class="well well-sm">

-             $ curl -X POST {{ custom_url }}

-           </div>

-           Note that the package of name 'PACKAGE_NAME' must exist within this

-           project, and that the 'POST' http method must be specified.

-         </p>

      </div>

  </div>

  

@@ -1,3 +1,5 @@ 

+ from typing import Optional

+ 

  import flask

  from functools import wraps

  
@@ -73,7 +75,8 @@ 

  

  

  @webhooks_ns.route("/bitbucket/<int:copr_id>/<uuid>/", methods=["POST"])

- def webhooks_bitbucket_push(copr_id, uuid):

+ @webhooks_ns.route("/bitbucket/<int:copr_id>/<uuid>/<string:pkg_name>/", methods=["POST"])

+ def webhooks_bitbucket_push(copr_id, uuid, pkg_name: Optional[str] = None):

      # For the documentation of the data we receive see:

      # https://confluence.atlassian.com/bitbucket/event-payloads-740262817.html

      copr = ComplexLogic.get_copr_by_id_safe(copr_id)
@@ -100,7 +103,7 @@ 

          return "Bad Request", 400

  

      packages = PackagesLogic.get_for_webhook_rebuild(

-         copr_id, uuid, clone_url, commits, ref_type, ref

+         copr_id, uuid, clone_url, commits, ref_type, ref, pkg_name

      )

  

      for package in packages:
@@ -113,7 +116,8 @@ 

  

  

  @webhooks_ns.route("/github/<int:copr_id>/<uuid>/", methods=["POST"])

- def webhooks_git_push(copr_id, uuid):

+ @webhooks_ns.route("/github/<int:copr_id>/<uuid>/<string:pkg_name>/", methods=["POST"])

+ def webhooks_git_push(copr_id: int, uuid, pkg_name: Optional[str] = None):

      if flask.request.headers["X-GitHub-Event"] == "ping":

          return "OK", 200

      # For the documentation of the data we receive see:
@@ -146,7 +150,9 @@ 

      except KeyError:

          return "Bad Request", 400

  

-     packages = PackagesLogic.get_for_webhook_rebuild(copr_id, uuid, clone_url, commits, ref_type, ref)

+     packages = PackagesLogic.get_for_webhook_rebuild(

+         copr_id, uuid, clone_url, commits, ref_type, ref, pkg_name

+     )

  

      committish = (ref if ref_type == 'tag' else payload.get('after', ''))

      for package in packages:
@@ -159,7 +165,8 @@ 

  

  

  @webhooks_ns.route("/gitlab/<int:copr_id>/<uuid>/", methods=["POST"])

- def webhooks_gitlab_push(copr_id, uuid):

+ @webhooks_ns.route("/gitlab/<int:copr_id>/<uuid>/<string:pkg_name>/", methods=["POST"])

+ def webhooks_gitlab_push(copr_id: int, uuid, pkg_name: Optional[str] = None):

      # For the documentation of the data we receive see:

      # https://gitlab.com/help/user/project/integrations/webhooks#events

      copr = ComplexLogic.get_copr_by_id_safe(copr_id)
@@ -192,7 +199,9 @@ 

      except KeyError:

          return "Bad Request", 400

  

-     packages = PackagesLogic.get_for_webhook_rebuild(copr_id, uuid, clone_url, commits, ref_type, ref)

+     packages = PackagesLogic.get_for_webhook_rebuild(

+         copr_id, uuid, clone_url, commits, ref_type, ref, pkg_name

+     )

  

      committish = (ref if ref_type == 'tag' else payload.get('after', ''))

      for package in packages:

@@ -1,3 +1,5 @@ 

+ import pytest

+ 

  from coprs.logic.packages_logic import PackagesLogic

  

  from tests.coprs_test_case import CoprsTestCase
@@ -11,3 +13,28 @@ 

          assert builds_p4 == {self.b6: self.b6_bc}

          assert builds_p5 == {self.b10: [self.b10_bc[0]],

                               self.b11: [self.b11_bc[1]]}

+ 

+     @staticmethod

+     @pytest.mark.parametrize(

+         "ref, copr_pkg_name, result",

+         [

+             pytest.param("copr-cli-1-1alpha", "copr-cli", True),

+             pytest.param("copr-cli-1", "copr-cli", True),

+             pytest.param("copr-cli-1", "copr", False),

+             pytest.param("copr_cli-1.1-1", "copr_cli", True),

+             pytest.param("copr_cli-1.1", "copr_cli", True),

+             pytest.param("copr-frontend-a1", "copr", False),

+             pytest.param("copr-frontend-a1", "copr-frontend", True),

+             pytest.param("copr_frontend_a1", "copr-frontend", True),

+             pytest.param("copr-frontend-a1", "copr_frontend", True),

+             pytest.param("copr_frontend_a1", "copr_frontend", True),

+             pytest.param("copr_frontend-a1", "copr-frontend", False),

+             pytest.param("copr-1.1alpha-1", "copr", True),

+             pytest.param("copr-1.1alpha-1beta", "copr", True),

+             pytest.param("copr-1.1-1", "copr", True),

+             pytest.param("copr-1", "copr", True),

+         ]

+     )

+     def test_ref_matches_copr_pkgname(ref, copr_pkg_name, result):

+         # pylint: disable-next=protected-access

+         assert PackagesLogic._ref_matches_copr_pkgname(ref, copr_pkg_name) == result

@@ -260,7 +260,7 @@ 

              "before": "9e6f31bead67d176a71a198f0c10fc764799a4a7",

              "after": "f956357fe84ba899faf9efadeed1f32c8c8cac85",

              "push": {

-                 "changes": [{"new": {"name": "{0}".format(package_name), "type": "tag",

+                 "changes": [{"new": {"name": "{0}-1".format(package_name), "type": "tag",

                                       "target": {"hash": "82c876a27ceafd80465c35e601afab604463ae86"}}}]

              },

              "commits": [{"added": ["{0}".format(package_name)], "removed": [], "modified": []}],

see commit messages for reasoning

Fix #2213

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

rebased onto 804e49cb9b4c359300b98684e790031f713beb62

2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

rebased onto e23e439be73ed47667973734bbc2ce9295d25f6e

2 years ago

Build succeeded.

If pkg_name is defined, this seems fine - but if it is not defined (the old webhook format) we still want to compare the tag, right?

If pkg_name is defined, this seems fine - but if it is not defined (the old webhook format) we still want to compare the tag, right?

I still compare the copr package name with tag name.
package.name in ref
I am just looking for {package.name} instead of exact nvr version and then matching the name

I am just looking for {package.name} instead of exact nvr version and then matching the name

Makes sense! Can you just drop an additional comment before the return X if Y else Z statement explaining what are you doing and why (what effects it has)?

Metadata Update from @praiskup:
- Pull-request tagged with: wip

2 years ago

Build succeeded.

rebased onto 29f07f748ec3ec0d7f67d6792e10831818213554

2 years ago

Build failed. More information on how to proceed and troubleshoot errors available at https://fedoraproject.org/wiki/Zuul-based-ci

2 new commits added

  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

Build succeeded.

Build succeeded.

2 new commits added

  • frontend: document in settings how to use optinal pkg_name argument in the webhooks
  • doc: document optional pkg_name argument in webhooks
2 years ago

Metadata Update from @nikromen:
- Pull-request untagged with: wip

2 years ago

Build succeeded.

The package.name might be e.g. copr, while the tags can be
copr-backend-1.1, copr-frontend-v1, or v1 or just 1.1. I would say we
need at least a bit more complicated matcher than just name in ref.

What about:

if pkg_name == package.name:
    return True

return tag_matches_package_name(tag, package.name)

Then, tt would be nice to match copr-frontend package with all those tags:

copr-frontend-<VERSION>-<RELEASE>
copr-frontend-<VERSION>
copr-frontend-<VERSION>

copr_frontend_<VERSION>_<RELEASE>
copr_frontend_<VERSION>
copr_frontend_<VERSION>

While <version> and <release> should contain at least one digit. Such a matching
method would be super trivial to unit-test.

This part is a bit misplaced. Would you mind adding the example into one box,
near the GitHub section, like:

https://localhost:55555/webhooks/github/5893/ea73ff58-645a-4887-a55f-1d7c818a95fe/
https://localhost:55555/webhooks/github/5893/ea73ff58-645a-4887-a55f-1d7c818a95fe/<PACKAGE_NAME>/

It would be nice to have a link to the full documentation related to webhooks
here, like PACKAGE_NAME is useful for tag events, see FULL DOCS for more info.

rebased onto e3b2bd33a45ff40e978f83589502feeb94e52ceb

2 years ago

Build succeeded.

1 new commit added

  • frontend: add version to the bitbucket webhook tag name
2 years ago

Build succeeded.

Most of the existing packages prefer - separators over _. Testing it (or both) would be better.

This needs double-backtick I guess.

Can you move this "generic" paragraph right after the "Simple guide" part, before we start describing concrete forges?

I would maybe name it like Triggerring builds by tag events. This is what users search for initially, without realizing that the tag has to have a specific format.

I miss some prologue here :-) like .... "One copr has multiple packages... one forge may have multiple packages .... Copr needs to know what package or set of packages should be rebuilt for tag event. Copr uses tags for this...."

This seems difficult to understand, what about just "PKGNAME-VERSION[-RELEASE], as a separator you can use a dash or underscore.

... just please answer "why":
In case you use different tag name patterns, Copr has no idea what package build should be triggered. You have to be explicit.

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

@praiskup I hope I applied your suggestions... It's not easy to write a documentation when I am not sure how the things exactly works :D

Build succeeded.

@praiskup I hope I applied your suggestions... It's not easy to write
documentation when I am not sure how the things exactly works :D

Hold on, you did a very good job - nice docs. Thank you for that!

rebased onto 24fab15e078e4a28d4b7608917b583ef390c9c23

2 years ago

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

Build succeeded.

Shouldn't we add the trailing slash in this route? It just seems inconsistent with the old routes, but dunno...

Unfortunately, the build statuses are not reported back for anything else than Pagure. Suggestion: `s/build statuses reported in/rebuilds for/

Two things to consider ^^ otherwise +1!

rebased onto 88303f2

2 years ago

If with the trailing slash, this example shoud be fixed ^^^.

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

4 new commits added

  • frontend: add version to the bitbucket webhook tag name
  • frontend, doc: document optional pkg_name argument in webhooks
  • frontend: loosen the rules of package matching in webhook tags
  • frontend: add optional argument pkg_name to webhooks API
2 years ago

Build succeeded.

Commit 5dd8a9a fixes this pull-request

Pull-Request has been merged by praiskup

2 years ago

Commit 02296b6 fixes this pull-request

Pull-Request has been merged by praiskup

2 years ago

Commit e7ebcbd fixes this pull-request

Pull-Request has been merged by praiskup

2 years ago

Commit 5a5c816 fixes this pull-request

Pull-Request has been merged by praiskup

2 years ago