[PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[Philippe Mathieu-Daudé]

Hi Michael,

2 more patches to avoid gitlab-ci mayhem when you push the
stable tags. See this cover for more info:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg846861.html

Based-on: <20211019140944.152419-1-michael.roth@amd.com>
"Patch Round-up for stable 6.0.1, freeze on 2021-10-26"

Daniel P. Berrangé (1):
  gitlab: only let pages be published from default branch

Philippe Mathieu-Daudé (1):
  gitlab-ci: Only push docker images to registry from /master branch

 .gitlab-ci.d/containers.yml | 11 ++++++++++-
 .gitlab-ci.d/edk2.yml       | 11 ++++++++++-
 .gitlab-ci.d/opensbi.yml    | 11 ++++++++++-
 .gitlab-ci.yml              | 18 ++++++++++++++++++
 4 files changed, 48 insertions(+), 3 deletions(-)

-- 
2.31.1

[PATCH-for-6.0.1 1/2] gitlab: only let pages be published from default branch

[Philippe Mathieu-Daudé]

From: Daniel P. Berrangé <berrange@redhat.com>

GitLab will happily publish pages generated by the latest CI pipeline
from any branch:

https://docs.gitlab.com/ee/user/project/pages/introduction.html

  "Remember that GitLab Pages are by default branch/tag agnostic
   and their deployment relies solely on what you specify in
   .gitlab-ci.yml. You can limit the pages job with the only
   parameter, whenever a new commit is pushed to a branch used
   specifically for your pages."

The current "pages" job is not limited, so it is happily publishing
docs content from any branch/tag in qemu.git that gets pushed to.
This means we're potentially publishing from the "staging" branch
or worse from outdated "stable-NNN" branches

This change restricts it to only publish from the default branch
in the main repository. For contributor forks, however, we allow
it to publish from any branch, since users will have arbitrarily
named topic branches in flight at any time.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210723113051.2792799-1-berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
(cherry picked from commit eafadbbbac06a8d72baa976f4d3c42b0e5f8cfc8)
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.yml | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 52d65d6c04f..f8cc67baad9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -817,6 +817,17 @@ build-tools-and-docs-debian:
 
 # Prepare for GitLab pages deployment. Anything copied into the
 # "public" directory will be deployed to $USER.gitlab.io/$PROJECT
+#
+# GitLab publishes from any branch that triggers a CI pipeline
+#
+# For the main repo we don't want to publish from 'staging'
+# since that content may not be pushed, nor do we wish to
+# publish from 'stable-NNN' branches as that content is outdated.
+# Thus we restrict to just the default branch
+#
+# For contributor forks we want to publish from any repo so
+# that users can see the results of their commits, regardless
+# of what topic branch they're currently using
 pages:
   image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
   stage: test
@@ -835,3 +846,10 @@ pages:
   artifacts:
     paths:
       - public
+  rules:
+    - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
+      when: on_success
+    - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
+      when: never
+    - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
+      when: on_success
-- 
2.31.1

[PATCH-for-6.0.1 2/2] gitlab-ci: Only push docker images to registry from /master branch

[Philippe Mathieu-Daudé]

Users expect images pulled from registry.gitlab.com/qemu-project/qemu/
to be stable. QEMU repository workflow pushes merge candidates to
the /staging branch, and on success the same commit is pushed as
/master. If /staging fails, we do not want to push the built images
to the registry. Therefore limit the 'docker push' command to the
/master branch on the mainstream CI. The fork behavior is unchanged.

Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20211026145509.1029274-1-philmd@redhat.com>
---
 .gitlab-ci.d/containers.yml | 11 ++++++++++-
 .gitlab-ci.d/edk2.yml       | 11 ++++++++++-
 .gitlab-ci.d/opensbi.yml    | 11 ++++++++++-
 3 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index 33e4046e233..57928e4fb11 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -17,7 +17,16 @@
           -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker"
           -r $CI_REGISTRY_IMAGE
     - docker tag "qemu/$NAME" "$TAG"
-    - docker push "$TAG"
+    # On mainstream CI, we only want to push images on the master branch,
+    # so skip the other cases (tag or non-master branch).
+    - if test "$CI_PROJECT_NAMESPACE" = "qemu-project" &&
+         test -n "$CI_COMMIT_TAG"
+              -o "$CI_COMMIT_BRANCH" != "$CI_DEFAULT_BRANCH";
+      then
+        :;
+      else
+        docker push "$TAG";
+      fi
   after_script:
     - docker logout
 
diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index ba7280605c4..b165cc95214 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -21,7 +21,16 @@ docker-edk2:
  - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
                                         --tag $IMAGE_TAG .gitlab-ci.d/edk2
  - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- - docker push $IMAGE_TAG
+ # On mainstream CI, we only want to push images on the master branch,
+ # so skip the other cases (tag or non-master branch).
+ - if test "$CI_PROJECT_NAMESPACE" = "qemu-project" &&
+      test -n "$CI_COMMIT_TAG"
+           -o "$CI_COMMIT_BRANCH" != "$CI_DEFAULT_BRANCH";
+   then
+      :;
+   else
+      docker push "$IMAGE_TAG";
+   fi
 
 build-edk2:
  stage: build
diff --git a/.gitlab-ci.d/opensbi.yml b/.gitlab-ci.d/opensbi.yml
index f66cd1d9089..908540daaf7 100644
--- a/.gitlab-ci.d/opensbi.yml
+++ b/.gitlab-ci.d/opensbi.yml
@@ -21,7 +21,16 @@ docker-opensbi:
  - docker build --cache-from $IMAGE_TAG --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
                                         --tag $IMAGE_TAG .gitlab-ci.d/opensbi
  - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- - docker push $IMAGE_TAG
+ # On mainstream CI, we only want to push images on the master branch,
+ # so skip the other cases (tag or non-master branch).
+ - if test "$CI_PROJECT_NAMESPACE" = "qemu-project" &&
+      test -n "$CI_COMMIT_TAG"
+           -o "$CI_COMMIT_BRANCH" != "$CI_DEFAULT_BRANCH";
+   then
+      :;
+   else
+      docker push "$IMAGE_TAG";
+   fi
 
 build-opensbi:
  stage: build
-- 
2.31.1

Re: [PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[Daniel P. Berrangé]

On Wed, Oct 27, 2021 at 07:26:54AM +0200, Philippe Mathieu-Daudé wrote:
> Hi Michael,
> 
> 2 more patches to avoid gitlab-ci mayhem when you push the
> stable tags. See this cover for more info:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg846861.html

Please don't push this to stable - Thomas points out that it is broken
when any changes to dockerfiles are made.

> 
> Based-on: <20211019140944.152419-1-michael.roth@amd.com>
> "Patch Round-up for stable 6.0.1, freeze on 2021-10-26"
> 
> Daniel P. Berrangé (1):
>   gitlab: only let pages be published from default branch
> 
> Philippe Mathieu-Daudé (1):
>   gitlab-ci: Only push docker images to registry from /master branch
> 
>  .gitlab-ci.d/containers.yml | 11 ++++++++++-
>  .gitlab-ci.d/edk2.yml       | 11 ++++++++++-
>  .gitlab-ci.d/opensbi.yml    | 11 ++++++++++-
>  .gitlab-ci.yml              | 18 ++++++++++++++++++
>  4 files changed, 48 insertions(+), 3 deletions(-)
> 
> -- 
> 2.31.1
> 
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[Philippe Mathieu-Daudé]

+Richard/Peter

On 10/27/21 10:49, Daniel P. Berrangé wrote:
> On Wed, Oct 27, 2021 at 07:26:54AM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Michael,
>>
>> 2 more patches to avoid gitlab-ci mayhem when you push the
>> stable tags. See this cover for more info:
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg846861.html
> 
> Please don't push this to stable - Thomas points out that it is broken
> when any changes to dockerfiles are made.

But we still don't want to update the registry with these old
images...

What is the plan then, hold the stable tag until we figure out
the best fix?

Otherwise Michael can use 'git-push --push-option=ci.skip' to
not trigger a CI pipeline when pushing stable tags (running
CI pipelines previously in his own gitlab namespace).

>> Based-on: <20211019140944.152419-1-michael.roth@amd.com>
>> "Patch Round-up for stable 6.0.1, freeze on 2021-10-26"
>>
>> Daniel P. Berrangé (1):
>>   gitlab: only let pages be published from default branch
>>
>> Philippe Mathieu-Daudé (1):
>>   gitlab-ci: Only push docker images to registry from /master branch
>>
>>  .gitlab-ci.d/containers.yml | 11 ++++++++++-
>>  .gitlab-ci.d/edk2.yml       | 11 ++++++++++-
>>  .gitlab-ci.d/opensbi.yml    | 11 ++++++++++-
>>  .gitlab-ci.yml              | 18 ++++++++++++++++++
>>  4 files changed, 48 insertions(+), 3 deletions(-)
>>
>> -- 
>> 2.31.1
>>
>>
>>
> 
> Regards,
> Daniel
> 

Re: [PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[Daniel P. Berrangé]

On Wed, Oct 27, 2021 at 12:09:39PM +0200, Philippe Mathieu-Daudé wrote:
> +Richard/Peter
> 
> On 10/27/21 10:49, Daniel P. Berrangé wrote:
> > On Wed, Oct 27, 2021 at 07:26:54AM +0200, Philippe Mathieu-Daudé wrote:
> >> Hi Michael,
> >>
> >> 2 more patches to avoid gitlab-ci mayhem when you push the
> >> stable tags. See this cover for more info:
> >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg846861.html
> > 
> > Please don't push this to stable - Thomas points out that it is broken
> > when any changes to dockerfiles are made.
> 
> But we still don't want to update the registry with these old
> images...
> 
> What is the plan then, hold the stable tag until we figure out
> the best fix?
> 
> Otherwise Michael can use 'git-push --push-option=ci.skip' to
> not trigger a CI pipeline when pushing stable tags (running
> CI pipelines previously in his own gitlab namespace).

Yes, I'd suggest that approach currently, until we figure out a
real long term solution.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH-for-6.0.1 0/2] gitlab-ci: Only push docker images to mainstream registry from /master

[Michael Roth via]

On Wed, Oct 27, 2021 at 12:09:39PM +0200, Philippe Mathieu-Daudé wrote:
> +Richard/Peter
> 
> On 10/27/21 10:49, Daniel P. Berrangé wrote:
> > On Wed, Oct 27, 2021 at 07:26:54AM +0200, Philippe Mathieu-Daudé wrote:
> >> Hi Michael,
> >>
> >> 2 more patches to avoid gitlab-ci mayhem when you push the
> >> stable tags. See this cover for more info:
> >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fqemu-devel%40nongnu.org%2Fmsg846861.html&data=04%7C01%7Cmichael.roth%40amd.com%7C3c19b44a450a4db8aa1c08d99931e741%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637709261892671750%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kPrWrqTCJSgz%2FLoCfYNDyIb6zqXY%2Fl8v1p4IgUg5psM%3D&reserved=0
> > 
> > Please don't push this to stable - Thomas points out that it is broken
> > when any changes to dockerfiles are made.
> 
> But we still don't want to update the registry with these old
> images...
> 
> What is the plan then, hold the stable tag until we figure out
> the best fix?
> 
> Otherwise Michael can use 'git-push --push-option=ci.skip' to
> not trigger a CI pipeline when pushing stable tags (running
> CI pipelines previously in his own gitlab namespace).

I can take this approach for now.

Thanks everyone for all the debugging/suggestions.

-Mike

> 
> >> Based-on: <20211019140944.152419-1-michael.roth@amd.com>
> >> "Patch Round-up for stable 6.0.1, freeze on 2021-10-26"
> >>
> >> Daniel P. Berrangé (1):
> >>   gitlab: only let pages be published from default branch
> >>
> >> Philippe Mathieu-Daudé (1):
> >>   gitlab-ci: Only push docker images to registry from /master branch
> >>
> >>  .gitlab-ci.d/containers.yml | 11 ++++++++++-
> >>  .gitlab-ci.d/edk2.yml       | 11 ++++++++++-
> >>  .gitlab-ci.d/opensbi.yml    | 11 ++++++++++-
> >>  .gitlab-ci.yml              | 18 ++++++++++++++++++
> >>  4 files changed, 48 insertions(+), 3 deletions(-)
> >>
> >> -- 
> >> 2.31.1
> >>
> >>
> >>
> > 
> > Regards,
> > Daniel
> > 
>