[PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Philippe Mathieu-Daudé]

All jobs depending on 'docker-edk2' job must use at most all
the rules that triggers it. The simplest way to ensure that
is to always use the same rules. Extract all the rules to a
reusable section, and include this section (with the 'extends'
keyword) in both 'docker-edk2' and 'build-edk2' jobs.

The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
Add jobs to build EDK2 firmware binaries"), but was revealed in
commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
container") and eventually failed on CI:
https://gitlab.com/qemu-project/qemu/-/pipelines/335995843

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
If this is a correct fix, I'll send a similar fix for the
OpenSBI jobs.
---
 .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
 1 file changed, 16 insertions(+), 11 deletions(-)

diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index ba7280605c4..aae2f7ad880 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -1,10 +1,22 @@
-docker-edk2:
- stage: containers
- rules: # Only run this job when the Dockerfile is modified
+# All jobs needing docker-edk2 must use the same rules it uses.
+.edk2_job_rules:
+ rules: # Only run this job when ...
  - changes:
+   # this file is modified
    - .gitlab-ci.d/edk2.yml
+   # or the Dockerfile is modified
    - .gitlab-ci.d/edk2/Dockerfile
+   # or roms/edk2/ is modified (submodule updated)
+   - roms/edk2/*
    when: always
+ - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
+   when: always
+ - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
+   when: always
+
+docker-edk2:
+ extends: .edk2_job_rules
+ stage: containers
  image: docker:19.03.1
  services:
  - docker:19.03.1-dind
@@ -24,16 +36,9 @@ docker-edk2:
  - docker push $IMAGE_TAG
 
 build-edk2:
+ extends: .edk2_job_rules
  stage: build
  needs: ['docker-edk2']
- rules: # Only run this job when ...
- - changes: # ... roms/edk2/ is modified (submodule updated)
-   - roms/edk2/*
-   when: always
- - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
-   when: always
- - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
-   when: always
  artifacts:
    paths: # 'artifacts.zip' will contains the following files:
    - pc-bios/edk2*bz2
-- 
2.31.1

Re: [PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Daniel P. Berrangé]

On Wed, Jul 14, 2021 at 12:10:03PM +0200, Philippe Mathieu-Daudé wrote:
> All jobs depending on 'docker-edk2' job must use at most all
> the rules that triggers it. The simplest way to ensure that
> is to always use the same rules. Extract all the rules to a
> reusable section, and include this section (with the 'extends'
> keyword) in both 'docker-edk2' and 'build-edk2' jobs.
> 
> The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
> Add jobs to build EDK2 firmware binaries"), but was revealed in
> commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
> container") and eventually failed on CI:
> https://gitlab.com/qemu-project/qemu/-/pipelines/335995843
> 
> Reported-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> If this is a correct fix, I'll send a similar fix for the
> OpenSBI jobs.
> ---
>  .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Willian Rampazzo]

On Wed, Jul 14, 2021 at 7:10 AM Philippe Mathieu-Daudé
<philmd@redhat.com> wrote:
>
> All jobs depending on 'docker-edk2' job must use at most all
> the rules that triggers it. The simplest way to ensure that
> is to always use the same rules. Extract all the rules to a
> reusable section, and include this section (with the 'extends'
> keyword) in both 'docker-edk2' and 'build-edk2' jobs.
>
> The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
> Add jobs to build EDK2 firmware binaries"), but was revealed in
> commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
> container") and eventually failed on CI:
> https://gitlab.com/qemu-project/qemu/-/pipelines/335995843
>
> Reported-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> If this is a correct fix, I'll send a similar fix for the
> OpenSBI jobs.
> ---
>  .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)
>

Reviewed-by: Willian Rampazzo <willianr@redhat.com>

Re: [PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Cleber Rosa]

Philippe Mathieu-Daudé writes:

> All jobs depending on 'docker-edk2' job must use at most all
> the rules that triggers it. The simplest way to ensure that
> is to always use the same rules. Extract all the rules to a
> reusable section, and include this section (with the 'extends'
> keyword) in both 'docker-edk2' and 'build-edk2' jobs.
>
> The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
> Add jobs to build EDK2 firmware binaries"), but was revealed in
> commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
> container") and eventually failed on CI:
> https://gitlab.com/qemu-project/qemu/-/pipelines/335995843
>
> Reported-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> If this is a correct fix, I'll send a similar fix for the
> OpenSBI jobs.
> ---
>  .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)
>
> diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
> index ba7280605c4..aae2f7ad880 100644
> --- a/.gitlab-ci.d/edk2.yml
> +++ b/.gitlab-ci.d/edk2.yml
> @@ -1,10 +1,22 @@
> -docker-edk2:
> - stage: containers
> - rules: # Only run this job when the Dockerfile is modified
> +# All jobs needing docker-edk2 must use the same rules it uses.
> +.edk2_job_rules:
> + rules: # Only run this job when ...
>   - changes:
> +   # this file is modified
>     - .gitlab-ci.d/edk2.yml
> +   # or the Dockerfile is modified
>     - .gitlab-ci.d/edk2/Dockerfile
> +   # or roms/edk2/ is modified (submodule updated)
> +   - roms/edk2/*
>     when: always
> + - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
> +   when: always
> + - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
> +   when: always
> +
> +docker-edk2:
> + extends: .edk2_job_rules
> + stage: containers
>   image: docker:19.03.1
>   services:
>   - docker:19.03.1-dind
> @@ -24,16 +36,9 @@ docker-edk2:
>   - docker push $IMAGE_TAG
>  
>  build-edk2:
> + extends: .edk2_job_rules
>   stage: build
>   needs: ['docker-edk2']
> - rules: # Only run this job when ...
> - - changes: # ... roms/edk2/ is modified (submodule updated)
> -   - roms/edk2/*
> -   when: always
> - - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
> -   when: always
> - - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
> -   when: always
>   artifacts:
>     paths: # 'artifacts.zip' will contains the following files:
>     - pc-bios/edk2*bz2


Hi Phil,

This looks good, but have you triggered a pipeline with these changes?
It'd be helpful for an extra level of assurance for the intended results.

Thanks!

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Philippe Mathieu-Daudé]

On 7/14/21 8:14 PM, Cleber Rosa wrote:
> 
> Philippe Mathieu-Daudé writes:
> 
>> All jobs depending on 'docker-edk2' job must use at most all
>> the rules that triggers it. The simplest way to ensure that
>> is to always use the same rules. Extract all the rules to a
>> reusable section, and include this section (with the 'extends'
>> keyword) in both 'docker-edk2' and 'build-edk2' jobs.
>>
>> The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
>> Add jobs to build EDK2 firmware binaries"), but was revealed in
>> commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
>> container") and eventually failed on CI:
>> https://gitlab.com/qemu-project/qemu/-/pipelines/335995843
>>
>> Reported-by: Daniel P. Berrangé <berrange@redhat.com>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>> If this is a correct fix, I'll send a similar fix for the
>> OpenSBI jobs.
>> ---
>>  .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
>>  1 file changed, 16 insertions(+), 11 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
>> index ba7280605c4..aae2f7ad880 100644
>> --- a/.gitlab-ci.d/edk2.yml
>> +++ b/.gitlab-ci.d/edk2.yml
>> @@ -1,10 +1,22 @@
>> -docker-edk2:
>> - stage: containers
>> - rules: # Only run this job when the Dockerfile is modified
>> +# All jobs needing docker-edk2 must use the same rules it uses.
>> +.edk2_job_rules:
>> + rules: # Only run this job when ...
>>   - changes:
>> +   # this file is modified
>>     - .gitlab-ci.d/edk2.yml
>> +   # or the Dockerfile is modified
>>     - .gitlab-ci.d/edk2/Dockerfile
>> +   # or roms/edk2/ is modified (submodule updated)
>> +   - roms/edk2/*
>>     when: always
>> + - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
>> +   when: always
>> + - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
>> +   when: always
>> +
>> +docker-edk2:
>> + extends: .edk2_job_rules
>> + stage: containers
>>   image: docker:19.03.1
>>   services:
>>   - docker:19.03.1-dind
>> @@ -24,16 +36,9 @@ docker-edk2:
>>   - docker push $IMAGE_TAG
>>  
>>  build-edk2:
>> + extends: .edk2_job_rules
>>   stage: build
>>   needs: ['docker-edk2']
>> - rules: # Only run this job when ...
>> - - changes: # ... roms/edk2/ is modified (submodule updated)
>> -   - roms/edk2/*
>> -   when: always
>> - - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 'edk2'
>> -   when: always
>> - - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 'EDK2'
>> -   when: always
>>   artifacts:
>>     paths: # 'artifacts.zip' will contains the following files:
>>     - pc-bios/edk2*bz2
> 
> 
> Hi Phil,
> 
> This looks good, but have you triggered a pipeline with these changes?
> It'd be helpful for an extra level of assurance for the intended results.

Yes, before sending this patch ;)

https://gitlab.com/philmd_rh/qemu/-/pipelines/336800660

Re: [PATCH-for-6.1] gitlab-ci: Extract EDK2 job rules to reusable section

[Philippe Mathieu-Daudé]

On 7/14/21 12:10 PM, Philippe Mathieu-Daudé wrote:
> All jobs depending on 'docker-edk2' job must use at most all
> the rules that triggers it. The simplest way to ensure that
> is to always use the same rules. Extract all the rules to a
> reusable section, and include this section (with the 'extends'
> keyword) in both 'docker-edk2' and 'build-edk2' jobs.
> 
> The problem was introduced in commit 71920809cea ("gitlab-ci.yml:
> Add jobs to build EDK2 firmware binaries"), but was revealed in
> commit 1925468ddbf ("docker: EDK2 build job depends on EDK2
> container") and eventually failed on CI:
> https://gitlab.com/qemu-project/qemu/-/pipelines/335995843
> 
> Reported-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
> If this is a correct fix, I'll send a similar fix for the
> OpenSBI jobs.
> ---
>  .gitlab-ci.d/edk2.yml | 27 ++++++++++++++++-----------
>  1 file changed, 16 insertions(+), 11 deletions(-)

Queued to fw-edk2 tree.