[PATCH] doc: Add basic information about running CI tests

[PATCH] doc: Add basic information about running CI tests

[Tom Rini]

Start out by documenting general expectations on when CI is run, how
anyone can run Azure pipelines, and how GitLab CI pipelines can be run.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 doc/develop/ci_testing.rst | 45 ++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst      |  1 +
 2 files changed, 46 insertions(+)
 create mode 100644 doc/develop/ci_testing.rst

diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
new file mode 100644
index 000000000000..48dbec4a4452
--- /dev/null
+++ b/doc/develop/ci_testing.rst
@@ -0,0 +1,45 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Continuous Integration testing
+==============================
+
+All changes require passing our continuous integration tests prior to being
+merged in to mainline.  To help facilitate merges being accepted quickly,
+custodians are encouraged but not required to run a pipeline prior to sending a
+pull request.  Individual developers submitting significant or widespread
+changes are encouraged to run a pipeline themselves prior to posting.
+
+In order to make this process as easy as possible, the ability to run a CI
+pipeline is provided in both Azure and GitLab.  Both of these pipelines perform
+their Linux build jobs on the same Docker container image.
+
+Azure Pipelines
+---------------
+
+Currently there are two ways to run a Microsoft Azure Pipeline test for U-Boot.
+The first way is to create an account with Microsoft at
+https://azure.microsoft.com/en-us/services/devops/ and then use the
+``.azure-pipelines.yml`` file in the U-Boot repository as the pipeline
+description.  The second way is to use GitHub and submit a pull request against
+https://github.com/u-boot/u-boot as this will trigger an Azure pipeline run.
+Clicking on your pull request on the list at
+https://github.com/u-boot/u-boot/pulls and then the "Checks" tab will show the
+results.
+
+GitLab CI Pipelines
+-------------------
+
+Currently, we support running GitLab CI pipelines only for custodians, due to
+the resources the project has available.  For Custodians, it is a matter of
+enabling the pipeline feature in your project repository following the standard
+GitLab documentation.  For non-custodians, the pipeline itself is part of the
+tree and should be able to be used on any GitLab instance, with whatever
+runners you are able to provide.
+
+Docker container
+----------------
+
+As previously states, both of the above pipelines build using the same Docker
+container image.  This is maintained in the repository at
+https://gitlab.denx.de/u-boot/gitlab-ci-runner and new images are made as
+needed to support new tests or features.
diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index 0a7e204b3438..d5619e8fce86 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -7,6 +7,7 @@ Develop U-Boot
 .. toctree::
    :maxdepth: 2
 
+   ci_testing
    coccinelle
    commands
    crash_dumps
-- 
2.17.1

[PATCH] doc: Add basic information about running CI tests

[Sean Anderson]

On 12/18/20 5:14 PM, Tom Rini wrote:
> Start out by documenting general expectations on when CI is run, how
> anyone can run Azure pipelines, and how GitLab CI pipelines can be run.

Should travis be documented as well?

--Sean

> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>   doc/develop/ci_testing.rst | 45 ++++++++++++++++++++++++++++++++++++++
>   doc/develop/index.rst      |  1 +
>   2 files changed, 46 insertions(+)
>   create mode 100644 doc/develop/ci_testing.rst
> 
> diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
> new file mode 100644
> index 000000000000..48dbec4a4452
> --- /dev/null
> +++ b/doc/develop/ci_testing.rst
> @@ -0,0 +1,45 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +Continuous Integration testing
> +==============================
> +
> +All changes require passing our continuous integration tests prior to being
> +merged in to mainline.  To help facilitate merges being accepted quickly,
> +custodians are encouraged but not required to run a pipeline prior to sending a
> +pull request.  Individual developers submitting significant or widespread
> +changes are encouraged to run a pipeline themselves prior to posting.
> +
> +In order to make this process as easy as possible, the ability to run a CI
> +pipeline is provided in both Azure and GitLab.  Both of these pipelines perform
> +their Linux build jobs on the same Docker container image.
> +
> +Azure Pipelines
> +---------------
> +
> +Currently there are two ways to run a Microsoft Azure Pipeline test for U-Boot.
> +The first way is to create an account with Microsoft at
> +https://azure.microsoft.com/en-us/services/devops/ and then use the
> +``.azure-pipelines.yml`` file in the U-Boot repository as the pipeline
> +description.  The second way is to use GitHub and submit a pull request against
> +https://github.com/u-boot/u-boot as this will trigger an Azure pipeline run.
> +Clicking on your pull request on the list at
> +https://github.com/u-boot/u-boot/pulls and then the "Checks" tab will show the
> +results.
> +
> +GitLab CI Pipelines
> +-------------------
> +
> +Currently, we support running GitLab CI pipelines only for custodians, due to
> +the resources the project has available.  For Custodians, it is a matter of
> +enabling the pipeline feature in your project repository following the standard
> +GitLab documentation.  For non-custodians, the pipeline itself is part of the
> +tree and should be able to be used on any GitLab instance, with whatever
> +runners you are able to provide.
> +
> +Docker container
> +----------------
> +
> +As previously states, both of the above pipelines build using the same Docker
> +container image.  This is maintained in the repository at
> +https://gitlab.denx.de/u-boot/gitlab-ci-runner and new images are made as
> +needed to support new tests or features.
> diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> index 0a7e204b3438..d5619e8fce86 100644
> --- a/doc/develop/index.rst
> +++ b/doc/develop/index.rst
> @@ -7,6 +7,7 @@ Develop U-Boot
>   .. toctree::
>      :maxdepth: 2
>   
> +   ci_testing
>      coccinelle
>      commands
>      crash_dumps
> 

[PATCH] doc: Add basic information about running CI tests

[Tom Rini]

On Fri, Dec 18, 2020 at 06:34:10PM -0500, Sean Anderson wrote:
> On 12/18/20 5:14 PM, Tom Rini wrote:
> > Start out by documenting general expectations on when CI is run, how
> > anyone can run Azure pipelines, and how GitLab CI pipelines can be run.
> 
> Should travis be documented as well?

Travis is being dropped as travis-ci.org is ending and travis-ci.com
offers too small of a number of build minutes for free to be useful to
us.

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20201218/f7b895d2/attachment.sig>

[PATCH] doc: Add basic information about running CI tests

[AKASHI Takahiro]

Tom,

Thank you for adding this valuable doc about CI.
I have a couple of comments from my real experience :)

On Fri, Dec 18, 2020 at 05:14:56PM -0500, Tom Rini wrote:
> Start out by documenting general expectations on when CI is run, how
> anyone can run Azure pipelines, and how GitLab CI pipelines can be run.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  doc/develop/ci_testing.rst | 45 ++++++++++++++++++++++++++++++++++++++
>  doc/develop/index.rst      |  1 +
>  2 files changed, 46 insertions(+)
>  create mode 100644 doc/develop/ci_testing.rst
> 
> diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
> new file mode 100644
> index 000000000000..48dbec4a4452
> --- /dev/null
> +++ b/doc/develop/ci_testing.rst
> @@ -0,0 +1,45 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +Continuous Integration testing
> +==============================
> +
> +All changes require passing our continuous integration tests prior to being
> +merged in to mainline.  To help facilitate merges being accepted quickly,
> +custodians are encouraged but not required to run a pipeline prior to sending a
> +pull request.  Individual developers submitting significant or widespread
> +changes are encouraged to run a pipeline themselves prior to posting.
> +
> +In order to make this process as easy as possible, the ability to run a CI
> +pipeline is provided in both Azure and GitLab.  Both of these pipelines perform
> +their Linux build jobs on the same Docker container image.

It would be nice to describe what is the difference between Azure and GitLab
in terms of test conditions/tested platforms if any.

> +
> +Azure Pipelines
> +---------------
> +
> +Currently there are two ways to run a Microsoft Azure Pipeline test for U-Boot.
> +The first way is to create an account with Microsoft at
> +https://azure.microsoft.com/en-us/services/devops/ and then use the
> +``.azure-pipelines.yml`` file in the U-Boot repository as the pipeline
> +description.  The second way is to use GitHub and submit a pull request against
> +https://github.com/u-boot/u-boot as this will trigger an Azure pipeline run.

To make a pull request, you will have to have your own repository which
was cloned from the original. I'm not sure that it is trivial for novice
users.

> +Clicking on your pull request on the list at
> +https://github.com/u-boot/u-boot/pulls and then the "Checks" tab will show the
> +results.
> +
> +GitLab CI Pipelines
> +-------------------
> +
> +Currently, we support running GitLab CI pipelines only for custodians, due to
> +the resources the project has available.  For Custodians, it is a matter of
> +enabling the pipeline feature in your project repository following the standard
> +GitLab documentation.  For non-custodians, the pipeline itself is part of the
> +tree and should be able to be used on any GitLab instance, with whatever
> +runners you are able to provide.
> +
> +Docker container
> +----------------
> +
> +As previously states, both of the above pipelines build using the same Docker
> +container image.  This is maintained in the repository at
> +https://gitlab.denx.de/u-boot/gitlab-ci-runner and new images are made as
> +needed to support new tests or features.

It will be required to add an additional package(s) if you want to
use extra commands in your tests (especially in pytest).
It would be worth noting it here.

It may also be useful to mention about files like
  .azure-pipelines.yml
  .gitlab-ci.yml
  .travis.yml
and suggest that people should take a look at them so that we can
understand how those CI's work for U-Boot.

Thanks,
-Takahiro Akashi

> diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> index 0a7e204b3438..d5619e8fce86 100644
> --- a/doc/develop/index.rst
> +++ b/doc/develop/index.rst
> @@ -7,6 +7,7 @@ Develop U-Boot
>  .. toctree::
>     :maxdepth: 2
>  
> +   ci_testing
>     coccinelle
>     commands
>     crash_dumps
> -- 
> 2.17.1
> 

[PATCH] doc: Add basic information about running CI tests

[AKASHI Takahiro]

On Fri, Dec 18, 2020 at 07:44:42PM -0500, Tom Rini wrote:
> On Fri, Dec 18, 2020 at 06:34:10PM -0500, Sean Anderson wrote:
> > On 12/18/20 5:14 PM, Tom Rini wrote:
> > > Start out by documenting general expectations on when CI is run, how
> > > anyone can run Azure pipelines, and how GitLab CI pipelines can be run.
> > 
> > Should travis be documented as well?
> 
> Travis is being dropped as travis-ci.org is ending and travis-ci.com
> offers too small of a number of build minutes for free to be useful to
> us.

It would make little sense to run the whole set of test cases on Travis,
but I have used Travis until recently in order to run a small set of test
scenario, for example, limiting the scope to
  buildman catch-all AArch64
  test/py sandbox
  test/py qemu-x86_64
by modifying .travis.yml.

Since going through the all test paths is a time-consuming task any way,
it always makes sense for me to run some critical tests first locally
(without revealing those activities in public).

-Takahiro Akashi

> 
> -- 
> Tom

[PATCH] doc: Add basic information about running CI tests

[Tom Rini]

On Mon, Dec 21, 2020 at 11:17:39AM +0900, AKASHI Takahiro wrote:
> On Fri, Dec 18, 2020 at 07:44:42PM -0500, Tom Rini wrote:
> > On Fri, Dec 18, 2020 at 06:34:10PM -0500, Sean Anderson wrote:
> > > On 12/18/20 5:14 PM, Tom Rini wrote:
> > > > Start out by documenting general expectations on when CI is run, how
> > > > anyone can run Azure pipelines, and how GitLab CI pipelines can be run.
> > > 
> > > Should travis be documented as well?
> > 
> > Travis is being dropped as travis-ci.org is ending and travis-ci.com
> > offers too small of a number of build minutes for free to be useful to
> > us.
> 
> It would make little sense to run the whole set of test cases on Travis,
> but I have used Travis until recently in order to run a small set of test
> scenario, for example, limiting the scope to
>   buildman catch-all AArch64
>   test/py sandbox
>   test/py qemu-x86_64
> by modifying .travis.yml.
> 
> Since going through the all test paths is a time-consuming task any way,
> it always makes sense for me to run some critical tests first locally
> (without revealing those activities in public).

Yes, but you can do that for Azure/GitLab too.  I'll expand on that in
v2, along with your other feedback.  Thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: not available
URL: <https://lists.denx.de/pipermail/u-boot/attachments/20201221/0c426f31/attachment.sig>

[PATCHv2] doc: Add basic information about running CI tests

[Tom Rini]

Start out by documenting general expectations on when CI is run, how
anyone can run Azure pipelines, and how GitLab CI pipelines can be run.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v2:
- Address some of Akashi-san's feedback
---
 doc/develop/ci_testing.rst | 45 ++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst      |  1 +
 2 files changed, 46 insertions(+)
 create mode 100644 doc/develop/ci_testing.rst

diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
new file mode 100644
index 000000000000..48dbec4a4452
--- /dev/null
+++ b/doc/develop/ci_testing.rst
@@ -0,0 +1,45 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Continuous Integration testing
+==============================
+
+All changes require passing our continuous integration tests prior to being
+merged in to mainline.  To help facilitate merges being accepted quickly,
+custodians are encouraged but not required to run a pipeline prior to sending a
+pull request.  Individual developers submitting significant or widespread
+changes are encouraged to run a pipeline themselves prior to posting.
+
+In order to make this process as easy as possible, the ability to run a CI
+pipeline is provided in both Azure and GitLab.  Both of these pipelines perform
+their Linux build jobs on the same Docker container image.
+
+Azure Pipelines
+---------------
+
+Currently there are two ways to run a Microsoft Azure Pipeline test for U-Boot.
+The first way is to create an account with Microsoft at
+https://azure.microsoft.com/en-us/services/devops/ and then use the
+``.azure-pipelines.yml`` file in the U-Boot repository as the pipeline
+description.  The second way is to use GitHub and submit a pull request against
+https://github.com/u-boot/u-boot as this will trigger an Azure pipeline run.
+Clicking on your pull request on the list at
+https://github.com/u-boot/u-boot/pulls and then the "Checks" tab will show the
+results.
+
+GitLab CI Pipelines
+-------------------
+
+Currently, we support running GitLab CI pipelines only for custodians, due to
+the resources the project has available.  For Custodians, it is a matter of
+enabling the pipeline feature in your project repository following the standard
+GitLab documentation.  For non-custodians, the pipeline itself is part of the
+tree and should be able to be used on any GitLab instance, with whatever
+runners you are able to provide.
+
+Docker container
+----------------
+
+As previously states, both of the above pipelines build using the same Docker
+container image.  This is maintained in the repository at
+https://gitlab.denx.de/u-boot/gitlab-ci-runner and new images are made as
+needed to support new tests or features.
diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index 54e14dd77b5e..3c0f2da44318 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -9,6 +9,7 @@ Implementation
 .. toctree::
    :maxdepth: 1
 
+   ci_testing
    commands
    driver-model/index
    global_data
-- 
2.17.1

[PATCH] doc: Add basic information about running CI tests

[Tom Rini]

Start out by documenting general expectations on when CI is run, how
anyone can run Azure pipelines, and how GitLab CI pipelines can be run.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
Changes in v3:
- Really address all of the feedback.

Changes in v2:
- Address some of Akashi-san's feedback
---
 doc/develop/ci_testing.rst | 70 ++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst      |  1 +
 2 files changed, 71 insertions(+)
 create mode 100644 doc/develop/ci_testing.rst

diff --git a/doc/develop/ci_testing.rst b/doc/develop/ci_testing.rst
new file mode 100644
index 000000000000..b9a9a516c1e6
--- /dev/null
+++ b/doc/develop/ci_testing.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0+
+
+Continuous Integration testing
+==============================
+
+All changes require passing our continuous integration tests prior to being
+merged in to mainline.  To help facilitate merges being accepted quickly,
+custodians are encouraged but not required to run a pipeline prior to sending a
+pull request.  Individual developers submitting significant or widespread
+changes are encouraged to run a pipeline themselves prior to posting.
+
+In order to make this process as easy as possible, the ability to run a CI
+pipeline is provided in both Azure and GitLab.  Both of these pipelines perform
+their Linux build jobs on the same Docker container image and to cover the same
+platforms.  In addition, Azure is also used to confirm that our host tools can
+be built with mingw to run on Windows.
+
+Each of the pipelines is written in such as way as to be a "world build" style
+test and as such we try and build all possible platforms.  In addition, for all
+platforms that support being run in QEMU we run them in QEMU and use our pytest
+suite.  See :doc:`py_testing` for more information about those tests.
+
+Azure Pipelines
+---------------
+
+This pipeline is defined in the top-level ``.azure-pipelines.yml`` file.
+Currently there are two ways to run a Microsoft Azure Pipeline test for U-Boot.
+
+The first way is to create an account with Microsoft at
+https://azure.microsoft.com/en-us/services/devops/ and then use the
+``.azure-pipelines.yml`` file in the U-Boot repository as the pipeline
+description.
+
+The second way is to use GitHub.  This requires a GitHub account
+and to fork the repository at https://github.com/u-boot/u-boot and to then
+submit a pull request as this will trigger an Azure pipeline run.  Clicking on
+your pull request on the list at https://github.com/u-boot/u-boot/pulls and
+then the "Checks" tab will show the results.
+
+GitLab CI Pipelines
+-------------------
+
+This pipeline is defined in the top-level ``.gitlab-ci.yml`` file.  Currently,
+we support running GitLab CI pipelines only for custodians, due to the
+resources the project has available.  For Custodians, it is a matter of
+enabling the pipeline feature in your project repository following the standard
+GitLab documentation.  For non-custodians, the pipeline itself is part of the
+tree and should be able to be used on any GitLab instance, with whatever
+runners you are able to provide.  While it is intended to be able to run this
+pipeline on the free public instances provided at https://gitlab.com/ a problem
+with our squashfs tests currently prevents this.
+
+Docker container
+----------------
+
+As previously stated, both of the above pipelines build using the same Docker
+container image.  This is maintained in the U-Boot source tree at
+``tools/docker/Dockerfile`` and new images are made as needed to support new
+tests or features.  This file needs to be updated whenever adding new external
+tool requirements to tests.
+
+Customizing CI
+--------------
+
+As noted above, the CI pipelines perform a world build.  While this is good for
+overall project testing, it can be less useful for testing specific cases or
+developing features.  In that case, it can be useful as part of your own
+testing cycle to edit these pipelines in separate local commits to pair them
+down to just the jobs you're interested in.  These changes must be removed
+prior to submission.
diff --git a/doc/develop/index.rst b/doc/develop/index.rst
index 54e14dd77b5e..3c0f2da44318 100644
--- a/doc/develop/index.rst
+++ b/doc/develop/index.rst
@@ -9,6 +9,7 @@ Implementation
 .. toctree::
    :maxdepth: 1
 
+   ci_testing
    commands
    driver-model/index
    global_data
-- 
2.17.1

Re: [PATCH] doc: Add basic information about running CI tests

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

On Tue, Jul 27, 2021 at 01:01:44PM -0400, Tom Rini wrote:

> Start out by documenting general expectations on when CI is run, how
> anyone can run Azure pipelines, and how GitLab CI pipelines can be run.
> 
> Signed-off-by: Tom Rini <trini@konsulko.com>

Applied to u-boot/master, thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]