Upcoming enhancements to `run-unit-test-docker`

Upcoming enhancements to `run-unit-test-docker`

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2891 bytes --]

Hello,

Inside openbmc-build-scripts is a tool which is used by Jenkins to run
repository-level CI and I think some developers also use it:
`run-unit-test-docker`.  I've rewritten a component of this, which is
now at `scripts/build-unit-test-docker`, in Python and done some
enhancements to it.  The [currently] last commit in this sequence is at
[1].

Important notices:

    - Location change: build-unit-test-docker.sh ->
      scripts/build-unit-test-docker
        + I suspect few people run this directly, but instead use
          `run-unit-test-docker` which hides the rename.

    - Python3.6 minimum.
        + Bitbake already requires this so it really shouldn't be a
          problem for anyone.

    - Python 'sh' module needed.
        + You can install this with the 'python3-sh' package on many
          distros or 'pip install sh'.

    - Docker stages are now tagged with different names / tag revs.
        + I wrote a `scripts/clean-unit-test-docker` which will clean
          out old Docker tags and can be ran periodically.

Major changes:

    - Rewrite in Python.
    - Significant speed improvements:
        - Docker stages all build in parallel to formulate the final image.
        - All Docker stages are tagged and reused from run to run.
        - CMake packages now use the Ninja backend.  This is the same as
          bitbake and faster than make.
    - Included packages are defined in a dictionary rather than as raw
      Dockerfiles.

Upcoming changes:

    - Shortly, I will add support to incorporate base package changes
      into the image itself.  We have had cases where a change to
      a common package like `sdbusplus` broke CI because a package that
      depended on it failed to compile with the changes.  I'll add code
      that uses some of the GERRIT_* environment variables to determine
      if the commit-under-test is part of one of the base packages and
      include it in the Docker image for the test run.

    - Topic-based testing.  We often run into co-dependencies between
      phosphor-dbus-interfaces and another repository.  Within the next
      month I'll enhance the script so that if you change PDI and your
      repository and tag the commit with the same topic, the PDI commit
      from that topic will be pulled into the Docker image so your
      repository commit can pass CI.

    - Some refactoring.  I did a lot of incremental changes from the
      original shell script to the Python version I currently have and
      didn't always rewrite in the most Pythonic way.  There will be
      some more, mostly non-functional, changes to fix this.

Feel free to comment on the review series if you have any concerns with
these changes.

1. https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40366

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Upcoming enhancements to `run-unit-test-docker`

[Johnathan Mantey]

[-- Attachment #1.1: Type: text/plain, Size: 4860 bytes --]

On 2/11/21 4:28 PM, Patrick Williams wrote:
> Hello,
>
> Inside openbmc-build-scripts is a tool which is used by Jenkins to run
> repository-level CI and I think some developers also use it:
> `run-unit-test-docker`.  I've rewritten a component of this, which is
> now at `scripts/build-unit-test-docker`, in Python and done some
> enhancements to it.  The [currently] last commit in this sequence is at
> [1].
>
> Important notices:
>
>      - Location change: build-unit-test-docker.sh ->
>        scripts/build-unit-test-docker
>          + I suspect few people run this directly, but instead use
>            `run-unit-test-docker` which hides the rename.
>
>      - Python3.6 minimum.
>          + Bitbake already requires this so it really shouldn't be a
>            problem for anyone.
>
>      - Python 'sh' module needed.
>          + You can install this with the 'python3-sh' package on many
>            distros or 'pip install sh'.
>
>      - Docker stages are now tagged with different names / tag revs.
>          + I wrote a `scripts/clean-unit-test-docker` which will clean
>            out old Docker tags and can be ran periodically.
>
> Major changes:
>
>      - Rewrite in Python.
>      - Significant speed improvements:
>          - Docker stages all build in parallel to formulate the final image.
>          - All Docker stages are tagged and reused from run to run.
>          - CMake packages now use the Ninja backend.  This is the same as
>            bitbake and faster than make.
>      - Included packages are defined in a dictionary rather than as raw
>        Dockerfiles.
>
> Upcoming changes:
>
>      - Shortly, I will add support to incorporate base package changes
>        into the image itself.  We have had cases where a change to
>        a common package like `sdbusplus` broke CI because a package that
>        depended on it failed to compile with the changes.  I'll add code
>        that uses some of the GERRIT_* environment variables to determine
>        if the commit-under-test is part of one of the base packages and
>        include it in the Docker image for the test run.
>
>      - Topic-based testing.  We often run into co-dependencies between
>        phosphor-dbus-interfaces and another repository.  Within the next
>        month I'll enhance the script so that if you change PDI and your
>        repository and tag the commit with the same topic, the PDI commit
>        from that topic will be pulled into the Docker image so your
>        repository commit can pass CI.
>
>      - Some refactoring.  I did a lot of incremental changes from the
>        original shell script to the Python version I currently have and
>        didn't always rewrite in the most Pythonic way.  There will be
>        some more, mostly non-functional, changes to fix this.
>
> Feel free to comment on the review series if you have any concerns with
> these changes.
>
> 1. https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40366
>
I tried using this script a day or two ago. I needed to fix some unit 
tests for a commit I pushed earlier this month. I had used the script in 
the past and couldn't remember how to use it.

A year ago I had to fix some unit tests, and I had to get Docker, and 
this test script configured. I have long since forgotten how I got the 
system setup, so I went back to try and run the unit tests I had fully 
configured and running from a year ago. In spite of having a fully 
configured environment I could not figure out how to run the tests any 
longer.

I decided to find the run-unit-test-docker.sh contents on the github 
repos, to see if I could read prepared content for how to configure, and 
run the unit test script. The readme.md is one line. The comments in the 
shell script assume you know what each settable parameter does, and how 
it is to be assigned.

I'm pretty sure I had 1:1 assistance from someone familiar with the test 
configuration to get this to work. The effort involved is more than "get 
docker, run script".

I request that someone familiar with the configuration and operation of 
this script please provide information about how to get the environment 
configured, and some example use cases for running the system. One place 
in particular that I found difficult/impossible was to run a test on 
code that required changes in other repositories. For example, if 
phosphor-network needed changes that are provided by 
phosphor-dbus-interfaces how would you configure the test run to be 
successful?

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Upcoming enhancements to `run-unit-test-docker`

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3317 bytes --]

On Fri, Feb 12, 2021 at 08:22:02AM -0800, Johnathan Mantey wrote:
> On 2/11/21 4:28 PM, Patrick Williams wrote:
> I tried using this script a day or two ago. I needed to fix some unit 
> tests for a commit I pushed earlier this month. I had used the script in 
> the past and couldn't remember how to use it.
> 
> A year ago I had to fix some unit tests, and I had to get Docker, and 
> this test script configured. I have long since forgotten how I got the 
> system setup, so I went back to try and run the unit tests I had fully 
> configured and running from a year ago. In spite of having a fully 
> configured environment I could not figure out how to run the tests any 
> longer.
> 
> I decided to find the run-unit-test-docker.sh contents on the github 
> repos, to see if I could read prepared content for how to configure, and 
> run the unit test script. The readme.md is one line. The comments in the 
> shell script assume you know what each settable parameter does, and how 
> it is to be assigned.

Andrew Jeffery pointed out this document last night:
    https://github.com/openbmc/docs/blob/master/testing/local-ci-build.md

I have all of my repositories cloned into a single subdirectory, which
makes using this script much easier than if they were spread out.
You'll see the doc above references as `/path/to/ci_test_area` and doing
additional clones / git-worktrees, but if you just keep everything
cloned in one sub-directory that *is* the ci_test_area / WORKSPACE.

I typically run this little helper script from a repo I want to test:

$ which lf-ut
lf-ut () {
	REPO="$(git rev-parse --show-toplevel)"
	UT_PATH="$(wd path obmcsrc)/openbmc-build-scripts"
	UNIT_TEST_PKG="$(basename $REPO)" WORKSPACE="$(dirname $REPO)" "$UT_PATH/run-unit-test-docker.sh"
}
$ EXTRA_UNIT_TEST_ARGS="--noformat" lf-ut

I use zsh, so ... Anyhow, `wd path obmcsrc` is the path to my workspace
where all the code is extracted.

> I'm pretty sure I had 1:1 assistance from someone familiar with the test 
> configuration to get this to work. The effort involved is more than "get 
> docker, run script".
> 
> I request that someone familiar with the configuration and operation of 
> this script please provide information about how to get the environment 
> configured, and some example use cases for running the system. One place 
> in particular that I found difficult/impossible was to run a test on 
> code that required changes in other repositories. For example, if 
> phosphor-network needed changes that are provided by 
> phosphor-dbus-interfaces how would you configure the test run to be 
> successful?

Right now there isn't a way to test two repositories together.  This is
an aspect I mentioned of wanting to improve (at least for Gerrit).  Your
only option right now is:

   1. Push your phosphor-dbus-interfaces change to Gerrit.
   2. Edit `scripts/build-unit-test-docker` and insert a "rev" keyword
      under "phosphor-dbus-interfaces" with your git commit hash from
      step 1 (this should hopefully be obvious when you look at the
      script).
   3. Run something like `lf-ut` above from inside phosphor-network.

** Step #2 assumes you're running my new versions of the script **

Hope this helps!

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Upcoming enhancements to `run-unit-test-docker`

[Andrew Geissler]

> On Feb 11, 2021, at 6:28 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> Hello,
> 
> Inside openbmc-build-scripts is a tool which is used by Jenkins to run
> repository-level CI and I think some developers also use it:
> `run-unit-test-docker`.  I've rewritten a component of this, which is
> now at `scripts/build-unit-test-docker`, in Python and done some
> enhancements to it.  The [currently] last commit in this sequence is at
> [1].
> 
> Important notices:
> 
>    - Location change: build-unit-test-docker.sh ->
>      scripts/build-unit-test-docker
>        + I suspect few people run this directly, but instead use
>          `run-unit-test-docker` which hides the rename.
> 
>    - Python3.6 minimum.
>        + Bitbake already requires this so it really shouldn't be a
>          problem for anyone.
> 
>    - Python 'sh' module needed.
>        + You can install this with the 'python3-sh' package on many
>          distros or 'pip install sh'.
> 
>    - Docker stages are now tagged with different names / tag revs.
>        + I wrote a `scripts/clean-unit-test-docker` which will clean
>          out old Docker tags and can be ran periodically.
> 
> Major changes:
> 
>    - Rewrite in Python.
>    - Significant speed improvements:
>        - Docker stages all build in parallel to formulate the final image.
>        - All Docker stages are tagged and reused from run to run.
>        - CMake packages now use the Ninja backend.  This is the same as
>          bitbake and faster than make.
>    - Included packages are defined in a dictionary rather than as raw
>      Dockerfiles.


Thanks for doing this Patrick! I submitted your first set of
commits this morning so we’re officially on the new python
based scripts in CI.

Let Patrick and I know if you see anything amiss.

Re: [External] Upcoming enhancements to `run-unit-test-docker`

[Lei Yu]

Reporting a bug on building with proxy:

    base: WARNING: Retrying (Retry(total=4, connect=None, read=None,
redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.',
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7efe419e99d0>: Failed to establish a new connection: [Errno -2]
Name or service not known'))': /simple/inflection/
    RAN: /usr/bin/docker build --build-arg http_proxy=http://<myproxy>
--build-arg 'https_proxy={https_proxy}' --network=host --force-rm
--no-cache=false -t
openbmc/ubuntu-unit-test-base:2021-W07-773f263e895d8ed6 -

The https_proxy is not correctly set.
Submitted https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40699
to fix the issue.

On Fri, Feb 12, 2021 at 8:51 AM Patrick Williams <patrick@stwcx.xyz> wrote:
>
> Hello,
>
> Inside openbmc-build-scripts is a tool which is used by Jenkins to run
> repository-level CI and I think some developers also use it:
> `run-unit-test-docker`.  I've rewritten a component of this, which is
> now at `scripts/build-unit-test-docker`, in Python and done some
> enhancements to it.  The [currently] last commit in this sequence is at
> [1].
>
> Important notices:
>
>     - Location change: build-unit-test-docker.sh ->
>       scripts/build-unit-test-docker
>         + I suspect few people run this directly, but instead use
>           `run-unit-test-docker` which hides the rename.
>
>     - Python3.6 minimum.
>         + Bitbake already requires this so it really shouldn't be a
>           problem for anyone.
>
>     - Python 'sh' module needed.
>         + You can install this with the 'python3-sh' package on many
>           distros or 'pip install sh'.
>
>     - Docker stages are now tagged with different names / tag revs.
>         + I wrote a `scripts/clean-unit-test-docker` which will clean
>           out old Docker tags and can be ran periodically.
>
> Major changes:
>
>     - Rewrite in Python.
>     - Significant speed improvements:
>         - Docker stages all build in parallel to formulate the final image.
>         - All Docker stages are tagged and reused from run to run.
>         - CMake packages now use the Ninja backend.  This is the same as
>           bitbake and faster than make.
>     - Included packages are defined in a dictionary rather than as raw
>       Dockerfiles.
>
> Upcoming changes:
>
>     - Shortly, I will add support to incorporate base package changes
>       into the image itself.  We have had cases where a change to
>       a common package like `sdbusplus` broke CI because a package that
>       depended on it failed to compile with the changes.  I'll add code
>       that uses some of the GERRIT_* environment variables to determine
>       if the commit-under-test is part of one of the base packages and
>       include it in the Docker image for the test run.
>
>     - Topic-based testing.  We often run into co-dependencies between
>       phosphor-dbus-interfaces and another repository.  Within the next
>       month I'll enhance the script so that if you change PDI and your
>       repository and tag the commit with the same topic, the PDI commit
>       from that topic will be pulled into the Docker image so your
>       repository commit can pass CI.
>
>     - Some refactoring.  I did a lot of incremental changes from the
>       original shell script to the Python version I currently have and
>       didn't always rewrite in the most Pythonic way.  There will be
>       some more, mostly non-functional, changes to fix this.
>
> Feel free to comment on the review series if you have any concerns with
> these changes.
>
> 1. https://gerrit.openbmc-project.xyz/c/openbmc/openbmc-build-scripts/+/40366
>
> --
> Patrick Williams



-- 
BRs,
Lei YU