Bizarre CI Failure

Bizarre CI Failure

[Patrick Venture]

https://gerrit.openbmc-project.xyz/18327 is failing:

07:17:43 /home/jenkins-op/workspace/openbmc-repository-ci/openbmc/openpower-debug-collector
> make -j 80 -l 80 -O
07:17:43 cp: cannot create regular file
'/usr/local/share/phosphor-dbus-yaml/yaml/org/open_power/Host/Boot.errors.yaml':
Permission denied
07:17:43 Makefile:1067: recipe for target
'phosphor-logging/elog-errors.hpp' failed
07:17:43 make: *** [phosphor-logging/elog-errors.hpp] Error 1
07:17:43 make: *** Waiting for unfinished jobs....
07:17:43 /usr/local/bin/sdbus++ -r . error exception-cpp
org.open_power.Host.Boot> org/open_power/Host/Boot/error.cpp
07:17:43 /usr/local/bin/sdbus++ -r . error exception-header
org.open_power.Host.Boot> org/open_power/Host/Boot/error.hpp
07:17:43 Traceback (most recent call last):
07:17:43   File
"/home/jenkins-op/workspace/openbmc-repository-ci/openbmc/unit-test.py",
line 796, in <module>
07:17:43     build_and_install(UNIT_TEST_PKG, True)
07:17:43   File
"/home/jenkins-op/workspace/openbmc-repository-ci/openbmc/unit-test.py",
line 447, in build_and_install
07:17:43     check_call_cmd(*make_parallel)
07:17:43   File
"/home/jenkins-op/workspace/openbmc-repository-ci/openbmc/unit-test.py",
line 229, in check_call_cmd
07:17:43     check_call(cmd)
07:17:43   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
07:17:43     raise CalledProcessError(retcode, cmd)
07:17:43 subprocess.CalledProcessError: Command '('make', '-j', '80',
'-l', '80', '-O')' returned non-zero exit status 2
07:17:43 Traceback (most recent call last):
07:17:43   File
"/home/jenkins-op/workspace/openbmc-repository-ci/openbmc/dbus-unit-test.py",
line 91, in <module>
07:17:43     check_call(UNIT_TEST.split(','), env=os.environ)
07:17:43   File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
07:17:43     raise CalledProcessError(retcode, cmd)
07:17:43 subprocess.CalledProcessError: Command
'['/home/jenkins-op/workspace/openbmc-repository-ci/openbmc/unit-test.py',
'-w', '/home/jenkins-op/workspace/openbmc-repository-ci/openbmc',
'-p', 'openpower-debug-collector', '-b', 'master', '-v']' returned
non-zero exit status 1
07:17:44 Build step 'Execute shell' marked build as failure
07:17:44 Set build name.
07:17:44 New build name is '#25702-Patrick Venture'
07:17:47 Finished: FAILURE

It works fine when I run the CI on my local system (using the same
scripts).  I even purged all my docker images this morning and re-ran
it.  Wow, that takes a while to build up, but then it was still fine.
Any thoughts?  My immediate reaction is that it could be a permissions
issue...

Patrick

Re: Bizarre CI Failure

[Andrew Geissler]

On Tue, Mar 5, 2019 at 10:48 AM Patrick Venture <venture@google.com> wrote:
>
> https://gerrit.openbmc-project.xyz/18327 is failing:

Yeah, better open an issue on
https://github.com/openbmc/openbmc-build-scripts/issues
Some sort of permission issue with the generated files from the docker container
build. This repo is pre-built into the docker container and then when CI tries
to rebuild the repo, it's hitting permission issues with the
pre-existing file. No idea
why it doesn't recreate in your env though. I verified it hits multiple of our
CI systems.

> Patrick

Re: Bizarre CI Failure

[Patrick Venture]

On Tue, Mar 5, 2019 at 9:26 AM Andrew Geissler <geissonator@gmail.com> wrote:
>
> On Tue, Mar 5, 2019 at 10:48 AM Patrick Venture <venture@google.com> wrote:
> >
> > https://gerrit.openbmc-project.xyz/18327 is failing:
>
> Yeah, better open an issue on
> https://github.com/openbmc/openbmc-build-scripts/issues

Done. https://github.com/openbmc/openbmc-build-scripts/issues/25

> Some sort of permission issue with the generated files from the docker container
> build. This repo is pre-built into the docker container and then when CI tries
> to rebuild the repo, it's hitting permission issues with the
> pre-existing file. No idea
> why it doesn't recreate in your env though.

Maybe it's a docker version issue, or a permissions issue associated
with the area where the docker stuff is installed.  Or a difference in
how the scripts are run, I'm using:

sudo WORKSPACE=$(pwd) UNIT_TEST_PKG=openpower-debug-collector
./openbmc-build-scripts/run-unit-test-docker.sh

> I verified it hits multiple of our
> CI systems.
>
> > Patrick

Re: Bizarre CI Failure

[Lei YU]

> Some sort of permission issue with the generated files from the docker container
> build. This repo is pre-built into the docker container and then when CI tries
> to rebuild the repo, it's hitting permission issues with the
> pre-existing file. No idea
> why it doesn't recreate in your env though. I verified it hits multiple of our
> CI systems.

When I was checking the build error in
https://gerrit.openbmc-project.xyz/#/c/openbmc/openpower-pnor-code-mgmt/+/18795/1,
I just notice that the auto-build-and-test happens partly in docker, that the
environment is in docker, but the source code, and the generated
scripts/objects are in host side.

That seems, a bit weird, because I would expect all the build (env, source,
objects) are in docker instead of host.