Robot Tests in CI

Robot Tests in CI

[Andrew Geissler]

I've enabled running our robot CI tests against all of the
openbmc/openbmc gerrit commits.  This is against the basic ARM based
QEMU model.

Reviews/Comments (or +1's) appreciated on the scripts to do this:

https://gerrit.openbmc-project.xyz/#/c/166/  (main script)
https://gerrit.openbmc-project.xyz/#/c/176/  (minor change to support
QEMU in docker)

The jenkins job can be found here:
https://openpower.xyz/job/openbmc-test-qemu-ci/

The test team is working on optimizing the test suite to ensure as
much coverage in as minimum amount of time.

Also, I've tweaked the build-setup.sh script to support ppc64le,
comments (or +1's) appreciated here as well:
https://gerrit.openbmc-project.xyz/#/c/196/

Next on the list is a real-hardware based CI job.

Andrew

Re: Robot Tests in CI

[Joel Stanley]

Hey Andrew,

On Tue, Aug 16, 2016 at 1:00 AM, Andrew Geissler <geissonator@gmail.com> wrote:
> I've enabled running our robot CI tests against all of the
> openbmc/openbmc gerrit commits.  This is against the basic ARM based
> QEMU model.

Thanks.

Do you intend to run it against the palmetto qemu? This would be
useful to ensure our kernel and userspace interactions are not
regressing.

>
> Reviews/Comments (or +1's) appreciated on the scripts to do this:
>
> https://gerrit.openbmc-project.xyz/#/c/166/  (main script)
> https://gerrit.openbmc-project.xyz/#/c/176/  (minor change to support
> QEMU in docker)

Andrew J has beat me to it. Look good in general.

>
> The jenkins job can be found here:
> https://openpower.xyz/job/openbmc-test-qemu-ci/
>
> The test team is working on optimizing the test suite to ensure as
> much coverage in as minimum amount of time.
>
> Also, I've tweaked the build-setup.sh script to support ppc64le,
> comments (or +1's) appreciated here as well:
> https://gerrit.openbmc-project.xyz/#/c/196/

Looks good, pending addressing Andrew J's feedback.

>
> Next on the list is a real-hardware based CI job.

Great!

Cheers,

Joel

Re: Robot Tests in CI

[Andrew Geissler]

On Mon, Aug 15, 2016 at 9:14 PM, Joel Stanley <joel@jms.id.au> wrote:
> Hey Andrew,

Hey Joel!

>
> On Tue, Aug 16, 2016 at 1:00 AM, Andrew Geissler <geissonator@gmail.com> wrote:
>> I've enabled running our robot CI tests against all of the
>> openbmc/openbmc gerrit commits.  This is against the basic ARM based
>> QEMU model.
>
> Thanks.
>
> Do you intend to run it against the palmetto qemu? This would be
> useful to ensure our kernel and userspace interactions are not
> regressing.

Definitely!  Just not in this commit :)  We wanted to start with the basics.
Running against Palmetto QEMU and running an enhanced CI bucket
against it are on the list.

>
>>
>> Reviews/Comments (or +1's) appreciated on the scripts to do this:
>>
>> https://gerrit.openbmc-project.xyz/#/c/166/  (main script)
>> https://gerrit.openbmc-project.xyz/#/c/176/  (minor change to support
>> QEMU in docker)
>
> Andrew J has beat me to it. Look good in general.
>
>>
>> The jenkins job can be found here:
>> https://openpower.xyz/job/openbmc-test-qemu-ci/
>>
>> The test team is working on optimizing the test suite to ensure as
>> much coverage in as minimum amount of time.
>>
>> Also, I've tweaked the build-setup.sh script to support ppc64le,
>> comments (or +1's) appreciated here as well:
>> https://gerrit.openbmc-project.xyz/#/c/196/
>
> Looks good, pending addressing Andrew J's feedback.
>
>>
>> Next on the list is a real-hardware based CI job.
>
> Great!
>
> Cheers,
>
> Joel

Unfortunately we've got a bit of a blocker with
https://github.com/openbmc/openbmc-test-automation/issues/78
We're still running the CI tests but we wont fail the job if the CI
test fails until we get this resolved.

Andrew

Re: Robot Tests in CI

[Andrew Geissler]

ok, all code review comments addressed and code is getting cherry
picked into our CI runs without issue.

Final +1's and merge appreciated from the team.

# New scripts to launch qemu and run robot tests
https://gerrit.openbmc-project.xyz/#/c/166/

# Disable uninative feature
https://gerrit.openbmc-project.xyz/#/c/176/

# Update build script to support ppc64le
https://gerrit.openbmc-project.xyz/#/c/196/

# Update build script to auto extract master if no code given
https://gerrit.openbmc-project.xyz/#/c/211/

Thanks,
Andrew

On Tue, Aug 16, 2016 at 3:34 PM, Andrew Geissler <geissonator@gmail.com> wrote:
> On Mon, Aug 15, 2016 at 9:14 PM, Joel Stanley <joel@jms.id.au> wrote:
>> Hey Andrew,
>
> Hey Joel!
>
>>
>> On Tue, Aug 16, 2016 at 1:00 AM, Andrew Geissler <geissonator@gmail.com> wrote:
>>> I've enabled running our robot CI tests against all of the
>>> openbmc/openbmc gerrit commits.  This is against the basic ARM based
>>> QEMU model.
>>
>> Thanks.
>>
>> Do you intend to run it against the palmetto qemu? This would be
>> useful to ensure our kernel and userspace interactions are not
>> regressing.
>
> Definitely!  Just not in this commit :)  We wanted to start with the basics.
> Running against Palmetto QEMU and running an enhanced CI bucket
> against it are on the list.
>
>>
>>>
>>> Reviews/Comments (or +1's) appreciated on the scripts to do this:
>>>
>>> https://gerrit.openbmc-project.xyz/#/c/166/  (main script)
>>> https://gerrit.openbmc-project.xyz/#/c/176/  (minor change to support
>>> QEMU in docker)
>>
>> Andrew J has beat me to it. Look good in general.
>>
>>>
>>> The jenkins job can be found here:
>>> https://openpower.xyz/job/openbmc-test-qemu-ci/
>>>
>>> The test team is working on optimizing the test suite to ensure as
>>> much coverage in as minimum amount of time.
>>>
>>> Also, I've tweaked the build-setup.sh script to support ppc64le,
>>> comments (or +1's) appreciated here as well:
>>> https://gerrit.openbmc-project.xyz/#/c/196/
>>
>> Looks good, pending addressing Andrew J's feedback.
>>
>>>
>>> Next on the list is a real-hardware based CI job.
>>
>> Great!
>>
>> Cheers,
>>
>> Joel
>
> Unfortunately we've got a bit of a blocker with
> https://github.com/openbmc/openbmc-test-automation/issues/78
> We're still running the CI tests but we wont fail the job if the CI
> test fails until we get this resolved.
>
> Andrew