CI to stop testing meta-* layers not in tested machine

CI to stop testing meta-* layers not in tested machine

[Andrew Geissler]

I took an action item from last weeks Infrastructure Workgroup.

The point was we're wasting CI resources by testing meta-*
commits that are not actually tested by any of the machines in the
CI job. We're also falsely marking those commits as Verified because
if they are not in any of the systems under test, they're not being
tested at all.

The systems currently run as a part of the meta-* CI jobs are here:
https://openpower.xyz/view/CI/job/run-meta-ci/

Some quick grepping indicates the following meta-* repos are not
being tested:

meta-arm
meta-evb
meta-google
meta-hxt
meta-inspur
meta-intel
meta-inventec
meta-mellanox
meta-nuvoton
meta-portwell
meta-qualcomm
meta-quanta
meta-raspberrypi
meta-security
meta-x86
meta-xilinx

This would mean the maintainers of the above repos would need to +1
Verify the changes to these layers before merging.

Are there any advantages to running CI against meta-* layers that
are not in a machine being built? Are there other machines we can
add to CI that would cover some of the meta layers above? The
general criteria for getting a machine added to CI is that it's actively
being developed and supported. We also need to balance our
CI compute resources so the overall goal (in my mind) would be
to pick the machines that cover the most meta layers.

Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Patrick Venture]

On Thu, Mar 14, 2019 at 6:39 AM Andrew Geissler <geissonator@gmail.com> wrote:
>
> I took an action item from last weeks Infrastructure Workgroup.
>
> The point was we're wasting CI resources by testing meta-*
> commits that are not actually tested by any of the machines in the
> CI job. We're also falsely marking those commits as Verified because
> if they are not in any of the systems under test, they're not being
> tested at all.
>
> The systems currently run as a part of the meta-* CI jobs are here:
> https://openpower.xyz/view/CI/job/run-meta-ci/
>
> Some quick grepping indicates the following meta-* repos are not
> being tested:
>
> meta-arm
> meta-evb
> meta-google
> meta-hxt
> meta-inspur
> meta-intel
> meta-inventec
> meta-mellanox
> meta-nuvoton
> meta-portwell
> meta-qualcomm
> meta-quanta
> meta-raspberrypi
> meta-security
> meta-x86
> meta-xilinx
>
> This would mean the maintainers of the above repos would need to +1
> Verify the changes to these layers before merging.
>
> Are there any advantages to running CI against meta-* layers that
> are not in a machine being built? Are there other machines we can
> add to CI that would cover some of the meta layers above? The
> general criteria for getting a machine added to CI is that it's actively
> being developed and supported. We also need to balance our
> CI compute resources so the overall goal (in my mind) would be
> to pick the machines that cover the most meta layers.

@OCP Summit today talking to Brad.  It's possible I'll end up pushing
up a machine configuration that'll use meta-google.  However, as far
as not testing it during a build, I think yeah, with limited compute
resources, there's only so much we can do.

>
> Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Andrew Geissler]

I'd like to revisit this topic. It looks like the meta-* layers I
would remove from
meta-CI are still the same as below. We've added the
meta-facebook/meta-tiogapass
since my last email to CI so that will be a meta layer (meta-facebook)
we continue
to test.

Please let me know if you have any questions or concerns.

Thanks,
Andrew

On Thu, Mar 14, 2019 at 8:38 AM Andrew Geissler <geissonator@gmail.com> wrote:
>
> I took an action item from last weeks Infrastructure Workgroup.
>
> The point was we're wasting CI resources by testing meta-*
> commits that are not actually tested by any of the machines in the
> CI job. We're also falsely marking those commits as Verified because
> if they are not in any of the systems under test, they're not being
> tested at all.
>
> The systems currently run as a part of the meta-* CI jobs are here:
> https://openpower.xyz/view/CI/job/run-meta-ci/
>
> Some quick grepping indicates the following meta-* repos are not
> being tested:
>
> meta-arm
> meta-evb
> meta-google
> meta-hxt
> meta-inspur
> meta-intel
> meta-inventec
> meta-mellanox
> meta-nuvoton
> meta-portwell
> meta-qualcomm
> meta-quanta
> meta-raspberrypi
> meta-security
> meta-x86
> meta-xilinx
>
> This would mean the maintainers of the above repos would need to +1
> Verify the changes to these layers before merging.
>
> Are there any advantages to running CI against meta-* layers that
> are not in a machine being built? Are there other machines we can
> add to CI that would cover some of the meta layers above? The
> general criteria for getting a machine added to CI is that it's actively
> being developed and supported. We also need to balance our
> CI compute resources so the overall goal (in my mind) would be
> to pick the machines that cover the most meta layers.
>
> Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Joel Stanley]

On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote:
>
> I took an action item from last weeks Infrastructure Workgroup.
>
> The point was we're wasting CI resources by testing meta-*
> commits that are not actually tested by any of the machines in the
> CI job. We're also falsely marking those commits as Verified because
> if they are not in any of the systems under test, they're not being
> tested at all.
>
> The systems currently run as a part of the meta-* CI jobs are here:
> https://openpower.xyz/view/CI/job/run-meta-ci/

> Are there any advantages to running CI against meta-* layers that
> are not in a machine being built? Are there other machines we can
> add to CI that would cover some of the meta layers above? The
> general criteria for getting a machine added to CI is that it's actively
> being developed and supported. We also need to balance our
> CI compute resources so the overall goal (in my mind) would be
> to pick the machines that cover the most meta layers.

I'd like to have a nuvoton based machine so we have some confidence
that kernel bumps aren't broken.

That would mean adding the evb-nuvoton or gsj machines to CI.

Cheers,

Joel

Re: CI to stop testing meta-* layers not in tested machine

[Patrick Venture]

From: Joel Stanley <joel@jms.id.au>
Date: Thu, May 9, 2019 at 6:11 PM
To: Andrew Geissler, Benjamin Fair
Cc: OpenBMC Maillist

> On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote:
> >
> > I took an action item from last weeks Infrastructure Workgroup.
> >
> > The point was we're wasting CI resources by testing meta-*
> > commits that are not actually tested by any of the machines in the
> > CI job. We're also falsely marking those commits as Verified because
> > if they are not in any of the systems under test, they're not being
> > tested at all.
> >
> > The systems currently run as a part of the meta-* CI jobs are here:
> > https://openpower.xyz/view/CI/job/run-meta-ci/
>
> > Are there any advantages to running CI against meta-* layers that
> > are not in a machine being built? Are there other machines we can
> > add to CI that would cover some of the meta layers above? The
> > general criteria for getting a machine added to CI is that it's actively
> > being developed and supported. We also need to balance our
> > CI compute resources so the overall goal (in my mind) would be
> > to pick the machines that cover the most meta layers.
>
> I'd like to have a nuvoton based machine so we have some confidence
> that kernel bumps aren't broken.
>
> That would mean adding the evb-nuvoton or gsj machines to CI.

I vote for the gsj machine.  Not that it's a democracy :)

>
> Cheers,
>
> Joel

Re: CI to stop testing meta-* layers not in tested machine

[Andrew Geissler]

On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote:
>
> From: Joel Stanley <joel@jms.id.au>
> Date: Thu, May 9, 2019 at 6:11 PM
> To: Andrew Geissler, Benjamin Fair
> Cc: OpenBMC Maillist
>
> > On Thu, 14 Mar 2019 at 13:39, Andrew Geissler <geissonator@gmail.com> wrote:
> > >
> > > I took an action item from last weeks Infrastructure Workgroup.
> > >
> > > The point was we're wasting CI resources by testing meta-*
> > > commits that are not actually tested by any of the machines in the
> > > CI job. We're also falsely marking those commits as Verified because
> > > if they are not in any of the systems under test, they're not being
> > > tested at all.
> > >
> > > The systems currently run as a part of the meta-* CI jobs are here:
> > > https://openpower.xyz/view/CI/job/run-meta-ci/
> >
> > > Are there any advantages to running CI against meta-* layers that
> > > are not in a machine being built? Are there other machines we can
> > > add to CI that would cover some of the meta layers above? The
> > > general criteria for getting a machine added to CI is that it's actively
> > > being developed and supported. We also need to balance our
> > > CI compute resources so the overall goal (in my mind) would be
> > > to pick the machines that cover the most meta layers.
> >
> > I'd like to have a nuvoton based machine so we have some confidence
> > that kernel bumps aren't broken.
> >
> > That would mean adding the evb-nuvoton or gsj machines to CI.
>
> I vote for the gsj machine.  Not that it's a democracy :)

I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542
Be great to get gsj into CI since it would give us a few new layers
for coverage.

google provides a good chunk of the CI build infrastructure for OpenBMC so
you definitely get a vote :)

Andrew

>
> >
> > Cheers,
> >
> > Joel

Re: CI to stop testing meta-* layers not in tested machine

[Joel Stanley]

On Fri, 10 May 2019 at 18:46, Andrew Geissler <geissonator@gmail.com> wrote:
>
> On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote:
> > > I'd like to have a nuvoton based machine so we have some confidence
> > > that kernel bumps aren't broken.
> > >
> > > That would mean adding the evb-nuvoton or gsj machines to CI.
> >
> > I vote for the gsj machine.  Not that it's a democracy :)
>
> I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542
> Be great to get gsj into CI since it would give us a few new layers
> for coverage.

gsj is now supported in the kernel.

Andrew tried to build the machine and ran into u-boot issues which is
still blocking the machine's addition to our CI. Patrick, are you able
to look into that?

 https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892

Once we get the u-boot issue sorted out, I propose the following changes:

 - drop qemu from CI. 'qemu' is actually testing on a generic arm
machine. A few of us at IBM have a side project that has resulted in a
high quality Qemu support for the aspeed boards, so if you would like
to test in qemu I recommend grabbing palmetto or romulus and doing
that. So consider this dropping the generic qemu image and instead
focusing on the aspeed one.

 - add gsj. This gives us coverage of the nuvoton kernel and u-boot,
as well as the nuvoton specific layers

 - add swift. This is an ast2500-based system that we're looking to
use emmc flash with, and having testing for those images will be
useful

Cheers,

Joel

Re: CI to stop testing meta-* layers not in tested machine

[Benjamin Fair]

On Thu, Jun 13, 2019 at 9:54 PM Joel Stanley <joel@jms.id.au> wrote:
>
> On Fri, 10 May 2019 at 18:46, Andrew Geissler <geissonator@gmail.com> wrote:
> >
> > On Fri, May 10, 2019 at 9:33 AM Patrick Venture <venture@google.com> wrote:
> > > > I'd like to have a nuvoton based machine so we have some confidence
> > > > that kernel bumps aren't broken.
> > > >
> > > > That would mean adding the evb-nuvoton or gsj machines to CI.
> > >
> > > I vote for the gsj machine.  Not that it's a democracy :)
> >
> > I gave this a try but ran into https://github.com/openbmc/openbmc/issues/3542
> > Be great to get gsj into CI since it would give us a few new layers
> > for coverage.
>
> gsj is now supported in the kernel.
>
> Andrew tried to build the machine and ran into u-boot issues which is
> still blocking the machine's addition to our CI. Patrick, are you able
> to look into that?
>
>  https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892

That issue will be resolved by switching to a 2019-based U-Boot branch:

https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556

>
>
>
> Once we get the u-boot issue sorted out, I propose the following changes:
>
>  - drop qemu from CI. 'qemu' is actually testing on a generic arm
> machine. A few of us at IBM have a side project that has resulted in a
> high quality Qemu support for the aspeed boards, so if you would like
> to test in qemu I recommend grabbing palmetto or romulus and doing
> that. So consider this dropping the generic qemu image and instead
> focusing on the aspeed one.

+1

Many things are already broken on QEMU, including phosphor-ipmi-host.
It's not a useful platform to test with.

>
>  - add gsj. This gives us coverage of the nuvoton kernel and u-boot,
> as well as the nuvoton specific layers

Agreed. I'd also like to switch to (or add) a Nuvoton system with a
host once such a machine is supported upstream. The gsj is only a
storage tray so it doesn't test IPMI bridges, power control, etc.

>
>  - add swift. This is an ast2500-based system that we're looking to
> use emmc flash with, and having testing for those images will be
> useful
>
> Cheers,
>
> Joel

Re: CI to stop testing meta-* layers not in tested machine

[Andrew Jeffery]

> >
> >
> >
> > Once we get the u-boot issue sorted out, I propose the following changes:
> >
> >  - drop qemu from CI. 'qemu' is actually testing on a generic arm
> > machine. A few of us at IBM have a side project that has resulted in a
> > high quality Qemu support for the aspeed boards, so if you would like
> > to test in qemu I recommend grabbing palmetto or romulus and doing
> > that. So consider this dropping the generic qemu image and instead
> > focusing on the aspeed one.
> 
> +1
> 
> Many things are already broken on QEMU, including phosphor-ipmi-host.
> It's not a useful platform to test with.
> 

Is that a general statement about QEMU, or are you talking about the generic
qemu image as Joel was?

It would be great if we had more contributions on the QEMU side. There's a
list of things that can be attacked in the wiki:

https://github.com/openbmc/qemu/wiki

Modelling I2C peripherals is usually a good first step.

Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Benjamin Fair]

On Sun, Jun 16, 2019 at 8:30 PM Andrew Jeffery <andrew@aj.id.au> wrote:
>
> > >
> > >
> > >
> > > Once we get the u-boot issue sorted out, I propose the following changes:
> > >
> > >  - drop qemu from CI. 'qemu' is actually testing on a generic arm
> > > machine. A few of us at IBM have a side project that has resulted in a
> > > high quality Qemu support for the aspeed boards, so if you would like
> > > to test in qemu I recommend grabbing palmetto or romulus and doing
> > > that. So consider this dropping the generic qemu image and instead
> > > focusing on the aspeed one.
> >
> > +1
> >
> > Many things are already broken on QEMU, including phosphor-ipmi-host.
> > It's not a useful platform to test with.
> >
>
> Is that a general statement about QEMU, or are you talking about the generic
> qemu image as Joel was?

That's specifically an issue with the generic QEMU image. It's because
the generic image doesn't include u-boot, which is a dependency of
phosphor-ipmi-host (transitively through clear-once).

>
> It would be great if we had more contributions on the QEMU side. There's a
> list of things that can be attacked in the wiki:
>
> https://github.com/openbmc/qemu/wiki
>
> Modelling I2C peripherals is usually a good first step.
>
> Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Andrew Jeffery]

On Wed, 19 Jun 2019, at 10:16, Benjamin Fair wrote:
> On Sun, Jun 16, 2019 at 8:30 PM Andrew Jeffery <andrew@aj.id.au> wrote:
> >
> > > >
> > > >
> > > >
> > > > Once we get the u-boot issue sorted out, I propose the following changes:
> > > >
> > > >  - drop qemu from CI. 'qemu' is actually testing on a generic arm
> > > > machine. A few of us at IBM have a side project that has resulted in a
> > > > high quality Qemu support for the aspeed boards, so if you would like
> > > > to test in qemu I recommend grabbing palmetto or romulus and doing
> > > > that. So consider this dropping the generic qemu image and instead
> > > > focusing on the aspeed one.
> > >
> > > +1
> > >
> > > Many things are already broken on QEMU, including phosphor-ipmi-host.
> > > It's not a useful platform to test with.
> > >
> >
> > Is that a general statement about QEMU, or are you talking about the generic
> > qemu image as Joel was?
> 
> That's specifically an issue with the generic QEMU image. It's because
> the generic image doesn't include u-boot, which is a dependency of
> phosphor-ipmi-host (transitively through clear-once).
> 

I've hit that before, and it's not an intuitive problem. I'm all for ditching the
generic qemu machine. Back when I wrote the original aspeed support for
qemu I was looking at integrating the machine configuration into the openbmc
tree, but it was a little awkward at the time. It would be good if someone could
pick that up, as upstream have rearranged how they invoke qemu and it should
be more flexible now.

Andrew

Re: CI to stop testing meta-* layers not in tested machine

[Joel Stanley]

On Fri, 14 Jun 2019 at 15:55, Benjamin Fair <benjaminfair@google.com> wrote:
> >
> > Andrew tried to build the machine and ran into u-boot issues which is
> > still blocking the machine's addition to our CI. Patrick, are you able
> > to look into that?
> >
> >  https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892
>
> That issue will be resolved by switching to a 2019-based U-Boot branch:
>
> https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556

This has been merged now.

Andrew G, are we able to turn on the CI?

I think we have consensus to drop qemu, and enable gsj. There were no
objections to enabling swift too.

Cheers,

Joel

>
> >
> >
> >
> > Once we get the u-boot issue sorted out, I propose the following changes:
> >
> >  - drop qemu from CI. 'qemu' is actually testing on a generic arm
> > machine. A few of us at IBM have a side project that has resulted in a
> > high quality Qemu support for the aspeed boards, so if you would like
> > to test in qemu I recommend grabbing palmetto or romulus and doing
> > that. So consider this dropping the generic qemu image and instead
> > focusing on the aspeed one.
>
> +1
>
> Many things are already broken on QEMU, including phosphor-ipmi-host.
> It's not a useful platform to test with.
>
> >
> >  - add gsj. This gives us coverage of the nuvoton kernel and u-boot,
> > as well as the nuvoton specific layers
>
> Agreed. I'd also like to switch to (or add) a Nuvoton system with a
> host once such a machine is supported upstream. The gsj is only a
> storage tray so it doesn't test IPMI bridges, power control, etc.
>
> >
> >  - add swift. This is an ast2500-based system that we're looking to
> > use emmc flash with, and having testing for those images will be
> > useful
> >
> > Cheers,
> >
> > Joel

Re: CI to stop testing meta-* layers not in tested machine

[Andrew Geissler]

On Thu, Jun 20, 2019 at 7:38 PM Joel Stanley <joel@jms.id.au> wrote:
>
> On Fri, 14 Jun 2019 at 15:55, Benjamin Fair <benjaminfair@google.com> wrote:
> > >
> > > Andrew tried to build the machine and ran into u-boot issues which is
> > > still blocking the machine's addition to our CI. Patrick, are you able
> > > to look into that?
> > >
> > >  https://github.com/openbmc/openbmc/issues/3542#issuecomment-501706892
> >
> > That issue will be resolved by switching to a 2019-based U-Boot branch:
> >
> > https://gerrit.openbmc-project.xyz/c/openbmc/meta-nuvoton/+/22556
>
> This has been merged now.
>
> Andrew G, are we able to turn on the CI?
>
> I think we have consensus to drop qemu, and enable gsj. There were no
> objections to enabling swift too.

I removed qemu and added in gsj but we still need to work a few things out
to get Swift going. I pinged you offline on that Joel.

>
> Cheers,
>
> Joel