Linux kernel updates and v6.0

Linux kernel updates and v6.0

[Joel Stanley]

Hello OpenBMC,

We've been using the v5.15 kernel as a base for almost 11 months. In
that time there's been 16 bumps to pull in stable fixes. We have
merged about 300 patches in that time to support new machines, and new
hardware, including MCTP, nct6775, lm25066, aspeed-adc and aspeed's
spi-nor devices.

It's time to move to a new base to ensure progress is made on our
mission to upstream all of the support. By rebasing on a new kernel
release we can see what work has been done, and what remains. Since
v5.15 we have upstream support for:

 - PECI, thanks to Jae and Iwona
 - MCTP, thanks to Jermey and Matt
 - spi-nor, thanks to Cédric
 - nct6775 i2c and lm25066, thanks to Zev
 - ast2600 adc, thanks to Billy
 - ast2600 gfx, thanks to Tommy

and others I have missed.

In addition to the ASPEED changes the Nuvton hackers have been hard at
work. We now have support for their latest generation  Cortex-A35 BMC,
the npcm845 "Arbel" and it's eval board. Likewise the HPE "GXP"
Cortex-A9 ASIC now has upstream support. Congratulations to both teams
for this work.

I have prepared a v6.0 tree that contains backports of the FSI and
Aspeed v6.1 patches, and a small set of existing patches. I will
publish this on Monday, or once v6.0 final has been tagged.

As promised the last time we rebased, the Nuvoton patches that have
not seen any updates since they were merged in 2019 have been dropped.
They are welcome to be resubmitted as long as they are also being
worked on upstream.

Please address any future patches to the dev-6.0 tree.

Cheers,

Joel

Re: Linux kernel updates and v6.0

[William Kennington]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

The ToF was talking about linux versions being used for OpenBMC users and
we were wondering if we could codify a policy around which versions we will
be developing against regularly. In the past it seems like you usually pick
LTS releases to base on (and Facebook / Google are interested in sticking
with LTS releases in order to reduce toil / number of potentially
incompatible kernel upgrades for each of our machines). It would seem like
kernel 6.0 should be an LTS release, although gregkh hasn't said anything
specifically about it yet. Can we get something written about the expected
alignment with upstream kernel release cadence?

On Tue, Sep 27, 2022 at 11:35 PM Joel Stanley <joel@jms.id.au> wrote:

> Hello OpenBMC,
>
> We've been using the v5.15 kernel as a base for almost 11 months. In
> that time there's been 16 bumps to pull in stable fixes. We have
> merged about 300 patches in that time to support new machines, and new
> hardware, including MCTP, nct6775, lm25066, aspeed-adc and aspeed's
> spi-nor devices.
>
> It's time to move to a new base to ensure progress is made on our
> mission to upstream all of the support. By rebasing on a new kernel
> release we can see what work has been done, and what remains. Since
> v5.15 we have upstream support for:
>
>  - PECI, thanks to Jae and Iwona
>  - MCTP, thanks to Jermey and Matt
>  - spi-nor, thanks to Cédric
>  - nct6775 i2c and lm25066, thanks to Zev
>  - ast2600 adc, thanks to Billy
>  - ast2600 gfx, thanks to Tommy
>
> and others I have missed.
>
> In addition to the ASPEED changes the Nuvton hackers have been hard at
> work. We now have support for their latest generation  Cortex-A35 BMC,
> the npcm845 "Arbel" and it's eval board. Likewise the HPE "GXP"
> Cortex-A9 ASIC now has upstream support. Congratulations to both teams
> for this work.
>
> I have prepared a v6.0 tree that contains backports of the FSI and
> Aspeed v6.1 patches, and a small set of existing patches. I will
> publish this on Monday, or once v6.0 final has been tagged.
>
> As promised the last time we rebased, the Nuvoton patches that have
> not seen any updates since they were merged in 2019 have been dropped.
> They are welcome to be resubmitted as long as they are also being
> worked on upstream.
>
> Please address any future patches to the dev-6.0 tree.
>
> Cheers,
>
> Joel
>

[-- Attachment #2: Type: text/html, Size: 2822 bytes --]

Re: Linux kernel updates and v6.0

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2857 bytes --]

Joel,

Any thoughts on this?

On Wed, Sep 28, 2022 at 03:30:23PM -0700, William Kennington wrote:
> The ToF was talking about linux versions being used for OpenBMC users and
> we were wondering if we could codify a policy around which versions we will
> be developing against regularly. In the past it seems like you usually pick
> LTS releases to base on (and Facebook / Google are interested in sticking
> with LTS releases in order to reduce toil / number of potentially
> incompatible kernel upgrades for each of our machines). It would seem like
> kernel 6.0 should be an LTS release, although gregkh hasn't said anything
> specifically about it yet. Can we get something written about the expected
> alignment with upstream kernel release cadence?

For minor clairification, I believe it was Intel / Google that preferred
LTS.  Meta does not have a preference as long as the OpenBMC tree is
well-supported and we can backport changes that have been sent
upstream.

> On Tue, Sep 27, 2022 at 11:35 PM Joel Stanley <joel@jms.id.au> wrote:
> 
> > Hello OpenBMC,
> >
> > We've been using the v5.15 kernel as a base for almost 11 months. In
> > that time there's been 16 bumps to pull in stable fixes. We have
> > merged about 300 patches in that time to support new machines, and new
> > hardware, including MCTP, nct6775, lm25066, aspeed-adc and aspeed's
> > spi-nor devices.
> >
> > It's time to move to a new base to ensure progress is made on our
> > mission to upstream all of the support. By rebasing on a new kernel
> > release we can see what work has been done, and what remains. Since
> > v5.15 we have upstream support for:
> >
> >  - PECI, thanks to Jae and Iwona
> >  - MCTP, thanks to Jermey and Matt
> >  - spi-nor, thanks to Cédric
> >  - nct6775 i2c and lm25066, thanks to Zev
> >  - ast2600 adc, thanks to Billy
> >  - ast2600 gfx, thanks to Tommy
> >
> > and others I have missed.
> >
> > In addition to the ASPEED changes the Nuvton hackers have been hard at
> > work. We now have support for their latest generation  Cortex-A35 BMC,
> > the npcm845 "Arbel" and it's eval board. Likewise the HPE "GXP"
> > Cortex-A9 ASIC now has upstream support. Congratulations to both teams
> > for this work.
> >
> > I have prepared a v6.0 tree that contains backports of the FSI and
> > Aspeed v6.1 patches, and a small set of existing patches. I will
> > publish this on Monday, or once v6.0 final has been tagged.
> >
> > As promised the last time we rebased, the Nuvoton patches that have
> > not seen any updates since they were merged in 2019 have been dropped.
> > They are welcome to be resubmitted as long as they are also being
> > worked on upstream.
> >
> > Please address any future patches to the dev-6.0 tree.
> >
> > Cheers,
> >
> > Joel
> >

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux kernel updates and v6.0

[Joel Stanley]

On Wed, 28 Sept 2022 at 06:34, Joel Stanley <joel@jms.id.au> wrote:
 for this work.
> I have prepared a v6.0 tree that contains backports of the FSI and
> Aspeed v6.1 patches, and a small set of existing patches. I will
> publish this on Monday, or once v6.0 final has been tagged.

This slipped as we had a public holiday on Monday, and there were a
few yocto issues to sort out before I could get the tree through CI.
I've now pushed dev-6.0 to openbmc/linux.

The +1 party is here:

 https://gerrit.openbmc.org/c/openbmc/openbmc/+/57706

Please join in.

> As promised the last time we rebased, the Nuvoton patches that have
> not seen any updates since they were merged in 2019 have been dropped.
> They are welcome to be resubmitted as long as they are also being
> worked on upstream.

Tomer has been working with me to get the recent Nuvoton work
backported to 6.0. There remains a chunk of work that hasn't been
posted to the upstream lists and therefore isn't in the openbmc tree
yet. I encourage those who maintain systems with a npcm7xx and npcm8xx
to help there.

> Please address any future patches to the dev-6.0 tree.

If you have pending patches then please let me know that you want them
merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
the list.

Cheers,

Joel

Re: Linux kernel updates and v6.0

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 556 bytes --]

On Thu, Oct 06, 2022 at 01:26:13AM +0000, Joel Stanley wrote:
> On Wed, 28 Sept 2022 at 06:34, Joel Stanley <joel@jms.id.au> wrote:
> > Please address any future patches to the dev-6.0 tree.
> 
> If you have pending patches then please let me know that you want them
> merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
> the list.

Hi Joel,

We'd like to have this series applied to your tree and backported to
dev-6.0.

https://lore.kernel.org/lkml/20220929013130.1916525-1-potin.lai.pt@gmail.com/

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Linux kernel updates and v6.0

[Quan Nguyen]

> 
>> Please address any future patches to the dev-6.0 tree.
> 
> If you have pending patches then please let me know that you want them
> merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
> the list.
> 

Hi Joel,

Could you help to pick this patchset to the dev-6.0 branch ?

https://lore.kernel.org/lkml/20221004093106.1653317-4-quan@os.amperecomputing.com/

Thank you,
- Quan

Re: Linux kernel updates and v6.0

[Joel Stanley]

On Thu, 6 Oct 2022 at 03:31, Patrick Williams <patrick@stwcx.xyz> wrote:
>
> On Thu, Oct 06, 2022 at 01:26:13AM +0000, Joel Stanley wrote:
> > On Wed, 28 Sept 2022 at 06:34, Joel Stanley <joel@jms.id.au> wrote:
> > > Please address any future patches to the dev-6.0 tree.
> >
> > If you have pending patches then please let me know that you want them
> > merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
> > the list.
>
> Hi Joel,
>
> We'd like to have this series applied to your tree and backported to
> dev-6.0.
>
> https://lore.kernel.org/lkml/20220929013130.1916525-1-potin.lai.pt@gmail.com/

I've merged to dev-6.0. This missed the merge window for 6.1; I'll try
to remember to apply it once 6.1-rc1 comes out for 6.2.

Re: Linux kernel updates and v6.0

[Joel Stanley]

On Thu, 6 Oct 2022 at 07:04, Quan Nguyen <quan@os.amperecomputing.com> wrote:
>
>
> >
> >> Please address any future patches to the dev-6.0 tree.
> >
> > If you have pending patches then please let me know that you want them
> > merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
> > the list.
> >
>
> Hi Joel,
>
> Could you help to pick this patchset to the dev-6.0 branch ?
>
> https://lore.kernel.org/lkml/20221004093106.1653317-4-quan@os.amperecomputing.com/

I merged this but it caused a build error:

drivers/char/ipmi/ssif_bmc.c:864:27: error: initialization of ‘int
(*)(struct i2c_client *)’ from incompatible pointer type ‘void
(*)(struct i2c_client *)’ [-Werror=incompatible-pointer-types]
  864 |         .remove         = ssif_bmc_remove,
      |                           ^~~~~~~~~~~~~~~

I think in 6.1 the i2c drivers will return void in their remove
callbacks, but before then they still need to return an int. I have
updated your change with this patch:

--- a/drivers/char/ipmi/ssif_bmc.c
+++ b/drivers/char/ipmi/ssif_bmc.c
@@ -835,12 +835,14 @@ static int ssif_bmc_probe(struct i2c_client
*client, const struct i2c_device_id
        return ret;
 }

-static void ssif_bmc_remove(struct i2c_client *client)
+static int ssif_bmc_remove(struct i2c_client *client)
 {
        struct ssif_bmc_ctx *ssif_bmc = i2c_get_clientdata(client);

        i2c_slave_unregister(client);
        misc_deregister(&ssif_bmc->miscdev);
+
+       return 0;
 }

Cheers,

Joel

Re: Linux kernel updates and v6.0

[Quan Nguyen]

Thanks Joel,
- Quan

On 06/10/2022 15:37, Joel Stanley wrote:
> On Thu, 6 Oct 2022 at 07:04, Quan Nguyen <quan@os.amperecomputing.com> wrote:
>>
>>
>>>
>>>> Please address any future patches to the dev-6.0 tree.
>>>
>>> If you have pending patches then please let me know that you want them
>>> merged to the dev-6.0 branch. Otherwise, rebase and re-send them to
>>> the list.
>>>
>>
>> Hi Joel,
>>
>> Could you help to pick this patchset to the dev-6.0 branch ?
>>
>> https://lore.kernel.org/lkml/20221004093106.1653317-4-quan@os.amperecomputing.com/
> 
> I merged this but it caused a build error:
> 
> drivers/char/ipmi/ssif_bmc.c:864:27: error: initialization of ‘int
> (*)(struct i2c_client *)’ from incompatible pointer type ‘void
> (*)(struct i2c_client *)’ [-Werror=incompatible-pointer-types]
>    864 |         .remove         = ssif_bmc_remove,
>        |                           ^~~~~~~~~~~~~~~
> 
> I think in 6.1 the i2c drivers will return void in their remove
> callbacks, but before then they still need to return an int. I have
> updated your change with this patch:
> 
> --- a/drivers/char/ipmi/ssif_bmc.c
> +++ b/drivers/char/ipmi/ssif_bmc.c
> @@ -835,12 +835,14 @@ static int ssif_bmc_probe(struct i2c_client
> *client, const struct i2c_device_id
>          return ret;
>   }
> 
> -static void ssif_bmc_remove(struct i2c_client *client)
> +static int ssif_bmc_remove(struct i2c_client *client)
>   {
>          struct ssif_bmc_ctx *ssif_bmc = i2c_get_clientdata(client);
> 
>          i2c_slave_unregister(client);
>          misc_deregister(&ssif_bmc->miscdev);
> +
> +       return 0;
>   }
> 
> Cheers,
> 
> Joel

Re: Linux kernel updates and v6.0

[Tao Ren]

Hi Joel,

Finally I added v6.0 to Meta/Facebook OpenBMC repository. Instead of
making a copy of dev-6.0 (like what I did before), I create a recipe to
fetch kernel from your repository directly this time. In other word, we
are sharing the same kernel tree now.

Thanks again for maintaining the tree, and it is very helpful. BTW, I
may ask your help to backport some upstream patches in the future :)


Cheers,

Tao

On Wed, Sep 28, 2022 at 06:34:53AM +0000, Joel Stanley wrote:
> Hello OpenBMC,
> 
> We've been using the v5.15 kernel as a base for almost 11 months. In
> that time there's been 16 bumps to pull in stable fixes. We have
> merged about 300 patches in that time to support new machines, and new
> hardware, including MCTP, nct6775, lm25066, aspeed-adc and aspeed's
> spi-nor devices.
> 
> It's time to move to a new base to ensure progress is made on our
> mission to upstream all of the support. By rebasing on a new kernel
> release we can see what work has been done, and what remains. Since
> v5.15 we have upstream support for:
> 
>  - PECI, thanks to Jae and Iwona
>  - MCTP, thanks to Jermey and Matt
>  - spi-nor, thanks to Cédric
>  - nct6775 i2c and lm25066, thanks to Zev
>  - ast2600 adc, thanks to Billy
>  - ast2600 gfx, thanks to Tommy
> 
> and others I have missed.
> 
> In addition to the ASPEED changes the Nuvton hackers have been hard at
> work. We now have support for their latest generation  Cortex-A35 BMC,
> the npcm845 "Arbel" and it's eval board. Likewise the HPE "GXP"
> Cortex-A9 ASIC now has upstream support. Congratulations to both teams
> for this work.
> 
> I have prepared a v6.0 tree that contains backports of the FSI and
> Aspeed v6.1 patches, and a small set of existing patches. I will
> publish this on Monday, or once v6.0 final has been tagged.
> 
> As promised the last time we rebased, the Nuvoton patches that have
> not seen any updates since they were merged in 2019 have been dropped.
> They are welcome to be resubmitted as long as they are also being
> worked on upstream.
> 
> Please address any future patches to the dev-6.0 tree.
> 
> Cheers,
> 
> Joel

Re: Linux kernel updates and v6.0

[Joel Stanley]

On Mon, 24 Oct 2022 at 00:08, Tao Ren <rentao.bupt@gmail.com> wrote:
>
> Hi Joel,
>
> Finally I added v6.0 to Meta/Facebook OpenBMC repository. Instead of
> making a copy of dev-6.0 (like what I did before), I create a recipe to
> fetch kernel from your repository directly this time. In other word, we
> are sharing the same kernel tree now.

That's great news. Congrats to you and your team for getting all of
your changes into the tree.

> Thanks again for maintaining the tree, and it is very helpful. BTW, I
> may ask your help to backport some upstream patches in the future :)

Thank you for the kind feedback.

Cheers,

Joel

>
>
> Cheers,
>
> Tao
>
> On Wed, Sep 28, 2022 at 06:34:53AM +0000, Joel Stanley wrote:
> > Hello OpenBMC,
> >
> > We've been using the v5.15 kernel as a base for almost 11 months. In
> > that time there's been 16 bumps to pull in stable fixes. We have
> > merged about 300 patches in that time to support new machines, and new
> > hardware, including MCTP, nct6775, lm25066, aspeed-adc and aspeed's
> > spi-nor devices.
> >
> > It's time to move to a new base to ensure progress is made on our
> > mission to upstream all of the support. By rebasing on a new kernel
> > release we can see what work has been done, and what remains. Since
> > v5.15 we have upstream support for:
> >
> >  - PECI, thanks to Jae and Iwona
> >  - MCTP, thanks to Jermey and Matt
> >  - spi-nor, thanks to Cédric
> >  - nct6775 i2c and lm25066, thanks to Zev
> >  - ast2600 adc, thanks to Billy
> >  - ast2600 gfx, thanks to Tommy
> >
> > and others I have missed.
> >
> > In addition to the ASPEED changes the Nuvton hackers have been hard at
> > work. We now have support for their latest generation  Cortex-A35 BMC,
> > the npcm845 "Arbel" and it's eval board. Likewise the HPE "GXP"
> > Cortex-A9 ASIC now has upstream support. Congratulations to both teams
> > for this work.
> >
> > I have prepared a v6.0 tree that contains backports of the FSI and
> > Aspeed v6.1 patches, and a small set of existing patches. I will
> > publish this on Monday, or once v6.0 final has been tagged.
> >
> > As promised the last time we rebased, the Nuvoton patches that have
> > not seen any updates since they were merged in 2019 have been dropped.
> > They are welcome to be resubmitted as long as they are also being
> > worked on upstream.
> >
> > Please address any future patches to the dev-6.0 tree.
> >
> > Cheers,
> >
> > Joel

Re: Linux kernel updates and v6.0

[Joel Stanley]

On Wed, 5 Oct 2022 at 11:33, Patrick Williams <patrick@stwcx.xyz> wrote:
>
> Joel,
>
> Any thoughts on this?
>
> On Wed, Sep 28, 2022 at 03:30:23PM -0700, William Kennington wrote:
> > The ToF was talking about linux versions being used for OpenBMC users and

In the future it would be preferable to bring the discussion to the
community first, instead of talking among the TOFs.

> > we were wondering if we could codify a policy around which versions we will
> > be developing against regularly. In the past it seems like you usually pick
> > LTS releases to base on (and Facebook / Google are interested in sticking
> > with LTS releases in order to reduce toil / number of potentially
> > incompatible kernel upgrades for each of our machines). It would seem like
> > kernel 6.0 should be an LTS release, although gregkh hasn't said anything
> > specifically about it yet. Can we get something written about the expected
> > alignment with upstream kernel release cadence?

The documented alignment is every 30 days:

 https://github.com/openbmc/linux/wiki/DevelopmentProcess#openbmc-kernel-maintainer

Obviously that doesn't happen, because we lack the developer power to
perform this work. The document could do with some updates in that
respect.

I intend to rebase on each upstream release as they come out. This
means every 90 days or so. There was a period where that worked well,
but we have been stuck for a while due to lack of developer (my) time.

Building on an upstream LTS makes sense if the following are true:

 - You're regularly pulling in the stable point releases and pushing
them out as firmware updates

 - All of your patches are upstream. The stable tree can only apply
patches to code that is upstream. If they're not upstream, then
they're not getting patches backported to fix the issues, so the
stable tree doesn't provide any benefit to you

 - You're actively submitting fixes to mainline to be backported to stable

Obviously there's code that we ship on the BMC that is upstream, and
gets a lot of attention from the wider kernel community. The
networking code, filesystems, scheduler.

Given we have a lot of downstream code, I would (and regularly do)
argue that we're just as well off shipping the latest upstream kernel
than we are staying on an old LTS with less upstream code.

As we strive to upstream more of our code, the benefits of being on a
LTS tree increase.

How do we encourage developers to upstream their code? One way is to
regularly rebase on upstream releases, and drop patches that have not
seen any development. This works as a forcing function and incentive
for upstream development, and you could argue it has worked. Since
dropping PECI in v5.15 we have seen it submitted and merged upstream,
for example.

I've had minimal trouble moving the IBM systems from kernel release to
kernel release. Perhaps you're relying on userspace interfaces that
are not upstream? If that is the case, I encourage you to get the
interfaces upstream earlier, so moving is less effort.

Cheers,

Joel

> For minor clairification, I believe it was Intel / Google that preferred
> LTS.  Meta does not have a preference as long as the OpenBMC tree is
> well-supported and we can backport changes that have been sent
> upstream.