Re: Reg the next LTS kernel (6.1?)

[Michal Simek <michal.simek@amd.com>]

On 1/13/23 17:22, Greg KH wrote:
> On Fri, Jan 13, 2023 at 01:54:29PM +0100, Michal Simek wrote:
>>
>>
>> On 1/13/23 12:27, Greg KH wrote:
>>> On Thu, Jan 12, 2023 at 01:26:28PM +0100, Michal Simek wrote:
>>>> Hi Greg,
>>>>
>>>> On 1/12/23 13:23, Kris Chaplin wrote:
>>>>> Hello Greg,
>>>>>
>>>>>> You tell me please.  How has your testing gone for 6.1 so far?  Does it
>>>>>> work properly for you?  Are you and/or your company willing to test out
>>>>>> the -rc releases and provide feedback if it works or not for your
>>>>>> systems?  Do you have problems with 6.1.y vs. older kernels?  Is there
>>>>>> anything missing in it that you feel needs to be addressed with a newer
>>>>>> kernel instead?
>>>>>
>>>>> We have been integrating and testing 6.1 on the Microblaze, ARM32 and
>>>>> ARM64-bit architectures over the past few weeks.  These builds have
>>>>> been successful and we are able to run our regression tests on hardware
>>>>> targeting our FPGA SoC devices.
>>>>>
>>>>> We're continuing our tests as new updates to the 6.1 kernel series
>>>>> appear.
>>>>
>>>> As Kris said AMD/Xilinx has already moved internal SOC tree to 6.1 based
>>>> kernel in expectation that 6.1 will become next LTS.
>>>> And I am not aware about any issue from testing team related to 6.1 kernel
>>>> version. And we are covering our AMD/Xilinx SOCs based on arm32/arm64 and
>>>> Microblaze CPUs.
>>>>
>>>> It would be good to continue with the same strategy which using the latest
>>>> kernel at that year which is what I am hearing all the time from others that
>>>> 6.1 was last kernel at that year and it should be LTS.
>>>
>>> Generally yes, I pick the last release of the year but we need people to
>>> verify and validate that both it works for them, and that they are going
>>> to be using it in their systems and can provide testing results to us
>>> (as well as providing a way for their devices to actually be updated to
>>> the new releases, we've had previous stable kernel releases that were
>>> never actually shipped out to devices...)
>>
>> We are preparing base (every year) for our customers which is what it is
>> also shipped in our distribution.
> 
> What distribution does AMD create?  Who uses it?  Where is it published
> and how is it updated?

AMD acquired Xilinx last year. That's why Xilinx PetaLinux distribution is under 
AMD wings. It means it is distribution for Xilinx SOCs (Microblaze, Zynq 
(arm32), ZynqMP/Versal/VersalNET(arm64).
https://www.xilinx.com/products/design-tools/embedded-software/petalinux-sdk.html


It is used initially by all Xilinx customers. Kernel itself is wired inside this 
distribution and also used by Xilinx meta Layers in Yocto.
Cannonical is also using it as a base for AMD/Xilinx derivative kernel in Ubuntu 
for supporting arm64 based SOCs.


>> Other distribution like Ubuntu with their derivative kernels prefer to stay
>> on the LTS kernels too.
> 
> I already discuss this with the various "normal" distributions all the
> time so they know what is going on here already.

ok

>> Also users will be getting fixes against our base for extended period with
>> fixes going from vanilla kernel.
> 
> What is "your base"?  Who uses this?  How long is it supported?

For 2022 it was 5.15 + Xilinx SOC patches.
For 2023 it is going to be 6.1 + Xilinx SOC patches.

As I said above. It is wired in petalinux, yocto and also partners are using it.

Xilinx is supporting this kernel for one year till the next upgrade.
After it 3rd party partners are continue to support these kernels if required.

> 
>> For code which is not upstream(which we unfortunately have it) we are
>> providing fixes only till the next LTS version base is created.
> 
> So you only support the image for one year?

 From AMD/Xilinx point of view unfortunately yes but in Ubuntu via derivative 
kernel newer LTS version are merged and supported base on their support strategy.


>> That's why it is very beneficial to use for our base kernel which is going
>> to be LTS that users will get that extended period support.
> 
> But you should be keeping this up to date for each normal release,
> right?

We have never had enough resources to support multiple kernels. In perfect world 
we should do it but there are all the time constraints.
I am keep pushing over years to do upstreaming as early as possible and then our 
changes will be spread to all kernels. But we are not there yet.

>  Never do only yearly jumps, that way causes way too much work
> and is proven to be a bad idea.

I know that for many years but not the person who is deciding the upgrade strategy.

> 
> And why not just update your "customers" to every normal release?  Why
> stick them with one a year?  What is preventing more frequent updates?

Questions are correct but I am not defining strategy.

My goal is very simple to push everything to upstream and then we don't need to 
have discussions about it.

>> Base Kernel version is taken after your decision about LTS. But this year
>> because of timing we couldn't wait for this decision. That's why 6.1 was
>> taken with expectation that your generic concept about picking up the last
>> release of the year is going to continue.
> 
> That's a great guess, but again, you should always be updating your
> out-of-tree changes against every release so it will not matter what
> kernel is picked as a stable one or not.

There are a lot of things what we should be doing but we don't have unlimited 
number of resources especially on testing side. For me it is not a problem to 
preparing that kernels but someone has to run regressions over it and fixing 
issues if any issue happens.


>> Tested wise. We have test result from 6.1 <.0> and I have also shared 6.1.5
>> based version for our distribution. As of today we can't see any issue with
>> 6.1 kernel in general on features which we are using on AMD/Xilinx SOCs.
> 
> Great, can you also test the -rc releases to make sure we don't break
> anything for your systems?

Request to test the HEAD of TF-A, U-Boot and Linux was sent long time ago.
And I hope that regressions are running based on available features internally.

And our ZynqMP board is the part of LTSI and KernelCI and I see that regression 
is running there properly

https://linux.kernelci.org/soc/zynqmp/job/next/
https://lava.ciplatform.org/scheduler/device_type/zynqmp-zcu102

Another regression is running on Canonical side.


>> Also Jonathan wrote at https://lwn.net/Articles/915435/
>> "Unless something extremely surprising happens, 6.1 will be the final kernel
>> release for 2022, and thus will become the next LTS kernel."
>> That's why I hope that 6.1 is going to be next LTS.
> 
> Testing and feedback is going to be what allows this to happen.  I can't
> happen in a vacuum, and nor should you want it to.

And that was the purpose of jumping to this thread to send you a message that we 
are going to use 6.1 kernel and we can't see any issue with it on our side.
If 6.1 is going to be LTS then good for us.
If different kernel is going to be pick up based on feedback you get then we 
will just stay on kernel which is not going to be LTS.
For me this discussion is worth as input for internal discussion for kernel 
selection for 2024.

As I said I prefer to push everything upstream, properly mark fixes and then 
kernel version decision is up to every distribution/project. I got there with 
TF-A project and we are pretty much there with U-Boot and the kernel is the last 
one to focus.


>>>> I didn't run any stats but normally also more patches are going to this
>>>> version to be the part of LTS.
>>>
>>> What do you mean by this?  The patches accepted so far since 6.1.0 was
>>> released, or up until 6.1.0 was released?  For the patches since 6.1.0
>>> was released, that's due to more developers/maintainers tagging patches
>>> during the -rc1 merge window for stable releases (honestly they should
>>> have gotten them into the -final release first), and due to us having
>>> better tools in digging up potential stable patches (i.e. Sasha's
>>> AUTOSEL bot work.)
>>
>> I am just saying that developers/driver owners can simple do calculation to
>> identify LTS version. When they know it they also know time when their
>> deadline is for upstreaming work. It means if patch is accepted between
>> 6.0-r1 and 6.0-rc5/6 they know that it will get to 6.1 merge window.
> 
> That is what I am afraid of and if it causes problems I will purposfully
> pick the previous release.  This has happened in the past and is never
> an excuse to get anything merged.  Code gets merged when it is ready,
> not based on a LTS release.

No doubt about it.

Thanks,
Michal