Xen 4.18 release: Reminder about last posting date

Xen 4.18 release: Reminder about last posting date

[Henry Wang]

Hi everyone,

Following the release schedule discussion in in April, I am sending this email
to remind that according to the release schedule [1], August 11 (this Friday)
will be the last posting date, when patches adding new features are expected
to be posted to the mailing list by this date.

Also, note that we currently have 1 release blocker [2] which might need
some attention.

[1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
[2] https://gitlab.com/xen-project/xen/-/issues/114

Kind regards,
Henry

Re: Xen 4.18 release: Reminder about last posting date

[Stefano Stabellini]

On Mon, 7 Aug 2023, Henry Wang wrote:
> Hi everyone,
> 
> Following the release schedule discussion in in April, I am sending this email
> to remind that according to the release schedule [1], August 11 (this Friday)
> will be the last posting date, when patches adding new features are expected
> to be posted to the mailing list by this date.
> 
> Also, note that we currently have 1 release blocker [2] which might need
> some attention.
> 
> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
> [2] https://gitlab.com/xen-project/xen/-/issues/114

We have another release blocker in my option:
https://marc.info/?l=xen-devel&m=168312468808977
https://marc.info/?l=xen-devel&m=168312687610283

Andrew, would you OK with making it a release blocker? We already have a
tentative fix from Julien.

Re: Xen 4.18 release: Reminder about last posting date

[Andrew Cooper]

On 07/08/2023 8:24 pm, Stefano Stabellini wrote:
> On Mon, 7 Aug 2023, Henry Wang wrote:
>> Hi everyone,
>>
>> Following the release schedule discussion in in April, I am sending this email
>> to remind that according to the release schedule [1], August 11 (this Friday)
>> will be the last posting date, when patches adding new features are expected
>> to be posted to the mailing list by this date.
>>
>> Also, note that we currently have 1 release blocker [2] which might need
>> some attention.
>>
>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
>> [2] https://gitlab.com/xen-project/xen/-/issues/114
> We have another release blocker in my option:
> https://marc.info/?l=xen-devel&m=168312468808977
> https://marc.info/?l=xen-devel&m=168312687610283
>
> Andrew, would you OK with making it a release blocker? We already have a
> tentative fix from Julien.

Yes that is a release critical/blocker.  No it's not a fix.

~Andrew

Re: Xen 4.18 release: Reminder about last posting date

[Henry Wang]

Hi Stefano,

> On Aug 8, 2023, at 03:24, Stefano Stabellini <sstabellini@kernel.org> wrote:
> 
> On Mon, 7 Aug 2023, Henry Wang wrote:
>> Hi everyone,
>> 
>> Following the release schedule discussion in in April, I am sending this email
>> to remind that according to the release schedule [1], August 11 (this Friday)
>> will be the last posting date, when patches adding new features are expected
>> to be posted to the mailing list by this date.
>> 
>> Also, note that we currently have 1 release blocker [2] which might need
>> some attention.
>> 
>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
>> [2] https://gitlab.com/xen-project/xen/-/issues/114
> 
> We have another release blocker in my option:
> https://marc.info/?l=xen-devel&m=168312468808977
> https://marc.info/?l=xen-devel&m=168312687610283

I’ve taken a note for this, thanks for the reminder.

Kind regards,
Henry

> 
> Andrew, would you OK with making it a release blocker? We already have a
> tentative fix from Julien.
> 

Re: Xen 4.18 release: Reminder about last posting date

[Andrew Cooper]

On 07/08/2023 2:24 am, Henry Wang wrote:
> Hi everyone,
>
> Following the release schedule discussion in in April, I am sending this email
> to remind that according to the release schedule [1], August 11 (this Friday)
> will be the last posting date, when patches adding new features are expected
> to be posted to the mailing list by this date.
>
> Also, note that we currently have 1 release blocker [2] which might need
> some attention.
>
> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
> [2] https://gitlab.com/xen-project/xen/-/issues/114

Off the top of my head.

There are still unaddressed Gitlab bugs from the Eclair integration, and
other Gitlab bugs (use of unstable containers) which I'd unwilling to
let 4.18 be released with, given the pain we've had on the stable trees
trying to keep CI working.

There's the XENVERSION_* hypercall ABI truncation fixes which is a
blocker for XenServer (I'm refusing to forward port a piece of technical
debt when it needs fixing upstream).

There's another thing on the back of Tuesday's mess which has
technically been posted years ago - the proper eIBRS support, but I want
to refresh and get sorted for 4.18.  I expect it to be an even bigger
perf win than the ARCH_CAPS work was.  But I'm not going to have time to
get that sorted this week.

At this point, I'm hoping to squeeze in before RC1, along with some
compelling perf numbers.

~Andrew

Re: Xen 4.18 release: Reminder about last posting date

[Stefano Stabellini]

On Wed, 9 Aug 2023, Andrew Cooper wrote:
> On 07/08/2023 2:24 am, Henry Wang wrote:
> > Hi everyone,
> >
> > Following the release schedule discussion in in April, I am sending this email
> > to remind that according to the release schedule [1], August 11 (this Friday)
> > will be the last posting date, when patches adding new features are expected
> > to be posted to the mailing list by this date.
> >
> > Also, note that we currently have 1 release blocker [2] which might need
> > some attention.
> >
> > [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
> > [2] https://gitlab.com/xen-project/xen/-/issues/114
> 
> Off the top of my head.
> 
> There are still unaddressed Gitlab bugs from the Eclair integration

The bug you managed to find it is now fixed (commit e55146071de9). I am
all for fixing Gitlab bugs so let me know if you find anything else! I
am not aware of any other issue with Eclair at the moment.


> and other Gitlab bugs (use of unstable containers) which I'd unwilling
> to let 4.18 be released with, given the pain we've had on the stable
> trees trying to keep CI working.

That is fair enough. To make this more concrete and easier to track, the
following would need to be changed to using stable containers:

- .qemu-arm64
- .qemu-arm32
(I am not counting .qemu-riscv64)

Andrew, is that what you meant? Am I missing anything?

Re: Xen 4.18 release: Reminder about last posting date

[Andrew Cooper]

On 09/08/2023 10:43 pm, Stefano Stabellini wrote:
> On Wed, 9 Aug 2023, Andrew Cooper wrote:
>> On 07/08/2023 2:24 am, Henry Wang wrote:
>>> Hi everyone,
>>>
>>> Following the release schedule discussion in in April, I am sending this email
>>> to remind that according to the release schedule [1], August 11 (this Friday)
>>> will be the last posting date, when patches adding new features are expected
>>> to be posted to the mailing list by this date.
>>>
>>> Also, note that we currently have 1 release blocker [2] which might need
>>> some attention.
>>>
>>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
>>> [2] https://gitlab.com/xen-project/xen/-/issues/114
>> Off the top of my head.
>>
>> There are still unaddressed Gitlab bugs from the Eclair integration
> The bug you managed to find it is now fixed (commit e55146071de9). I am
> all for fixing Gitlab bugs so let me know if you find anything else! I
> am not aware of any other issue with Eclair at the moment.

I meant the one where Eclair is still running on `smoke` and twiddling
its thumbs for 1h doing so each time OSSTest says yes to a push.  It
will be a missing 'exclude' somewhere, but I haven't hand enough time to
look.

>> and other Gitlab bugs (use of unstable containers) which I'd unwilling
>> to let 4.18 be released with, given the pain we've had on the stable
>> trees trying to keep CI working.
> That is fair enough. To make this more concrete and easier to track, the
> following would need to be changed to using stable containers:
>
> - .qemu-arm64
> - .qemu-arm32
> (I am not counting .qemu-riscv64)
>
> Andrew, is that what you meant? Am I missing anything?

Every debian unstable container, and the other containers (OpenSUSE)
which are using an non-specific upstream version.  Upstreams which
really are rolling distros (Arch, Tumbleweed) need to be made non-fatal.

Then the second part which is also a blocker.  Update the distros we
test with to be less obsolete.  e.g. the majority of our CI testing runs
with Alpine-3.12 when EOL more than a year ago. (2022-05-01)

~Andrew

Re: Xen 4.18 release: Reminder about last posting date

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2758 bytes --]

On Wed, 9 Aug 2023, Andrew Cooper wrote:
> On 09/08/2023 10:43 pm, Stefano Stabellini wrote:
> > On Wed, 9 Aug 2023, Andrew Cooper wrote:
> >> On 07/08/2023 2:24 am, Henry Wang wrote:
> >>> Hi everyone,
> >>>
> >>> Following the release schedule discussion in in April, I am sending this email
> >>> to remind that according to the release schedule [1], August 11 (this Friday)
> >>> will be the last posting date, when patches adding new features are expected
> >>> to be posted to the mailing list by this date.
> >>>
> >>> Also, note that we currently have 1 release blocker [2] which might need
> >>> some attention.
> >>>
> >>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
> >>> [2] https://gitlab.com/xen-project/xen/-/issues/114
> >> Off the top of my head.
> >>
> >> There are still unaddressed Gitlab bugs from the Eclair integration
> > The bug you managed to find it is now fixed (commit e55146071de9). I am
> > all for fixing Gitlab bugs so let me know if you find anything else! I
> > am not aware of any other issue with Eclair at the moment.
> 
> I meant the one where Eclair is still running on `smoke` and twiddling
> its thumbs for 1h doing so each time OSSTest says yes to a push.  It
> will be a missing 'exclude' somewhere, but I haven't hand enough time to
> look.

Ah, yes, got it. I would call it "Eclair lagging behind". I am happy for
this to be a blocker.


> >> and other Gitlab bugs (use of unstable containers) which I'd unwilling
> >> to let 4.18 be released with, given the pain we've had on the stable
> >> trees trying to keep CI working.
> > That is fair enough. To make this more concrete and easier to track, the
> > following would need to be changed to using stable containers:
> >
> > - .qemu-arm64
> > - .qemu-arm32
> > (I am not counting .qemu-riscv64)
> >
> > Andrew, is that what you meant? Am I missing anything?
> 
> Every debian unstable container, and the other containers (OpenSUSE)
> which are using an non-specific upstream version.  Upstreams which
> really are rolling distros (Arch, Tumbleweed) need to be made non-fatal.

I was more asking whether for 4.18 you want to fix only test.yaml or
also build.yaml. (Typically it is test.yaml that causes most issues
with rolling containers.)

.qemu-arm64 and .qemu-arm32 are the only ones using unstable/rolling
containers in automation/gitlab-ci/test.yaml (excluding .qemu-riscv64),
that's why I was asking about those two specifically.


> Then the second part which is also a blocker.  Update the distros we
> test with to be less obsolete.  e.g. the majority of our CI testing runs
> with Alpine-3.12 when EOL more than a year ago. (2022-05-01)

OK. Hopefully it will be trivial.

Re: Xen 4.18 release: Reminder about last posting date

[Andrew Cooper]

On 09/08/2023 11:34 pm, Stefano Stabellini wrote:
> On Wed, 9 Aug 2023, Andrew Cooper wrote:
>> On 09/08/2023 10:43 pm, Stefano Stabellini wrote:
>>> On Wed, 9 Aug 2023, Andrew Cooper wrote:
>>>> On 07/08/2023 2:24 am, Henry Wang wrote:
>>>>> Hi everyone,
>>>>>
>>>>> Following the release schedule discussion in in April, I am sending this email
>>>>> to remind that according to the release schedule [1], August 11 (this Friday)
>>>>> will be the last posting date, when patches adding new features are expected
>>>>> to be posted to the mailing list by this date.
>>>>>
>>>>> Also, note that we currently have 1 release blocker [2] which might need
>>>>> some attention.
>>>>>
>>>>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
>>>>> [2] https://gitlab.com/xen-project/xen/-/issues/114
>>>> Off the top of my head.
>>>>
>>>> There are still unaddressed Gitlab bugs from the Eclair integration
>>> The bug you managed to find it is now fixed (commit e55146071de9). I am
>>> all for fixing Gitlab bugs so let me know if you find anything else! I
>>> am not aware of any other issue with Eclair at the moment.
>> I meant the one where Eclair is still running on `smoke` and twiddling
>> its thumbs for 1h doing so each time OSSTest says yes to a push.  It
>> will be a missing 'exclude' somewhere, but I haven't hand enough time to
>> look.
> Ah, yes, got it. I would call it "Eclair lagging behind". I am happy for
> this to be a blocker.

Something else for the list is to see about trimming down the testing
we're doing.

I had to cancel 7 pipelines (mostly patchew) in order to get a gitlab
run on a late-breaking tweak on the security content yesterday.

Take
https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/959800176
as an example.  Nearly 2h to run, and queued for 2h before that waiting
to start.  This is not a rare occurrence right now.

>>>> and other Gitlab bugs (use of unstable containers) which I'd unwilling
>>>> to let 4.18 be released with, given the pain we've had on the stable
>>>> trees trying to keep CI working.
>>> That is fair enough. To make this more concrete and easier to track, the
>>> following would need to be changed to using stable containers:
>>>
>>> - .qemu-arm64
>>> - .qemu-arm32
>>> (I am not counting .qemu-riscv64)
>>>
>>> Andrew, is that what you meant? Am I missing anything?
>> Every debian unstable container, and the other containers (OpenSUSE)
>> which are using an non-specific upstream version.  Upstreams which
>> really are rolling distros (Arch, Tumbleweed) need to be made non-fatal.
> I was more asking whether for 4.18 you want to fix only test.yaml or
> also build.yaml. (Typically it is test.yaml that causes most issues
> with rolling containers.)

Everything.  Because both bite equally hard in stable trees.

It's not hard to do - just needs some time.

~Andrew

Re: Xen 4.18 release: Reminder about last posting date

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3957 bytes --]

On Wed, 9 Aug 2023, Andrew Cooper wrote:
> On 09/08/2023 11:34 pm, Stefano Stabellini wrote:
> > On Wed, 9 Aug 2023, Andrew Cooper wrote:
> >> On 09/08/2023 10:43 pm, Stefano Stabellini wrote:
> >>> On Wed, 9 Aug 2023, Andrew Cooper wrote:
> >>>> On 07/08/2023 2:24 am, Henry Wang wrote:
> >>>>> Hi everyone,
> >>>>>
> >>>>> Following the release schedule discussion in in April, I am sending this email
> >>>>> to remind that according to the release schedule [1], August 11 (this Friday)
> >>>>> will be the last posting date, when patches adding new features are expected
> >>>>> to be posted to the mailing list by this date.
> >>>>>
> >>>>> Also, note that we currently have 1 release blocker [2] which might need
> >>>>> some attention.
> >>>>>
> >>>>> [1] https://lore.kernel.org/xen-devel/AS8PR08MB79919F9CE0B2BF80E7103FB592609@AS8PR08MB7991.eurprd08.prod.outlook.com/
> >>>>> [2] https://gitlab.com/xen-project/xen/-/issues/114
> >>>> Off the top of my head.
> >>>>
> >>>> There are still unaddressed Gitlab bugs from the Eclair integration
> >>> The bug you managed to find it is now fixed (commit e55146071de9). I am
> >>> all for fixing Gitlab bugs so let me know if you find anything else! I
> >>> am not aware of any other issue with Eclair at the moment.
> >> I meant the one where Eclair is still running on `smoke` and twiddling
> >> its thumbs for 1h doing so each time OSSTest says yes to a push.  It
> >> will be a missing 'exclude' somewhere, but I haven't hand enough time to
> >> look.
> > Ah, yes, got it. I would call it "Eclair lagging behind". I am happy for
> > this to be a blocker.
> 
> Something else for the list is to see about trimming down the testing
> we're doing.
> 
> I had to cancel 7 pipelines (mostly patchew) in order to get a gitlab
> run on a late-breaking tweak on the security content yesterday.
> 
> Take
> https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/959800176
> as an example.  Nearly 2h to run, and queued for 2h before that waiting
> to start.  This is not a rare occurrence right now.

Yeah...

So, an easy fix that would solve the problem for the rest of the tests
but not Eclair would be to reduce the number of tests. It is easy
because we can make them conditional on a CI/CD variable and *not*
define the variable in patchew. That way patchew gets a reduced set of
tests. I think that would be fine.

Another option we can also pursue in parallel is to throw more hardware
at the problem. I am not sure where we are on reusing Colo hardware for
this.

Eclair is a different issue because we cannot increase the hardware and
I don't think we want to reduce the set of rules to be scanned. So the
only idea I can come up with is to limit the number of Eclair jobs in
patchew to 1-2 per day. However it would be pretty random which ones are
getting executed. Or find a wat to prioritize non-patchew jobs? If you
or anyone else has better idea feel free to suggest.


> >>>> and other Gitlab bugs (use of unstable containers) which I'd unwilling
> >>>> to let 4.18 be released with, given the pain we've had on the stable
> >>>> trees trying to keep CI working.
> >>> That is fair enough. To make this more concrete and easier to track, the
> >>> following would need to be changed to using stable containers:
> >>>
> >>> - .qemu-arm64
> >>> - .qemu-arm32
> >>> (I am not counting .qemu-riscv64)
> >>>
> >>> Andrew, is that what you meant? Am I missing anything?
> >> Every debian unstable container, and the other containers (OpenSUSE)
> >> which are using an non-specific upstream version.  Upstreams which
> >> really are rolling distros (Arch, Tumbleweed) need to be made non-fatal.
> > I was more asking whether for 4.18 you want to fix only test.yaml or
> > also build.yaml. (Typically it is test.yaml that causes most issues
> > with rolling containers.)
> 
> Everything.  Because both bite equally hard in stable trees.
> 
> It's not hard to do - just needs some time.
 
OK