[ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Ian Jackson]

I think things are looking in reasonable shape.

I intend to branch off the 4.15 stable branch tomorrow.  I will then
turn off debug on that branch.  There will be a commit moratorium in
force for much of the afternoon while the branching is done -
commmitters please check your mail or irc.

Any outstanding patches that have a release-ack but have not yet been
committed should go in ASAP, and certainly by Friday.

I have reviewed my list of blockers and the conversation that followed
and there are just three areas that are still of concern to me:

* io-apic issue on Ryzen 1800X

   Related Qubes issue tracking this:
   https://github.com/QubesOS/qubes-issues/issues/6423
   Information from
     Jan Beulich <jbeulich@suse.com>
     Andrew Cooper <andrew.cooper3@citrix.com>
     Frédéric Pierret <frederic.pierret@qubes-os.org>

  Are we likely to get a fix in the next few days ?

  I think it may be time to reconcile ourselves to not fixing this,
  and deciding on a suitable plan B.  Do we need to put something in
  the release notes, or SUPPORT.md, or implement a mitigation of some
  kind ?

* Subject: Re: xenstore_lib.h and libxenstore API/ABI problems

   In the last mail in that thread, I wrote:   

   | I suggest, instead, that we:
   |
   | In 4.15:
   |
   |  * Retain the current soname, but:
   |  * Delete the tdb internals from the header file and cease to export
   |    those symbols.
   |  * Rename the expanding_buffer and sanitise_value functions, to
   |    properly namespace them, and move them to a private header.
   |
   | This is of course technically a breach of the ABI stability rules but
   | for the reasons I [give above] I don't think it will cause anyone any
   | trouble.

   I don't think I have seen any patches in this area.  I'm concerned
   that this is getting late.  I suspect we may have to punt this to
   xen-next.

 * Release notes (feature list), SUPPORT.md.  This is on my plate,
   although George is helping with the feature list (thanks!)

Thanks,
Ian.

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Roger Pau Monné]

On Tue, Mar 23, 2021 at 03:17:38PM +0000, Ian Jackson wrote:
> I think things are looking in reasonable shape.
> 
> I intend to branch off the 4.15 stable branch tomorrow.  I will then
> turn off debug on that branch.  There will be a commit moratorium in
> force for much of the afternoon while the branching is done -
> commmitters please check your mail or irc.
> 
> Any outstanding patches that have a release-ack but have not yet been
> committed should go in ASAP, and certainly by Friday.
> 
> I have reviewed my list of blockers and the conversation that followed
> and there are just three areas that are still of concern to me:
> 
> * io-apic issue on Ryzen 1800X
> 
>    Related Qubes issue tracking this:
>    https://github.com/QubesOS/qubes-issues/issues/6423
>    Information from
>      Jan Beulich <jbeulich@suse.com>
>      Andrew Cooper <andrew.cooper3@citrix.com>
>      Frédéric Pierret <frederic.pierret@qubes-os.org>
> 
>   Are we likely to get a fix in the next few days ?
> 
>   I think it may be time to reconcile ourselves to not fixing this,
>   and deciding on a suitable plan B.  Do we need to put something in
>   the release notes, or SUPPORT.md, or implement a mitigation of some
>   kind ?
> 
> * Subject: Re: xenstore_lib.h and libxenstore API/ABI problems
> 
>    In the last mail in that thread, I wrote:   
> 
>    | I suggest, instead, that we:
>    |
>    | In 4.15:
>    |
>    |  * Retain the current soname, but:
>    |  * Delete the tdb internals from the header file and cease to export
>    |    those symbols.
>    |  * Rename the expanding_buffer and sanitise_value functions, to
>    |    properly namespace them, and move them to a private header.
>    |
>    | This is of course technically a breach of the ABI stability rules but
>    | for the reasons I [give above] I don't think it will cause anyone any
>    | trouble.
> 
>    I don't think I have seen any patches in this area.  I'm concerned
>    that this is getting late.  I suspect we may have to punt this to
>    xen-next.
> 
>  * Release notes (feature list), SUPPORT.md.  This is on my plate,
>    although George is helping with the feature list (thanks!)

So there's also the series from Andrew to allow Solaris to boot
without resorting to use the 'msr_relaxed' option:

https://lore.kernel.org/xen-devel/20210316161844.1658-1-andrew.cooper3@citrix.com/

This has been R-A:

https://lore.kernel.org/xen-devel/24658.7471.309734.168120@mariner.uk.xensource.com/

But AFAICT it's missing a repost with some minimal adjustments?

If we don't get this in we should document on the release notes that
Solaris guests will likely require 'msr_relaxed=1' option added to the
configuration file in order to work.

Thanks, Roger.

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Ian Jackson]

(dropping Frédéric Pierret of Qubes, who was CC'd becausd of the Ryzen
issue.)

Roger Pau Monné writes ("Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow"):
> So there's also the series from Andrew to allow Solaris to boot
> without resorting to use the 'msr_relaxed' option:
> 
> https://lore.kernel.org/xen-devel/20210316161844.1658-1-andrew.cooper3@citrix.com/
> 
> This has been R-A:
> 
> https://lore.kernel.org/xen-devel/24658.7471.309734.168120@mariner.uk.xensource.com/

Thanks for pointing this out.  You seem to be right.

> But AFAICT it's missing a repost with some minimal adjustments?
> 
> If we don't get this in we should document on the release notes that
> Solaris guests will likely require 'msr_relaxed=1' option added to the
> configuration file in order to work.

Right.  I still would like to see this fixed.  But I think I would
like to see it finished and committed ASAP.  Can we manage that by
the end of Thursday ?

Ian.

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Jan Beulich]

On 23.03.2021 16:17, Ian Jackson wrote:
> I have reviewed my list of blockers and the conversation that followed
> and there are just three areas that are still of concern to me:
> 
> * io-apic issue on Ryzen 1800X
> 
>    Related Qubes issue tracking this:
>    https://github.com/QubesOS/qubes-issues/issues/6423
>    Information from
>      Jan Beulich <jbeulich@suse.com>
>      Andrew Cooper <andrew.cooper3@citrix.com>
>      Frédéric Pierret <frederic.pierret@qubes-os.org>
> 
>   Are we likely to get a fix in the next few days ?

Afaic - I'm still lacking certain bits of information to even think
of possible solutions.

>   I think it may be time to reconcile ourselves to not fixing this,
>   and deciding on a suitable plan B.  Do we need to put something in
>   the release notes, or SUPPORT.md, or implement a mitigation of some
>   kind ?

One option of course is, like was just done for 4.13.3, to revert.
Iirc Andrew had some thoughts towards making the new piece of code
conditional upon the original issue actually hitting. Another
(somewhat similar) option might be to hide the new piece of code
behind a default-off command line option.

Jan

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Ian Jackson]

Jan Beulich writes ("Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow"):
> On 23.03.2021 16:17, Ian Jackson wrote:
> >   I think it may be time to reconcile ourselves to not fixing this,
> >   and deciding on a suitable plan B.  Do we need to put something in
> >   the release notes, or SUPPORT.md, or implement a mitigation of some
> >   kind ?
> 
> One option of course is, like was just done for 4.13.3, to revert.
> Iirc Andrew had some thoughts towards making the new piece of code
> conditional upon the original issue actually hitting.

I would be very happy to consider a revert it someone would give me
references and explain to me the implications in words of one
syllable.

> Another
> (somewhat similar) option might be to hide the new piece of code
> behind a default-off command line option.

Likewise.

Sorry to be vague but I feel quite ignorant here!

Ian.

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Jan Beulich]

On 23.03.2021 17:53, Ian Jackson wrote:
> Jan Beulich writes ("Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow"):
>> On 23.03.2021 16:17, Ian Jackson wrote:
>>>   I think it may be time to reconcile ourselves to not fixing this,
>>>   and deciding on a suitable plan B.  Do we need to put something in
>>>   the release notes, or SUPPORT.md, or implement a mitigation of some
>>>   kind ?
>>
>> One option of course is, like was just done for 4.13.3, to revert.
>> Iirc Andrew had some thoughts towards making the new piece of code
>> conditional upon the original issue actually hitting.
> 
> I would be very happy to consider a revert it someone would give me
> references and explain to me the implications in words of one
> syllable.

Reference: e1de4c196a2e "x86/timer: Fix boot on Intel systems using
ITSSPRC static PIT clock gating"

Reverting would unbreak Xen on the Ryzen 1800X system where the
breakage was reported for, and likely a few others. It would at the
same time re-introduce Xen failing to boot on at least some Icelake
(and yet newer) systems.

>> Another
>> (somewhat similar) option might be to hide the new piece of code
>> behind a default-off command line option.
> 
> Likewise.

Well, not sure what to say here. Introducing a command line option
to allow making Icelake systems boot (by use of the option) while
keeping things working by default on older hardware is about as
simple as an explanation here can get, I guess.

Jan

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Ian Jackson]

Jan Beulich writes ("Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow"):
> On 23.03.2021 17:53, Ian Jackson wrote:
> >> One option of course is, like was just done for 4.13.3, to revert.
> >> Iirc Andrew had some thoughts towards making the new piece of code
> >> conditional upon the original issue actually hitting.
> > 
> > I would be very happy to consider a revert it someone would give me
> > references and explain to me the implications in words of one
> > syllable.
> 
> Reference: e1de4c196a2e "x86/timer: Fix boot on Intel systems using
> ITSSPRC static PIT clock gating"
> 
> Reverting would unbreak Xen on the Ryzen 1800X system where the
> breakage was reported for, and likely a few others. It would at the
> same time re-introduce Xen failing to boot on at least some Icelake
> (and yet newer) systems.

Thanks.  That explanation and the reference makes everything clear for
me.

> >> Another
> >> (somewhat similar) option might be to hide the new piece of code
> >> behind a default-off command line option.
> > 
> > Likewise.
> 
> Well, not sure what to say here. Introducing a command line option
> to allow making Icelake systems boot (by use of the option) while
> keeping things working by default on older hardware is about as
> simple as an explanation here can get, I guess.

Right.

The revert seems unattractive.  Your suggested command line option
sounds like a good workaround to me.  Under the circumstances it seems
like it should default to the old behaviour, as I think you are
suggesting.

So I am be inclined to ask if you, Jan, would prepare a patch
implementing such an option.  Anyone else have any opinions ?

Thanks,
Ian.

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Tamas K Lengyel]

> The revert seems unattractive.  Your suggested command line option
> sounds like a good workaround to me.  Under the circumstances it seems
> like it should default to the old behaviour, as I think you are
> suggesting.
>
> So I am be inclined to ask if you, Jan, would prepare a patch
> implementing such an option.  Anyone else have any opinions ?

I've replied to Jan's patch as well. IMHO having the option but
leaving things broken without the option set is a bad user experience
as we don't have a way to tell the user when the option is needing to
be set during install. Asking users to see if Xen crashes during boot
is a bad user experience as part of the setup. There should be an
automatic fallback to try enabling the legacy option if things don't
work without it.

Tamas

Re: [ANNOUNCE] Xen 4.15 - release status, branching tomorrow

[Andrew Cooper]

On 24/03/2021 11:48, Tamas K Lengyel wrote:
>> The revert seems unattractive.  Your suggested command line option
>> sounds like a good workaround to me.  Under the circumstances it seems
>> like it should default to the old behaviour, as I think you are
>> suggesting.
>>
>> So I am be inclined to ask if you, Jan, would prepare a patch
>> implementing such an option.  Anyone else have any opinions ?
> I've replied to Jan's patch as well. IMHO having the option but
> leaving things broken without the option set is a bad user experience
> as we don't have a way to tell the user when the option is needing to
> be set during install. Asking users to see if Xen crashes during boot
> is a bad user experience as part of the setup. There should be an
> automatic fallback to try enabling the legacy option if things don't
> work without it.

I agree.  I will try to make a fix with does the right thing and doesn't
require users to play blind guesswork to get their system booting.

~Andrew