Boot time and 3 sec in warning_print

Boot time and 3 sec in warning_print

[Anders Törnqvist]

Hi,

I would like to shorten the boot time in our system if possible.

In xen/common/warning.c there is warning_print() which contains a 3 
seconds loop that calls  process_pending_softirqs().

What would the potential problems be if that loop is radically shortened?

/Anders

Re: Boot time and 3 sec in warning_print

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 561 bytes --]

On 15.02.21 08:37, Anders Törnqvist wrote:
> Hi,
> 
> I would like to shorten the boot time in our system if possible.
> 
> In xen/common/warning.c there is warning_print() which contains a 3 
> seconds loop that calls  process_pending_softirqs().
> 
> What would the potential problems be if that loop is radically shortened?

A user might not notice the warning(s) printed.

But I can see your point. I think adding a boot option for setting
another timeout value (e.g. 0) would do the job without compromising
the default case.


Juergen

[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Boot time and 3 sec in warning_print

[Jan Beulich]

On 15.02.2021 09:13, Jürgen Groß wrote:
> On 15.02.21 08:37, Anders Törnqvist wrote:
>> I would like to shorten the boot time in our system if possible.
>>
>> In xen/common/warning.c there is warning_print() which contains a 3 
>> seconds loop that calls  process_pending_softirqs().
>>
>> What would the potential problems be if that loop is radically shortened?
> 
> A user might not notice the warning(s) printed.
> 
> But I can see your point. I think adding a boot option for setting
> another timeout value (e.g. 0) would do the job without compromising
> the default case.

I don't think I agree - the solution to this is to eliminate the
reason leading to the warning. The delay is intentionally this way
to annoy the admin and force them to take measures.

Jan

Re: Boot time and 3 sec in warning_print

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 1340 bytes --]

On 15.02.21 09:38, Jan Beulich wrote:
> On 15.02.2021 09:13, Jürgen Groß wrote:
>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>> I would like to shorten the boot time in our system if possible.
>>>
>>> In xen/common/warning.c there is warning_print() which contains a 3
>>> seconds loop that calls  process_pending_softirqs().
>>>
>>> What would the potential problems be if that loop is radically shortened?
>>
>> A user might not notice the warning(s) printed.
>>
>> But I can see your point. I think adding a boot option for setting
>> another timeout value (e.g. 0) would do the job without compromising
>> the default case.
> 
> I don't think I agree - the solution to this is to eliminate the
> reason leading to the warning. The delay is intentionally this way
> to annoy the admin and force them to take measures.

OTOH there are some warnings which can't be dealt with via boot
parameters or which can be solved by other means, e.g.:

"WARNING: SILO mode is not enabled.\n"
"It has implications on the security of the system,\n"
"unless the communications have been forbidden between\n"
"untrusted domains.\n"

"WARNING: HMP COMPUTING HAS BEEN ENABLED.\n"
"It has implications on the security and stability of the system,\n"
"unless the cpu affinity of all domains is specified.\n"


Juergen

[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Boot time and 3 sec in warning_print

[Julien Grall]

Hi Juergen,

On 15/02/2021 08:51, Jürgen Groß wrote:
> On 15.02.21 09:38, Jan Beulich wrote:
>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>>> I would like to shorten the boot time in our system if possible.
>>>>
>>>> In xen/common/warning.c there is warning_print() which contains a 3
>>>> seconds loop that calls  process_pending_softirqs().
>>>>
>>>> What would the potential problems be if that loop is radically 
>>>> shortened?
>>>
>>> A user might not notice the warning(s) printed.
>>>
>>> But I can see your point. I think adding a boot option for setting
>>> another timeout value (e.g. 0) would do the job without compromising
>>> the default case.
>>
>> I don't think I agree - the solution to this is to eliminate the
>> reason leading to the warning. The delay is intentionally this way
>> to annoy the admin and force them to take measures.
> 
> OTOH there are some warnings which can't be dealt with via boot
> parameters or which can be solved by other means, e.g
Both of the warning can be removed...

> "WARNING: SILO mode is not enabled.\n"
> "It has implications on the security of the system,\n"
> "unless the communications have been forbidden between\n"
> "untrusted domains.\n"

The default Arm config will use SILO mode. You would have needed to 
tweak your .config in order to get this message.

There are current discussion to allow again FLASK on Armv8.1+ platform 
(see [1]).

> 
> "WARNING: HMP COMPUTING HAS BEEN ENABLED.\n"
> "It has implications on the security and stability of the system,\n"
> "unless the cpu affinity of all domains is specified.\n"

This is printed if the admin add "hmp-unsafe=yes" on the command line.

Cheers,

[1]
https://lore.kernel.org/xen-devel/20201111215203.80336-1-ash.j.wilding@gmail.com/

-- 
Julien Grall

Re: Boot time and 3 sec in warning_print

[Julien Grall]

Hi Jan,

On 15/02/2021 08:38, Jan Beulich wrote:
> On 15.02.2021 09:13, Jürgen Groß wrote:
>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>> I would like to shorten the boot time in our system if possible.
>>>
>>> In xen/common/warning.c there is warning_print() which contains a 3
>>> seconds loop that calls  process_pending_softirqs().
>>>
>>> What would the potential problems be if that loop is radically shortened?
>>
>> A user might not notice the warning(s) printed.
>>
>> But I can see your point. I think adding a boot option for setting
>> another timeout value (e.g. 0) would do the job without compromising
>> the default case.
> 
> I don't think I agree - the solution to this is to eliminate the
> reason leading to the warning. 
 >
> The delay is intentionally this way
> to annoy the admin and force them to take measures.
Given they are warning, an admin may have assessed them and considered 
there is no remediation necessary.

We encountered the same problem with LiveUpdate. If you happen to have a 
warning (e.g. sync_console for debugging), then you are adding 3s in 
your downtime (this can more than double-up the actual one).

What was just an "annoyance" for boot can now completely wreck your 
guests and system (not many software can tolerate large downtime).

So I think we either want to drop the 3s pause completely or allow the 
user to decide whether he/she cares about it via a command line option.

I am leaning towards the former at the moment.

Cheers,

-- 
Julien Grall

Re: Boot time and 3 sec in warning_print

[Jan Beulich]

On 15.02.2021 11:35, Julien Grall wrote:
> On 15/02/2021 08:38, Jan Beulich wrote:
>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>>> I would like to shorten the boot time in our system if possible.
>>>>
>>>> In xen/common/warning.c there is warning_print() which contains a 3
>>>> seconds loop that calls  process_pending_softirqs().
>>>>
>>>> What would the potential problems be if that loop is radically shortened?
>>>
>>> A user might not notice the warning(s) printed.
>>>
>>> But I can see your point. I think adding a boot option for setting
>>> another timeout value (e.g. 0) would do the job without compromising
>>> the default case.
>>
>> I don't think I agree - the solution to this is to eliminate the
>> reason leading to the warning. 
>  >
>> The delay is intentionally this way
>> to annoy the admin and force them to take measures.
> Given they are warning, an admin may have assessed them and considered 
> there is no remediation necessary.
> 
> We encountered the same problem with LiveUpdate. If you happen to have a 
> warning (e.g. sync_console for debugging), then you are adding 3s in 
> your downtime (this can more than double-up the actual one).

One very explicitly should not run with sync_console in production.

> What was just an "annoyance" for boot can now completely wreck your 
> guests and system (not many software can tolerate large downtime).
> 
> So I think we either want to drop the 3s pause completely or allow the 
> user to decide whether he/she cares about it via a command line option.
> 
> I am leaning towards the former at the moment.

I'm afraid I'm -2 towards complete removal. I'm at least -1 towards
shortening of the pause, as already indicated.

Jan

Re: Boot time and 3 sec in warning_print

[Julien Grall]

Hi Jan,

On 15/02/2021 10:41, Jan Beulich wrote:
> On 15.02.2021 11:35, Julien Grall wrote:
>> On 15/02/2021 08:38, Jan Beulich wrote:
>>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>>>> I would like to shorten the boot time in our system if possible.
>>>>>
>>>>> In xen/common/warning.c there is warning_print() which contains a 3
>>>>> seconds loop that calls  process_pending_softirqs().
>>>>>
>>>>> What would the potential problems be if that loop is radically shortened?
>>>>
>>>> A user might not notice the warning(s) printed.
>>>>
>>>> But I can see your point. I think adding a boot option for setting
>>>> another timeout value (e.g. 0) would do the job without compromising
>>>> the default case.
>>>
>>> I don't think I agree - the solution to this is to eliminate the
>>> reason leading to the warning.
>>   >
>>> The delay is intentionally this way
>>> to annoy the admin and force them to take measures.
>> Given they are warning, an admin may have assessed them and considered
>> there is no remediation necessary.
>>
>> We encountered the same problem with LiveUpdate. If you happen to have a
>> warning (e.g. sync_console for debugging), then you are adding 3s in
>> your downtime (this can more than double-up the actual one).
> 
> One very explicitly should not run with sync_console in production.

I knew it would be misinterpreted ;). I agree that sync_console must not 
be used in production.

I gave the example of sync_console because this is something impacting 
debugging of LiveUpdate. If you have a console issue and need to add 
sync_console, then you may end up to wreck completely your platform when 
LiveUpdating.

Without the 3s delay, then you have a chance to LiveUpdate and figure 
out the problem.

> 
>> What was just an "annoyance" for boot can now completely wreck your
>> guests and system (not many software can tolerate large downtime).
>>
>> So I think we either want to drop the 3s pause completely or allow the
>> user to decide whether he/she cares about it via a command line option.
>>
>> I am leaning towards the former at the moment.
> 
> I'm afraid I'm -2 towards complete removal. I'm at least -1 towards
> shortening of the pause, as already indicated.
So how do you suggest to approach the issues I discussed above?

Cheers,

-- 
Julien Grall

Re: Boot time and 3 sec in warning_print

[Jan Beulich]

On 15.02.2021 11:50, Julien Grall wrote:
> Hi Jan,
> 
> On 15/02/2021 10:41, Jan Beulich wrote:
>> On 15.02.2021 11:35, Julien Grall wrote:
>>> On 15/02/2021 08:38, Jan Beulich wrote:
>>>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>>>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>>>>> I would like to shorten the boot time in our system if possible.
>>>>>>
>>>>>> In xen/common/warning.c there is warning_print() which contains a 3
>>>>>> seconds loop that calls  process_pending_softirqs().
>>>>>>
>>>>>> What would the potential problems be if that loop is radically shortened?
>>>>>
>>>>> A user might not notice the warning(s) printed.
>>>>>
>>>>> But I can see your point. I think adding a boot option for setting
>>>>> another timeout value (e.g. 0) would do the job without compromising
>>>>> the default case.
>>>>
>>>> I don't think I agree - the solution to this is to eliminate the
>>>> reason leading to the warning.
>>>   >
>>>> The delay is intentionally this way
>>>> to annoy the admin and force them to take measures.
>>> Given they are warning, an admin may have assessed them and considered
>>> there is no remediation necessary.
>>>
>>> We encountered the same problem with LiveUpdate. If you happen to have a
>>> warning (e.g. sync_console for debugging), then you are adding 3s in
>>> your downtime (this can more than double-up the actual one).
>>
>> One very explicitly should not run with sync_console in production.
> 
> I knew it would be misinterpreted ;). I agree that sync_console must not 
> be used in production.
> 
> I gave the example of sync_console because this is something impacting 
> debugging of LiveUpdate. If you have a console issue and need to add 
> sync_console, then you may end up to wreck completely your platform when 
> LiveUpdating.
> 
> Without the 3s delay, then you have a chance to LiveUpdate and figure 
> out the problem.

I'm afraid I don't see how LU comes into the picture here: We're
talking about a boot time delay. The delay doesn't recur at any
point at runtime.

Jan

Re: Boot time and 3 sec in warning_print

[Julien Grall]

Hi Jan,

On 15/02/2021 12:29, Jan Beulich wrote:
> On 15.02.2021 11:50, Julien Grall wrote:
>> On 15/02/2021 10:41, Jan Beulich wrote:
>>> On 15.02.2021 11:35, Julien Grall wrote:
>>>> On 15/02/2021 08:38, Jan Beulich wrote:
>>>>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>>>>> On 15.02.21 08:37, Anders Törnqvist wrote:
>>>>>>> I would like to shorten the boot time in our system if possible.
>>>>>>>
>>>>>>> In xen/common/warning.c there is warning_print() which contains a 3
>>>>>>> seconds loop that calls  process_pending_softirqs().
>>>>>>>
>>>>>>> What would the potential problems be if that loop is radically shortened?
>>>>>>
>>>>>> A user might not notice the warning(s) printed.
>>>>>>
>>>>>> But I can see your point. I think adding a boot option for setting
>>>>>> another timeout value (e.g. 0) would do the job without compromising
>>>>>> the default case.
>>>>>
>>>>> I don't think I agree - the solution to this is to eliminate the
>>>>> reason leading to the warning.
>>>>    >
>>>>> The delay is intentionally this way
>>>>> to annoy the admin and force them to take measures.
>>>> Given they are warning, an admin may have assessed them and considered
>>>> there is no remediation necessary.
>>>>
>>>> We encountered the same problem with LiveUpdate. If you happen to have a
>>>> warning (e.g. sync_console for debugging), then you are adding 3s in
>>>> your downtime (this can more than double-up the actual one).
>>>
>>> One very explicitly should not run with sync_console in production.
>>
>> I knew it would be misinterpreted ;). I agree that sync_console must not
>> be used in production.
>>
>> I gave the example of sync_console because this is something impacting
>> debugging of LiveUpdate. If you have a console issue and need to add
>> sync_console, then you may end up to wreck completely your platform when
>> LiveUpdating.
>>
>> Without the 3s delay, then you have a chance to LiveUpdate and figure
>> out the problem.
> 
> I'm afraid I don't see how LU comes into the picture here: We're
> talking about a boot time delay. The delay doesn't recur at any
> point at runtime.

Live Update is a reboot of the hypervisor. The main difference is we are 
carrying the state of each domain states to the new Xen along with some 
strictly necessary global state (e.g. IOMMU).

So the new Xen will mostly followed the same boot path as you would from 
a "classic" (re)boot. There are only a few twist necessary for 
LiveUpdate (e.g. reserving memory, creating/restoring multiple domains).

We would need to gate the 3s pause in the case of LiveUpdate. At which 
point, it seems we may want to take an approach that also benefits other 
users.

Cheers,

-- 
Julien Grall

Re: Boot time and 3 sec in warning_print

[Andrew Cooper]

On 15/02/2021 10:41, Jan Beulich wrote:
> On 15.02.2021 11:35, Julien Grall wrote:
>> On 15/02/2021 08:38, Jan Beulich wrote:
>>> On 15.02.2021 09:13, Jürgen Groß wrote:
>> What was just an "annoyance" for boot can now completely wreck your 
>> guests and system (not many software can tolerate large downtime).
>>
>> So I think we either want to drop the 3s pause completely or allow the 
>> user to decide whether he/she cares about it via a command line option.
>>
>> I am leaning towards the former at the moment.
> I'm afraid I'm -2 towards complete removal. I'm at least -1 towards
> shortening of the pause, as already indicated.

A 3s delay on boot doesn't even cause most people to notice.  The
infrastructure has failed at its intended purpose.

Therefore, we should consider now to replace this largely-failed
experiment with something better.


Personally, I think ARM is abusing this in the first place.  Adding a 3
second delay for someone who's explicitly chosen hmp_unsafe is petty. 
So is adding a 3 second delay for anyone who's explicitly chosen a
non-default configuration.  In retrospect, I think the delay for hvm_fep
is also wrong, especially as we also have a taint for it.


The *only* way to make users deal with the warnings is to surface them
very obviously in the UI they use to interact with their Xen system. 
That is XenCenter/XenOrchestra/virt-manager/etc, and possibly the SSH
MOTD - not a logfile that approximately noone reads.

To make this happen, warnings need to be available somewhere which isn't
the dmesg ring.  hypfs would be the obvious candidate at the moment.

~Andrew

Re: Boot time and 3 sec in warning_print

[Julien Grall]

Hi Andrew,

On 15/02/2021 15:00, Andrew Cooper wrote:
> On 15/02/2021 10:41, Jan Beulich wrote:
>> On 15.02.2021 11:35, Julien Grall wrote:
>>> On 15/02/2021 08:38, Jan Beulich wrote:
>>>> On 15.02.2021 09:13, Jürgen Groß wrote:
>>> What was just an "annoyance" for boot can now completely wreck your
>>> guests and system (not many software can tolerate large downtime).
>>>
>>> So I think we either want to drop the 3s pause completely or allow the
>>> user to decide whether he/she cares about it via a command line option.
>>>
>>> I am leaning towards the former at the moment.
>> I'm afraid I'm -2 towards complete removal. I'm at least -1 towards
>> shortening of the pause, as already indicated.
> 
> A 3s delay on boot doesn't even cause most people to notice.  The
> infrastructure has failed at its intended purpose.
> 
> Therefore, we should consider now to replace this largely-failed
> experiment with something better.
> 
> 
> Personally, I think ARM is abusing this in the first place.  Adding a 3
> second delay for someone who's explicitly chosen hmp_unsafe is petty.

For Arm, the original goal was to get all the important warning in a 
single place so they are easy to find.

The 3 seconds is unfortunately an unintended consequence of using 
warning_add().

> So is adding a 3 second delay for anyone who's explicitly chosen a
> non-default configuration.  In retrospect, I think the delay for hvm_fep
> is also wrong, especially as we also have a taint for it. >
> 
> The *only* way to make users deal with the warnings is to surface them
> very obviously in the UI they use to interact with their Xen system.
> That is XenCenter/XenOrchestra/virt-manager/etc, and possibly the SSH
> MOTD - not a logfile that approximately noone reads.
> 
> To make this happen, warnings need to be available somewhere which isn't
> the dmesg ring.  hypfs would be the obvious candidate at the moment.

Could we also taint Xen if there is a major warning?

Cheers,

-- 
Julien Grall