Checking for network online

Checking for network online

[Johnathan Mantey]

[-- Attachment #1.1: Type: text/plain, Size: 1696 bytes --]

/I/ntel has recently run into an issue regarding a systemd service, and 
we're interested in soliciting feedback from the community.

Issue: systemd-networkd-wait-online.service stalls for 120 seconds when 
the managed NICs do not have a network connection.

TLDR: Should OpenBMC remove systemd-networkd-wait-online.service 
universally?

System Config: All NICs in the system are not connected to an active 
network.

Test Process: The system under test (SUT) has AC removed, and some time 
later AC applied. Wait for BMC/BIOS to boot

Behavior: U-Boot hands control to the Linux boot process, and the 
systemd services are started. When systemd-networkd-wait-online.service 
starts it stalls waiting for the NICs to enter a fully functional state. 
This never happens during the default 120 second timeout period for this 
service. When the timeout elapses, an error message is logged to the 
journal reporting the service exited unsuccessfully.

Issues: This service blocks entry to multi-user.target.
phosphor-state-manager uses multi-user.target to report the BMC is ready 
to use.
This is reported via IPMI Get Device ID.
The Intel BIOS is blocked from booting until 
systemd-networkd-wait-online times out.
BMC entry to multi-user.target is delayed. Journal entries are created.

Question for the community: Given the negative side effects caused by 
running this service does the community want to have this service 
collectively removed from global build image?

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Checking for network online

[Jeremy Kerr]

Hi Johnathan,

> Issue: systemd-networkd-wait-online.service stalls for 120 seconds
> when the managed NICs do not have a network connection.
> 
> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service 
> universally?

Probably not, it's required to implement network-online,target, which
is standard, and may be referred to by arbitrary packages.

There's some good background on the issues you're experiencing here:

 https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

in short: most services should be able to start before network-
online.target, and be able to adapt to changes in network configuration
after that point.

For your specific issue there:

> Issues: This service blocks entry to multi-user.target.
> phosphor-state-manager uses multi-user.target to report the BMC is
> ready to use.
> This is reported via IPMI Get Device ID.

That sounds like more of an issue of whether that reported state
actually represents the expected BMC state...

Regards,


Jeremy

Re: Checking for network online

[Lei Yu]

On Fri, Feb 18, 2022 at 8:11 AM Jeremy Kerr <jk@codeconstruct.com.au> wrote:
>
> Hi Johnathan,
>
> > Issue: systemd-networkd-wait-online.service stalls for 120 seconds
> > when the managed NICs do not have a network connection.
> >
> > TLDR: Should OpenBMC remove systemd-networkd-wait-online.service
> > universally?
>
> Probably not, it's required to implement network-online,target, which
> is standard, and may be referred to by arbitrary packages.
>
> There's some good background on the issues you're experiencing here:
>
>  https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
>
> in short: most services should be able to start before network-
> online.target, and be able to adapt to changes in network configuration
> after that point.
>
> For your specific issue there:
>
> > Issues: This service blocks entry to multi-user.target.
> > phosphor-state-manager uses multi-user.target to report the BMC is
> > ready to use.
> > This is reported via IPMI Get Device ID.
>
> That sounds like more of an issue of whether that reported state
> actually represents the expected BMC state...

We have an internal "override" config to start
systemd-networkd-wait-online with --any option:

 # override.conf
 [Service]
 ExecStart=
 ExecStart=/lib/systemd/systemd-networkd-wait-online --any

This is mostly about fixing the QEMU CI.
In the real environment the network *should* be up and online so the
above makes the systemd-networkd-wait-online starts OK in both cases.

Re: Checking for network online

[Johnathan Mantey]

[-- Attachment #1.1.1: Type: text/plain, Size: 2675 bytes --]

Reading the --any switch in the systemd-networkd-wait-online man page 
doesn't look like it would be helpful. That flag permits the service to 
move on when one of the NICs achieves 'online' functionality. In the 
case of a NIC w/o a cable connection 'online' never happens. Thus the 
default 120 second timeout is still going to elapse, BMC ready is going 
to be held off, BIOS is going to delay completion (in our BIOS), and an 
error message is still going to be logged.

It appears, based on comments so far, that my best way forward with the 
current implementation of wait-online is to assign "--timeout 
<number-smaller-than-120> -q" to reduce the amount of time for testing 
the NIC state, and to never log an error because the NIC was unplugged.

Gating on operational state, and issuing --ignore flags didn't work, 
leaving a large blunt instrument for a solution.

On 2/17/22 18:29, Lei Yu wrote:
> On Fri, Feb 18, 2022 at 8:11 AM Jeremy Kerr<jk@codeconstruct.com.au>  wrote:
>> Hi Johnathan,
>>
>>> Issue: systemd-networkd-wait-online.service stalls for 120 seconds
>>> when the managed NICs do not have a network connection.
>>>
>>> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service
>>> universally?
>> Probably not, it's required to implement network-online,target, which
>> is standard, and may be referred to by arbitrary packages.
>>
>> There's some good background on the issues you're experiencing here:
>>
>>   https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
>>
>> in short: most services should be able to start before network-
>> online.target, and be able to adapt to changes in network configuration
>> after that point.
>>
>> For your specific issue there:
>>
>>> Issues: This service blocks entry to multi-user.target.
>>> phosphor-state-manager uses multi-user.target to report the BMC is
>>> ready to use.
>>> This is reported via IPMI Get Device ID.
>> That sounds like more of an issue of whether that reported state
>> actually represents the expected BMC state...
> We have an internal "override" config to start
> systemd-networkd-wait-online with --any option:
>
>   # override.conf
>   [Service]
>   ExecStart=
>   ExecStart=/lib/systemd/systemd-networkd-wait-online --any
>
> This is mostly about fixing the QEMU CI.
> In the real environment the network *should* be up and online so the
> above makes the systemd-networkd-wait-online starts OK in both cases.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com


[-- Attachment #1.1.2: Type: text/html, Size: 4368 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Checking for network online

[Doman, Jonathan]

On Thu, 2022-02-17 at 14:54 -0800, Johnathan Mantey wrote:
> /I/ntel has recently run into an issue regarding a systemd service, and 
> we're interested in soliciting feedback from the community.
> 
> Issue: systemd-networkd-wait-online.service stalls for 120 seconds when 
> the managed NICs do not have a network connection.
> 
> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service 
> universally?
> 
> System Config: All NICs in the system are not connected to an active 
> network.
> 
> Test Process: The system under test (SUT) has AC removed, and some time 
> later AC applied. Wait for BMC/BIOS to boot
> 
> Behavior: U-Boot hands control to the Linux boot process, and the 
> systemd services are started. When systemd-networkd-wait-online.service 
> starts it stalls waiting for the NICs to enter a fully functional state. 
> This never happens during the default 120 second timeout period for this 
> service. When the timeout elapses, an error message is logged to the 
> journal reporting the service exited unsuccessfully.
> 
> Issues: This service blocks entry to multi-user.target.
> phosphor-state-manager uses multi-user.target to report the BMC is ready 
> to use.
> This is reported via IPMI Get Device ID.
> The Intel BIOS is blocked from booting until 
> systemd-networkd-wait-online times out.
> BMC entry to multi-user.target is delayed. Journal entries are created.
> 
> Question for the community: Given the negative side effects caused by 
> running this service does the community want to have this service 
> collectively removed from global build image?

I think the initial discussion in #general got to the root of the
issue: multi-user.target Wants rsyslog.service, which in turn is
ordered After network-online.target. rsyslog seems to be the only thing
tying multi-user to network-online.

Did you try removing the Wants/After=network-online.target from
rsyslog.service to see if the situation improves? If it does, then we
can discuss removing that dependency or making it configurable.

Re: Checking for network online

[Johnathan Mantey]

[-- Attachment #1.1: Type: text/plain, Size: 3137 bytes --]



On 2/18/22 11:04, Doman, Jonathan wrote:
> On Thu, 2022-02-17 at 14:54 -0800, Johnathan Mantey wrote:
>> /I/ntel has recently run into an issue regarding a systemd service, and
>> we're interested in soliciting feedback from the community.
>>
>> Issue: systemd-networkd-wait-online.service stalls for 120 seconds when
>> the managed NICs do not have a network connection.
>>
>> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service
>> universally?
>>
>> System Config: All NICs in the system are not connected to an active
>> network.
>>
>> Test Process: The system under test (SUT) has AC removed, and some time
>> later AC applied. Wait for BMC/BIOS to boot
>>
>> Behavior: U-Boot hands control to the Linux boot process, and the
>> systemd services are started. When systemd-networkd-wait-online.service
>> starts it stalls waiting for the NICs to enter a fully functional state.
>> This never happens during the default 120 second timeout period for this
>> service. When the timeout elapses, an error message is logged to the
>> journal reporting the service exited unsuccessfully.
>>
>> Issues: This service blocks entry to multi-user.target.
>> phosphor-state-manager uses multi-user.target to report the BMC is ready
>> to use.
>> This is reported via IPMI Get Device ID.
>> The Intel BIOS is blocked from booting until
>> systemd-networkd-wait-online times out.
>> BMC entry to multi-user.target is delayed. Journal entries are created.
>>
>> Question for the community: Given the negative side effects caused by
>> running this service does the community want to have this service
>> collectively removed from global build image?
> 
> I think the initial discussion in #general got to the root of the
> issue: multi-user.target Wants rsyslog.service, which in turn is
> ordered After network-online.target. rsyslog seems to be the only thing
> tying multi-user to network-online.

I assume you mean OpenBMC Discord #general channel?

> 
> Did you try removing the Wants/After=network-online.target from
> rsyslog.service to see if the situation improves? If it does, then we
> can discuss removing that dependency or making it configurable.

No, I had not tried that. My take on doing so is that it'll be like 
playing whack a mole. Some other service may decide to rely on 
systemd-networkd-wait-online. The issue is now compounded as a result.

I basically took it on faith that rsyslog needed this service. I did not 
investigate what issues arise in rsyslog when no network is present.

Ultimately there will be times where the BMC will have to operate sans 
network. It's unfortunate that the wait-online service doesn't seem to 
perform the expected operation for the --ignore and the min/max 
operational state functions. This may be a mismatch between my 
expectations and the actual implementation of wait-online.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Checking for network online

[Jiaqing Zhao]

I think a solution is to set RequiredForOnline=no (https://www.freedesktop.org/software/systemd/man/systemd.network.html#RequiredForOnline=) in all network interface config. This option skips the interface when running systemd-networkd-wait-online.service. Canonical netplan (used in ubuntu server) also uses this option to skip the online check for given interface (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L639).

I'll submit a patch to phosphor-networkd later.

On 2022-02-19 00:11, Johnathan Mantey wrote:
> Reading the --any switch in the systemd-networkd-wait-online man page doesn't look like it would be helpful. That flag permits the service to move on when one of the NICs achieves 'online' functionality. In the case of a NIC w/o a cable connection 'online' never happens. Thus the default 120 second timeout is still going to elapse, BMC ready is going to be held off, BIOS is going to delay completion (in our BIOS), and an error message is still going to be logged.
> 
> It appears, based on comments so far, that my best way forward with the current implementation of wait-online is to assign "--timeout <number-smaller-than-120> -q" to reduce the amount of time for testing the NIC state, and to never log an error because the NIC was unplugged.
> 
> Gating on operational state, and issuing --ignore flags didn't work, leaving a large blunt instrument for a solution.
> 
> On 2/17/22 18:29, Lei Yu wrote:
>> On Fri, Feb 18, 2022 at 8:11 AM Jeremy Kerr<jk@codeconstruct.com.au>  wrote:
>>> Hi Johnathan,
>>>
>>>> Issue: systemd-networkd-wait-online.service stalls for 120 seconds
>>>> when the managed NICs do not have a network connection.
>>>>
>>>> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service
>>>> universally?
>>> Probably not, it's required to implement network-online,target, which
>>> is standard, and may be referred to by arbitrary packages.
>>>
>>> There's some good background on the issues you're experiencing here:
>>>
>>>   https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
>>>
>>> in short: most services should be able to start before network-
>>> online.target, and be able to adapt to changes in network configuration
>>> after that point.
>>>
>>> For your specific issue there:
>>>
>>>> Issues: This service blocks entry to multi-user.target.
>>>> phosphor-state-manager uses multi-user.target to report the BMC is
>>>> ready to use.
>>>> This is reported via IPMI Get Device ID.
>>> That sounds like more of an issue of whether that reported state
>>> actually represents the expected BMC state...
>> We have an internal "override" config to start
>> systemd-networkd-wait-online with --any option:
>>
>>   # override.conf
>>   [Service]
>>   ExecStart=
>>   ExecStart=/lib/systemd/systemd-networkd-wait-online --any
>>
>> This is mostly about fixing the QEMU CI.
>> In the real environment the network *should* be up and online so the
>> above makes the systemd-networkd-wait-online starts OK in both cases.
> 

Re: Checking for network online

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2223 bytes --]

On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
> I think a solution is to set RequiredForOnline=no (https://www.freedesktop.org/software/systemd/man/systemd.network.html#RequiredForOnline=) in all network interface config. This option skips the interface when running systemd-networkd-wait-online.service. Canonical netplan (used in ubuntu server) also uses this option to skip the online check for given interface (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L639).
> 
> I'll submit a patch to phosphor-networkd later.

I really don't think this is appropriate for all systems.  Services have
dependencies on network-online.target for a reason.  If the side-effect of
having the BMC network cable unplugged is that the host doesn't boot, that might
be entirely reasonable behavior in some environments.

We use rsyslog as the mechanism to offload our BMC logging data to an
aggregation point.  When you have a very large scale deployment, it is actually
better for the system to not come online than for us to lose out on that data,
since we have spare capacity to take its place.

Note that the Canonical netplan only applies this option if the configuration
indicates that the interface is optional, which is entirely appropriate.  The
way you wrote it could have been interpreted that they set this on *every*
interface, which is what it seems like you're proposing to do to
phosphor-networkd.

If this is desired behavior for someone, can't you supply a wildcard .network
file that adds this option, rather than modifying phosphor-networkd to manually
add it to each network interface that it is managing?

I believe some designs use a USB network device to connect two internal pieces
of the system and those interfaces are not necessarily managed by
phosphor-networkd (interfaces that, for example connect BMC-to-BMC or
BMC-to-Host).  While it is obviously up to the system designer to work through
this bug, by applying this configuration as you proposed you are causing
unusual default behavior in that networkd is going to start waiting for these
internal connections to come online instead of the external interface.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Checking for network online

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2278 bytes --]

On Fri, Feb 18, 2022 at 11:39:03AM -0800, Johnathan Mantey wrote:
> On 2/18/22 11:04, Doman, Jonathan wrote:
> > On Thu, 2022-02-17 at 14:54 -0800, Johnathan Mantey wrote:

> > 
> > Did you try removing the Wants/After=network-online.target from
> > rsyslog.service to see if the situation improves? If it does, then we
> > can discuss removing that dependency or making it configurable.
> 
> No, I had not tried that. My take on doing so is that it'll be like 
> playing whack a mole. Some other service may decide to rely on 
> systemd-networkd-wait-online. The issue is now compounded as a result.

I don't understand why this is a particularly difficult problem.  You asked me
on Discord how I figured this out and I said "grep".  You literally just look
for a service that depends on `network-online.target`.

No service should depend on systemd-networkd-wait-online directly.  If they do,
this is a bug.  They are always depending on network-online.target.

Services depend on network-online.target for valid reasons.  As far as I can
tell, currently, only rsyslog* has this dependency (at least on Witherspoon and
Bletchley).  If a new service adds this dependency, and you have an issue with
it, I think you should take some time to reason about why this dependency was
added rather than simply ignoring it (by a force-disable on
systemd-networkd-wait-online).

"Fixing" the rsyslog dependency in your system is a pretty trivial
`do_install:append` with a `sed` in it to strip out the line.  Catching new
dependencies is a fairly simple `ROOTFS_POSTPROCESS_COMMAND` to check all the
service files in `/lib/systemd/system` for a `network-online.target` dependency.

* As I mentioned in Discord, rsyslog having this dependency by default makes
  good sense from the upstream's perspective.  rsyslog is _typically_
  configured for remote offload of the syslog (hence, r in rsyslog).  It just
  happens that we have this unusual Rube Goldberg transformation using rsyslog
  for formulating the SEL and Redfish log files.  For that transformation the
  dependency isn't necessary, but for people who use rsyslog as it is intended
  they probably want the dependency (as I've mentioned in another email we do).

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Checking for network online

[Jiaqing Zhao]

On 2022-02-23 21:48, Patrick Williams wrote:
> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
>> I think a solution is to set RequiredForOnline=no (https://www.freedesktop.org/software/systemd/man/systemd.network.html#RequiredForOnline=) in all network interface config. This option skips the interface when running systemd-networkd-wait-online.service. Canonical netplan (used in ubuntu server) also uses this option to skip the online check for given interface (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L639).
>>
>> I'll submit a patch to phosphor-networkd later.
> 
> I really don't think this is appropriate for all systems.  Services have
> dependencies on network-online.target for a reason.  If the side-effect of
> having the BMC network cable unplugged is that the host doesn't boot, that might
> be entirely reasonable behavior in some environments.
> 
> We use rsyslog as the mechanism to offload our BMC logging data to an
> aggregation point.  When you have a very large scale deployment, it is actually
> better for the system to not come online than for us to lose out on that data,
> since we have spare capacity to take its place.

My understanding is that in OpenBMC, the propose to use rsyslog is to format the Redfish and IPMI SEL logs from system journal. The "r" of rsyslogd is not used in most cases. I think the "network not available" can be handled same as "server misconfigured" in rsyslogd, as in both cases it fails to connect to the server, and may exit or print some error messages? (not tried yet)

Jonathan mentions that the 120s wait blocks multi-user.target in his initial email. Considering that there is no BMC serial port in most production hardware, when BMC has no network connection, the only way to interact with BMC is to use IPMI in host. However, IPMI services are started in multi-user.target, if BMC infinitely waits network online, there would be no way to debug the issue. 

> Note that the Canonical netplan only applies this option if the configuration
> indicates that the interface is optional, which is entirely appropriate.  The
> way you wrote it could have been interpreted that they set this on *every*
> interface, which is what it seems like you're proposing to do to
> phosphor-networkd
>
> If this is desired behavior for someone, can't you supply a wildcard .network
> file that adds this option, rather than modifying phosphor-networkd to manually
> add it to each network interface that it is managing?

Maybe we can add a similar DBus property like how netplan does? Reading/writing systemd-networkd config files is feasible in phosphor-networkd. Default value can be assigned via build option.
 
> I believe some designs use a USB network device to connect two internal pieces
> of the system and those interfaces are not necessarily managed by
> phosphor-networkd (interfaces that, for example connect BMC-to-BMC or
> BMC-to-Host).  While it is obviously up to the system designer to work through
> this bug, by applying this configuration as you proposed you are causing
> unusual default behavior in that networkd is going to start waiting for these
> internal connections to come online instead of the external interface.

I think this is a extremely rare case, internal interfaces should be configurable. For example, host OS can change the IP of its BMC-Host virtual interface, BMC should also be able to change its, and for BMC-to-BMC interfaces, it is impossible to assign a fixed LAN IP without conflicts in manufacturing. The easiest way to configure it is to utilize the phosphor-networkd.

Even it is not managed by phosphor-networkd, keeping default RequiredForOnline=yes will cause the 120s wait on BMC boot. Developers can simply search it and find out the solution. I remember it will show a timer with message on BMC serial console, that's how I found I should set the "optional" on my ubuntu server.

Re: Checking for network online

[Bills, Jason M]

On 2/23/2022 10:44 AM, Jiaqing Zhao wrote:
> On 2022-02-23 21:48, Patrick Williams wrote:
>> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
<snip>...
> 
> My understanding is that in OpenBMC, the propose to use rsyslog is to format the Redfish and IPMI SEL logs from system journal. The "r" of rsyslogd is not used in most cases.
> 

Just a nit-picky side-note: The "r" in rsyslogd is for "rocket-fast".  I 
don't believe rsyslogd is inherently designed for remote logging.
https://www.rsyslog.com/

Re: Checking for network online

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 6143 bytes --]

On Thu, Feb 24, 2022 at 01:44:18AM +0800, Jiaqing Zhao wrote:
> On 2022-02-23 21:48, Patrick Williams wrote:
> > On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
> >> I think a solution is to set RequiredForOnline=no (https://www.freedesktop.org/software/systemd/man/systemd.network.html#RequiredForOnline=) in all network interface config. This option skips the interface when running systemd-networkd-wait-online.service. Canonical netplan (used in ubuntu server) also uses this option to skip the online check for given interface (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L639).
> >>
> >> I'll submit a patch to phosphor-networkd later.
> > 
> > I really don't think this is appropriate for all systems.  Services have
> > dependencies on network-online.target for a reason.  If the side-effect of
> > having the BMC network cable unplugged is that the host doesn't boot, that might
> > be entirely reasonable behavior in some environments.
> > 
> > We use rsyslog as the mechanism to offload our BMC logging data to an
> > aggregation point.  When you have a very large scale deployment, it is actually
> > better for the system to not come online than for us to lose out on that data,
> > since we have spare capacity to take its place.
> 
> My understanding is that in OpenBMC, the propose to use rsyslog is to format the Redfish and IPMI SEL logs from system journal. The "r" of rsyslogd is not used in most cases. 

I might have left some ambiguity in 'we' in this context.  I meant 'the
deployments I am working on'.  I believe at least one other company leverages
this as well.

> I think the "network not available" can be handled same as "server misconfigured" in rsyslogd, as in both cases it fails to connect to the server, and may exit or print some error messages? (not tried yet)

That is probably true, but it means that I can't offload any data about the
system in the meantime.  Like I said, I'd rather leave the system out of my
deployment if it is degraded.

> 
> Jonathan mentions that the 120s wait blocks multi-user.target in his initial email. Considering that there is no BMC serial port in most production hardware, when BMC has no network connection, the only way to interact with BMC is to use IPMI in host.

Your assertion "no BMC serial port in most production hardware" might be true
globally speaking.  It isn't necessarily true for any particular deployment.

With the 120s wait time, is rsyslog actually running after that? Or is it
failed?  I guess since it has a Wants and not a Requires on network-online,
it'll still start up after the 120s timeout of systemd-networkd-wait-online.

My understanding of systemd-networkd's defaults here is that it waits for DHCP
in order for network-online.target to pass.  You can have the IPv6-LL address
configured still, which can allow remote access, even if the IPv6-DHCP address
is not assigned.

> However, IPMI services are started in multi-user.target, if BMC infinitely waits network online, there would be no way to debug the issue. 

Sure, but the BMC doesn't wait forever, does it?  It just waits 120s.

I'm not suggesting this isn't the right solution for your systems, or even that
it might not be the right solution for most systems, but I don't think it is the
right solution for _all_ systems so we need to ensure it can be opt-out.

> 
> > Note that the Canonical netplan only applies this option if the configuration
> > indicates that the interface is optional, which is entirely appropriate.  The
> > way you wrote it could have been interpreted that they set this on *every*
> > interface, which is what it seems like you're proposing to do to
> > phosphor-networkd
> >
> > If this is desired behavior for someone, can't you supply a wildcard .network
> > file that adds this option, rather than modifying phosphor-networkd to manually
> > add it to each network interface that it is managing?
> 
> Maybe we can add a similar DBus property like how netplan does? Reading/writing systemd-networkd config files is feasible in phosphor-networkd. Default value can be assigned via build option.

I'm not sure if it belongs as a DBus property or not.  I'd have to see what
you're proposing and think about it.  I think this is a system design constraint
and not really configurable by users (hence why exposing a DBus property might
not make sense) but maybe I'm wrong on this.

> > I believe some designs use a USB network device to connect two internal pieces
> > of the system and those interfaces are not necessarily managed by
> > phosphor-networkd (interfaces that, for example connect BMC-to-BMC or
> > BMC-to-Host).  While it is obviously up to the system designer to work through
> > this bug, by applying this configuration as you proposed you are causing
> > unusual default behavior in that networkd is going to start waiting for these
> > internal connections to come online instead of the external interface.
> 
> I think this is a extremely rare case, internal interfaces should be configurable. For example, host OS can change the IP of its BMC-Host virtual interface, BMC should also be able to change its, and for BMC-to-BMC interfaces, it is impossible to assign a fixed LAN IP without conflicts in manufacturing. 

I don't follow your concern here.  We can (and do) easily assign a static IP
address for the BMC-to-BMC interfaces based on position information fed into the
BMC via GPIO signals.

> The easiest way to configure it is to utilize the phosphor-networkd.
> 
> Even it is not managed by phosphor-networkd, keeping default RequiredForOnline=yes will cause the 120s wait on BMC boot. Developers can simply search it and find out the solution. I remember it will show a timer with message on BMC serial console, that's how I found I should set the "optional" on my ubuntu server.

Agreed.  Someone _can_ find it and debug it.  It is to me not an obvious or easy
thing to work out though because automated "network down" test cases are not
often done in my experience.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Checking for network online

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Wed, Feb 23, 2022 at 11:36:50AM -0700, Bills, Jason M wrote:
> > My understanding is that in OpenBMC, the propose to use rsyslog is to format the Redfish and IPMI SEL logs from system journal. The "r" of rsyslogd is not used in most cases.
> > 
> 
> Just a nit-picky side-note: The "r" in rsyslogd is for "rocket-fast".  I 
> don't believe rsyslogd is inherently designed for remote logging.
> https://www.rsyslog.com/

#TIL

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Checking for network online

[Johnathan Mantey]

[-- Attachment #1.1: Type: text/plain, Size: 6238 bytes --]



On 2/23/22 09:44, Jiaqing Zhao wrote:
> On 2022-02-23 21:48, Patrick Williams wrote:
>> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
>>> I think a solution is to set RequiredForOnline=no (https://www.freedesktop.org/software/systemd/man/systemd.network.html#RequiredForOnline=) in all network interface config. This option skips the interface when running systemd-networkd-wait-online.service. Canonical netplan (used in ubuntu server) also uses this option to skip the online check for given interface (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L639).
>>>
>>> I'll submit a patch to phosphor-networkd later.
>>
>> I really don't think this is appropriate for all systems.  Services have
>> dependencies on network-online.target for a reason.  If the side-effect of
>> having the BMC network cable unplugged is that the host doesn't boot, that might
>> be entirely reasonable behavior in some environments.
>>
>> We use rsyslog as the mechanism to offload our BMC logging data to an
>> aggregation point.  When you have a very large scale deployment, it is actually
>> better for the system to not come online than for us to lose out on that data,
>> since we have spare capacity to take its place.
> 
> My understanding is that in OpenBMC, the propose to use rsyslog is to format the Redfish and IPMI SEL logs from system journal. The "r" of rsyslogd is not used in most cases. I think the "network not available" can be handled same as "server misconfigured" in rsyslogd, as in both cases it fails to connect to the server, and may exit or print some error messages? (not tried yet)
> 
> Jonathan mentions that the 120s wait blocks multi-user.target in his initial email. Considering that there is no BMC serial port in most production hardware, when BMC has no network connection, the only way to interact with BMC is to use IPMI in host. However, IPMI services are started in multi-user.target, if BMC infinitely waits network online, there would be no way to debug the issue.
> 
>> Note that the Canonical netplan only applies this option if the configuration
>> indicates that the interface is optional, which is entirely appropriate.  The
>> way you wrote it could have been interpreted that they set this on *every*
>> interface, which is what it seems like you're proposing to do to
>> phosphor-networkd
>>
>> If this is desired behavior for someone, can't you supply a wildcard .network
>> file that adds this option, rather than modifying phosphor-networkd to manually
>> add it to each network interface that it is managing?
> 
> Maybe we can add a similar DBus property like how netplan does? Reading/writing systemd-networkd config files is feasible in phosphor-networkd. Default value can be assigned via build option.
>   
>> I believe some designs use a USB network device to connect two internal pieces
>> of the system and those interfaces are not necessarily managed by
>> phosphor-networkd (interfaces that, for example connect BMC-to-BMC or
>> BMC-to-Host).  While it is obviously up to the system designer to work through
>> this bug, by applying this configuration as you proposed you are causing
>> unusual default behavior in that networkd is going to start waiting for these
>> internal connections to come online instead of the external interface.
> 
> I think this is a extremely rare case, internal interfaces should be configurable. For example, host OS can change the IP of its BMC-Host virtual interface, BMC should also be able to change its, and for BMC-to-BMC interfaces, it is impossible to assign a fixed LAN IP without conflicts in manufacturing. The easiest way to configure it is to utilize the phosphor-networkd.
> 
> Even it is not managed by phosphor-networkd, keeping default RequiredForOnline=yes will cause the 120s wait on BMC boot. Developers can simply search it and find out the solution. I remember it will show a timer with message on BMC serial console, that's how I found I should set the "optional" on my ubuntu server.

FWIW, my experimentation with systemd-networkd-wait-online was not 
successful in doing much to change the 120 second timeout.

Setting the RequiredForOnline entry to false in systemd.network did not 
prevent the 120 second timeout from elapsing.

Setting any of the following switches in the service file failed to 
eliminate the timeout:
--ignore=eth0
--interface=eth0:no-carrier            # overrides RequiredForOnline
--interface=eth0:no-carrier:no-carrier # <- probably a bad setting in
                                        # hindsight

It appears systemd-networkd-wait-online expects some state greater than 
no-carrier to consider the link online, thus allowing it to exit with a 
SUCCESS error code. This even when explicitly instructed no-carrier is 
defined as "online".

The only switch that seemed to perform as expected in this instance was 
--timeout. Assigning a value less than 120 to the --timeout control did 
reduce the wait period. It does assign a SUCCESS error code upon timing 
out, which is expected behavior.

systemd-networkd-wait-online appears to have logic preventing no-carrier 
state from being assigned as the "network online" value.

rsyslogd has both a network and network-online target. If the 
network-online target is removed then systemd-networkd-wait-online 
doesn't run, and any configuation of that service appears to be 
pointless. The conclusion I have from that is that network-online.target 
is a valid configuration option for a service to assign.

There may be openbmc powered servers that do use the distributed logging 
provided by rsyslogd. If there are then globally removing network-online 
from the rsyslog service file is undesirable. I consider the same to be 
true of assigning a default RequiredForOnline=false.

Based on the above, it's my opinion this is a vendor based decision for 
how to configure rsyslog/systemd-networkd-wait-online.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com <mailto:johnathanx.mantey@intel.com>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Checking for network online

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 975 bytes --]

On Wed, Feb 23, 2022 at 12:04:12PM -0800, Johnathan Mantey wrote:
> On 2/23/22 09:44, Jiaqing Zhao wrote:
> > On 2022-02-23 21:48, Patrick Williams wrote:
> >> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:

> There may be openbmc powered servers that do use the distributed logging 
> provided by rsyslogd. If there are then globally removing network-online 
> from the rsyslog service file is undesirable. I consider the same to be 
> true of assigning a default RequiredForOnline=false.
> 
> Based on the above, it's my opinion this is a vendor based decision for 
> how to configure rsyslog/systemd-networkd-wait-online.

I agree we shouldn't enable this globally, but that doesn't mean we can't add
a simple PKGCONFIG that allows it to be enabled/disabled as needed.  That way
we only have the single `PKGCONFIG:append` line in vendor layers and vendors
that have a problem with it can leave it same as upstream.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: Checking for network online

[Milton Miller II]

On Feb 23, 2022,  Johnathan Mantey wrote:
>On 2/23/22 09:44, Jiaqing Zhao wrote:
>> On 2022-02-23 21:48, Patrick Williams wrote:
>>> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
>>>> I think a solution is to set RequiredForOnline=no
>>>> (https://www.freedesktop.org/software/systemd/man/systemd.network.htm
>>>> l#RequiredForOnline=) in all network interface config. This option
>>>> skips the interface when running
>>>> systemd-networkd-wait-online.service. Canonical netplan (used in
>>>> ubuntu server) also uses this option to skip the online check for
>>>> given interface
>>>> (https://github.com/canonical/netplan/blob/main/src/networkd.c#L636-L
>>>> 639).
>>>>
>>>> I'll submit a patch to phosphor-networkd later.
>>>
>>> I really don't think this is appropriate for all systems.
>>> Services have
>>> dependencies on network-online.target for a reason. If the
>>> side-effect of
>>> having the BMC network cable unplugged is that the host doesn't
>>> boot, that might
>>> be entirely reasonable behavior in some environments.
>>>
>>> We use rsyslog as the mechanism to offload our BMC logging data to
>>> an
>>> aggregation point. When you have a very large scale deployment,
>>> it is actually
>>> better for the system to not come online than for us to lose out
>>> on that data,
>>> since we have spare capacity to take its place.
>>
>> My understanding is that in OpenBMC, the propose to use rsyslog is
>> to format the Redfish and IPMI SEL logs from system journal. The "r"
>> of rsyslogd is not used in most cases. I think the "network not
>> available" can be handled same as "server misconfigured" in rsyslogd,
>> as in both cases it fails to connect to the server, and may exit or
>> print some error messages? (not tried yet)
>>
>> Jonathan mentions that the 120s wait blocks multi-user.target in
>> his initial email. Considering that there is no BMC serial port in
>> most production hardware, when BMC has no network connection, the
>> only way to interact with BMC is to use IPMI in host. However, IPMI
>> services are started in multi-user.target, if BMC infinitely waits
>> network online, there would be no way to debug the issue.
>>
>>> Note that the Canonical netplan only applies this option if the
>>> configuration
>>> indicates that the interface is optional, which is entirely
>>> appropriate. The
>>> way you wrote it could have been interpreted that they set this on
>>> *every*
>>> interface, which is what it seems like you're proposing to do to
>>> phosphor-networkd
>>>
>>> If this is desired behavior for someone, can't you supply a
>>> wildcard .network
>>> file that adds this option, rather than modifying
>>> phosphor-networkd to manually
>>> add it to each network interface that it is managing?
>>
>> Maybe we can add a similar DBus property like how netplan does?
>> Reading/writing systemd-networkd config files is feasible in
>> phosphor-networkd. Default value can be assigned via build option.
>>
>>> I believe some designs use a USB network device to connect two
>>> internal pieces
>>> of the system and those interfaces are not necessarily managed by
>>> phosphor-networkd (interfaces that, for example connect BMC-to-BMC
>>> or
>>> BMC-to-Host). While it is obviously up to the system designer to
>>> work through
>>> this bug, by applying this configuration as you proposed you are
>>> causing
>>> unusual default behavior in that networkd is going to start
>>> waiting for these
>>> internal connections to come online instead of the external
>>> interface.
>>
>> I think this is a extremely rare case, internal interfaces should
>> be configurable. For example, host OS can change the IP of its
>> BMC-Host virtual interface, BMC should also be able to change its,
>> and for BMC-to-BMC interfaces, it is impossible to assign a fixed LAN
>> IP without conflicts in manufacturing. The easiest way to configure
>> it is to utilize the phosphor-networkd.
>>
>> Even it is not managed by phosphor-networkd, keeping default
>> RequiredForOnline=yes will cause the 120s wait on BMC boot.
>> Developers can simply search it and find out the solution. I remember
>> it will show a timer with message on BMC serial console, that's how I
>>found I should set the "optional" on my ubuntu server.
>
>FWIW, my experimentation with systemd-networkd-wait-online was not
>successful in doing much to change the 120 second timeout.
>
>Setting the RequiredForOnline entry to false in systemd.network did
>not
>prevent the 120 second timeout from elapsing.
>
>Setting any of the following switches in the service file failed to
>eliminate the timeout:
>--ignore=eth0
>--interface=eth0:no-carrier # overrides RequiredForOnline
>--interface=eth0:no-carrier:no-carrier # <- probably a bad setting in
> # hindsight
>
>It appears systemd-networkd-wait-online expects some state greater
>than
>no-carrier to consider the link online, thus allowing it to exit with
>a
>SUCCESS error code. This even when explicitly instructed no-carrier
>is
>defined as "online".
>
>The only switch that seemed to perform as expected in this instance
>was
>--timeout. Assigning a value less than 120 to the --timeout control
>did
>reduce the wait period. It does assign a SUCCESS error code upon
>timing
>out, which is expected behavior.
>
>systemd-networkd-wait-online appears to have logic preventing
>no-carrier
>state from being assigned as the "network online" value.
>
>rsyslogd has both a network and network-online target. If the
>network-online target is removed then systemd-networkd-wait-online
>doesn't run, and any configuation of that service appears to be
>pointless. The conclusion I have from that is that
>network-online.target
>is a valid configuration option for a service to assign.
>
>There may be openbmc powered servers that do use the distributed
>logging
>provided by rsyslogd. If there are then globally removing
>network-online
>from the rsyslog service file is undesirable. I consider the same to
>be
>true of assigning a default RequiredForOnline=false.
>
>Based on the above, it's my opinion this is a vendor based decision
>for
>how to configure rsyslog/systemd-networkd-wait-online.
>


I just wanted to point out that for those using the kernel NCSI stack, 
the networks are always showing on line and link up because of how 
the stack was created.  My reading is it would take a new slave 
interface to overcome this limitation.

Milton

Re: Checking for network online

[Jiaqing Zhao]

On 2022-02-25 04:09, Patrick Williams wrote:
> On Wed, Feb 23, 2022 at 12:04:12PM -0800, Johnathan Mantey wrote:
>> On 2/23/22 09:44, Jiaqing Zhao wrote:
>>> On 2022-02-23 21:48, Patrick Williams wrote:
>>>> On Wed, Feb 23, 2022 at 10:09:19AM +0800, Jiaqing Zhao wrote:
> 
>> There may be openbmc powered servers that do use the distributed logging 
>> provided by rsyslogd. If there are then globally removing network-online 
>> from the rsyslog service file is undesirable. I consider the same to be 
>> true of assigning a default RequiredForOnline=false.
>>
>> Based on the above, it's my opinion this is a vendor based decision for 
>> how to configure rsyslog/systemd-networkd-wait-online.
> 
> I agree we shouldn't enable this globally, but that doesn't mean we can't add
> a simple PKGCONFIG that allows it to be enabled/disabled as needed.  That way
> we only have the single `PKGCONFIG:append` line in vendor layers and vendors
> that have a problem with it can leave it same as upstream.

I am also in favor of this solution, let the vendor decide whether rsyslog depends on network-online.target or not with the PKGCONFIG option.

Re: Checking for network online

[Ratan Gupta]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Hi Johnathan,

Can you ask this in the systemd community?Any workaround if they can offer.

Ratan



On Fri, Feb 18, 2022 at 4:26 AM Johnathan Mantey <
johnathanx.mantey@intel.com> wrote:

> /I/ntel has recently run into an issue regarding a systemd service, and
> we're interested in soliciting feedback from the community.
>
> Issue: systemd-networkd-wait-online.service stalls for 120 seconds when
> the managed NICs do not have a network connection.
>
> TLDR: Should OpenBMC remove systemd-networkd-wait-online.service
> universally?
>
> System Config: All NICs in the system are not connected to an active
> network.
>
> Test Process: The system under test (SUT) has AC removed, and some time
> later AC applied. Wait for BMC/BIOS to boot
>
> Behavior: U-Boot hands control to the Linux boot process, and the
> systemd services are started. When systemd-networkd-wait-online.service
> starts it stalls waiting for the NICs to enter a fully functional state.
> This never happens during the default 120 second timeout period for this
> service. When the timeout elapses, an error message is logged to the
> journal reporting the service exited unsuccessfully.
>
> Issues: This service blocks entry to multi-user.target.
> phosphor-state-manager uses multi-user.target to report the BMC is ready
> to use.
> This is reported via IPMI Get Device ID.
> The Intel BIOS is blocked from booting until
> systemd-networkd-wait-online times out.
> BMC entry to multi-user.target is delayed. Journal entries are created.
>
> Question for the community: Given the negative side effects caused by
> running this service does the community want to have this service
> collectively removed from global build image?
>
> --
> Johnathan Mantey
> Senior Software Engineer
> *azad te**chnology partners*
> Contributing to Technology Innovation since 1992
> Phone: (503) 712-6764
> Email: johnathanx.mantey@intel.com
>
>

[-- Attachment #2: Type: text/html, Size: 2473 bytes --]