[Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

[Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

[Bell, Oren]

[-- Attachment #1.1: Type: text/plain, Size: 616 bytes --]

I've encountered an issue where installing Xen >4.10 on a Dell Optiplex will break the onboard NIC. This issue persists if the computer is booted without Xen, after OS reinstall, and even if removing the SSD and HDD completely to boot from a LiveUSB. The only way to fix the issue is to install Windows 10 on the machine. This appears to "fix" the firmware of the NIC. After reinstalling Ubuntu, the NIC continues to work (until Xen is installed again).

This bug was confirmed with both Xen 4.10 and 4.12 installed on Ubuntu 18.04.

If this is a known issue, is there some "in-work patch" I can be pointed to?

[-- Attachment #1.2: Type: text/html, Size: 1761 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

[Jan Beulich]

On 27.10.2019 17:09,  Bell, Oren  wrote:
> I've encountered an issue where installing Xen >4.10 on a Dell Optiplex will break the onboard NIC. This issue persists if the computer is booted without Xen, after OS reinstall, and even if removing the SSD and HDD completely to boot from a LiveUSB. The only way to fix the issue is to install Windows 10 on the machine. This appears to "fix" the firmware of the NIC. After reinstalling Ubuntu, the NIC continues to work (until Xen is installed again).
> 
> This bug was confirmed with both Xen 4.10 and 4.12 installed on Ubuntu 18.04.
> 
> If this is a known issue, is there some "in-work patch" I can be pointed to?

This is a rather strange problem you're facing - Xen itself doesn't
do anything to NICs. Therefore I'm afraid some more experimenting
may be needed to somehow narrow where things go wrong. In particular
I'd be curious to understand whether it's indeed Xen that breaks
things, or whether e.g. other software misbehaves if run on top of
Xen. As a first step, could you boot
- Xen without a Dom0 kernel,
- Xen with a Dom0 kernel, but without a driver for the NIC,
- Xen with a Dom0 kernel and with a driver for the NIC, but without
  actually configuring/using the NIC?
Could you further check whether Xen using the presumably present
IOMMU matters? (Providing maximum verbosity hypervisor and kernel
logs would of course also help, in particular e.g. to know whether
there is an IOMMU in the system, and also to see whether any
anomalies get logged.)

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

[Bell, Oren]

[-- Attachment #1.1: Type: text/plain, Size: 2429 bytes --]

Running Xen Dom0-less leaves the NIC intact, so you're correct in assessing that Xen by itself is not the cause.
As for running without the driver, I'm not sure that's possible (at least for my competency). It uses the Intel Base Gigabit driver that's built into the kernel.
And running the machine without using the NIC will still break it.

As for the IOMMU suggestion: we still got basic pinging to work, assuming an IP address was statically allocated, so I don't think IOMMU is a valid route for investigation, as any aberrations there should leave the NIC totally non-functional.
________________________________
From: Jan Beulich <jbeulich@suse.com>
Sent: Monday, October 28, 2019 4:03 AM
To: Bell, Oren <oren.bell@wustl.edu>
Cc: xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>
Subject: Re: [Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

On 27.10.2019 17:09,  Bell, Oren  wrote:
> I've encountered an issue where installing Xen >4.10 on a Dell Optiplex will break the onboard NIC. This issue persists if the computer is booted without Xen, after OS reinstall, and even if removing the SSD and HDD completely to boot from a LiveUSB. The only way to fix the issue is to install Windows 10 on the machine. This appears to "fix" the firmware of the NIC. After reinstalling Ubuntu, the NIC continues to work (until Xen is installed again).
>
> This bug was confirmed with both Xen 4.10 and 4.12 installed on Ubuntu 18.04.
>
> If this is a known issue, is there some "in-work patch" I can be pointed to?

This is a rather strange problem you're facing - Xen itself doesn't
do anything to NICs. Therefore I'm afraid some more experimenting
may be needed to somehow narrow where things go wrong. In particular
I'd be curious to understand whether it's indeed Xen that breaks
things, or whether e.g. other software misbehaves if run on top of
Xen. As a first step, could you boot
- Xen without a Dom0 kernel,
- Xen with a Dom0 kernel, but without a driver for the NIC,
- Xen with a Dom0 kernel and with a driver for the NIC, but without
  actually configuring/using the NIC?
Could you further check whether Xen using the presumably present
IOMMU matters? (Providing maximum verbosity hypervisor and kernel
logs would of course also help, in particular e.g. to know whether
there is an IOMMU in the system, and also to see whether any
anomalies get logged.)

Jan

[-- Attachment #1.2: Type: text/html, Size: 3758 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen >4.10 bricks onboard NIC of Dell Optiplex 7060

[Jan Beulich]

On 30.10.2019 19:24, Bell, Oren wrote:
> Running Xen Dom0-less leaves the NIC intact, so you're correct in assessing that Xen by itself is not the cause.
> As for running without the driver, I'm not sure that's possible (at least for my competency). It uses the Intel Base Gigabit driver that's built into the kernel.

Well, if there's a kernel side aspect to it (which now seems
pretty likely), then playing with the kernel will be unavoidable.
A first step would be to build a kernel with the driver not built
in, but as a module, such that its loading could be prevented (an
alternative would be a kernel without the driver). Then again I
seem to vaguely recall there being means to suppress the binding
of a driver to particular devices via the kernel command line. I'm
not going to exclude though that this might be distro specific, or
be restricted to even more special cases like distro installation.

> And running the machine without using the NIC will still break it.
> 
> As for the IOMMU suggestion: we still got basic pinging to work, assuming an IP address was statically allocated, so I don't think IOMMU is a valid route for investigation, as any aberrations there should leave the NIC totally non-functional.

I'm confused - basic pinging seems to contradict your unconditional
"bricks" statement. This would rather suggest something during
normal operation breaks things, rather than (as assumed so far) a
specific initialization step. Yet even above you state "And running
the machine without using the NIC will still break it" - this again
suggests some setup step to be at fault. Or perhaps we don't share
an understanding of what "not using the NIC" means: I imply here
that the NIC doesn't get initialized, just its driver loaded (i.e.
no IP address assigned, and hence in particular also no working
pings).

And btw - please don't top-post.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel