X58 Virtualization w/ Linux

X58 Virtualization w/ Linux

[Steve Novakov]

Hello,

I was instructed to send an email to the KVM-devel group about this. I 
made a post on reddit about some issues I've had virtualizating an X58 
environment. The details are here:

https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/

I'm asking around to see if anyone has a straightforward solution, or 
any advice on how to approach this problem. Also (please read the reddit 
post first):
     - can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on 
vfio_iommu_type1.allow_unsafe_interrupts=1" safely?
     - is there a way to fix the DMAR table for my BIOS (see post)? How 
might I dump it and fix it?

I'm running barebones Arch and KVM/QEMU. My progress is outlined in that 
reddit post and discussion. At the moment, I'm after "safe" passthrough 
of a PCIe NIC and video card to separate VMs. "Safe" may mean totally 
secure (secure passthrough), or just stable enough that I don't have to 
debug problems every kernel update *whichever is possible*.  Thank you 
in advance for any help.

Sincerely,
-- 
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/12 5:34, Steve Novakov wrote:
> Hello,
>
> I was instructed to send an email to the KVM-devel group about this. I
> made a post on reddit about some issues I've had virtualizating an X58
> environment. The details are here:
>
> https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/
>
>
> I'm asking around to see if anyone has a straightforward solution, or
> any advice on how to approach this problem. Also (please read the reddit
> post first):
>     - can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1" safely?

allow_unsafe_interupts actually means the interrupt remapping on Intel 
IOMMU which is a security feature. Without it, a malicious VM can attack 
the host, see below document for more details:
http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

>     - is there a way to fix the DMAR table for my BIOS (see post)? How
> might I dump it and fix it?

You can dump it from /sys/firmware/acpi/tables/DMAR. I remember linux 
kernel allow you to use the customized ACPI table but i am not sure 
whether DMAR is suitable for it. Also, you can try to upgrade your BIOS 
to fix it.

>
> I'm running barebones Arch and KVM/QEMU. My progress is outlined in that
> reddit post and discussion. At the moment, I'm after "safe" passthrough
> of a PCIe NIC and video card to separate VMs. "Safe" may mean totally
> secure (secure passthrough), or just stable enough that I don't have to
> debug problems every kernel update *whichever is possible*.  Thank you
> in advance for any help.
>
> Sincerely,


-- 
best regards
yang

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Hello Yang,

> allow_unsafe_interupts actually means the interrupt remapping on Intel 
> IOMMU which is a security feature. Without it, a malicious VM can 
> attack the host, see below document for more details:
> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

Should I take that to mean that "allow_unsafe_interrupts" is actually a 
security feature???  Is this the discussed "interrupt remapping" in the 
whitepaper? I am interpreting that paper as saying that this interrupt 
remapping does *not* use the supplied DMAR table. Is that correct?

> Also, you can try to upgrade your BIOS to fix it. 

I'll take a look but I think I have the latest (which means, from ~2011 
probably) BIOS version.

Could I also ask you outright what entire set of boot options you would 
pass when booting into a kvm system with IOMMU enabled? I would love 
some "default" set of boot options to compare to, as opposed to random 
ones from assorted forums.

Thank you for the prompt reply!

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/11/2016 9:46 PM, Yang Zhang wrote:

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Hello Yang,

To add to this, a list of mandatory packages would be helpful as well 
(like if I need some additional virtio- packages). Thank you,

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor

On 6/11/2016 9:55 PM, Steve Novakov wrote:
> Hello Yang,
>
>> allow_unsafe_interupts actually means the interrupt remapping on 
>> Intel IOMMU which is a security feature. Without it, a malicious VM 
>> can attack the host, see below document for more details:
>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf 
>>
>
> Should I take that to mean that "allow_unsafe_interrupts" is actually 
> a security feature???  Is this the discussed "interrupt remapping" in 
> the whitepaper? I am interpreting that paper as saying that this 
> interrupt remapping does *not* use the supplied DMAR table. Is that 
> correct?
>
>> Also, you can try to upgrade your BIOS to fix it. 
>
> I'll take a look but I think I have the latest (which means, from 
> ~2011 probably) BIOS version.
>
> Could I also ask you outright what entire set of boot options you 
> would pass when booting into a kvm system with IOMMU enabled? I would 
> love some "default" set of boot options to compare to, as opposed to 
> random ones from assorted forums.
>
> Thank you for the prompt reply!
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Sorry, I was thinking of kernel modules and said packages (virtio-X are 
kernel modules) . Thanks,

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/11/2016 10:54 PM, Steve Novakov wrote:
> Hello Yang,
>
> To add to this, a list of mandatory packages would be helpful as well 
> (like if I need some additional virtio- packages). Thank you,
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
>
> On 6/11/2016 9:55 PM, Steve Novakov wrote:
>> Hello Yang,
>>
>>> allow_unsafe_interupts actually means the interrupt remapping on 
>>> Intel IOMMU which is a security feature. Without it, a malicious VM 
>>> can attack the host, see below document for more details:
>>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf 
>>>
>>
>> Should I take that to mean that "allow_unsafe_interrupts" is actually 
>> a security feature???  Is this the discussed "interrupt remapping" in 
>> the whitepaper? I am interpreting that paper as saying that this 
>> interrupt remapping does *not* use the supplied DMAR table. Is that 
>> correct?
>>
>>> Also, you can try to upgrade your BIOS to fix it. 
>>
>> I'll take a look but I think I have the latest (which means, from 
>> ~2011 probably) BIOS version.
>>
>> Could I also ask you outright what entire set of boot options you 
>> would pass when booting into a kvm system with IOMMU enabled? I would 
>> love some "default" set of boot options to compare to, as opposed to 
>> random ones from assorted forums.
>>
>> Thank you for the prompt reply!
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>>
>

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/12 9:55, Steve Novakov wrote:
> Hello Yang,
>
>> allow_unsafe_interupts actually means the interrupt remapping on Intel
>> IOMMU which is a security feature. Without it, a malicious VM can
>> attack the host, see below document for more details:
>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
>>
>
> Should I take that to mean that "allow_unsafe_interrupts" is actually a
> security feature???  Is this the discussed "interrupt remapping" in the

Interrupt remapping not only a security feature, it also supports more 
than 255 CPUs associate with x2apic. allow_unsafe_interrupts allows you 
to enable IOMMU on the platform even without interrupt remapping because 
first platform supporting IOMMU doesn't have interrupt remapping.

> whitepaper? I am interpreting that paper as saying that this interrupt
> remapping does *not* use the supplied DMAR table. Is that correct?

All the necessary information for IOMMU is located in ACPI tables not 
only DMAR table. OS need to parse it before enabling the IOMMU.

>
>> Also, you can try to upgrade your BIOS to fix it.
>
> I'll take a look but I think I have the latest (which means, from ~2011
> probably) BIOS version.
>
> Could I also ask you outright what entire set of boot options you would
> pass when booting into a kvm system with IOMMU enabled? I would love
> some "default" set of boot options to compare to, as opposed to random
> ones from assorted forums.

Usually, i am using intel_iommu=on and everything works well. But in 
your case, i guess you may also need intremap=off.

>
> Thank you for the prompt reply!
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>


-- 
best regards
yang

Re: X58 Virtualization w/ Linux

[Steve Novakov]

 > Usually, i am using intel_iommu=on and everything works well. But in 
your case, i guess you may also need intremap=off.

Passing EITHER   "intel_iommu=on intremap=off"  OR " "intel_iommu=on 
vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a 
freeze at boot. This happens at the loading initramfs step ("initrd 
/boot/initramfs-linux.img").

Any further suggestions? Thanks,

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Interestingly, when I disable VT-D in the motherboard BIOS. I can boot 
with "intel_iommu=on pci-stub.ids=DEVICEID"

and the DEVICE is successfully loaded by the pci-stub driver, according 
to $lspci -nnk   readout

additionally, I see:

$dmesg | grep -e IOMMU

     [0.000000] DMAR: IOMMU enabled

even though I actually DISABLED VT-D in the motherboard BIOS. Going back 
now and enabling it in the BIOS causes freezing during boot again.

I'm also having trouble assigning this device to the vfio-pci driver, 
but that might be a separate issue.

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/12/2016 11:32 PM, Steve Novakov wrote:
> > Usually, i am using intel_iommu=on and everything works well. But in 
> your case, i guess you may also need intremap=off.
>
> Passing EITHER   "intel_iommu=on intremap=off"  OR " "intel_iommu=on 
> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a 
> freeze at boot. This happens at the loading initramfs step ("initrd 
> /boot/initramfs-linux.img").
>
> Any further suggestions? Thanks,
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
>
>
>

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Some more info here from someone having exactly the same problem I am 
and at exactly the same stage

  http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/13/2016 4:11 PM, Steve Novakov wrote:
> Interestingly, when I disable VT-D in the motherboard BIOS. I can boot 
> with "intel_iommu=on pci-stub.ids=DEVICEID"
>
> and the DEVICE is successfully loaded by the pci-stub driver, 
> according to $lspci -nnk   readout
>
> additionally, I see:
>
> $dmesg | grep -e IOMMU
>
>     [0.000000] DMAR: IOMMU enabled
>
> even though I actually DISABLED VT-D in the motherboard BIOS. Going 
> back now and enabling it in the BIOS causes freezing during boot again.
>
> I'm also having trouble assigning this device to the vfio-pci driver, 
> but that might be a separate issue.
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/12/2016 11:32 PM, Steve Novakov wrote:
>> > Usually, i am using intel_iommu=on and everything works well. But 
>> in your case, i guess you may also need intremap=off.
>>
>> Passing EITHER   "intel_iommu=on intremap=off"  OR " "intel_iommu=on 
>> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a 
>> freeze at boot. This happens at the loading initramfs step ("initrd 
>> /boot/initramfs-linux.img").
>>
>> Any further suggestions? Thanks,
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>>
>>
>>
>

Re: X58 Virtualization w/ Linux

[Steve Novakov]

So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:

[0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on 
a chipset that contains an erratum making that feature unstable. To 
maintain system stability interrupt remapping is being disabled.please 
contact your BIOS vendor for an update.

This line was actually present while using arch as well I just never 
took the time to read past the first line (which just says enabled 
interrupt remapping). This seems to be an issue with northbridge 
stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).

I wonder if it's even possible to get this to work with my motherboard 
now....**though ESXi can manage just fine...**

Maybe there's some way to trick linux into just trying to do it. I mean, 
it seems to pass devices to pci-stub just fine.

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/14/2016 6:00 PM, Steve Novakov wrote:
> Some more info here from someone having exactly the same problem I am 
> and at exactly the same stage
>
>  http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605 
>
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/13/2016 4:11 PM, Steve Novakov wrote:
>> Interestingly, when I disable VT-D in the motherboard BIOS. I can 
>> boot with "intel_iommu=on pci-stub.ids=DEVICEID"
>>
>> and the DEVICE is successfully loaded by the pci-stub driver, 
>> according to $lspci -nnk   readout
>>
>> additionally, I see:
>>
>> $dmesg | grep -e IOMMU
>>
>>     [0.000000] DMAR: IOMMU enabled
>>
>> even though I actually DISABLED VT-D in the motherboard BIOS. Going 
>> back now and enabling it in the BIOS causes freezing during boot again.
>>
>> I'm also having trouble assigning this device to the vfio-pci driver, 
>> but that might be a separate issue.
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>> On 6/12/2016 11:32 PM, Steve Novakov wrote:
>>> > Usually, i am using intel_iommu=on and everything works well. But 
>>> in your case, i guess you may also need intremap=off.
>>>
>>> Passing EITHER   "intel_iommu=on intremap=off"  OR " "intel_iommu=on 
>>> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a 
>>> freeze at boot. This happens at the loading initramfs step ("initrd 
>>> /boot/initramfs-linux.img").
>>>
>>> Any further suggestions? Thanks,
>>>
>>> Steve Novakov
>>> B.A.Sc Engineering Physics
>>> PhD Student - Physics
>>> University of Michigan - Ann Arbor
>>>
>>>
>>>
>>
>

Re: X58 Virtualization w/ Linux

[Paolo Bonzini]

On 15/06/2016 07:04, Steve Novakov wrote:
> So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:
> 
> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
> a chipset that contains an erratum making that feature unstable. To
> maintain system stability interrupt remapping is being disabled.please
> contact your BIOS vendor for an update.
> 
> This line was actually present while using arch as well I just never
> took the time to read past the first line (which just says enabled
> interrupt remapping). This seems to be an issue with northbridge
> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
> 
> I wonder if it's even possible to get this to work with my motherboard
> now....**though ESXi can manage just fine...**
> 
> Maybe there's some way to trick linux into just trying to do it. I mean,
> it seems to pass devices to pci-stub just fine.

Passing devices to pci-stub doesn't require the IOMMU, it's just
swapping the driver that binds to the device.  I think this is why with
VT-d disabled you cannot assign the device to vfio-pci.

Does it still freeze with Ubuntu 16.04 (VT-d enabled,
vfio_iommu_type1.allow_unsafe_interrupts=1)?

Thanks,

Paolo

Re: X58 Virtualization w/ Linux

[Steve Novakov]

Hello Paolo, thank you for joining in,

Yeah absolutely, it freezes no matter what I do. I get all sorts of 
interesting error messages during boot. Dmesg goes to like [64.4XXXX] or 
higher and spits out random kernel functions, addresses, etc. then 
freezes. I've tried

intel_iommu=on
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1

all sorts of combinations. I've verified that all of the relevant 
modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that 
"DMAR-IR" message is suspicious though, no? That there is some setting 
in the kernel that disables remapping by default for my BIOS. I've seen 
posts with working vt-d even with an i7-950. I wonder if they have the 
C3 stepping of the northbridge. Does that sound like it might be an issue?

I can't believe that there is not some way to make this work, as it 
works flawlessly with ESXi (which sits on top of a linux kernel, though 
it's vmware's proprietary fork I think). If my system boots in "Plug and 
play" mode, the BIOS configuration for non critical devices shouldn't 
matter, right?

Steve Novakov
B.A.Sc. Engineering Physics
PhD Student - Physics
University of Michigan
On 6/15/2016 2:59 AM, Paolo Bonzini wrote:
>
> On 15/06/2016 07:04, Steve Novakov wrote:
>> So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:
>>
>> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
>> a chipset that contains an erratum making that feature unstable. To
>> maintain system stability interrupt remapping is being disabled.please
>> contact your BIOS vendor for an update.
>>
>> This line was actually present while using arch as well I just never
>> took the time to read past the first line (which just says enabled
>> interrupt remapping). This seems to be an issue with northbridge
>> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
>>
>> I wonder if it's even possible to get this to work with my motherboard
>> now....**though ESXi can manage just fine...**
>>
>> Maybe there's some way to trick linux into just trying to do it. I mean,
>> it seems to pass devices to pci-stub just fine.
> Passing devices to pci-stub doesn't require the IOMMU, it's just
> swapping the driver that binds to the device.  I think this is why with
> VT-d disabled you cannot assign the device to vfio-pci.
>
> Does it still freeze with Ubuntu 16.04 (VT-d enabled,
> vfio_iommu_type1.allow_unsafe_interrupts=1)?
>
> Thanks,
>
> Paolo

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/15 23:40, Steve Novakov wrote:
> Hello Paolo, thank you for joining in,
>
> Yeah absolutely, it freezes no matter what I do. I get all sorts of
> interesting error messages during boot. Dmesg goes to like [64.4XXXX] or
> higher and spits out random kernel functions, addresses, etc. then
> freezes. I've tried
>
> intel_iommu=on
> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>
> all sorts of combinations. I've verified that all of the relevant
> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
> "DMAR-IR" message is suspicious though, no? That there is some setting
> in the kernel that disables remapping by default for my BIOS. I've seen
> posts with working vt-d even with an i7-950. I wonder if they have the
> C3 stepping of the northbridge. Does that sound like it might be an issue?
>
> I can't believe that there is not some way to make this work, as it
> works flawlessly with ESXi (which sits on top of a linux kernel, though
> it's vmware's proprietary fork I think). If my system boots in "Plug and
> play" mode, the BIOS configuration for non critical devices shouldn't
> matter, right?

It seems there are something wrong with vfio. Have you tried with legacy 
device pass through? It should be disabled on new distribution(like 
ubuntu 16.04) but RHEL6 may still support it.

>
> Steve Novakov
> B.A.Sc. Engineering Physics
> PhD Student - Physics
> University of Michigan
> On 6/15/2016 2:59 AM, Paolo Bonzini wrote:
>>
>> On 15/06/2016 07:04, Steve Novakov wrote:
>>> So I swapped over to ubuntu 16.04 server and I just saw this line in
>>> dmesg:
>>>
>>> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
>>> a chipset that contains an erratum making that feature unstable. To
>>> maintain system stability interrupt remapping is being disabled.please
>>> contact your BIOS vendor for an update.
>>>
>>> This line was actually present while using arch as well I just never
>>> took the time to read past the first line (which just says enabled
>>> interrupt remapping). This seems to be an issue with northbridge
>>> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
>>>
>>> I wonder if it's even possible to get this to work with my motherboard
>>> now....**though ESXi can manage just fine...**
>>>
>>> Maybe there's some way to trick linux into just trying to do it. I mean,
>>> it seems to pass devices to pci-stub just fine.
>> Passing devices to pci-stub doesn't require the IOMMU, it's just
>> swapping the driver that binds to the device.  I think this is why with
>> VT-d disabled you cannot assign the device to vfio-pci.
>>
>> Does it still freeze with Ubuntu 16.04 (VT-d enabled,
>> vfio_iommu_type1.allow_unsafe_interrupts=1)?
>>
>> Thanks,
>>
>> Paolo
>


-- 
best regards
yang

Re: X58 Virtualization w/ Linux

[Steve Novakov]

 > It seems there are something wrong with vfio. Have you tried with 
legacy device pass through? It should be disabled on new 
distribution(like ubuntu 16.04) but RHEL6 may still support it.

Hello Yang, would you mind elaborating on this?  I should get RHEL 6.8? 
What is the "legacy pass through"?

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/15/2016 9:19 PM, Yang Zhang wrote:

Re: X58 Virtualization w/ Linux

[Paolo Bonzini]

On 16/06/2016 03:19, Yang Zhang wrote:
>> intel_iommu=on
>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>>
>> all sorts of combinations. I've verified that all of the relevant
>> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
>> "DMAR-IR" message is suspicious though, no? That there is some setting
>> in the kernel that disables remapping by default for my BIOS. I've seen
>> posts with working vt-d even with an i7-950. I wonder if they have the
>> C3 stepping of the northbridge. Does that sound like it might be an
>> issue?
>>
>> I can't believe that there is not some way to make this work, as it
>> works flawlessly with ESXi (which sits on top of a linux kernel, though
>> it's vmware's proprietary fork I think). If my system boots in "Plug and
>> play" mode, the BIOS configuration for non critical devices shouldn't
>> matter, right?
> 
> It seems there are something wrong with vfio. Have you tried with legacy
> device pass through? It should be disabled on new distribution(like
> ubuntu 16.04) but RHEL6 may still support it.

There shouldn't be any difference.  He gets a freeze with
intel_iommu=on, even before going to device assignment.

Paolo

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/16 15:17, Paolo Bonzini wrote:
>
>
> On 16/06/2016 03:19, Yang Zhang wrote:
>>> intel_iommu=on
>>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>>>
>>> all sorts of combinations. I've verified that all of the relevant
>>> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
>>> "DMAR-IR" message is suspicious though, no? That there is some setting
>>> in the kernel that disables remapping by default for my BIOS. I've seen
>>> posts with working vt-d even with an i7-950. I wonder if they have the
>>> C3 stepping of the northbridge. Does that sound like it might be an
>>> issue?
>>>
>>> I can't believe that there is not some way to make this work, as it
>>> works flawlessly with ESXi (which sits on top of a linux kernel, though
>>> it's vmware's proprietary fork I think). If my system boots in "Plug and
>>> play" mode, the BIOS configuration for non critical devices shouldn't
>>> matter, right?
>>
>> It seems there are something wrong with vfio. Have you tried with legacy
>> device pass through? It should be disabled on new distribution(like
>> ubuntu 16.04) but RHEL6 may still support it.
>
> There shouldn't be any difference.  He gets a freeze with
> intel_iommu=on, even before going to device assignment.

Yes, if system freeze on boot, then it seems something wrong in linux 
kernel. Steve, can you post the dmesg log and check whether there are 
some additional devices on your board? I remember some legacy device 
drivers are not friendly to IOMMU.


-- 
best regards
yang

Re: X58 Virtualization w/ Linux

[Steve Novakov]

 > Yes, if system freeze on boot, then it seems something wrong in linux 
kernel. Steve, can you post the dmesg log and check whether there are 
some additional devices on your board? I remember some legacy device 
drivers are not friendly to IOMMU.

I kind of gave up on this and moved back to ESXi, though I am now 
fighting HID device passthrough >___> (prosumer virtualization tech not 
as robust as I imagined....). ESXi is passing through USB controllers (I 
get non HID device, like a USB-ethernet adapter passthrough no problem) 
and an Intel PCIe NIC, but I'm having issues with graphics cards. Tough 
to say whether it's a driver or X server issue at the moment, the cards 
are seemingly passed through in ESXi. I have an NVIDIA GT 730 and Radeon 
HD5450 to play with. I'm worried a bit about the replacement of fglrx 
and the use of "radeon" module in newer kernels, and how it deals with 
passthrough, etc, with such an old card, but we'll see how it goes. I'm 
also fighting being able to show the VM console and a graphics card 
output at the same time (that's an X server configuration thing I think).

I can support some debugging though. I'll reinstall Ubuntu 16.04 server 
on a spare drive and get it back to the state it was in. I'll send the 
entire dmesg, lspci -nnk from just a base install. Should I also give 
you dumps with "intel_iommu=on" and VT-d disabled in the BIOS? (I can't 
boot if both things are on)

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/19/2016 10:07 PM, Yang Zhang wrote:

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/21 1:50, Steve Novakov wrote:
>> I can support some debugging though. I'll reinstall Ubuntu 16.04
> server on a spare drive and get it back to the state it was in. I'll
> send the entire dmesg, lspci -nnk from just a base install. Should I
> also give you dumps with "intel_iommu=on" and VT-d disabled in the BIOS?
> (I can't boot if both things are on)
>
> I've attached a zip file with some dumps. There is a text file in there
> called "install_sequence.txt" that explains important steps and what
> each of the dump files are from.

There is no error message from the dmesg. But there do have lots of 
additional devices on your board. Do you see any error when booting 
fail? Does system stuck while enabling a device?


-- 
best regards
yang

Re: X58 Virtualization w/ Linux

[Steve Novakov]

 >There is no error message from the dmesg. But there do have lots of 
additional devices on your board. Do you see any error when booting 
fail? Does system stuck while enabling a device?


All I see now is a completely blank screen right after the grub screen 
(with BIOS VT-d on, and with intel_iommu=on..etc  boot options). There 
seems to be no logging happening (syslog has no entries for these boots).

The blank screen persists and there is no response from any inputs. I 
waited 3-5 minutes and then rebooted manually.

Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/20/2016 9:21 PM, Yang Zhang wrote:

Re: X58 Virtualization w/ Linux

[Yang Zhang]

On 2016/6/21 9:24, Steve Novakov wrote:
>>There is no error message from the dmesg. But there do have lots of
> additional devices on your board. Do you see any error when booting
> fail? Does system stuck while enabling a device?
>
>
> All I see now is a completely blank screen right after the grub screen
> (with BIOS VT-d on, and with intel_iommu=on..etc  boot options). There
> seems to be no logging happening (syslog has no entries for these boots).
>
> The blank screen persists and there is no response from any inputs. I
> waited 3-5 minutes and then rebooted manually.

You may need to config the serial port to get the boot message. Without 
enough message, it's hard to tell what is wrong. Alternatively, you can 
remove all additional devices to see whether system can boot.


-- 
best regards
yang