* X58 Virtualization w/ Linux
@ 2016-06-11 21:34 Steve Novakov
2016-06-12 1:46 ` Yang Zhang
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-11 21:34 UTC (permalink / raw)
To: kvm
Hello,
I was instructed to send an email to the KVM-devel group about this. I
made a post on reddit about some issues I've had virtualizating an X58
environment. The details are here:
https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/
I'm asking around to see if anyone has a straightforward solution, or
any advice on how to approach this problem. Also (please read the reddit
post first):
- can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on
vfio_iommu_type1.allow_unsafe_interrupts=1" safely?
- is there a way to fix the DMAR table for my BIOS (see post)? How
might I dump it and fix it?
I'm running barebones Arch and KVM/QEMU. My progress is outlined in that
reddit post and discussion. At the moment, I'm after "safe" passthrough
of a PCIe NIC and video card to separate VMs. "Safe" may mean totally
secure (secure passthrough), or just stable enough that I don't have to
debug problems every kernel update *whichever is possible*. Thank you
in advance for any help.
Sincerely,
--
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-11 21:34 X58 Virtualization w/ Linux Steve Novakov
@ 2016-06-12 1:46 ` Yang Zhang
2016-06-12 1:55 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Yang Zhang @ 2016-06-12 1:46 UTC (permalink / raw)
To: Steve Novakov, kvm
On 2016/6/12 5:34, Steve Novakov wrote:
> Hello,
>
> I was instructed to send an email to the KVM-devel group about this. I
> made a post on reddit about some issues I've had virtualizating an X58
> environment. The details are here:
>
> https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/
>
>
> I'm asking around to see if anyone has a straightforward solution, or
> any advice on how to approach this problem. Also (please read the reddit
> post first):
> - can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1" safely?
allow_unsafe_interupts actually means the interrupt remapping on Intel
IOMMU which is a security feature. Without it, a malicious VM can attack
the host, see below document for more details:
http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
> - is there a way to fix the DMAR table for my BIOS (see post)? How
> might I dump it and fix it?
You can dump it from /sys/firmware/acpi/tables/DMAR. I remember linux
kernel allow you to use the customized ACPI table but i am not sure
whether DMAR is suitable for it. Also, you can try to upgrade your BIOS
to fix it.
>
> I'm running barebones Arch and KVM/QEMU. My progress is outlined in that
> reddit post and discussion. At the moment, I'm after "safe" passthrough
> of a PCIe NIC and video card to separate VMs. "Safe" may mean totally
> secure (secure passthrough), or just stable enough that I don't have to
> debug problems every kernel update *whichever is possible*. Thank you
> in advance for any help.
>
> Sincerely,
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-12 1:46 ` Yang Zhang
@ 2016-06-12 1:55 ` Steve Novakov
2016-06-12 2:54 ` Steve Novakov
2016-06-13 1:46 ` Yang Zhang
0 siblings, 2 replies; 20+ messages in thread
From: Steve Novakov @ 2016-06-12 1:55 UTC (permalink / raw)
To: Yang Zhang, kvm
Hello Yang,
> allow_unsafe_interupts actually means the interrupt remapping on Intel
> IOMMU which is a security feature. Without it, a malicious VM can
> attack the host, see below document for more details:
> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
Should I take that to mean that "allow_unsafe_interrupts" is actually a
security feature??? Is this the discussed "interrupt remapping" in the
whitepaper? I am interpreting that paper as saying that this interrupt
remapping does *not* use the supplied DMAR table. Is that correct?
> Also, you can try to upgrade your BIOS to fix it.
I'll take a look but I think I have the latest (which means, from ~2011
probably) BIOS version.
Could I also ask you outright what entire set of boot options you would
pass when booting into a kvm system with IOMMU enabled? I would love
some "default" set of boot options to compare to, as opposed to random
ones from assorted forums.
Thank you for the prompt reply!
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/11/2016 9:46 PM, Yang Zhang wrote:
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-12 1:55 ` Steve Novakov
@ 2016-06-12 2:54 ` Steve Novakov
2016-06-12 2:57 ` Steve Novakov
2016-06-13 1:46 ` Yang Zhang
1 sibling, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-12 2:54 UTC (permalink / raw)
To: Yang Zhang, kvm
Hello Yang,
To add to this, a list of mandatory packages would be helpful as well
(like if I need some additional virtio- packages). Thank you,
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/11/2016 9:55 PM, Steve Novakov wrote:
> Hello Yang,
>
>> allow_unsafe_interupts actually means the interrupt remapping on
>> Intel IOMMU which is a security feature. Without it, a malicious VM
>> can attack the host, see below document for more details:
>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
>>
>
> Should I take that to mean that "allow_unsafe_interrupts" is actually
> a security feature??? Is this the discussed "interrupt remapping" in
> the whitepaper? I am interpreting that paper as saying that this
> interrupt remapping does *not* use the supplied DMAR table. Is that
> correct?
>
>> Also, you can try to upgrade your BIOS to fix it.
>
> I'll take a look but I think I have the latest (which means, from
> ~2011 probably) BIOS version.
>
> Could I also ask you outright what entire set of boot options you
> would pass when booting into a kvm system with IOMMU enabled? I would
> love some "default" set of boot options to compare to, as opposed to
> random ones from assorted forums.
>
> Thank you for the prompt reply!
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-12 2:54 ` Steve Novakov
@ 2016-06-12 2:57 ` Steve Novakov
0 siblings, 0 replies; 20+ messages in thread
From: Steve Novakov @ 2016-06-12 2:57 UTC (permalink / raw)
To: Yang Zhang, kvm
Sorry, I was thinking of kernel modules and said packages (virtio-X are
kernel modules) . Thanks,
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/11/2016 10:54 PM, Steve Novakov wrote:
> Hello Yang,
>
> To add to this, a list of mandatory packages would be helpful as well
> (like if I need some additional virtio- packages). Thank you,
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
>
> On 6/11/2016 9:55 PM, Steve Novakov wrote:
>> Hello Yang,
>>
>>> allow_unsafe_interupts actually means the interrupt remapping on
>>> Intel IOMMU which is a security feature. Without it, a malicious VM
>>> can attack the host, see below document for more details:
>>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
>>>
>>
>> Should I take that to mean that "allow_unsafe_interrupts" is actually
>> a security feature??? Is this the discussed "interrupt remapping" in
>> the whitepaper? I am interpreting that paper as saying that this
>> interrupt remapping does *not* use the supplied DMAR table. Is that
>> correct?
>>
>>> Also, you can try to upgrade your BIOS to fix it.
>>
>> I'll take a look but I think I have the latest (which means, from
>> ~2011 probably) BIOS version.
>>
>> Could I also ask you outright what entire set of boot options you
>> would pass when booting into a kvm system with IOMMU enabled? I would
>> love some "default" set of boot options to compare to, as opposed to
>> random ones from assorted forums.
>>
>> Thank you for the prompt reply!
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-12 1:55 ` Steve Novakov
2016-06-12 2:54 ` Steve Novakov
@ 2016-06-13 1:46 ` Yang Zhang
2016-06-13 3:32 ` Steve Novakov
1 sibling, 1 reply; 20+ messages in thread
From: Yang Zhang @ 2016-06-13 1:46 UTC (permalink / raw)
To: Steve Novakov, kvm
On 2016/6/12 9:55, Steve Novakov wrote:
> Hello Yang,
>
>> allow_unsafe_interupts actually means the interrupt remapping on Intel
>> IOMMU which is a security feature. Without it, a malicious VM can
>> attack the host, see below document for more details:
>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf
>>
>
> Should I take that to mean that "allow_unsafe_interrupts" is actually a
> security feature??? Is this the discussed "interrupt remapping" in the
Interrupt remapping not only a security feature, it also supports more
than 255 CPUs associate with x2apic. allow_unsafe_interrupts allows you
to enable IOMMU on the platform even without interrupt remapping because
first platform supporting IOMMU doesn't have interrupt remapping.
> whitepaper? I am interpreting that paper as saying that this interrupt
> remapping does *not* use the supplied DMAR table. Is that correct?
All the necessary information for IOMMU is located in ACPI tables not
only DMAR table. OS need to parse it before enabling the IOMMU.
>
>> Also, you can try to upgrade your BIOS to fix it.
>
> I'll take a look but I think I have the latest (which means, from ~2011
> probably) BIOS version.
>
> Could I also ask you outright what entire set of boot options you would
> pass when booting into a kvm system with IOMMU enabled? I would love
> some "default" set of boot options to compare to, as opposed to random
> ones from assorted forums.
Usually, i am using intel_iommu=on and everything works well. But in
your case, i guess you may also need intremap=off.
>
> Thank you for the prompt reply!
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/11/2016 9:46 PM, Yang Zhang wrote:
>
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-13 1:46 ` Yang Zhang
@ 2016-06-13 3:32 ` Steve Novakov
2016-06-13 20:11 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-13 3:32 UTC (permalink / raw)
To: Yang Zhang, kvm
> Usually, i am using intel_iommu=on and everything works well. But in
your case, i guess you may also need intremap=off.
Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on
vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a
freeze at boot. This happens at the loading initramfs step ("initrd
/boot/initramfs-linux.img").
Any further suggestions? Thanks,
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-13 3:32 ` Steve Novakov
@ 2016-06-13 20:11 ` Steve Novakov
2016-06-14 22:00 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-13 20:11 UTC (permalink / raw)
To: Yang Zhang, kvm
Interestingly, when I disable VT-D in the motherboard BIOS. I can boot
with "intel_iommu=on pci-stub.ids=DEVICEID"
and the DEVICE is successfully loaded by the pci-stub driver, according
to $lspci -nnk readout
additionally, I see:
$dmesg | grep -e IOMMU
[0.000000] DMAR: IOMMU enabled
even though I actually DISABLED VT-D in the motherboard BIOS. Going back
now and enabling it in the BIOS causes freezing during boot again.
I'm also having trouble assigning this device to the vfio-pci driver,
but that might be a separate issue.
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/12/2016 11:32 PM, Steve Novakov wrote:
> > Usually, i am using intel_iommu=on and everything works well. But in
> your case, i guess you may also need intremap=off.
>
> Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on
> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a
> freeze at boot. This happens at the loading initramfs step ("initrd
> /boot/initramfs-linux.img").
>
> Any further suggestions? Thanks,
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
>
>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-13 20:11 ` Steve Novakov
@ 2016-06-14 22:00 ` Steve Novakov
2016-06-15 5:04 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-14 22:00 UTC (permalink / raw)
To: Yang Zhang, kvm
Some more info here from someone having exactly the same problem I am
and at exactly the same stage
http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/13/2016 4:11 PM, Steve Novakov wrote:
> Interestingly, when I disable VT-D in the motherboard BIOS. I can boot
> with "intel_iommu=on pci-stub.ids=DEVICEID"
>
> and the DEVICE is successfully loaded by the pci-stub driver,
> according to $lspci -nnk readout
>
> additionally, I see:
>
> $dmesg | grep -e IOMMU
>
> [0.000000] DMAR: IOMMU enabled
>
> even though I actually DISABLED VT-D in the motherboard BIOS. Going
> back now and enabling it in the BIOS causes freezing during boot again.
>
> I'm also having trouble assigning this device to the vfio-pci driver,
> but that might be a separate issue.
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/12/2016 11:32 PM, Steve Novakov wrote:
>> > Usually, i am using intel_iommu=on and everything works well. But
>> in your case, i guess you may also need intremap=off.
>>
>> Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on
>> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a
>> freeze at boot. This happens at the loading initramfs step ("initrd
>> /boot/initramfs-linux.img").
>>
>> Any further suggestions? Thanks,
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>>
>>
>>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-14 22:00 ` Steve Novakov
@ 2016-06-15 5:04 ` Steve Novakov
2016-06-15 6:59 ` Paolo Bonzini
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-15 5:04 UTC (permalink / raw)
To: Yang Zhang, kvm
So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:
[0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
a chipset that contains an erratum making that feature unstable. To
maintain system stability interrupt remapping is being disabled.please
contact your BIOS vendor for an update.
This line was actually present while using arch as well I just never
took the time to read past the first line (which just says enabled
interrupt remapping). This seems to be an issue with northbridge
stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
I wonder if it's even possible to get this to work with my motherboard
now....**though ESXi can manage just fine...**
Maybe there's some way to trick linux into just trying to do it. I mean,
it seems to pass devices to pci-stub just fine.
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/14/2016 6:00 PM, Steve Novakov wrote:
> Some more info here from someone having exactly the same problem I am
> and at exactly the same stage
>
> http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605
>
>
> Steve Novakov
> B.A.Sc Engineering Physics
> PhD Student - Physics
> University of Michigan - Ann Arbor
> On 6/13/2016 4:11 PM, Steve Novakov wrote:
>> Interestingly, when I disable VT-D in the motherboard BIOS. I can
>> boot with "intel_iommu=on pci-stub.ids=DEVICEID"
>>
>> and the DEVICE is successfully loaded by the pci-stub driver,
>> according to $lspci -nnk readout
>>
>> additionally, I see:
>>
>> $dmesg | grep -e IOMMU
>>
>> [0.000000] DMAR: IOMMU enabled
>>
>> even though I actually DISABLED VT-D in the motherboard BIOS. Going
>> back now and enabling it in the BIOS causes freezing during boot again.
>>
>> I'm also having trouble assigning this device to the vfio-pci driver,
>> but that might be a separate issue.
>>
>> Steve Novakov
>> B.A.Sc Engineering Physics
>> PhD Student - Physics
>> University of Michigan - Ann Arbor
>> On 6/12/2016 11:32 PM, Steve Novakov wrote:
>>> > Usually, i am using intel_iommu=on and everything works well. But
>>> in your case, i guess you may also need intremap=off.
>>>
>>> Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on
>>> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a
>>> freeze at boot. This happens at the loading initramfs step ("initrd
>>> /boot/initramfs-linux.img").
>>>
>>> Any further suggestions? Thanks,
>>>
>>> Steve Novakov
>>> B.A.Sc Engineering Physics
>>> PhD Student - Physics
>>> University of Michigan - Ann Arbor
>>>
>>>
>>>
>>
>
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-15 5:04 ` Steve Novakov
@ 2016-06-15 6:59 ` Paolo Bonzini
2016-06-15 15:40 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Paolo Bonzini @ 2016-06-15 6:59 UTC (permalink / raw)
To: Steve Novakov, Yang Zhang, kvm
On 15/06/2016 07:04, Steve Novakov wrote:
> So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:
>
> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
> a chipset that contains an erratum making that feature unstable. To
> maintain system stability interrupt remapping is being disabled.please
> contact your BIOS vendor for an update.
>
> This line was actually present while using arch as well I just never
> took the time to read past the first line (which just says enabled
> interrupt remapping). This seems to be an issue with northbridge
> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
>
> I wonder if it's even possible to get this to work with my motherboard
> now....**though ESXi can manage just fine...**
>
> Maybe there's some way to trick linux into just trying to do it. I mean,
> it seems to pass devices to pci-stub just fine.
Passing devices to pci-stub doesn't require the IOMMU, it's just
swapping the driver that binds to the device. I think this is why with
VT-d disabled you cannot assign the device to vfio-pci.
Does it still freeze with Ubuntu 16.04 (VT-d enabled,
vfio_iommu_type1.allow_unsafe_interrupts=1)?
Thanks,
Paolo
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-15 6:59 ` Paolo Bonzini
@ 2016-06-15 15:40 ` Steve Novakov
2016-06-16 1:19 ` Yang Zhang
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-15 15:40 UTC (permalink / raw)
To: Paolo Bonzini, Yang Zhang, kvm
Hello Paolo, thank you for joining in,
Yeah absolutely, it freezes no matter what I do. I get all sorts of
interesting error messages during boot. Dmesg goes to like [64.4XXXX] or
higher and spits out random kernel functions, addresses, etc. then
freezes. I've tried
intel_iommu=on
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
all sorts of combinations. I've verified that all of the relevant
modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
"DMAR-IR" message is suspicious though, no? That there is some setting
in the kernel that disables remapping by default for my BIOS. I've seen
posts with working vt-d even with an i7-950. I wonder if they have the
C3 stepping of the northbridge. Does that sound like it might be an issue?
I can't believe that there is not some way to make this work, as it
works flawlessly with ESXi (which sits on top of a linux kernel, though
it's vmware's proprietary fork I think). If my system boots in "Plug and
play" mode, the BIOS configuration for non critical devices shouldn't
matter, right?
Steve Novakov
B.A.Sc. Engineering Physics
PhD Student - Physics
University of Michigan
On 6/15/2016 2:59 AM, Paolo Bonzini wrote:
>
> On 15/06/2016 07:04, Steve Novakov wrote:
>> So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg:
>>
>> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
>> a chipset that contains an erratum making that feature unstable. To
>> maintain system stability interrupt remapping is being disabled.please
>> contact your BIOS vendor for an update.
>>
>> This line was actually present while using arch as well I just never
>> took the time to read past the first line (which just says enabled
>> interrupt remapping). This seems to be an issue with northbridge
>> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
>>
>> I wonder if it's even possible to get this to work with my motherboard
>> now....**though ESXi can manage just fine...**
>>
>> Maybe there's some way to trick linux into just trying to do it. I mean,
>> it seems to pass devices to pci-stub just fine.
> Passing devices to pci-stub doesn't require the IOMMU, it's just
> swapping the driver that binds to the device. I think this is why with
> VT-d disabled you cannot assign the device to vfio-pci.
>
> Does it still freeze with Ubuntu 16.04 (VT-d enabled,
> vfio_iommu_type1.allow_unsafe_interrupts=1)?
>
> Thanks,
>
> Paolo
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-15 15:40 ` Steve Novakov
@ 2016-06-16 1:19 ` Yang Zhang
2016-06-16 1:22 ` Steve Novakov
2016-06-16 7:17 ` Paolo Bonzini
0 siblings, 2 replies; 20+ messages in thread
From: Yang Zhang @ 2016-06-16 1:19 UTC (permalink / raw)
To: Steve Novakov, Paolo Bonzini, kvm
On 2016/6/15 23:40, Steve Novakov wrote:
> Hello Paolo, thank you for joining in,
>
> Yeah absolutely, it freezes no matter what I do. I get all sorts of
> interesting error messages during boot. Dmesg goes to like [64.4XXXX] or
> higher and spits out random kernel functions, addresses, etc. then
> freezes. I've tried
>
> intel_iommu=on
> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>
> all sorts of combinations. I've verified that all of the relevant
> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
> "DMAR-IR" message is suspicious though, no? That there is some setting
> in the kernel that disables remapping by default for my BIOS. I've seen
> posts with working vt-d even with an i7-950. I wonder if they have the
> C3 stepping of the northbridge. Does that sound like it might be an issue?
>
> I can't believe that there is not some way to make this work, as it
> works flawlessly with ESXi (which sits on top of a linux kernel, though
> it's vmware's proprietary fork I think). If my system boots in "Plug and
> play" mode, the BIOS configuration for non critical devices shouldn't
> matter, right?
It seems there are something wrong with vfio. Have you tried with legacy
device pass through? It should be disabled on new distribution(like
ubuntu 16.04) but RHEL6 may still support it.
>
> Steve Novakov
> B.A.Sc. Engineering Physics
> PhD Student - Physics
> University of Michigan
> On 6/15/2016 2:59 AM, Paolo Bonzini wrote:
>>
>> On 15/06/2016 07:04, Steve Novakov wrote:
>>> So I swapped over to ubuntu 16.04 server and I just saw this line in
>>> dmesg:
>>>
>>> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on
>>> a chipset that contains an erratum making that feature unstable. To
>>> maintain system stability interrupt remapping is being disabled.please
>>> contact your BIOS vendor for an update.
>>>
>>> This line was actually present while using arch as well I just never
>>> took the time to read past the first line (which just says enabled
>>> interrupt remapping). This seems to be an issue with northbridge
>>> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ).
>>>
>>> I wonder if it's even possible to get this to work with my motherboard
>>> now....**though ESXi can manage just fine...**
>>>
>>> Maybe there's some way to trick linux into just trying to do it. I mean,
>>> it seems to pass devices to pci-stub just fine.
>> Passing devices to pci-stub doesn't require the IOMMU, it's just
>> swapping the driver that binds to the device. I think this is why with
>> VT-d disabled you cannot assign the device to vfio-pci.
>>
>> Does it still freeze with Ubuntu 16.04 (VT-d enabled,
>> vfio_iommu_type1.allow_unsafe_interrupts=1)?
>>
>> Thanks,
>>
>> Paolo
>
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-16 1:19 ` Yang Zhang
@ 2016-06-16 1:22 ` Steve Novakov
2016-06-16 7:17 ` Paolo Bonzini
1 sibling, 0 replies; 20+ messages in thread
From: Steve Novakov @ 2016-06-16 1:22 UTC (permalink / raw)
To: Yang Zhang, Paolo Bonzini, kvm
> It seems there are something wrong with vfio. Have you tried with
legacy device pass through? It should be disabled on new
distribution(like ubuntu 16.04) but RHEL6 may still support it.
Hello Yang, would you mind elaborating on this? I should get RHEL 6.8?
What is the "legacy pass through"?
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/15/2016 9:19 PM, Yang Zhang wrote:
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-16 1:19 ` Yang Zhang
2016-06-16 1:22 ` Steve Novakov
@ 2016-06-16 7:17 ` Paolo Bonzini
2016-06-20 2:07 ` Yang Zhang
1 sibling, 1 reply; 20+ messages in thread
From: Paolo Bonzini @ 2016-06-16 7:17 UTC (permalink / raw)
To: Yang Zhang, Steve Novakov, kvm
On 16/06/2016 03:19, Yang Zhang wrote:
>> intel_iommu=on
>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>>
>> all sorts of combinations. I've verified that all of the relevant
>> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
>> "DMAR-IR" message is suspicious though, no? That there is some setting
>> in the kernel that disables remapping by default for my BIOS. I've seen
>> posts with working vt-d even with an i7-950. I wonder if they have the
>> C3 stepping of the northbridge. Does that sound like it might be an
>> issue?
>>
>> I can't believe that there is not some way to make this work, as it
>> works flawlessly with ESXi (which sits on top of a linux kernel, though
>> it's vmware's proprietary fork I think). If my system boots in "Plug and
>> play" mode, the BIOS configuration for non critical devices shouldn't
>> matter, right?
>
> It seems there are something wrong with vfio. Have you tried with legacy
> device pass through? It should be disabled on new distribution(like
> ubuntu 16.04) but RHEL6 may still support it.
There shouldn't be any difference. He gets a freeze with
intel_iommu=on, even before going to device assignment.
Paolo
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-16 7:17 ` Paolo Bonzini
@ 2016-06-20 2:07 ` Yang Zhang
2016-06-20 4:05 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Yang Zhang @ 2016-06-20 2:07 UTC (permalink / raw)
To: Paolo Bonzini, Steve Novakov, kvm
On 2016/6/16 15:17, Paolo Bonzini wrote:
>
>
> On 16/06/2016 03:19, Yang Zhang wrote:
>>> intel_iommu=on
>>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
>>>
>>> all sorts of combinations. I've verified that all of the relevant
>>> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that
>>> "DMAR-IR" message is suspicious though, no? That there is some setting
>>> in the kernel that disables remapping by default for my BIOS. I've seen
>>> posts with working vt-d even with an i7-950. I wonder if they have the
>>> C3 stepping of the northbridge. Does that sound like it might be an
>>> issue?
>>>
>>> I can't believe that there is not some way to make this work, as it
>>> works flawlessly with ESXi (which sits on top of a linux kernel, though
>>> it's vmware's proprietary fork I think). If my system boots in "Plug and
>>> play" mode, the BIOS configuration for non critical devices shouldn't
>>> matter, right?
>>
>> It seems there are something wrong with vfio. Have you tried with legacy
>> device pass through? It should be disabled on new distribution(like
>> ubuntu 16.04) but RHEL6 may still support it.
>
> There shouldn't be any difference. He gets a freeze with
> intel_iommu=on, even before going to device assignment.
Yes, if system freeze on boot, then it seems something wrong in linux
kernel. Steve, can you post the dmesg log and check whether there are
some additional devices on your board? I remember some legacy device
drivers are not friendly to IOMMU.
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-20 2:07 ` Yang Zhang
@ 2016-06-20 4:05 ` Steve Novakov
[not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com>
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-20 4:05 UTC (permalink / raw)
To: Yang Zhang, Paolo Bonzini, kvm
> Yes, if system freeze on boot, then it seems something wrong in linux
kernel. Steve, can you post the dmesg log and check whether there are
some additional devices on your board? I remember some legacy device
drivers are not friendly to IOMMU.
I kind of gave up on this and moved back to ESXi, though I am now
fighting HID device passthrough >___> (prosumer virtualization tech not
as robust as I imagined....). ESXi is passing through USB controllers (I
get non HID device, like a USB-ethernet adapter passthrough no problem)
and an Intel PCIe NIC, but I'm having issues with graphics cards. Tough
to say whether it's a driver or X server issue at the moment, the cards
are seemingly passed through in ESXi. I have an NVIDIA GT 730 and Radeon
HD5450 to play with. I'm worried a bit about the replacement of fglrx
and the use of "radeon" module in newer kernels, and how it deals with
passthrough, etc, with such an old card, but we'll see how it goes. I'm
also fighting being able to show the VM console and a graphics card
output at the same time (that's an X server configuration thing I think).
I can support some debugging though. I'll reinstall Ubuntu 16.04 server
on a spare drive and get it back to the state it was in. I'll send the
entire dmesg, lspci -nnk from just a base install. Should I also give
you dumps with "intel_iommu=on" and VT-d disabled in the BIOS? (I can't
boot if both things are on)
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/19/2016 10:07 PM, Yang Zhang wrote:
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
[not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com>
@ 2016-06-21 1:21 ` Yang Zhang
2016-06-21 1:24 ` Steve Novakov
0 siblings, 1 reply; 20+ messages in thread
From: Yang Zhang @ 2016-06-21 1:21 UTC (permalink / raw)
To: Steve Novakov, Paolo Bonzini, kvm
On 2016/6/21 1:50, Steve Novakov wrote:
>> I can support some debugging though. I'll reinstall Ubuntu 16.04
> server on a spare drive and get it back to the state it was in. I'll
> send the entire dmesg, lspci -nnk from just a base install. Should I
> also give you dumps with "intel_iommu=on" and VT-d disabled in the BIOS?
> (I can't boot if both things are on)
>
> I've attached a zip file with some dumps. There is a text file in there
> called "install_sequence.txt" that explains important steps and what
> each of the dump files are from.
There is no error message from the dmesg. But there do have lots of
additional devices on your board. Do you see any error when booting
fail? Does system stuck while enabling a device?
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-21 1:21 ` Yang Zhang
@ 2016-06-21 1:24 ` Steve Novakov
2016-06-27 2:21 ` Yang Zhang
0 siblings, 1 reply; 20+ messages in thread
From: Steve Novakov @ 2016-06-21 1:24 UTC (permalink / raw)
To: Yang Zhang, Paolo Bonzini, kvm
>There is no error message from the dmesg. But there do have lots of
additional devices on your board. Do you see any error when booting
fail? Does system stuck while enabling a device?
All I see now is a completely blank screen right after the grub screen
(with BIOS VT-d on, and with intel_iommu=on..etc boot options). There
seems to be no logging happening (syslog has no entries for these boots).
The blank screen persists and there is no response from any inputs. I
waited 3-5 minutes and then rebooted manually.
Steve Novakov
B.A.Sc Engineering Physics
PhD Student - Physics
University of Michigan - Ann Arbor
On 6/20/2016 9:21 PM, Yang Zhang wrote:
^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux
2016-06-21 1:24 ` Steve Novakov
@ 2016-06-27 2:21 ` Yang Zhang
0 siblings, 0 replies; 20+ messages in thread
From: Yang Zhang @ 2016-06-27 2:21 UTC (permalink / raw)
To: Steve Novakov, Paolo Bonzini, kvm
On 2016/6/21 9:24, Steve Novakov wrote:
>>There is no error message from the dmesg. But there do have lots of
> additional devices on your board. Do you see any error when booting
> fail? Does system stuck while enabling a device?
>
>
> All I see now is a completely blank screen right after the grub screen
> (with BIOS VT-d on, and with intel_iommu=on..etc boot options). There
> seems to be no logging happening (syslog has no entries for these boots).
>
> The blank screen persists and there is no response from any inputs. I
> waited 3-5 minutes and then rebooted manually.
You may need to config the serial port to get the boot message. Without
enough message, it's hard to tell what is wrong. Alternatively, you can
remove all additional devices to see whether system can boot.
--
best regards
yang
^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-06-27 2:21 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-06-11 21:34 X58 Virtualization w/ Linux Steve Novakov
2016-06-12 1:46 ` Yang Zhang
2016-06-12 1:55 ` Steve Novakov
2016-06-12 2:54 ` Steve Novakov
2016-06-12 2:57 ` Steve Novakov
2016-06-13 1:46 ` Yang Zhang
2016-06-13 3:32 ` Steve Novakov
2016-06-13 20:11 ` Steve Novakov
2016-06-14 22:00 ` Steve Novakov
2016-06-15 5:04 ` Steve Novakov
2016-06-15 6:59 ` Paolo Bonzini
2016-06-15 15:40 ` Steve Novakov
2016-06-16 1:19 ` Yang Zhang
2016-06-16 1:22 ` Steve Novakov
2016-06-16 7:17 ` Paolo Bonzini
2016-06-20 2:07 ` Yang Zhang
2016-06-20 4:05 ` Steve Novakov
[not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com>
2016-06-21 1:21 ` Yang Zhang
2016-06-21 1:24 ` Steve Novakov
2016-06-27 2:21 ` Yang Zhang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.