* X58 Virtualization w/ Linux @ 2016-06-11 21:34 Steve Novakov 2016-06-12 1:46 ` Yang Zhang 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-11 21:34 UTC (permalink / raw) To: kvm Hello, I was instructed to send an email to the KVM-devel group about this. I made a post on reddit about some issues I've had virtualizating an X58 environment. The details are here: https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/ I'm asking around to see if anyone has a straightforward solution, or any advice on how to approach this problem. Also (please read the reddit post first): - can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1" safely? - is there a way to fix the DMAR table for my BIOS (see post)? How might I dump it and fix it? I'm running barebones Arch and KVM/QEMU. My progress is outlined in that reddit post and discussion. At the moment, I'm after "safe" passthrough of a PCIe NIC and video card to separate VMs. "Safe" may mean totally secure (secure passthrough), or just stable enough that I don't have to debug problems every kernel update *whichever is possible*. Thank you in advance for any help. Sincerely, -- Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-11 21:34 X58 Virtualization w/ Linux Steve Novakov @ 2016-06-12 1:46 ` Yang Zhang 2016-06-12 1:55 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Yang Zhang @ 2016-06-12 1:46 UTC (permalink / raw) To: Steve Novakov, kvm On 2016/6/12 5:34, Steve Novakov wrote: > Hello, > > I was instructed to send an email to the KVM-devel group about this. I > made a post on reddit about some issues I've had virtualizating an X58 > environment. The details are here: > > https://www.reddit.com/r/homelab/comments/4njtoi/x58_virtualization_w_linux_xpost_rlinux4noobs/ > > > I'm asking around to see if anyone has a straightforward solution, or > any advice on how to approach this problem. Also (please read the reddit > post first): > - can I just pass "GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on > vfio_iommu_type1.allow_unsafe_interrupts=1" safely? allow_unsafe_interupts actually means the interrupt remapping on Intel IOMMU which is a security feature. Without it, a malicious VM can attack the host, see below document for more details: http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf > - is there a way to fix the DMAR table for my BIOS (see post)? How > might I dump it and fix it? You can dump it from /sys/firmware/acpi/tables/DMAR. I remember linux kernel allow you to use the customized ACPI table but i am not sure whether DMAR is suitable for it. Also, you can try to upgrade your BIOS to fix it. > > I'm running barebones Arch and KVM/QEMU. My progress is outlined in that > reddit post and discussion. At the moment, I'm after "safe" passthrough > of a PCIe NIC and video card to separate VMs. "Safe" may mean totally > secure (secure passthrough), or just stable enough that I don't have to > debug problems every kernel update *whichever is possible*. Thank you > in advance for any help. > > Sincerely, -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-12 1:46 ` Yang Zhang @ 2016-06-12 1:55 ` Steve Novakov 2016-06-12 2:54 ` Steve Novakov 2016-06-13 1:46 ` Yang Zhang 0 siblings, 2 replies; 20+ messages in thread From: Steve Novakov @ 2016-06-12 1:55 UTC (permalink / raw) To: Yang Zhang, kvm Hello Yang, > allow_unsafe_interupts actually means the interrupt remapping on Intel > IOMMU which is a security feature. Without it, a malicious VM can > attack the host, see below document for more details: > http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf Should I take that to mean that "allow_unsafe_interrupts" is actually a security feature??? Is this the discussed "interrupt remapping" in the whitepaper? I am interpreting that paper as saying that this interrupt remapping does *not* use the supplied DMAR table. Is that correct? > Also, you can try to upgrade your BIOS to fix it. I'll take a look but I think I have the latest (which means, from ~2011 probably) BIOS version. Could I also ask you outright what entire set of boot options you would pass when booting into a kvm system with IOMMU enabled? I would love some "default" set of boot options to compare to, as opposed to random ones from assorted forums. Thank you for the prompt reply! Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/11/2016 9:46 PM, Yang Zhang wrote: ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-12 1:55 ` Steve Novakov @ 2016-06-12 2:54 ` Steve Novakov 2016-06-12 2:57 ` Steve Novakov 2016-06-13 1:46 ` Yang Zhang 1 sibling, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-12 2:54 UTC (permalink / raw) To: Yang Zhang, kvm Hello Yang, To add to this, a list of mandatory packages would be helpful as well (like if I need some additional virtio- packages). Thank you, Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/11/2016 9:55 PM, Steve Novakov wrote: > Hello Yang, > >> allow_unsafe_interupts actually means the interrupt remapping on >> Intel IOMMU which is a security feature. Without it, a malicious VM >> can attack the host, see below document for more details: >> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf >> > > Should I take that to mean that "allow_unsafe_interrupts" is actually > a security feature??? Is this the discussed "interrupt remapping" in > the whitepaper? I am interpreting that paper as saying that this > interrupt remapping does *not* use the supplied DMAR table. Is that > correct? > >> Also, you can try to upgrade your BIOS to fix it. > > I'll take a look but I think I have the latest (which means, from > ~2011 probably) BIOS version. > > Could I also ask you outright what entire set of boot options you > would pass when booting into a kvm system with IOMMU enabled? I would > love some "default" set of boot options to compare to, as opposed to > random ones from assorted forums. > > Thank you for the prompt reply! > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > On 6/11/2016 9:46 PM, Yang Zhang wrote: > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-12 2:54 ` Steve Novakov @ 2016-06-12 2:57 ` Steve Novakov 0 siblings, 0 replies; 20+ messages in thread From: Steve Novakov @ 2016-06-12 2:57 UTC (permalink / raw) To: Yang Zhang, kvm Sorry, I was thinking of kernel modules and said packages (virtio-X are kernel modules) . Thanks, Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/11/2016 10:54 PM, Steve Novakov wrote: > Hello Yang, > > To add to this, a list of mandatory packages would be helpful as well > (like if I need some additional virtio- packages). Thank you, > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > > On 6/11/2016 9:55 PM, Steve Novakov wrote: >> Hello Yang, >> >>> allow_unsafe_interupts actually means the interrupt remapping on >>> Intel IOMMU which is a security feature. Without it, a malicious VM >>> can attack the host, see below document for more details: >>> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf >>> >> >> Should I take that to mean that "allow_unsafe_interrupts" is actually >> a security feature??? Is this the discussed "interrupt remapping" in >> the whitepaper? I am interpreting that paper as saying that this >> interrupt remapping does *not* use the supplied DMAR table. Is that >> correct? >> >>> Also, you can try to upgrade your BIOS to fix it. >> >> I'll take a look but I think I have the latest (which means, from >> ~2011 probably) BIOS version. >> >> Could I also ask you outright what entire set of boot options you >> would pass when booting into a kvm system with IOMMU enabled? I would >> love some "default" set of boot options to compare to, as opposed to >> random ones from assorted forums. >> >> Thank you for the prompt reply! >> >> Steve Novakov >> B.A.Sc Engineering Physics >> PhD Student - Physics >> University of Michigan - Ann Arbor >> On 6/11/2016 9:46 PM, Yang Zhang wrote: >> > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-12 1:55 ` Steve Novakov 2016-06-12 2:54 ` Steve Novakov @ 2016-06-13 1:46 ` Yang Zhang 2016-06-13 3:32 ` Steve Novakov 1 sibling, 1 reply; 20+ messages in thread From: Yang Zhang @ 2016-06-13 1:46 UTC (permalink / raw) To: Steve Novakov, kvm On 2016/6/12 9:55, Steve Novakov wrote: > Hello Yang, > >> allow_unsafe_interupts actually means the interrupt remapping on Intel >> IOMMU which is a security feature. Without it, a malicious VM can >> attack the host, see below document for more details: >> http://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf >> > > Should I take that to mean that "allow_unsafe_interrupts" is actually a > security feature??? Is this the discussed "interrupt remapping" in the Interrupt remapping not only a security feature, it also supports more than 255 CPUs associate with x2apic. allow_unsafe_interrupts allows you to enable IOMMU on the platform even without interrupt remapping because first platform supporting IOMMU doesn't have interrupt remapping. > whitepaper? I am interpreting that paper as saying that this interrupt > remapping does *not* use the supplied DMAR table. Is that correct? All the necessary information for IOMMU is located in ACPI tables not only DMAR table. OS need to parse it before enabling the IOMMU. > >> Also, you can try to upgrade your BIOS to fix it. > > I'll take a look but I think I have the latest (which means, from ~2011 > probably) BIOS version. > > Could I also ask you outright what entire set of boot options you would > pass when booting into a kvm system with IOMMU enabled? I would love > some "default" set of boot options to compare to, as opposed to random > ones from assorted forums. Usually, i am using intel_iommu=on and everything works well. But in your case, i guess you may also need intremap=off. > > Thank you for the prompt reply! > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > On 6/11/2016 9:46 PM, Yang Zhang wrote: > -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-13 1:46 ` Yang Zhang @ 2016-06-13 3:32 ` Steve Novakov 2016-06-13 20:11 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-13 3:32 UTC (permalink / raw) To: Yang Zhang, kvm > Usually, i am using intel_iommu=on and everything works well. But in your case, i guess you may also need intremap=off. Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a freeze at boot. This happens at the loading initramfs step ("initrd /boot/initramfs-linux.img"). Any further suggestions? Thanks, Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-13 3:32 ` Steve Novakov @ 2016-06-13 20:11 ` Steve Novakov 2016-06-14 22:00 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-13 20:11 UTC (permalink / raw) To: Yang Zhang, kvm Interestingly, when I disable VT-D in the motherboard BIOS. I can boot with "intel_iommu=on pci-stub.ids=DEVICEID" and the DEVICE is successfully loaded by the pci-stub driver, according to $lspci -nnk readout additionally, I see: $dmesg | grep -e IOMMU [0.000000] DMAR: IOMMU enabled even though I actually DISABLED VT-D in the motherboard BIOS. Going back now and enabling it in the BIOS causes freezing during boot again. I'm also having trouble assigning this device to the vfio-pci driver, but that might be a separate issue. Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/12/2016 11:32 PM, Steve Novakov wrote: > > Usually, i am using intel_iommu=on and everything works well. But in > your case, i guess you may also need intremap=off. > > Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on > vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a > freeze at boot. This happens at the loading initramfs step ("initrd > /boot/initramfs-linux.img"). > > Any further suggestions? Thanks, > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > > > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-13 20:11 ` Steve Novakov @ 2016-06-14 22:00 ` Steve Novakov 2016-06-15 5:04 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-14 22:00 UTC (permalink / raw) To: Yang Zhang, kvm Some more info here from someone having exactly the same problem I am and at exactly the same stage http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605 Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/13/2016 4:11 PM, Steve Novakov wrote: > Interestingly, when I disable VT-D in the motherboard BIOS. I can boot > with "intel_iommu=on pci-stub.ids=DEVICEID" > > and the DEVICE is successfully loaded by the pci-stub driver, > according to $lspci -nnk readout > > additionally, I see: > > $dmesg | grep -e IOMMU > > [0.000000] DMAR: IOMMU enabled > > even though I actually DISABLED VT-D in the motherboard BIOS. Going > back now and enabling it in the BIOS causes freezing during boot again. > > I'm also having trouble assigning this device to the vfio-pci driver, > but that might be a separate issue. > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > On 6/12/2016 11:32 PM, Steve Novakov wrote: >> > Usually, i am using intel_iommu=on and everything works well. But >> in your case, i guess you may also need intremap=off. >> >> Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on >> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a >> freeze at boot. This happens at the loading initramfs step ("initrd >> /boot/initramfs-linux.img"). >> >> Any further suggestions? Thanks, >> >> Steve Novakov >> B.A.Sc Engineering Physics >> PhD Student - Physics >> University of Michigan - Ann Arbor >> >> >> > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-14 22:00 ` Steve Novakov @ 2016-06-15 5:04 ` Steve Novakov 2016-06-15 6:59 ` Paolo Bonzini 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-15 5:04 UTC (permalink / raw) To: Yang Zhang, kvm So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg: [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on a chipset that contains an erratum making that feature unstable. To maintain system stability interrupt remapping is being disabled.please contact your BIOS vendor for an update. This line was actually present while using arch as well I just never took the time to read past the first line (which just says enabled interrupt remapping). This seems to be an issue with northbridge stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ). I wonder if it's even possible to get this to work with my motherboard now....**though ESXi can manage just fine...** Maybe there's some way to trick linux into just trying to do it. I mean, it seems to pass devices to pci-stub just fine. Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/14/2016 6:00 PM, Steve Novakov wrote: > Some more info here from someone having exactly the same problem I am > and at exactly the same stage > > http://www.tomshardware.com/answers/id-3049002/socket-1366-iomuu-compatibility-decent.html#18122605 > > > Steve Novakov > B.A.Sc Engineering Physics > PhD Student - Physics > University of Michigan - Ann Arbor > On 6/13/2016 4:11 PM, Steve Novakov wrote: >> Interestingly, when I disable VT-D in the motherboard BIOS. I can >> boot with "intel_iommu=on pci-stub.ids=DEVICEID" >> >> and the DEVICE is successfully loaded by the pci-stub driver, >> according to $lspci -nnk readout >> >> additionally, I see: >> >> $dmesg | grep -e IOMMU >> >> [0.000000] DMAR: IOMMU enabled >> >> even though I actually DISABLED VT-D in the motherboard BIOS. Going >> back now and enabling it in the BIOS causes freezing during boot again. >> >> I'm also having trouble assigning this device to the vfio-pci driver, >> but that might be a separate issue. >> >> Steve Novakov >> B.A.Sc Engineering Physics >> PhD Student - Physics >> University of Michigan - Ann Arbor >> On 6/12/2016 11:32 PM, Steve Novakov wrote: >>> > Usually, i am using intel_iommu=on and everything works well. But >>> in your case, i guess you may also need intremap=off. >>> >>> Passing EITHER "intel_iommu=on intremap=off" OR " "intel_iommu=on >>> vfio_iommu_type1.allow_unsafe_interrupts=1" during boot results in a >>> freeze at boot. This happens at the loading initramfs step ("initrd >>> /boot/initramfs-linux.img"). >>> >>> Any further suggestions? Thanks, >>> >>> Steve Novakov >>> B.A.Sc Engineering Physics >>> PhD Student - Physics >>> University of Michigan - Ann Arbor >>> >>> >>> >> > ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-15 5:04 ` Steve Novakov @ 2016-06-15 6:59 ` Paolo Bonzini 2016-06-15 15:40 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Paolo Bonzini @ 2016-06-15 6:59 UTC (permalink / raw) To: Steve Novakov, Yang Zhang, kvm On 15/06/2016 07:04, Steve Novakov wrote: > So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg: > > [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on > a chipset that contains an erratum making that feature unstable. To > maintain system stability interrupt remapping is being disabled.please > contact your BIOS vendor for an update. > > This line was actually present while using arch as well I just never > took the time to read past the first line (which just says enabled > interrupt remapping). This seems to be an issue with northbridge > stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ). > > I wonder if it's even possible to get this to work with my motherboard > now....**though ESXi can manage just fine...** > > Maybe there's some way to trick linux into just trying to do it. I mean, > it seems to pass devices to pci-stub just fine. Passing devices to pci-stub doesn't require the IOMMU, it's just swapping the driver that binds to the device. I think this is why with VT-d disabled you cannot assign the device to vfio-pci. Does it still freeze with Ubuntu 16.04 (VT-d enabled, vfio_iommu_type1.allow_unsafe_interrupts=1)? Thanks, Paolo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-15 6:59 ` Paolo Bonzini @ 2016-06-15 15:40 ` Steve Novakov 2016-06-16 1:19 ` Yang Zhang 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-15 15:40 UTC (permalink / raw) To: Paolo Bonzini, Yang Zhang, kvm Hello Paolo, thank you for joining in, Yeah absolutely, it freezes no matter what I do. I get all sorts of interesting error messages during boot. Dmesg goes to like [64.4XXXX] or higher and spits out random kernel functions, addresses, etc. then freezes. I've tried intel_iommu=on intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 all sorts of combinations. I've verified that all of the relevant modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that "DMAR-IR" message is suspicious though, no? That there is some setting in the kernel that disables remapping by default for my BIOS. I've seen posts with working vt-d even with an i7-950. I wonder if they have the C3 stepping of the northbridge. Does that sound like it might be an issue? I can't believe that there is not some way to make this work, as it works flawlessly with ESXi (which sits on top of a linux kernel, though it's vmware's proprietary fork I think). If my system boots in "Plug and play" mode, the BIOS configuration for non critical devices shouldn't matter, right? Steve Novakov B.A.Sc. Engineering Physics PhD Student - Physics University of Michigan On 6/15/2016 2:59 AM, Paolo Bonzini wrote: > > On 15/06/2016 07:04, Steve Novakov wrote: >> So I swapped over to ubuntu 16.04 server and I just saw this line in dmesg: >> >> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on >> a chipset that contains an erratum making that feature unstable. To >> maintain system stability interrupt remapping is being disabled.please >> contact your BIOS vendor for an update. >> >> This line was actually present while using arch as well I just never >> took the time to read past the first line (which just says enabled >> interrupt remapping). This seems to be an issue with northbridge >> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ). >> >> I wonder if it's even possible to get this to work with my motherboard >> now....**though ESXi can manage just fine...** >> >> Maybe there's some way to trick linux into just trying to do it. I mean, >> it seems to pass devices to pci-stub just fine. > Passing devices to pci-stub doesn't require the IOMMU, it's just > swapping the driver that binds to the device. I think this is why with > VT-d disabled you cannot assign the device to vfio-pci. > > Does it still freeze with Ubuntu 16.04 (VT-d enabled, > vfio_iommu_type1.allow_unsafe_interrupts=1)? > > Thanks, > > Paolo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-15 15:40 ` Steve Novakov @ 2016-06-16 1:19 ` Yang Zhang 2016-06-16 1:22 ` Steve Novakov 2016-06-16 7:17 ` Paolo Bonzini 0 siblings, 2 replies; 20+ messages in thread From: Yang Zhang @ 2016-06-16 1:19 UTC (permalink / raw) To: Steve Novakov, Paolo Bonzini, kvm On 2016/6/15 23:40, Steve Novakov wrote: > Hello Paolo, thank you for joining in, > > Yeah absolutely, it freezes no matter what I do. I get all sorts of > interesting error messages during boot. Dmesg goes to like [64.4XXXX] or > higher and spits out random kernel functions, addresses, etc. then > freezes. I've tried > > intel_iommu=on > intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 > > all sorts of combinations. I've verified that all of the relevant > modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that > "DMAR-IR" message is suspicious though, no? That there is some setting > in the kernel that disables remapping by default for my BIOS. I've seen > posts with working vt-d even with an i7-950. I wonder if they have the > C3 stepping of the northbridge. Does that sound like it might be an issue? > > I can't believe that there is not some way to make this work, as it > works flawlessly with ESXi (which sits on top of a linux kernel, though > it's vmware's proprietary fork I think). If my system boots in "Plug and > play" mode, the BIOS configuration for non critical devices shouldn't > matter, right? It seems there are something wrong with vfio. Have you tried with legacy device pass through? It should be disabled on new distribution(like ubuntu 16.04) but RHEL6 may still support it. > > Steve Novakov > B.A.Sc. Engineering Physics > PhD Student - Physics > University of Michigan > On 6/15/2016 2:59 AM, Paolo Bonzini wrote: >> >> On 15/06/2016 07:04, Steve Novakov wrote: >>> So I swapped over to ubuntu 16.04 server and I just saw this line in >>> dmesg: >>> >>> [0.100541] DMAR-IR: This system BIOS has enabled interrupt remapping on >>> a chipset that contains an erratum making that feature unstable. To >>> maintain system stability interrupt remapping is being disabled.please >>> contact your BIOS vendor for an update. >>> >>> This line was actually present while using arch as well I just never >>> took the time to read past the first line (which just says enabled >>> interrupt remapping). This seems to be an issue with northbridge >>> stepping B3, and was apparently fixed in stepping C2 ( I have B3 : ( ). >>> >>> I wonder if it's even possible to get this to work with my motherboard >>> now....**though ESXi can manage just fine...** >>> >>> Maybe there's some way to trick linux into just trying to do it. I mean, >>> it seems to pass devices to pci-stub just fine. >> Passing devices to pci-stub doesn't require the IOMMU, it's just >> swapping the driver that binds to the device. I think this is why with >> VT-d disabled you cannot assign the device to vfio-pci. >> >> Does it still freeze with Ubuntu 16.04 (VT-d enabled, >> vfio_iommu_type1.allow_unsafe_interrupts=1)? >> >> Thanks, >> >> Paolo > -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-16 1:19 ` Yang Zhang @ 2016-06-16 1:22 ` Steve Novakov 2016-06-16 7:17 ` Paolo Bonzini 1 sibling, 0 replies; 20+ messages in thread From: Steve Novakov @ 2016-06-16 1:22 UTC (permalink / raw) To: Yang Zhang, Paolo Bonzini, kvm > It seems there are something wrong with vfio. Have you tried with legacy device pass through? It should be disabled on new distribution(like ubuntu 16.04) but RHEL6 may still support it. Hello Yang, would you mind elaborating on this? I should get RHEL 6.8? What is the "legacy pass through"? Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/15/2016 9:19 PM, Yang Zhang wrote: ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-16 1:19 ` Yang Zhang 2016-06-16 1:22 ` Steve Novakov @ 2016-06-16 7:17 ` Paolo Bonzini 2016-06-20 2:07 ` Yang Zhang 1 sibling, 1 reply; 20+ messages in thread From: Paolo Bonzini @ 2016-06-16 7:17 UTC (permalink / raw) To: Yang Zhang, Steve Novakov, kvm On 16/06/2016 03:19, Yang Zhang wrote: >> intel_iommu=on >> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 >> >> all sorts of combinations. I've verified that all of the relevant >> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that >> "DMAR-IR" message is suspicious though, no? That there is some setting >> in the kernel that disables remapping by default for my BIOS. I've seen >> posts with working vt-d even with an i7-950. I wonder if they have the >> C3 stepping of the northbridge. Does that sound like it might be an >> issue? >> >> I can't believe that there is not some way to make this work, as it >> works flawlessly with ESXi (which sits on top of a linux kernel, though >> it's vmware's proprietary fork I think). If my system boots in "Plug and >> play" mode, the BIOS configuration for non critical devices shouldn't >> matter, right? > > It seems there are something wrong with vfio. Have you tried with legacy > device pass through? It should be disabled on new distribution(like > ubuntu 16.04) but RHEL6 may still support it. There shouldn't be any difference. He gets a freeze with intel_iommu=on, even before going to device assignment. Paolo ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-16 7:17 ` Paolo Bonzini @ 2016-06-20 2:07 ` Yang Zhang 2016-06-20 4:05 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Yang Zhang @ 2016-06-20 2:07 UTC (permalink / raw) To: Paolo Bonzini, Steve Novakov, kvm On 2016/6/16 15:17, Paolo Bonzini wrote: > > > On 16/06/2016 03:19, Yang Zhang wrote: >>> intel_iommu=on >>> intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 >>> >>> all sorts of combinations. I've verified that all of the relevant >>> modules load (vfio* kvm kvm_intel pci_stub) through dmesg. That that >>> "DMAR-IR" message is suspicious though, no? That there is some setting >>> in the kernel that disables remapping by default for my BIOS. I've seen >>> posts with working vt-d even with an i7-950. I wonder if they have the >>> C3 stepping of the northbridge. Does that sound like it might be an >>> issue? >>> >>> I can't believe that there is not some way to make this work, as it >>> works flawlessly with ESXi (which sits on top of a linux kernel, though >>> it's vmware's proprietary fork I think). If my system boots in "Plug and >>> play" mode, the BIOS configuration for non critical devices shouldn't >>> matter, right? >> >> It seems there are something wrong with vfio. Have you tried with legacy >> device pass through? It should be disabled on new distribution(like >> ubuntu 16.04) but RHEL6 may still support it. > > There shouldn't be any difference. He gets a freeze with > intel_iommu=on, even before going to device assignment. Yes, if system freeze on boot, then it seems something wrong in linux kernel. Steve, can you post the dmesg log and check whether there are some additional devices on your board? I remember some legacy device drivers are not friendly to IOMMU. -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-20 2:07 ` Yang Zhang @ 2016-06-20 4:05 ` Steve Novakov [not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com> 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-20 4:05 UTC (permalink / raw) To: Yang Zhang, Paolo Bonzini, kvm > Yes, if system freeze on boot, then it seems something wrong in linux kernel. Steve, can you post the dmesg log and check whether there are some additional devices on your board? I remember some legacy device drivers are not friendly to IOMMU. I kind of gave up on this and moved back to ESXi, though I am now fighting HID device passthrough >___> (prosumer virtualization tech not as robust as I imagined....). ESXi is passing through USB controllers (I get non HID device, like a USB-ethernet adapter passthrough no problem) and an Intel PCIe NIC, but I'm having issues with graphics cards. Tough to say whether it's a driver or X server issue at the moment, the cards are seemingly passed through in ESXi. I have an NVIDIA GT 730 and Radeon HD5450 to play with. I'm worried a bit about the replacement of fglrx and the use of "radeon" module in newer kernels, and how it deals with passthrough, etc, with such an old card, but we'll see how it goes. I'm also fighting being able to show the VM console and a graphics card output at the same time (that's an X server configuration thing I think). I can support some debugging though. I'll reinstall Ubuntu 16.04 server on a spare drive and get it back to the state it was in. I'll send the entire dmesg, lspci -nnk from just a base install. Should I also give you dumps with "intel_iommu=on" and VT-d disabled in the BIOS? (I can't boot if both things are on) Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/19/2016 10:07 PM, Yang Zhang wrote: ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com>]
* Re: X58 Virtualization w/ Linux [not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com> @ 2016-06-21 1:21 ` Yang Zhang 2016-06-21 1:24 ` Steve Novakov 0 siblings, 1 reply; 20+ messages in thread From: Yang Zhang @ 2016-06-21 1:21 UTC (permalink / raw) To: Steve Novakov, Paolo Bonzini, kvm On 2016/6/21 1:50, Steve Novakov wrote: >> I can support some debugging though. I'll reinstall Ubuntu 16.04 > server on a spare drive and get it back to the state it was in. I'll > send the entire dmesg, lspci -nnk from just a base install. Should I > also give you dumps with "intel_iommu=on" and VT-d disabled in the BIOS? > (I can't boot if both things are on) > > I've attached a zip file with some dumps. There is a text file in there > called "install_sequence.txt" that explains important steps and what > each of the dump files are from. There is no error message from the dmesg. But there do have lots of additional devices on your board. Do you see any error when booting fail? Does system stuck while enabling a device? -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-21 1:21 ` Yang Zhang @ 2016-06-21 1:24 ` Steve Novakov 2016-06-27 2:21 ` Yang Zhang 0 siblings, 1 reply; 20+ messages in thread From: Steve Novakov @ 2016-06-21 1:24 UTC (permalink / raw) To: Yang Zhang, Paolo Bonzini, kvm >There is no error message from the dmesg. But there do have lots of additional devices on your board. Do you see any error when booting fail? Does system stuck while enabling a device? All I see now is a completely blank screen right after the grub screen (with BIOS VT-d on, and with intel_iommu=on..etc boot options). There seems to be no logging happening (syslog has no entries for these boots). The blank screen persists and there is no response from any inputs. I waited 3-5 minutes and then rebooted manually. Steve Novakov B.A.Sc Engineering Physics PhD Student - Physics University of Michigan - Ann Arbor On 6/20/2016 9:21 PM, Yang Zhang wrote: ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: X58 Virtualization w/ Linux 2016-06-21 1:24 ` Steve Novakov @ 2016-06-27 2:21 ` Yang Zhang 0 siblings, 0 replies; 20+ messages in thread From: Yang Zhang @ 2016-06-27 2:21 UTC (permalink / raw) To: Steve Novakov, Paolo Bonzini, kvm On 2016/6/21 9:24, Steve Novakov wrote: >>There is no error message from the dmesg. But there do have lots of > additional devices on your board. Do you see any error when booting > fail? Does system stuck while enabling a device? > > > All I see now is a completely blank screen right after the grub screen > (with BIOS VT-d on, and with intel_iommu=on..etc boot options). There > seems to be no logging happening (syslog has no entries for these boots). > > The blank screen persists and there is no response from any inputs. I > waited 3-5 minutes and then rebooted manually. You may need to config the serial port to get the boot message. Without enough message, it's hard to tell what is wrong. Alternatively, you can remove all additional devices to see whether system can boot. -- best regards yang ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2016-06-27 2:21 UTC | newest] Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-06-11 21:34 X58 Virtualization w/ Linux Steve Novakov 2016-06-12 1:46 ` Yang Zhang 2016-06-12 1:55 ` Steve Novakov 2016-06-12 2:54 ` Steve Novakov 2016-06-12 2:57 ` Steve Novakov 2016-06-13 1:46 ` Yang Zhang 2016-06-13 3:32 ` Steve Novakov 2016-06-13 20:11 ` Steve Novakov 2016-06-14 22:00 ` Steve Novakov 2016-06-15 5:04 ` Steve Novakov 2016-06-15 6:59 ` Paolo Bonzini 2016-06-15 15:40 ` Steve Novakov 2016-06-16 1:19 ` Yang Zhang 2016-06-16 1:22 ` Steve Novakov 2016-06-16 7:17 ` Paolo Bonzini 2016-06-20 2:07 ` Yang Zhang 2016-06-20 4:05 ` Steve Novakov [not found] ` <97e6a96d-4139-2469-a8c9-f79df48727a6@stevenovakov.com> 2016-06-21 1:21 ` Yang Zhang 2016-06-21 1:24 ` Steve Novakov 2016-06-27 2:21 ` Yang Zhang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.