Issue with Ath9k/PCI passthrough

Issue with Ath9k/PCI passthrough

[James Prestwood]

Hi,

I am passing through PCI wireless adapters into a qemu VM and I am
seeing my host machine lock up/freeze when starting qemu if I try and
pass through an Atheros AR5B22 PCI card. After reboot I don't see
anything suspicious in /var/log/kern.log, although I don't really know
what to look for either (or maybe there is another log to look at?). I
have successfully done PCI passthrough with both an Intel 7260 and
3160. Its whenever I add the Atheros card into the mix (or by itself) I
get this lockup when starting the VM.

I have enabled the Ath9k drivers when building the kernel (same as with
Intel cards). This page I read online (https://wiki.debian.org/ath9k)
said the Ath9k cards don't require firmware like the Intel cards do, so
I have not added any firmware binaries for this card into the kernel
build. I also tried turning on the Ath9k debugging but saw no
additional prints in kern.log.

With PCI passthrough there is some configuration required, like
substituting the drivers for the vfio-pci driver on the host machine,
so it could be completely unrelated to the Ath9k driver. Still, I was
hoping that someone more knowledgeable than me may know whats going on,
or at least where to look. The fact that the Intel cards work fine was
what made me think it could be a driver problem.

I am more or less following this guide (except with wifi adapters
rather than GPU):
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Using a custom built kernel based on 4.19.

Thanks,
James

Re: Issue with Ath9k/PCI passthrough

[Christian Lamparter]

On Monday, January 7, 2019 6:55:48 PM CET James Prestwood wrote:
> Hi,
> 
> I am passing through PCI wireless adapters into a qemu VM and I am
> seeing my host machine lock up/freeze when starting qemu if I try and
> pass through an Atheros AR5B22 PCI card. After reboot I don't see
> anything suspicious in /var/log/kern.log, although I don't really know
> what to look for either (or maybe there is another log to look at?). I
> have successfully done PCI passthrough with both an Intel 7260 and
> 3160. Its whenever I add the Atheros card into the mix (or by itself) I
> get this lockup when starting the VM.
> 
> I have enabled the Ath9k drivers when building the kernel (same as with
> Intel cards). This page I read online (https://wiki.debian.org/ath9k)
> said the Ath9k cards don't require firmware like the Intel cards do, so
> I have not added any firmware binaries for this card into the kernel
> build. I also tried turning on the Ath9k debugging but saw no
> additional prints in kern.log.
> 
> With PCI passthrough there is some configuration required, like
> substituting the drivers for the vfio-pci driver on the host machine,
> so it could be completely unrelated to the Ath9k driver. Still, I was
> hoping that someone more knowledgeable than me may know whats going on,
> or at least where to look. The fact that the Intel cards work fine was
> what made me think it could be a driver problem.
> 
> I am more or less following this guide (except with wifi adapters
> rather than GPU):
> https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> 

You could be a victim of:

|/*
| * Some Atheros AR9xxx and QCA988x chips do not behave after a bus reset.
| * The device will throw a Link Down error on AER-capable systems and
| * regardless of AER, config space of the device is never accessible again
| * and typically causes the system to hang or reset when access is attempted.
| * http://www.spinics.net/lists/linux-pci/msg34797.html
| */
|DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030, quirk_no_bus_reset);
|DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032, quirk_no_bus_reset);
|DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c, quirk_no_bus_reset);
|DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033, quirk_no_bus_reset);
<https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3400>

I know that the AR93xx and AR94xx cards have problems with PCIe Passthrough:
<https://patchwork.kernel.org/patch/9141239/> and
<https://www.spinics.net/lists/linux-pci/msg34797.html> so maybe you can get
it to work, once you add a entry to the fixup list.

Regards,
Christian

Re: Issue with Ath9k/PCI passthrough

[James Prestwood]

On Mon, 2019-01-07 at 19:24 +0100, Christian Lamparter wrote:
> On Monday, January 7, 2019 6:55:48 PM CET James Prestwood wrote:
> > Hi,
> > 
> > I am passing through PCI wireless adapters into a qemu VM and I am
> > seeing my host machine lock up/freeze when starting qemu if I try
> > and
> > pass through an Atheros AR5B22 PCI card. After reboot I don't see
> > anything suspicious in /var/log/kern.log, although I don't really
> > know
> > what to look for either (or maybe there is another log to look
> > at?). I
> > have successfully done PCI passthrough with both an Intel 7260 and
> > 3160. Its whenever I add the Atheros card into the mix (or by
> > itself) I
> > get this lockup when starting the VM.
> > 
> > I have enabled the Ath9k drivers when building the kernel (same as
> > with
> > Intel cards). This page I read online (
> > https://wiki.debian.org/ath9k)
> > said the Ath9k cards don't require firmware like the Intel cards
> > do, so
> > I have not added any firmware binaries for this card into the
> > kernel
> > build. I also tried turning on the Ath9k debugging but saw no
> > additional prints in kern.log.
> > 
> > With PCI passthrough there is some configuration required, like
> > substituting the drivers for the vfio-pci driver on the host
> > machine,
> > so it could be completely unrelated to the Ath9k driver. Still, I
> > was
> > hoping that someone more knowledgeable than me may know whats going
> > on,
> > or at least where to look. The fact that the Intel cards work fine
> > was
> > what made me think it could be a driver problem.
> > 
> > I am more or less following this guide (except with wifi adapters
> > rather than GPU):
> > https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> > 
> 
> You could be a victim of:
> 
> > /*
> > * Some Atheros AR9xxx and QCA988x chips do not behave after a bus
> > reset.
> > * The device will throw a Link Down error on AER-capable systems
> > and
> > * regardless of AER, config space of the device is never accessible
> > again
> > * and typically causes the system to hang or reset when access is
> > attempted.
> > * http://www.spinics.net/lists/linux-pci/msg34797.html
> > */
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030,
> > quirk_no_bus_reset);
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032,
> > quirk_no_bus_reset);
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c,
> > quirk_no_bus_reset);
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033,
> > quirk_no_bus_reset);

Adding this to my guest kernel and rebuilding didn't change anything:

DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034,
quirk_no_bus_reset);

But would I need to make this change on the host kernel as well?

> 
> <
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3400
> >
> 
> I know that the AR93xx and AR94xx cards have problems with PCIe
> Passthrough:
> <https://patchwork.kernel.org/patch/9141239/> and
> <https://www.spinics.net/lists/linux-pci/msg34797.html> so maybe you
> can get
> it to work, once you add a entry to the fixup list.
> 
> Regards,
> Christian
> 
> 

Re: Issue with Ath9k/PCI passthrough

[Christian Lamparter]

On Monday, January 7, 2019 7:53:49 PM CET James Prestwood wrote:
> On Mon, 2019-01-07 at 19:24 +0100, Christian Lamparter wrote:
> > On Monday, January 7, 2019 6:55:48 PM CET James Prestwood wrote:
> > > Hi,
> > > 
> > > I am passing through PCI wireless adapters into a qemu VM and I am
> > > seeing my host machine lock up/freeze when starting qemu if I try
> > > and
> > > pass through an Atheros AR5B22 PCI card. After reboot I don't see
> > > anything suspicious in /var/log/kern.log, although I don't really
> > > know
> > > what to look for either (or maybe there is another log to look
> > > at?). I
> > > have successfully done PCI passthrough with both an Intel 7260 and
> > > 3160. Its whenever I add the Atheros card into the mix (or by
> > > itself) I
> > > get this lockup when starting the VM.
> > > 
> > > I have enabled the Ath9k drivers when building the kernel (same as
> > > with
> > > Intel cards). This page I read online (
> > > https://wiki.debian.org/ath9k)
> > > said the Ath9k cards don't require firmware like the Intel cards
> > > do, so
> > > I have not added any firmware binaries for this card into the
> > > kernel
> > > build. I also tried turning on the Ath9k debugging but saw no
> > > additional prints in kern.log.
> > > 
> > > With PCI passthrough there is some configuration required, like
> > > substituting the drivers for the vfio-pci driver on the host
> > > machine,
> > > so it could be completely unrelated to the Ath9k driver. Still, I
> > > was
> > > hoping that someone more knowledgeable than me may know whats going
> > > on,
> > > or at least where to look. The fact that the Intel cards work fine
> > > was
> > > what made me think it could be a driver problem.
> > > 
> > > I am more or less following this guide (except with wifi adapters
> > > rather than GPU):
> > > https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> > > 
> > 
> > You could be a victim of:
> > 
> > > /*
> > > * Some Atheros AR9xxx and QCA988x chips do not behave after a bus
> > > reset.
> > > * The device will throw a Link Down error on AER-capable systems
> > > and
> > > * regardless of AER, config space of the device is never accessible
> > > again
> > > * and typically causes the system to hang or reset when access is
> > > attempted.
> > > * http://www.spinics.net/lists/linux-pci/msg34797.html
> > > */
> > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030,
> > > quirk_no_bus_reset);
> > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032,
> > > quirk_no_bus_reset);
> > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c,
> > > quirk_no_bus_reset);
> > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033,
> > > quirk_no_bus_reset);
> 
> Adding this to my guest kernel and rebuilding didn't change anything:
> 
> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034,
> quirk_no_bus_reset);
> 
> But would I need to make this change on the host kernel as well?

It can matter. I have a mainboard that does a bus reset before
booting the kernel and as a result I was never able to get a
ath10k/ath9k card working on the machine. The kernel would just
get suck either during decompressing or shortly after in random
places. It had no issues with intel or broadcom cards.

You should also follow the thread at least a bit:

<https://www.spinics.net/lists/linux-pci/msg34797.html>

I know Bjorn Helgaas provided instructions on how to do a bus reset
from userspace with setpci tool.

> 
> > 
> > <
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/quirks.c#n3400
> > >
> > 
> > I know that the AR93xx and AR94xx cards have problems with PCIe
> > Passthrough:
> > <https://patchwork.kernel.org/patch/9141239/> and
> > <https://www.spinics.net/lists/linux-pci/msg34797.html> so maybe you
> > can get
> > it to work, once you add a entry to the fixup list.
> > 
> > Regards,
> > Christian
> > 
> > 
> 
> 

Re: Issue with Ath9k/PCI passthrough

[James Prestwood]

On Mon, 2019-01-07 at 21:00 +0100, Christian Lamparter wrote:
> On Monday, January 7, 2019 7:53:49 PM CET James Prestwood wrote:
> > On Mon, 2019-01-07 at 19:24 +0100, Christian Lamparter wrote:
> > > On Monday, January 7, 2019 6:55:48 PM CET James Prestwood wrote:
> > > > Hi,
> > > > 
> > > > I am passing through PCI wireless adapters into a qemu VM and I
> > > > am
> > > > seeing my host machine lock up/freeze when starting qemu if I
> > > > try
> > > > and
> > > > pass through an Atheros AR5B22 PCI card. After reboot I don't
> > > > see
> > > > anything suspicious in /var/log/kern.log, although I don't
> > > > really
> > > > know
> > > > what to look for either (or maybe there is another log to look
> > > > at?). I
> > > > have successfully done PCI passthrough with both an Intel 7260
> > > > and
> > > > 3160. Its whenever I add the Atheros card into the mix (or by
> > > > itself) I
> > > > get this lockup when starting the VM.
> > > > 
> > > > I have enabled the Ath9k drivers when building the kernel (same
> > > > as
> > > > with
> > > > Intel cards). This page I read online (
> > > > https://wiki.debian.org/ath9k)
> > > > said the Ath9k cards don't require firmware like the Intel
> > > > cards
> > > > do, so
> > > > I have not added any firmware binaries for this card into the
> > > > kernel
> > > > build. I also tried turning on the Ath9k debugging but saw no
> > > > additional prints in kern.log.
> > > > 
> > > > With PCI passthrough there is some configuration required, like
> > > > substituting the drivers for the vfio-pci driver on the host
> > > > machine,
> > > > so it could be completely unrelated to the Ath9k driver. Still,
> > > > I
> > > > was
> > > > hoping that someone more knowledgeable than me may know whats
> > > > going
> > > > on,
> > > > or at least where to look. The fact that the Intel cards work
> > > > fine
> > > > was
> > > > what made me think it could be a driver problem.
> > > > 
> > > > I am more or less following this guide (except with wifi
> > > > adapters
> > > > rather than GPU):
> > > > https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> > > > 
> > > 
> > > You could be a victim of:
> > > 
> > > > /*
> > > > * Some Atheros AR9xxx and QCA988x chips do not behave after a
> > > > bus
> > > > reset.
> > > > * The device will throw a Link Down error on AER-capable
> > > > systems
> > > > and
> > > > * regardless of AER, config space of the device is never
> > > > accessible
> > > > again
> > > > * and typically causes the system to hang or reset when access
> > > > is
> > > > attempted.
> > > > * http://www.spinics.net/lists/linux-pci/msg34797.html
> > > > */
> > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030,
> > > > quirk_no_bus_reset);
> > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032,
> > > > quirk_no_bus_reset);
> > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c,
> > > > quirk_no_bus_reset);
> > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033,
> > > > quirk_no_bus_reset);
> > 
> > Adding this to my guest kernel and rebuilding didn't change
> > anything:
> > 
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034,
> > quirk_no_bus_reset);
> > 
> > But would I need to make this change on the host kernel as well?
> 
> It can matter. I have a mainboard that does a bus reset before
> booting the kernel and as a result I was never able to get a
> ath10k/ath9k card working on the machine. The kernel would just
> get suck either during decompressing or shortly after in random
> places. It had no issues with intel or broadcom cards.

Looks like I did indeed need the change in the host kernel! No lockup
and I see the Ath9k inside the VM. Thanks!

Is this something that should be upstreamed (adding the vendor ID to
that list)? If so do you know the list by chance? A quick google it
would seem https://patchwork.kernel.org/project/linux-pci/list/ is the
right place?

Thanks,
James

Re: Issue with Ath9k/PCI passthrough

[Christian Lamparter]

On Monday, January 7, 2019 9:22:59 PM CET James Prestwood wrote:
> On Mon, 2019-01-07 at 21:00 +0100, Christian Lamparter wrote:
> > On Monday, January 7, 2019 7:53:49 PM CET James Prestwood wrote:
> > > On Mon, 2019-01-07 at 19:24 +0100, Christian Lamparter wrote:
> > > > On Monday, January 7, 2019 6:55:48 PM CET James Prestwood wrote:
> > > > > Hi,
> > > > > 
> > > > > I am passing through PCI wireless adapters into a qemu VM and I
> > > > > am
> > > > > seeing my host machine lock up/freeze when starting qemu if I
> > > > > try
> > > > > and
> > > > > pass through an Atheros AR5B22 PCI card. After reboot I don't
> > > > > see
> > > > > anything suspicious in /var/log/kern.log, although I don't
> > > > > really
> > > > > know
> > > > > what to look for either (or maybe there is another log to look
> > > > > at?). I
> > > > > have successfully done PCI passthrough with both an Intel 7260
> > > > > and
> > > > > 3160. Its whenever I add the Atheros card into the mix (or by
> > > > > itself) I
> > > > > get this lockup when starting the VM.
> > > > > 
> > > > > I have enabled the Ath9k drivers when building the kernel (same
> > > > > as
> > > > > with
> > > > > Intel cards). This page I read online (
> > > > > https://wiki.debian.org/ath9k)
> > > > > said the Ath9k cards don't require firmware like the Intel
> > > > > cards
> > > > > do, so
> > > > > I have not added any firmware binaries for this card into the
> > > > > kernel
> > > > > build. I also tried turning on the Ath9k debugging but saw no
> > > > > additional prints in kern.log.
> > > > > 
> > > > > With PCI passthrough there is some configuration required, like
> > > > > substituting the drivers for the vfio-pci driver on the host
> > > > > machine,
> > > > > so it could be completely unrelated to the Ath9k driver. Still,
> > > > > I
> > > > > was
> > > > > hoping that someone more knowledgeable than me may know whats
> > > > > going
> > > > > on,
> > > > > or at least where to look. The fact that the Intel cards work
> > > > > fine
> > > > > was
> > > > > what made me think it could be a driver problem.
> > > > > 
> > > > > I am more or less following this guide (except with wifi
> > > > > adapters
> > > > > rather than GPU):
> > > > > https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> > > > > 
> > > > 
> > > > You could be a victim of:
> > > > 
> > > > > /*
> > > > > * Some Atheros AR9xxx and QCA988x chips do not behave after a
> > > > > bus
> > > > > reset.
> > > > > * The device will throw a Link Down error on AER-capable
> > > > > systems
> > > > > and
> > > > > * regardless of AER, config space of the device is never
> > > > > accessible
> > > > > again
> > > > > * and typically causes the system to hang or reset when access
> > > > > is
> > > > > attempted.
> > > > > * http://www.spinics.net/lists/linux-pci/msg34797.html
> > > > > */
> > > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030,
> > > > > quirk_no_bus_reset);
> > > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0032,
> > > > > quirk_no_bus_reset);
> > > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x003c,
> > > > > quirk_no_bus_reset);
> > > > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0033,
> > > > > quirk_no_bus_reset);
> > > 
> > > Adding this to my guest kernel and rebuilding didn't change
> > > anything:
> > > 
> > > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0034,
> > > quirk_no_bus_reset);
> > > 
> > > But would I need to make this change on the host kernel as well?
> > 
> > It can matter. I have a mainboard that does a bus reset before
> > booting the kernel and as a result I was never able to get a
> > ath10k/ath9k card working on the machine. The kernel would just
> > get suck either during decompressing or shortly after in random
> > places. It had no issues with intel or broadcom cards.
> 
> Looks like I did indeed need the change in the host kernel! No lockup
> and I see the Ath9k inside the VM. Thanks!
> 
> Is this something that should be upstreamed (adding the vendor ID to
> that list)? If so do you know the list by chance? A quick google it
> would seem https://patchwork.kernel.org/project/linux-pci/list/ is the
> right place?

That. Or if you already have the patch, you could just run it through the
scripts/get_maintainer.pl [0]

 
script