All of lore.kernel.org
 help / color / mirror / Atom feed
From: Shawn Anastasio <shawn@anastas.io>
To: Alexey Kardashevskiy <aik@ozlabs.ru>,
	linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org
Cc: bhelgaas@google.com, oohall@gmail.com, sbobroff@linux.ibm.com
Subject: Re: [PATCH 0/2] Fix IOMMU setup for hotplugged devices on pseries
Date: Thu, 5 Sep 2019 12:59:44 -0500	[thread overview]
Message-ID: <7fb9fb72-2afd-451c-1411-54c7bb865d56@anastas.io> (raw)
In-Reply-To: <7a41184c-9b30-8d91-9d78-9d60c8d128ef@ozlabs.ru>

On 9/5/19 4:08 AM, Alexey Kardashevskiy wrot>
> I just tried hotplugging 3 virtio-net devices into a guest system with 
> v5.2 kernel and it seems working (i.e. BARs mapped, a driver is bound):
>
> 
> root@le-dbg:~# lspci -v | egrep -i '(virtio|Memory)'
> 00:00.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080040000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000000000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 00:01.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080041000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000004000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 00:02.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080042000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000008000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 
> Can you explain in detail what you are doing exactly and what is failing 
> and what qemu/guest kernel/guest distro is used? Thanks,

Sure. I'm on host kernel 5.2.8, guest on 5.3-rc7 (also tested on 5.1.16)
and I'm hotplugging ivshmem devices to a separate spapr-pci-host-bridge
defined as follows:

-device spapr-pci-host-bridge,index=1,id=pci.1

Device hotplug and BAR assignment does work, but IOMMU group assignment
seems to fail. This is evidenced by the kernel log which shows the
following message for the first device but not the second:

[  136.849448] pci 0001:00:00.0: Adding to iommu group 1

Trying to bind the second device to vfio-pci as a result of this
fails:

[  471.691948] vfio-pci: probe of 0001:00:01.0 failed with error -22

I traced that failure to a call to iommu_group_get() which returns
NULL for the second device. I then traced that back to the ordering
issue I described.

For your second and third virtio-net devices, was the
"Adding to iommu group N" kernel message printed?

WARNING: multiple messages have this Message-ID (diff)
From: Shawn Anastasio <shawn@anastas.io>
To: Alexey Kardashevskiy <aik@ozlabs.ru>,
	linux-pci@vger.kernel.org, linuxppc-dev@lists.ozlabs.org
Cc: bhelgaas@google.com, sbobroff@linux.ibm.com, oohall@gmail.com
Subject: Re: [PATCH 0/2] Fix IOMMU setup for hotplugged devices on pseries
Date: Thu, 5 Sep 2019 12:59:44 -0500	[thread overview]
Message-ID: <7fb9fb72-2afd-451c-1411-54c7bb865d56@anastas.io> (raw)
In-Reply-To: <7a41184c-9b30-8d91-9d78-9d60c8d128ef@ozlabs.ru>

On 9/5/19 4:08 AM, Alexey Kardashevskiy wrot>
> I just tried hotplugging 3 virtio-net devices into a guest system with 
> v5.2 kernel and it seems working (i.e. BARs mapped, a driver is bound):
>
> 
> root@le-dbg:~# lspci -v | egrep -i '(virtio|Memory)'
> 00:00.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080040000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000000000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 00:01.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080041000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000004000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 00:02.0 Ethernet controller: Red Hat, Inc Virtio network device
>          Memory at 200080042000 (32-bit, non-prefetchable) [size=4K]
>          Memory at 210000008000 (64-bit, prefetchable) [size=16K]
>          Kernel driver in use: virtio-pci
> 
> Can you explain in detail what you are doing exactly and what is failing 
> and what qemu/guest kernel/guest distro is used? Thanks,

Sure. I'm on host kernel 5.2.8, guest on 5.3-rc7 (also tested on 5.1.16)
and I'm hotplugging ivshmem devices to a separate spapr-pci-host-bridge
defined as follows:

-device spapr-pci-host-bridge,index=1,id=pci.1

Device hotplug and BAR assignment does work, but IOMMU group assignment
seems to fail. This is evidenced by the kernel log which shows the
following message for the first device but not the second:

[  136.849448] pci 0001:00:00.0: Adding to iommu group 1

Trying to bind the second device to vfio-pci as a result of this
fails:

[  471.691948] vfio-pci: probe of 0001:00:01.0 failed with error -22

I traced that failure to a call to iommu_group_get() which returns
NULL for the second device. I then traced that back to the ordering
issue I described.

For your second and third virtio-net devices, was the
"Adding to iommu group N" kernel message printed?

  reply	other threads:[~2019-09-05 17:59 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-05  4:22 [PATCH 0/2] Fix IOMMU setup for hotplugged devices on pseries Shawn Anastasio
2019-09-05  4:22 ` Shawn Anastasio
2019-09-05  4:22 ` [PATCH 1/2] PCI: Introduce pcibios_fixup_dev() Shawn Anastasio
2019-09-05  4:22   ` Shawn Anastasio
2019-09-05  4:22 ` [PATCH 2/2] powerpc/pci: Fix IOMMU setup for hotplugged devices on pseries Shawn Anastasio
2019-09-05  4:22   ` Shawn Anastasio
2019-09-05  9:08 ` [PATCH 0/2] " Alexey Kardashevskiy
2019-09-05  9:08   ` Alexey Kardashevskiy
2019-09-05 17:59   ` Shawn Anastasio [this message]
2019-09-05 17:59     ` Shawn Anastasio
2019-09-05  9:38 ` Lukas Wunner
2019-09-05 18:42   ` Shawn Anastasio
2019-09-05 18:42     ` Shawn Anastasio

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7fb9fb72-2afd-451c-1411-54c7bb865d56@anastas.io \
    --to=shawn@anastas.io \
    --cc=aik@ozlabs.ru \
    --cc=bhelgaas@google.com \
    --cc=linux-pci@vger.kernel.org \
    --cc=linuxppc-dev@lists.ozlabs.org \
    --cc=oohall@gmail.com \
    --cc=sbobroff@linux.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.