[Question] rk3399 vfio-pci/sr-iov support

[Question] rk3399 vfio-pci/sr-iov support

[Peter Geis]

Good Morning,

I'm back with more pcie fun on the rk3399.
I'm trying to get pcie passthrough working for a vm on the rk3399, and
have encountered some roadblocks.

First, vfio-pci doesn't work on the rk3399, as the pcie controller
doesn't bind explicitly to a iommu.
[37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
[37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/1/devices/ff8f0000.vop
/sys/kernel/iommu_groups/2/devices/ff900000.vop

# virsh start openwrt
error: Failed to start domain openwrt
error: internal error: Process exited prior to exec: libvirt:  error :
internal error: Invalid device 0000:01:00.0 iommu_group file
/sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink

Second, sr-iov support is broken.
root@rockpro64:/sys/bus/pci/devices/0000:01:00.0# echo 1 > sriov_numvfs
bash: echo: write error: Input/output error
[37352.907558] pci 0000:01:10.0: [8086:1520] type 7f class 0xffffff
[37352.907578] pci 0000:01:10.0: unknown header type 7f, ignoring device

Do any of y'all have some insight into these issues?

[Question] rk3399 vfio-pci/sr-iov support

[Peter Geis]

Good Morning,

I'm back with more pcie fun on the rk3399.
I'm trying to get pcie passthrough working for a vm on the rk3399, and
have encountered some roadblocks.

First, vfio-pci doesn't work on the rk3399, as the pcie controller
doesn't bind explicitly to a iommu.
[37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
[37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/1/devices/ff8f0000.vop
/sys/kernel/iommu_groups/2/devices/ff900000.vop

# virsh start openwrt
error: Failed to start domain openwrt
error: internal error: Process exited prior to exec: libvirt:  error :
internal error: Invalid device 0000:01:00.0 iommu_group file
/sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink

Second, sr-iov support is broken.
root@rockpro64:/sys/bus/pci/devices/0000:01:00.0# echo 1 > sriov_numvfs
bash: echo: write error: Input/output error
[37352.907558] pci 0000:01:10.0: [8086:1520] type 7f class 0xffffff
[37352.907578] pci 0000:01:10.0: unknown header type 7f, ignoring device

Do any of y'all have some insight into these issues?

Re: [Question] rk3399 vfio-pci/sr-iov support

[Robin Murphy]

On 09/12/2019 1:28 pm, Peter Geis wrote:
> Good Morning,
> 
> I'm back with more pcie fun on the rk3399.
> I'm trying to get pcie passthrough working for a vm on the rk3399, and
> have encountered some roadblocks.
> 
> First, vfio-pci doesn't work on the rk3399, as the pcie controller
> doesn't bind explicitly to a iommu.
> [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> 
> # find /sys/kernel/iommu_groups/ -type l
> /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> /sys/kernel/iommu_groups/2/devices/ff900000.vop
> 
> # virsh start openwrt
> error: Failed to start domain openwrt
> error: internal error: Process exited prior to exec: libvirt:  error :
> internal error: Invalid device 0000:01:00.0 iommu_group file
> /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink

That much I can help with somewhat: the major impediment is that RK3399 
doesn't have an IOMMU in front of PCIe. As far as I'm aware your only 
option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode 
(which I don't know an awful lot about beyond that it's a thing).

Robin.

> Second, sr-iov support is broken.
> root@rockpro64:/sys/bus/pci/devices/0000:01:00.0# echo 1 > sriov_numvfs
> bash: echo: write error: Input/output error
> [37352.907558] pci 0000:01:10.0: [8086:1520] type 7f class 0xffffff
> [37352.907578] pci 0000:01:10.0: unknown header type 7f, ignoring device
> 
> Do any of y'all have some insight into these issues?
> 
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip
> 

Re: [Question] rk3399 vfio-pci/sr-iov support

[Robin Murphy]

On 09/12/2019 1:28 pm, Peter Geis wrote:
> Good Morning,
> 
> I'm back with more pcie fun on the rk3399.
> I'm trying to get pcie passthrough working for a vm on the rk3399, and
> have encountered some roadblocks.
> 
> First, vfio-pci doesn't work on the rk3399, as the pcie controller
> doesn't bind explicitly to a iommu.
> [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> 
> # find /sys/kernel/iommu_groups/ -type l
> /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> /sys/kernel/iommu_groups/2/devices/ff900000.vop
> 
> # virsh start openwrt
> error: Failed to start domain openwrt
> error: internal error: Process exited prior to exec: libvirt:  error :
> internal error: Invalid device 0000:01:00.0 iommu_group file
> /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink

That much I can help with somewhat: the major impediment is that RK3399 
doesn't have an IOMMU in front of PCIe. As far as I'm aware your only 
option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode 
(which I don't know an awful lot about beyond that it's a thing).

Robin.

> Second, sr-iov support is broken.
> root@rockpro64:/sys/bus/pci/devices/0000:01:00.0# echo 1 > sriov_numvfs
> bash: echo: write error: Input/output error
> [37352.907558] pci 0000:01:10.0: [8086:1520] type 7f class 0xffffff
> [37352.907578] pci 0000:01:10.0: unknown header type 7f, ignoring device
> 
> Do any of y'all have some insight into these issues?
> 
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip
> 

Re: [Question] rk3399 vfio-pci/sr-iov support

[Alex Williamson]

On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:

> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > Good Morning,
> > 
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > have encountered some roadblocks.
> > 
> > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > doesn't bind explicitly to a iommu.
> > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > 
> > # find /sys/kernel/iommu_groups/ -type l
> > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > 
> > # virsh start openwrt
> > error: Failed to start domain openwrt
> > error: internal error: Process exited prior to exec: libvirt:  error :
> > internal error: Invalid device 0000:01:00.0 iommu_group file
> > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink  
> 
> That much I can help with somewhat: the major impediment is that RK3399 
> doesn't have an IOMMU in front of PCIe. As far as I'm aware your only 
> option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode 
> (which I don't know an awful lot about beyond that it's a thing).

And it's a thing that's really only useful if your motivation is to run
something like DPDK in the host and you're not concerned about
isolation between userspace drivers and the host kernel, and you don't
mind tainting the kernel.  It's not useful for things like assigning a
device to a VM as we can't readily do that without an IOMMU for
translation.  Thanks,

Alex

Re: [Question] rk3399 vfio-pci/sr-iov support

[Alex Williamson]

On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:

> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > Good Morning,
> > 
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > have encountered some roadblocks.
> > 
> > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > doesn't bind explicitly to a iommu.
> > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > 
> > # find /sys/kernel/iommu_groups/ -type l
> > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > 
> > # virsh start openwrt
> > error: Failed to start domain openwrt
> > error: internal error: Process exited prior to exec: libvirt:  error :
> > internal error: Invalid device 0000:01:00.0 iommu_group file
> > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink  
> 
> That much I can help with somewhat: the major impediment is that RK3399 
> doesn't have an IOMMU in front of PCIe. As far as I'm aware your only 
> option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode 
> (which I don't know an awful lot about beyond that it's a thing).

And it's a thing that's really only useful if your motivation is to run
something like DPDK in the host and you're not concerned about
isolation between userspace drivers and the host kernel, and you don't
mind tainting the kernel.  It's not useful for things like assigning a
device to a VM as we can't readily do that without an IOMMU for
translation.  Thanks,

Alex

Re: [Question] rk3399 vfio-pci/sr-iov support

[Peter Geis]

On Mon, Dec 9, 2019 at 12:05 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>
> > On 09/12/2019 1:28 pm, Peter Geis wrote:
> > > Good Morning,
> > >
> > > I'm back with more pcie fun on the rk3399.
> > > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > > have encountered some roadblocks.
> > >
> > > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > > doesn't bind explicitly to a iommu.
> > > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > >
> > > # find /sys/kernel/iommu_groups/ -type l
> > > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > >
> > > # virsh start openwrt
> > > error: Failed to start domain openwrt
> > > error: internal error: Process exited prior to exec: libvirt:  error :
> > > internal error: Invalid device 0000:01:00.0 iommu_group file
> > > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
> >
> > That much I can help with somewhat: the major impediment is that RK3399
> > doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> > option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> > (which I don't know an awful lot about beyond that it's a thing).
>
> And it's a thing that's really only useful if your motivation is to run
> something like DPDK in the host and you're not concerned about
> isolation between userspace drivers and the host kernel, and you don't
> mind tainting the kernel.  It's not useful for things like assigning a
> device to a VM as we can't readily do that without an IOMMU for
> translation.  Thanks,
>
> Alex

Thanks Robin,

That is an unfortunate limitation, especially considering the device
is supposed to support sr-iov.
I did try getting VFIO_NOIOMMU to work, but qemu does not like it, at
least for pci devices.
>

Re: [Question] rk3399 vfio-pci/sr-iov support

[Peter Geis]

On Mon, Dec 9, 2019 at 12:05 PM Alex Williamson
<alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:
>
> > On 09/12/2019 1:28 pm, Peter Geis wrote:
> > > Good Morning,
> > >
> > > I'm back with more pcie fun on the rk3399.
> > > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > > have encountered some roadblocks.
> > >
> > > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > > doesn't bind explicitly to a iommu.
> > > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > >
> > > # find /sys/kernel/iommu_groups/ -type l
> > > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > >
> > > # virsh start openwrt
> > > error: Failed to start domain openwrt
> > > error: internal error: Process exited prior to exec: libvirt:  error :
> > > internal error: Invalid device 0000:01:00.0 iommu_group file
> > > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
> >
> > That much I can help with somewhat: the major impediment is that RK3399
> > doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> > option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> > (which I don't know an awful lot about beyond that it's a thing).
>
> And it's a thing that's really only useful if your motivation is to run
> something like DPDK in the host and you're not concerned about
> isolation between userspace drivers and the host kernel, and you don't
> mind tainting the kernel.  It's not useful for things like assigning a
> device to a VM as we can't readily do that without an IOMMU for
> translation.  Thanks,
>
> Alex

Thanks Robin,

That is an unfortunate limitation, especially considering the device
is supposed to support sr-iov.
I did try getting VFIO_NOIOMMU to work, but qemu does not like it, at
least for pci devices.
>

Re: [Question] rk3399 vfio-pci/sr-iov support

[Adam Joseph]

On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399,
> > and have encountered some roadblocks.
> 
> That much I can help with somewhat: the major impediment is that
> RK3399 doesn't have an IOMMU in front of PCIe.

For the more limited case of defending against attacks from
hostile/buggy firmware on PCIe devices: is it possible to use the
RK3399 PCIe "inbound address translation" support instead of an IOMMU?

The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
configure address translation (including a base/bounds check) for
inbound-to-SoC memory writes, but details are quite sparse.

Linux appears to not use this functionality; from
drivers/pci/controller/pcie-rockchip-host.c we can see that it
disables the base/bounds (sets them to the entire 32-bit space, which
is all RAM on RK3399 since it supports only 4GB) and passes all 32
bits of incoming memory writes:

static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
                                     int region_no, u8 num_pass_bits,
	                             u32 lower_addr, u32 upper_addr)
...

static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
{
...
	err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);

Is this a dead end?  If not I might pursue it, if I can get the
necessary documentation.  I couldn't find any mention of ATS in the
RK3399 manual; if the PCIe RC allows that then all bets are off anyways.

  - a

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [Question] rk3399 vfio-pci/sr-iov support

[Adam Joseph]

On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399,
> > and have encountered some roadblocks.
> 
> That much I can help with somewhat: the major impediment is that
> RK3399 doesn't have an IOMMU in front of PCIe.

For the more limited case of defending against attacks from
hostile/buggy firmware on PCIe devices: is it possible to use the
RK3399 PCIe "inbound address translation" support instead of an IOMMU?

The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
configure address translation (including a base/bounds check) for
inbound-to-SoC memory writes, but details are quite sparse.

Linux appears to not use this functionality; from
drivers/pci/controller/pcie-rockchip-host.c we can see that it
disables the base/bounds (sets them to the entire 32-bit space, which
is all RAM on RK3399 since it supports only 4GB) and passes all 32
bits of incoming memory writes:

static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
                                     int region_no, u8 num_pass_bits,
	                             u32 lower_addr, u32 upper_addr)
...

static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
{
...
	err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);

Is this a dead end?  If not I might pursue it, if I can get the
necessary documentation.  I couldn't find any mention of ATS in the
RK3399 manual; if the PCIe RC allows that then all bets are off anyways.

  - a

Re: [Question] rk3399 vfio-pci/sr-iov support

[Robin Murphy]

On 2021-11-15 19:50, Adam Joseph wrote:
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>> On 09/12/2019 1:28 pm, Peter Geis wrote:
>>> I'm back with more pcie fun on the rk3399.
>>> I'm trying to get pcie passthrough working for a vm on the rk3399,
>>> and have encountered some roadblocks.
>>
>> That much I can help with somewhat: the major impediment is that
>> RK3399 doesn't have an IOMMU in front of PCIe.
> 
> For the more limited case of defending against attacks from
> hostile/buggy firmware on PCIe devices: is it possible to use the
> RK3399 PCIe "inbound address translation" support instead of an IOMMU?

I guess it might be possible to reserve some memory with the new 
"restricted-dma-pool" semantics then tweak the PCIe setup to match that. 
It will mostly depend on what the RC does with transactions that don't 
match any ATU entry. These kinds of registers typically only decode a 
small number of naturally-aligned power-of-two-sized address ranges so 
would not be able to support the kind of arbitrary translation/page 
protection you'd need for something like VFIO.

> The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
> configure address translation (including a base/bounds check) for
> inbound-to-SoC memory writes, but details are quite sparse.
> 
> Linux appears to not use this functionality; from
> drivers/pci/controller/pcie-rockchip-host.c we can see that it
> disables the base/bounds (sets them to the entire 32-bit space, which
> is all RAM on RK3399 since it supports only 4GB) and passes all 32
> bits of incoming memory writes:
> 
> static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
>                                       int region_no, u8 num_pass_bits,
> 	                             u32 lower_addr, u32 upper_addr)
> ...
> 
> static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
> {
> ...
> 	err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
> 
> Is this a dead end?  If not I might pursue it, if I can get the
> necessary documentation.  I couldn't find any mention of ATS in the
> RK3399 manual; if the PCIe RC allows that then all bets are off anyways.

Knowing this RC, if it ever did see an ATS translated transaction header 
it would probably SError the system, like it does for such outlandish 
things as trying to probe config space behind bridges :)

Robin.

_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip

Re: [Question] rk3399 vfio-pci/sr-iov support

[Robin Murphy]

On 2021-11-15 19:50, Adam Joseph wrote:
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>> On 09/12/2019 1:28 pm, Peter Geis wrote:
>>> I'm back with more pcie fun on the rk3399.
>>> I'm trying to get pcie passthrough working for a vm on the rk3399,
>>> and have encountered some roadblocks.
>>
>> That much I can help with somewhat: the major impediment is that
>> RK3399 doesn't have an IOMMU in front of PCIe.
> 
> For the more limited case of defending against attacks from
> hostile/buggy firmware on PCIe devices: is it possible to use the
> RK3399 PCIe "inbound address translation" support instead of an IOMMU?

I guess it might be possible to reserve some memory with the new 
"restricted-dma-pool" semantics then tweak the PCIe setup to match that. 
It will mostly depend on what the RC does with transactions that don't 
match any ATU entry. These kinds of registers typically only decode a 
small number of naturally-aligned power-of-two-sized address ranges so 
would not be able to support the kind of arbitrary translation/page 
protection you'd need for something like VFIO.

> The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
> configure address translation (including a base/bounds check) for
> inbound-to-SoC memory writes, but details are quite sparse.
> 
> Linux appears to not use this functionality; from
> drivers/pci/controller/pcie-rockchip-host.c we can see that it
> disables the base/bounds (sets them to the entire 32-bit space, which
> is all RAM on RK3399 since it supports only 4GB) and passes all 32
> bits of incoming memory writes:
> 
> static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
>                                       int region_no, u8 num_pass_bits,
> 	                             u32 lower_addr, u32 upper_addr)
> ...
> 
> static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
> {
> ...
> 	err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
> 
> Is this a dead end?  If not I might pursue it, if I can get the
> necessary documentation.  I couldn't find any mention of ATS in the
> RK3399 manual; if the PCIe RC allows that then all bets are off anyways.

Knowing this RC, if it ever did see an ATS translated transaction header 
it would probably SError the system, like it does for such outlandish 
things as trying to probe config space behind bridges :)

Robin.