Re: Question: KVM: Failed to bind vfio with PCI-e / SMMU on Juno-r2

From: Leo Yan <leo.yan@linaro.org>
To: Robin Murphy <robin.murphy@arm.com>
Cc: Daniel Thompson <daniel.thompson@linaro.org>,
	kvmarm@lists.cs.columbia.edu
Subject: Re: Question: KVM: Failed to bind vfio with PCI-e / SMMU on Juno-r2
Date: Tue, 19 Mar 2019 09:33:58 +0800	[thread overview]
Message-ID: <20190319013358.GA3374@leoy-ThinkPad-X240s> (raw)
In-Reply-To: <f245a401-163c-7819-9af5-4226b0ac5f25@arm.com>

Hi Robin,

On Mon, Mar 18, 2019 at 12:25:33PM +0000, Robin Murphy wrote:

[...]

> > diff --git a/arm/include/arm-common/kvm-arch.h b/arm/include/arm-common/kvm-arch.h
> > index b9d486d..43f78b1 100644
> > --- a/arm/include/arm-common/kvm-arch.h
> > +++ b/arm/include/arm-common/kvm-arch.h
> > @@ -7,10 +7,10 @@
> > 
> >   #include "arm-common/gic.h"
> > 
> > -#define ARM_IOPORT_AREA                _AC(0x0000000000000000, UL)
> > -#define ARM_MMIO_AREA          _AC(0x0000000000010000, UL)
> > -#define ARM_AXI_AREA           _AC(0x0000000040000000, UL)
> > -#define ARM_MEMORY_AREA                _AC(0x0000000080000000, UL)
> > +#define ARM_IOPORT_AREA                _AC(0x0000000080000000, UL)
> > +#define ARM_MMIO_AREA          _AC(0x0000000080010000, UL)
> > +#define ARM_AXI_AREA           _AC(0x0000000088000000, UL)
> > +#define ARM_MEMORY_AREA                _AC(0x0000000090000000, UL)
> > 
> > Anyway, very appreciate for the suggestions; it's sufficent for me to
> > dig more for memory related information (e.g. PCIe configurations,
> > IOMMU, etc) and will keep posted if I make any progress.
> 
> None of those should need to change (all the MMIO emulation stuff is
> irrelevant to PCIe DMA anyway) - provided you don't give the guest more than
> 2GB of RAM, passthrough with legacy INTx ought to work out-of-the-box. For
> MSIs to get through, you'll further need to change the host kernel to place
> its software MSI region[2] within any of the host bridge windows as well.

>From PCI configurations dumping, I can see after launch the guest with
kvmtool, the host receives the first interrupt (checked with the
function vfio_intx_handler() has been invoked once) and then PCI sent
command with PCI_COMMAND_INTX_DISABLE to disable interrupt line.  So
this flow is very likely the interrupt is not forwarded properly and
guest doesn't receive interrupt.

It's lucky that I found below flow can let interrupt forwarding from
host to guest after I always set "sky2.disable_msi=1" for both kernel
command lines:

    host                    guest

  INTx mode               INTx mode

So far, it still cannot work well if I only set "sky2.disable_msi=1"
for host kernel command line, with this config it runs with below flow
and which cannot forward interrupt properly from host to guest:

    host                    guest

  INTx mode               msi enable
                          msi disable
                          Switch back to INTx mode

I am so happy now I can use pure INTx mode on Juno board for NIC
enabling and pinged successfully from guest OS to my router :)

Will look into the issue in the second secnario; and if I have more
time I will look into msi mode as well (I confirmed msi mode can work
with host OS but failed in guest OS).

Very appreciate you & Eric helping!

Thanks,
Leo Yan