iommu.lists.linux-foundation.org archive mirror
 help / color / mirror / Atom feed
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
Cc: "devicetree@vger.kernel.org" <devicetree@vger.kernel.org>,
	"kevin.tian@intel.com" <kevin.tian@intel.com>,
	"fenghua.yu@intel.com" <fenghua.yu@intel.com>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"felix.kuehling@amd.com" <felix.kuehling@amd.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"christian.koenig@amd.com" <christian.koenig@amd.com>,
	"hch@infradead.org" <hch@infradead.org>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"zhangfei.gao@linaro.org" <zhangfei.gao@linaro.org>,
	"will@kernel.org" <will@kernel.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	"linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices
Date: Tue, 2 Jun 2020 13:46:11 +0200	[thread overview]
Message-ID: <20200602114611.GB1029680@myrica> (raw)
In-Reply-To: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com>

[-- Attachment #1: Type: text/plain, Size: 5830 bytes --]

On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote:
> Hi Jean,
> 
> > -----Original Message-----
> > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org]
> > On Behalf Of Jean-Philippe Brucker
> > Sent: 02 June 2020 10:39
> > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org;
> > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org;
> > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org;
> > iommu@lists.linux-foundation.org; catalin.marinas@arm.com;
> > zhangfei.gao@linaro.org; robin.murphy@arm.com;
> > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org
> > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for
> > platform devices
> > 
> > Hi Shameer,
> > 
> > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi
> > wrote:
> > > >  /* IRQ and event handlers */
> > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64
> > > > +*evt) {
> > > > +	int ret;
> > > > +	u32 perm = 0;
> > > > +	struct arm_smmu_master *master;
> > > > +	bool ssid_valid = evt[0] & EVTQ_0_SSV;
> > > > +	u8 type = FIELD_GET(EVTQ_0_ID, evt[0]);
> > > > +	u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]);
> > > > +	struct iommu_fault_event fault_evt = { };
> > > > +	struct iommu_fault *flt = &fault_evt.fault;
> > > > +
> > > > +	/* Stage-2 is always pinned at the moment */
> > > > +	if (evt[1] & EVTQ_1_S2)
> > > > +		return -EFAULT;
> > > > +
> > > > +	master = arm_smmu_find_master(smmu, sid);
> > > > +	if (!master)
> > > > +		return -EINVAL;
> > > > +
> > > > +	if (evt[1] & EVTQ_1_READ)
> > > > +		perm |= IOMMU_FAULT_PERM_READ;
> > > > +	else
> > > > +		perm |= IOMMU_FAULT_PERM_WRITE;
> > > > +
> > > > +	if (evt[1] & EVTQ_1_EXEC)
> > > > +		perm |= IOMMU_FAULT_PERM_EXEC;
> > > > +
> > > > +	if (evt[1] & EVTQ_1_PRIV)
> > > > +		perm |= IOMMU_FAULT_PERM_PRIV;
> > > > +
> > > > +	if (evt[1] & EVTQ_1_STALL) {
> > > > +		flt->type = IOMMU_FAULT_PAGE_REQ;
> > > > +		flt->prm = (struct iommu_fault_page_request) {
> > > > +			.flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE,
> > > > +			.pasid = FIELD_GET(EVTQ_0_SSID, evt[0]),
> > > > +			.grpid = FIELD_GET(EVTQ_1_STAG, evt[1]),
> > > > +			.perm = perm,
> > > > +			.addr = FIELD_GET(EVTQ_2_ADDR, evt[2]),
> > > > +		};
> > > > +
> > >
> > > > +		if (ssid_valid)
> > > > +			flt->prm.flags |=
> > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
> > >
> > > Do we need to set this for STALL mode only support? I had an issue
> > > with this being set on a vSVA POC based on our D06 zip device(which is
> > > a "fake " pci dev that supports STALL mode but no PRI). The issue is,
> > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid
> > and stag only.
> > 
> > I don't understand the problem, arm_smmu_page_response() doesn't set SSID
> > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall
> > event and RESUME command in your prototype?  Are you getting issues with
> > the host driver or the guest driver?
> 
> The issue is on the host side iommu_page_response(). The flow is something like
> below.
> 
> Stall: Host:-
> 
> arm_smmu_handle_evt()
>   iommu_report_device_fault()
>     vfio_pci_iommu_dev_fault_handler()
>       
> Stall: Qemu:-
> 
> vfio_dma_fault_notifier_handler()
>   inject_faults()
>     smmuv3_inject_faults()
> 
> Stall: Guest:-
> 
> arm_smmu_handle_evt()
>   iommu_report_device_fault()
>     iommu_queue_iopf
>   ...
>   iopf_handle_group()
>     iopf_handle_single()
>       handle_mm_fault()
>         iopf_complete()
>            iommu_page_response()
>              arm_smmu_page_response()
>                arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME)
> 
> Resume: Qemu:-
> 
> smmuv3_cmdq_consume(SMMU_CMD_RESUME)
>   smmuv3_notify_page_resp()
>     vfio:ioctl(page_response)  --> struct iommu_page_response is filled
>                              with only version, grpid and code.
> 
> Resume: Host:-
>   ioctl(page_response)
>     iommu_page_response()  --> fails as the pending req has PASID_VALID flag
>                              set and it checks for a match.

I believe the fix needs to be here. It's also wrong for PRI since not all
PCIe endpoint require a PASID in the page response. Could you try the
attached patch?

Thanks,
Jean

>       arm_smmu_page_response()
> 
> Hope the above is clear.
> 
> > We do need to forward the SSV flag all the way to the guest driver, so the guest
> > can find the faulting address space using the SSID. Once the guest handled the
> > fault, then we don't send the SSID back to the host as part of the RESUME
> > command.
> 
> True, the guest requires SSV flag to handle the page fault. But, as shown in the
> flow above, the issue is on the host side iommu_page_response() where it
> searches for a matching pending req based on pasid. Not sure we can bypass
> that and call arm_smmu_page_response() directly but then have to delete the
> pending req from the list as well.
> 
> Please let me know if there is a better way to handle the host side page
> response.
> 
> Thanks,
> Shameer
> 
> > Thanks,
> > Jean
> > 
> > > Hence, it is difficult for
> > > Qemu SMMUv3 to populate this fields while preparing a page response. I
> > > can see that this flag is being checked in iopf_handle_single() (patch
> > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let
> > me know your thoughts.
> > >
> > > Thanks,
> > > Shameer
> > >
> > > 1.
> > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a
> > > 5897e4becfa378d15
> > >
> > 
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[-- Attachment #2: 0001-iommu-Allow-page-responses-without-PASID.patch --]
[-- Type: text/plain, Size: 1682 bytes --]

From 9baf5b9894d4e4be05e665d80fd0ebac8b621aa4 Mon Sep 17 00:00:00 2001
From: Jean-Philippe Brucker <jean-philippe@linaro.org>
Date: Tue, 2 Jun 2020 13:13:27 +0200
Subject: [PATCH] iommu: Allow page responses without PASID

Some PCIe devices do not expect a PASID value in PRI Page Responses. If
the "PRG Response PASID Required" bit in the PRI capability is zero,
then the OS should not set the PASID field. Similarly on Arm SMMU,
responses to stall events do not have a PASID.

Currently iommu_page_response() checks that the PASID in the page
response corresponds to the one in the page request without first
checking the "PASID valid" bit. A page response coming from a guest OS
does not necessarily have a PASID, if the passed-through device does not
require one.

Allow page responses without PASID. The page request corresponding to a
page response is identified by device and by Page Response Group Index
(or stall tag).

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 drivers/iommu/iommu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c
index e61a9fc65b7e4..e481fdfafb77c 100644
--- a/drivers/iommu/iommu.c
+++ b/drivers/iommu/iommu.c
@@ -1296,7 +1296,8 @@ int iommu_page_response(struct device *dev,
 	 */
 	list_for_each_entry(evt, &param->fault_param->faults, list) {
 		prm = &evt->fault.prm;
-		pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID;
+		pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID
+			   && msg->flags & IOMMU_PAGE_RESP_PASID_VALID;
 
 		if ((pasid_valid && prm->pasid != msg->pasid) ||
 		    prm->grpid != msg->grpid)
-- 
2.26.2


[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

  reply	other threads:[~2020-06-02 11:46 UTC|newest]

Thread overview: 51+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-19 17:54 [PATCH v7 00/24] iommu: Shared Virtual Addressing for SMMUv3 Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 01/24] mm: Add a PASID field to mm_struct Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 02/24] iommu/ioasid: Add ioasid references Jean-Philippe Brucker
2020-05-20  2:31   ` Lu Baolu
2020-05-19 17:54 ` [PATCH v7 03/24] iommu/sva: Add PASID helpers Jean-Philippe Brucker
2020-05-20  2:41   ` Lu Baolu
2020-05-19 17:54 ` [PATCH v7 04/24] iommu: Add a page fault handler Jean-Philippe Brucker
2020-05-20  6:42   ` Lu Baolu
2020-11-11 13:57     ` Jean-Philippe Brucker
2020-11-11 23:11       ` Lu Baolu
2020-05-29  9:18   ` Xiang Zheng
2020-11-11 13:57     ` Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 05/24] arm64: mm: Add asid_gen_match() helper Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 06/24] arm64: mm: Pin down ASIDs for sharing mm with devices Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 07/24] iommu/io-pgtable-arm: Move some definitions to a header Jean-Philippe Brucker
2020-05-21 14:16   ` Will Deacon
2020-05-19 17:54 ` [PATCH v7 08/24] iommu/arm-smmu-v3: Manage ASIDs with xarray Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 09/24] arm64: cpufeature: Export symbol read_sanitised_ftr_reg() Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 10/24] iommu/arm-smmu-v3: Share process page tables Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 11/24] iommu/arm-smmu-v3: Seize private ASID Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 12/24] iommu/arm-smmu-v3: Add support for VHE Jean-Philippe Brucker
2020-05-21 14:16   ` Will Deacon
2020-05-19 17:54 ` [PATCH v7 13/24] iommu/arm-smmu-v3: Enable broadcast TLB maintenance Jean-Philippe Brucker
2020-05-21 14:17   ` Will Deacon
2020-05-21 14:38     ` Marc Zyngier
2020-05-22 10:17       ` Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 14/24] iommu/arm-smmu-v3: Add SVA feature checking Jean-Philippe Brucker
2020-05-21 14:17   ` Will Deacon
2020-05-19 17:54 ` [PATCH v7 15/24] iommu/arm-smmu-v3: Add SVA device feature Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 16/24] iommu/arm-smmu-v3: Implement iommu_sva_bind/unbind() Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 17/24] iommu/arm-smmu-v3: Hook up ATC invalidation to mm ops Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 18/24] iommu/arm-smmu-v3: Add support for Hardware Translation Table Update Jean-Philippe Brucker
2020-05-21 11:12   ` Will Deacon
2020-05-27  3:00   ` Xiang Zheng
2020-05-27  8:41     ` Jean-Philippe Brucker
2020-08-28  9:28   ` Zenghui Yu
2020-09-16 14:11     ` Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 19/24] iommu/arm-smmu-v3: Maintain a SID->device structure Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 20/24] dt-bindings: document stall property for IOMMU masters Jean-Philippe Brucker
2020-05-19 17:54 ` [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Jean-Philippe Brucker
2020-06-01 12:42   ` Shameerali Kolothum Thodi
2020-06-02  9:38     ` Jean-Philippe Brucker
2020-06-02 10:31       ` Shameerali Kolothum Thodi
2020-06-02 11:46         ` Jean-Philippe Brucker [this message]
2020-06-02 12:12           ` Shameerali Kolothum Thodi
2020-06-03  7:38             ` Jean-Philippe Brucker
2020-05-19 17:55 ` [PATCH v7 22/24] PCI/ATS: Add PRI stubs Jean-Philippe Brucker
2020-05-19 17:55 ` [PATCH v7 23/24] PCI/ATS: Export PRI functions Jean-Philippe Brucker
2020-05-19 17:55 ` [PATCH v7 24/24] iommu/arm-smmu-v3: Add support for PRI Jean-Philippe Brucker
2020-05-21 10:35 ` [PATCH v7 00/24] iommu: Shared Virtual Addressing for SMMUv3 Will Deacon
2020-05-21 14:17   ` Will Deacon

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200602114611.GB1029680@myrica \
    --to=jean-philippe@linaro.org \
    --cc=catalin.marinas@arm.com \
    --cc=christian.koenig@amd.com \
    --cc=devicetree@vger.kernel.org \
    --cc=felix.kuehling@amd.com \
    --cc=fenghua.yu@intel.com \
    --cc=hch@infradead.org \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jgg@ziepe.ca \
    --cc=kevin.tian@intel.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=robin.murphy@arm.com \
    --cc=shameerali.kolothum.thodi@huawei.com \
    --cc=will@kernel.org \
    --cc=zhangfei.gao@linaro.org \
    --subject='Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).