From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43EF7C433DF for ; Tue, 2 Jun 2020 12:12:42 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 155BE205CB for ; Tue, 2 Jun 2020 12:12:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 155BE205CB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id D7EF920500; Tue, 2 Jun 2020 12:12:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id isf9XUCdPgVK; Tue, 2 Jun 2020 12:12:37 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 90E3F204E2; Tue, 2 Jun 2020 12:12:37 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7E41AC0178; Tue, 2 Jun 2020 12:12:37 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 05D56C016E for ; Tue, 2 Jun 2020 12:12:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E7E5887A6E for ; Tue, 2 Jun 2020 12:12:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sUmwgRcUvWz6 for ; Tue, 2 Jun 2020 12:12:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from huawei.com (lhrrgout.huawei.com [185.176.76.210]) by hemlock.osuosl.org (Postfix) with ESMTPS id 4CD1D87934 for ; Tue, 2 Jun 2020 12:12:34 +0000 (UTC) Received: from lhreml721-chm.china.huawei.com (unknown [172.18.7.107]) by Forcepoint Email with ESMTP id 3829E36D873DF3B208A8; Tue, 2 Jun 2020 13:12:31 +0100 (IST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by lhreml721-chm.china.huawei.com (10.201.108.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Tue, 2 Jun 2020 13:12:31 +0100 Received: from lhreml710-chm.china.huawei.com ([169.254.81.184]) by lhreml710-chm.china.huawei.com ([169.254.81.184]) with mapi id 15.01.1913.007; Tue, 2 Jun 2020 13:12:31 +0100 From: Shameerali Kolothum Thodi To: Jean-Philippe Brucker Subject: RE: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Thread-Topic: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Thread-Index: AQHWLgezl67to0Fq/kugWrIzT8tlbajDuzCQgAFbKgCAABJrcIAAETqAgAAUUMA= Date: Tue, 2 Jun 2020 12:12:30 +0000 Message-ID: References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> <20200602114611.GB1029680@myrica> In-Reply-To: <20200602114611.GB1029680@myrica> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.47.94.73] MIME-Version: 1.0 X-CFilter-Loop: Reflected Cc: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "will@kernel.org" , "fenghua.yu@intel.com" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "hch@infradead.org" , "jgg@ziepe.ca" , "iommu@lists.linux-foundation.org" , "linux-mm@kvack.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "linux-arm-kernel@lists.infradead.org" X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" > -----Original Message----- > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > On Behalf Of Jean-Philippe Brucker > Sent: 02 June 2020 12:46 > To: Shameerali Kolothum Thodi > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; fenghua.yu@intel.com; > linux-pci@vger.kernel.org; felix.kuehling@amd.com; robin.murphy@arm.com; > christian.koenig@amd.com; hch@infradead.org; jgg@ziepe.ca; > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > zhangfei.gao@linaro.org; will@kernel.org; linux-mm@kvack.org; > linux-arm-kernel@lists.infradead.org > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > platform devices > > On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote: > > Hi Jean, > > > > > -----Original Message----- > > > From: linux-arm-kernel > > > [mailto:linux-arm-kernel-bounces@lists.infradead.org] > > > On Behalf Of Jean-Philippe Brucker > > > Sent: 02 June 2020 10:39 > > > To: Shameerali Kolothum Thodi > > > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; > > > will@kernel.org; fenghua.yu@intel.com; jgg@ziepe.ca; > > > linux-pci@vger.kernel.org; felix.kuehling@amd.com; > > > hch@infradead.org; linux-mm@kvack.org; > > > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > > > zhangfei.gao@linaro.org; robin.murphy@arm.com; > > > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > > > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support > > > for platform devices > > > > > > Hi Shameer, > > > > > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > > > wrote: > > > > > /* IRQ and event handlers */ > > > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, > > > > > +u64 > > > > > +*evt) { > > > > > + int ret; > > > > > + u32 perm = 0; > > > > > + struct arm_smmu_master *master; > > > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > > > + struct iommu_fault_event fault_evt = { }; > > > > > + struct iommu_fault *flt = &fault_evt.fault; > > > > > + > > > > > + /* Stage-2 is always pinned at the moment */ > > > > > + if (evt[1] & EVTQ_1_S2) > > > > > + return -EFAULT; > > > > > + > > > > > + master = arm_smmu_find_master(smmu, sid); > > > > > + if (!master) > > > > > + return -EINVAL; > > > > > + > > > > > + if (evt[1] & EVTQ_1_READ) > > > > > + perm |= IOMMU_FAULT_PERM_READ; > > > > > + else > > > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > > > + > > > > > + if (evt[1] & EVTQ_1_EXEC) > > > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > > > + > > > > > + if (evt[1] & EVTQ_1_PRIV) > > > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > > > + > > > > > + if (evt[1] & EVTQ_1_STALL) { > > > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > > > + flt->prm = (struct iommu_fault_page_request) { > > > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > > > + .perm = perm, > > > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > > > + }; > > > > > + > > > > > > > > > + if (ssid_valid) > > > > > + flt->prm.flags |= > > > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > > > > > Do we need to set this for STALL mode only support? I had an issue > > > > with this being set on a vSVA POC based on our D06 zip > > > > device(which is a "fake " pci dev that supports STALL mode but no > > > > PRI). The issue is, CMDQ_OP_RESUME doesn't have any ssid or SSV > > > > params and works on sid > > > and stag only. > > > > > > I don't understand the problem, arm_smmu_page_response() doesn't set > > > SSID or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow > > > of a stall event and RESUME command in your prototype? Are you > > > getting issues with the host driver or the guest driver? > > > > The issue is on the host side iommu_page_response(). The flow is > > something like below. > > > > Stall: Host:- > > > > arm_smmu_handle_evt() > > iommu_report_device_fault() > > vfio_pci_iommu_dev_fault_handler() > > > > Stall: Qemu:- > > > > vfio_dma_fault_notifier_handler() > > inject_faults() > > smmuv3_inject_faults() > > > > Stall: Guest:- > > > > arm_smmu_handle_evt() > > iommu_report_device_fault() > > iommu_queue_iopf > > ... > > iopf_handle_group() > > iopf_handle_single() > > handle_mm_fault() > > iopf_complete() > > iommu_page_response() > > arm_smmu_page_response() > > arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) > > > > Resume: Qemu:- > > > > smmuv3_cmdq_consume(SMMU_CMD_RESUME) > > smmuv3_notify_page_resp() > > vfio:ioctl(page_response) --> struct iommu_page_response is filled > > with only version, grpid and code. > > > > Resume: Host:- > > ioctl(page_response) > > iommu_page_response() --> fails as the pending req has PASID_VALID > flag > > set and it checks for a match. > > I believe the fix needs to be here. It's also wrong for PRI since not all PCIe > endpoint require a PASID in the page response. Could you try the attached > patch? Going through the patch, yes, that will definitely fix the issue. But isn't it better if the request itself indicate whether it expects a response msg with a valid pasid or not? The response msg can come from userspace as well(vSVA) and if for some reason doesn't set it for a req that expects pasid then it should be an error, right? In the temp fix I had, I introduced another flag to indicate the endpoint has PRI support or not and used that to verify the pasid requirement. But for the PRI case you mentioned above, not sure it is easy to get that information or not. May be I am complicating things here :) Thanks, Shameer > Thanks, > Jean > > > arm_smmu_page_response() > > > > Hope the above is clear. > > > > > We do need to forward the SSV flag all the way to the guest driver, > > > so the guest can find the faulting address space using the SSID. > > > Once the guest handled the fault, then we don't send the SSID back > > > to the host as part of the RESUME command. > > > > True, the guest requires SSV flag to handle the page fault. But, as > > shown in the flow above, the issue is on the host side > > iommu_page_response() where it searches for a matching pending req > > based on pasid. Not sure we can bypass that and call > > arm_smmu_page_response() directly but then have to delete the pending req > from the list as well. > > > > Please let me know if there is a better way to handle the host side > > page response. > > > > Thanks, > > Shameer > > > > > Thanks, > > > Jean > > > > > > > Hence, it is difficult for > > > > Qemu SMMUv3 to populate this fields while preparing a page > > > > response. I can see that this flag is being checked in > > > > iopf_handle_single() (patch > > > > 04/24) as well. For POC, I used a temp fix[1] to work around this. > > > > Please let > > > me know your thoughts. > > > > > > > > Thanks, > > > > Shameer > > > > > > > > 1. > > > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38 > > > > d97a > > > > 5897e4becfa378d15 > > > > > > > > > > _______________________________________________ > > > linux-arm-kernel mailing list > > > linux-arm-kernel@lists.infradead.org > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu