From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51EA1C433DF for ; Tue, 2 Jun 2020 10:31:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 237CC206C3 for ; Tue, 2 Jun 2020 10:31:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 237CC206C3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C31528004A; Tue, 2 Jun 2020 06:31:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9750B8E0006; Tue, 2 Jun 2020 06:31:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8899128004A; Tue, 2 Jun 2020 06:31:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 71EA58E0006 for ; Tue, 2 Jun 2020 06:31:43 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 27891181AEF0B for ; Tue, 2 Jun 2020 10:31:43 +0000 (UTC) X-FDA: 76883905686.20.owl81_822f60f2da717 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 1A52F180C08EF for ; Tue, 2 Jun 2020 10:31:35 +0000 (UTC) X-HE-Tag: owl81_822f60f2da717 X-Filterd-Recvd-Size: 8075 Received: from huawei.com (lhrrgout.huawei.com [185.176.76.210]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 2 Jun 2020 10:31:34 +0000 (UTC) Received: from lhreml710-chm.china.huawei.com (unknown [172.18.7.107]) by Forcepoint Email with ESMTP id 9DA1C2F3133232790D75; Tue, 2 Jun 2020 11:31:32 +0100 (IST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Tue, 2 Jun 2020 11:31:29 +0100 Received: from lhreml710-chm.china.huawei.com ([169.254.81.184]) by lhreml710-chm.china.huawei.com ([169.254.81.184]) with mapi id 15.01.1913.007; Tue, 2 Jun 2020 11:31:29 +0100 From: Shameerali Kolothum Thodi To: Jean-Philippe Brucker CC: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "will@kernel.org" , "fenghua.yu@intel.com" , "jgg@ziepe.ca" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "hch@infradead.org" , "linux-mm@kvack.org" , "iommu@lists.linux-foundation.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "linux-arm-kernel@lists.infradead.org" Subject: RE: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Thread-Topic: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Thread-Index: AQHWLgezl67to0Fq/kugWrIzT8tlbajDuzCQgAFbKgCAABJrcA== Date: Tue, 2 Jun 2020 10:31:29 +0000 Message-ID: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> In-Reply-To: <20200602093836.GA1029680@myrica> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.47.94.73] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 1A52F180C08EF X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Jean, > -----Original Message----- > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.o= rg] > On Behalf Of Jean-Philippe Brucker > Sent: 02 June 2020 10:39 > To: Shameerali Kolothum Thodi > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org; > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org; > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org; > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > zhangfei.gao@linaro.org; robin.murphy@arm.com; > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > platform devices >=20 > Hi Shameer, >=20 > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > wrote: > > > /* IRQ and event handlers */ > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > +*evt) { > > > + int ret; > > > + u32 perm =3D 0; > > > + struct arm_smmu_master *master; > > > + bool ssid_valid =3D evt[0] & EVTQ_0_SSV; > > > + u8 type =3D FIELD_GET(EVTQ_0_ID, evt[0]); > > > + u32 sid =3D FIELD_GET(EVTQ_0_SID, evt[0]); > > > + struct iommu_fault_event fault_evt =3D { }; > > > + struct iommu_fault *flt =3D &fault_evt.fault; > > > + > > > + /* Stage-2 is always pinned at the moment */ > > > + if (evt[1] & EVTQ_1_S2) > > > + return -EFAULT; > > > + > > > + master =3D arm_smmu_find_master(smmu, sid); > > > + if (!master) > > > + return -EINVAL; > > > + > > > + if (evt[1] & EVTQ_1_READ) > > > + perm |=3D IOMMU_FAULT_PERM_READ; > > > + else > > > + perm |=3D IOMMU_FAULT_PERM_WRITE; > > > + > > > + if (evt[1] & EVTQ_1_EXEC) > > > + perm |=3D IOMMU_FAULT_PERM_EXEC; > > > + > > > + if (evt[1] & EVTQ_1_PRIV) > > > + perm |=3D IOMMU_FAULT_PERM_PRIV; > > > + > > > + if (evt[1] & EVTQ_1_STALL) { > > > + flt->type =3D IOMMU_FAULT_PAGE_REQ; > > > + flt->prm =3D (struct iommu_fault_page_request) { > > > + .flags =3D IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > + .pasid =3D FIELD_GET(EVTQ_0_SSID, evt[0]), > > > + .grpid =3D FIELD_GET(EVTQ_1_STAG, evt[1]), > > > + .perm =3D perm, > > > + .addr =3D FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > + }; > > > + > > > > > + if (ssid_valid) > > > + flt->prm.flags |=3D > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > Do we need to set this for STALL mode only support? I had an issue > > with this being set on a vSVA POC based on our D06 zip device(which is > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > and stag only. >=20 > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stal= l > event and RESUME command in your prototype? Are you getting issues with > the host driver or the guest driver? The issue is on the host side iommu_page_response(). The flow is something = like below. Stall: Host:- arm_smmu_handle_evt() iommu_report_device_fault() vfio_pci_iommu_dev_fault_handler() =20 Stall: Qemu:- vfio_dma_fault_notifier_handler() inject_faults() smmuv3_inject_faults() Stall: Guest:- arm_smmu_handle_evt() iommu_report_device_fault() iommu_queue_iopf ... iopf_handle_group() iopf_handle_single() handle_mm_fault() iopf_complete() iommu_page_response() arm_smmu_page_response() arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) Resume: Qemu:- smmuv3_cmdq_consume(SMMU_CMD_RESUME) smmuv3_notify_page_resp() vfio:ioctl(page_response) --> struct iommu_page_response is filled with only version, grpid and code. Resume: Host:- ioctl(page_response) iommu_page_response() --> fails as the pending req has PASID_VALID fla= g set and it checks for a match. arm_smmu_page_response() Hope the above is clear. > We do need to forward the SSV flag all the way to the guest driver, so th= e guest > can find the faulting address space using the SSID. Once the guest handle= d the > fault, then we don't send the SSID back to the host as part of the RESUME > command. True, the guest requires SSV flag to handle the page fault. But, as shown i= n the flow above, the issue is on the host side iommu_page_response() where it searches for a matching pending req based on pasid. Not sure we can bypass that and call arm_smmu_page_response() directly but then have to delete the pending req from the list as well. Please let me know if there is a better way to handle the host side page response. Thanks, Shameer > Thanks, > Jean >=20 > > Hence, it is difficult for > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > can see that this flag is being checked in iopf_handle_single() (patch > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Plea= se let > me know your thoughts. > > > > Thanks, > > Shameer > > > > 1. > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > 5897e4becfa378d15 > > >=20 > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel