From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BB1DC433DF for ; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53288206A2 for ; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="yijCmqf0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727773AbgFBLq1 (ORCPT ); Tue, 2 Jun 2020 07:46:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726782AbgFBLq0 (ORCPT ); Tue, 2 Jun 2020 07:46:26 -0400 Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54D0AC08C5C0 for ; Tue, 2 Jun 2020 04:46:25 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id o26so6916775edq.0 for ; Tue, 02 Jun 2020 04:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=yijCmqf0VG0ntw+uiPMpdUH4LMWIvqHsCVmInI53fK9mX51SGYwD72IA6DhJmlD97a UNj+wr1xk+MpFkSEXLQv8eoO3cAwX2PJr2eu2gMt+Wdvj5/sAsHXbV5SzeGPc9ipSP0Q iHfDaFS6bvC9IQ/hqK24QfP+vlNw2yFbtAIcg3kNx4x1y/EZKku2MJwvmhK6aMevkQFs 8lcMkEWWV5PLBUJ0P4IRbkLYfbsmxy3mAD17T5a69W7H5qUaRUYgujxCLI7dNY/j2CSu DluWkc/YGeWTqDa51l8CLmHNfPGrr46903qlPJBuohlTXpBNuzAlqELce9wbsiG2I2+U /T8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=epP8htppa2kmCXB8ErXDeU3Tnqx/YzoS3xCEShDYQAcKCwOZG/9EhYYLZpxHFl+Tmi 1YP245GfiWbx9tfkBSFChn19VRH56S+BGqlHHZDxYPJZJpHcVkFojClYgIZR0Sdx+TI1 5oPLxLvBElQwrirrpnrAfHA2ehTe3R6o2ILQutt8w4veRfQZwbaMN/0DYm7Ma59lqh9n TWj1j0m/mixtVTtGmDAvmbFthgw5r5i5+uRVlM6zTnNymVJOTp1RXtYIZi7adPVF2esm Wvgm+jP69t0jilfSkiFU3kW9fosalXvLD4o8ZTkbF2zImpK77iZeH7xvmOris/na56wP N6Xg== X-Gm-Message-State: AOAM532j9CZjL/gHGUxa5JBIRQRhkHDt3WnnusxSUda0QCWxm/uKnRHX t075C9kSZpNioT3976ZNzQz4QA== X-Google-Smtp-Source: ABdhPJzk9qpnI9yUmwOaxqU5ALMvmfWfI4N5ixuLoClwbRn/ueEUAkkLlMA8Jw8WKqsKsrA7069JbA== X-Received: by 2002:a05:6402:34e:: with SMTP id r14mr14619562edw.351.1591098383894; Tue, 02 Jun 2020 04:46:23 -0700 (PDT) Received: from myrica ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id h17sm1047269edw.92.2020.06.02.04.46.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 04:46:22 -0700 (PDT) Date: Tue, 2 Jun 2020 13:46:11 +0200 From: Jean-Philippe Brucker To: Shameerali Kolothum Thodi Cc: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "will@kernel.org" , "fenghua.yu@intel.com" , "jgg@ziepe.ca" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "hch@infradead.org" , "linux-mm@kvack.org" , "iommu@lists.linux-foundation.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "linux-arm-kernel@lists.infradead.org" Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Message-ID: <20200602114611.GB1029680@myrica> References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> Sender: devicetree-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: devicetree@vger.kernel.org --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote: > Hi Jean, > > > -----Original Message----- > > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > > On Behalf Of Jean-Philippe Brucker > > Sent: 02 June 2020 10:39 > > To: Shameerali Kolothum Thodi > > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org; > > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org; > > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org; > > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > > zhangfei.gao@linaro.org; robin.murphy@arm.com; > > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > > platform devices > > > > Hi Shameer, > > > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > > wrote: > > > > /* IRQ and event handlers */ > > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > > +*evt) { > > > > + int ret; > > > > + u32 perm = 0; > > > > + struct arm_smmu_master *master; > > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > > + struct iommu_fault_event fault_evt = { }; > > > > + struct iommu_fault *flt = &fault_evt.fault; > > > > + > > > > + /* Stage-2 is always pinned at the moment */ > > > > + if (evt[1] & EVTQ_1_S2) > > > > + return -EFAULT; > > > > + > > > > + master = arm_smmu_find_master(smmu, sid); > > > > + if (!master) > > > > + return -EINVAL; > > > > + > > > > + if (evt[1] & EVTQ_1_READ) > > > > + perm |= IOMMU_FAULT_PERM_READ; > > > > + else > > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > > + > > > > + if (evt[1] & EVTQ_1_EXEC) > > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > > + > > > > + if (evt[1] & EVTQ_1_PRIV) > > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > > + > > > > + if (evt[1] & EVTQ_1_STALL) { > > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > > + flt->prm = (struct iommu_fault_page_request) { > > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > > + .perm = perm, > > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > > + }; > > > > + > > > > > > > + if (ssid_valid) > > > > + flt->prm.flags |= > > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > > > Do we need to set this for STALL mode only support? I had an issue > > > with this being set on a vSVA POC based on our D06 zip device(which is > > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > > and stag only. > > > > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall > > event and RESUME command in your prototype? Are you getting issues with > > the host driver or the guest driver? > > The issue is on the host side iommu_page_response(). The flow is something like > below. > > Stall: Host:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > vfio_pci_iommu_dev_fault_handler() > > Stall: Qemu:- > > vfio_dma_fault_notifier_handler() > inject_faults() > smmuv3_inject_faults() > > Stall: Guest:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > iommu_queue_iopf > ... > iopf_handle_group() > iopf_handle_single() > handle_mm_fault() > iopf_complete() > iommu_page_response() > arm_smmu_page_response() > arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) > > Resume: Qemu:- > > smmuv3_cmdq_consume(SMMU_CMD_RESUME) > smmuv3_notify_page_resp() > vfio:ioctl(page_response) --> struct iommu_page_response is filled > with only version, grpid and code. > > Resume: Host:- > ioctl(page_response) > iommu_page_response() --> fails as the pending req has PASID_VALID flag > set and it checks for a match. I believe the fix needs to be here. It's also wrong for PRI since not all PCIe endpoint require a PASID in the page response. Could you try the attached patch? Thanks, Jean > arm_smmu_page_response() > > Hope the above is clear. > > > We do need to forward the SSV flag all the way to the guest driver, so the guest > > can find the faulting address space using the SSID. Once the guest handled the > > fault, then we don't send the SSID back to the host as part of the RESUME > > command. > > True, the guest requires SSV flag to handle the page fault. But, as shown in the > flow above, the issue is on the host side iommu_page_response() where it > searches for a matching pending req based on pasid. Not sure we can bypass > that and call arm_smmu_page_response() directly but then have to delete the > pending req from the list as well. > > Please let me know if there is a better way to handle the host side page > response. > > Thanks, > Shameer > > > Thanks, > > Jean > > > > > Hence, it is difficult for > > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > > can see that this flag is being checked in iopf_handle_single() (patch > > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let > > me know your thoughts. > > > > > > Thanks, > > > Shameer > > > > > > 1. > > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > > 5897e4becfa378d15 > > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-iommu-Allow-page-responses-without-PASID.patch" >From 9baf5b9894d4e4be05e665d80fd0ebac8b621aa4 Mon Sep 17 00:00:00 2001 From: Jean-Philippe Brucker Date: Tue, 2 Jun 2020 13:13:27 +0200 Subject: [PATCH] iommu: Allow page responses without PASID Some PCIe devices do not expect a PASID value in PRI Page Responses. If the "PRG Response PASID Required" bit in the PRI capability is zero, then the OS should not set the PASID field. Similarly on Arm SMMU, responses to stall events do not have a PASID. Currently iommu_page_response() checks that the PASID in the page response corresponds to the one in the page request without first checking the "PASID valid" bit. A page response coming from a guest OS does not necessarily have a PASID, if the passed-through device does not require one. Allow page responses without PASID. The page request corresponding to a page response is identified by device and by Page Response Group Index (or stall tag). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e61a9fc65b7e4..e481fdfafb77c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1296,7 +1296,8 @@ int iommu_page_response(struct device *dev, */ list_for_each_entry(evt, ¶m->fault_param->faults, list) { prm = &evt->fault.prm; - pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID + && msg->flags & IOMMU_PAGE_RESP_PASID_VALID; if ((pasid_valid && prm->pasid != msg->pasid) || prm->grpid != msg->grpid) -- 2.26.2 --3MwIy2ne0vdjdPXF-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2158BC433E2 for ; Tue, 2 Jun 2020 11:46:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF589206A2 for ; Tue, 2 Jun 2020 11:46:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="yijCmqf0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF589206A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5D75528006F; Tue, 2 Jun 2020 07:46:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 586CF8E0006; Tue, 2 Jun 2020 07:46:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4758228006F; Tue, 2 Jun 2020 07:46:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0109.hostedemail.com [216.40.44.109]) by kanga.kvack.org (Postfix) with ESMTP id 2FD548E0006 for ; Tue, 2 Jun 2020 07:46:26 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D94A28248047 for ; Tue, 2 Jun 2020 11:46:25 +0000 (UTC) X-FDA: 76884093930.24.fly64_38114c554e72a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id BAAAC1A4A5 for ; Tue, 2 Jun 2020 11:46:25 +0000 (UTC) X-HE-Tag: fly64_38114c554e72a X-Filterd-Recvd-Size: 11899 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Tue, 2 Jun 2020 11:46:25 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id s19so9799007edt.12 for ; Tue, 02 Jun 2020 04:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=yijCmqf0VG0ntw+uiPMpdUH4LMWIvqHsCVmInI53fK9mX51SGYwD72IA6DhJmlD97a UNj+wr1xk+MpFkSEXLQv8eoO3cAwX2PJr2eu2gMt+Wdvj5/sAsHXbV5SzeGPc9ipSP0Q iHfDaFS6bvC9IQ/hqK24QfP+vlNw2yFbtAIcg3kNx4x1y/EZKku2MJwvmhK6aMevkQFs 8lcMkEWWV5PLBUJ0P4IRbkLYfbsmxy3mAD17T5a69W7H5qUaRUYgujxCLI7dNY/j2CSu DluWkc/YGeWTqDa51l8CLmHNfPGrr46903qlPJBuohlTXpBNuzAlqELce9wbsiG2I2+U /T8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=i+fPR2kdY0m2ebSIpbpS2+csBwCM1TXNNWKgFXOf3XAYM6wCYhCQsxnRcvkIgTpL14 SpktOix4NT+UtEb7ilO+glAFlugkohHMaaWe3dxU6z1J1bVRExgEsPUDN0t9l31F4elQ 0REVDS+2vNDcAx4YBQuHjPI0h/1qTwb+VfzqxSkWlxzM3hI1Fssx4yZYKOVuPI2uoUfP mBi8GgDnXIMVoqizUTJ1UtErz0cAHDfJeYwL/pHYsCRIsLvj65TnJ5P5dQ6wgkZM3nlZ 6w8tt82+5QqQPVHaz47S6fjG74+j5L5Cab8cAXqGxsWn4Zq6+PqKEwOqc7yGL+QBPIz6 J5LA== X-Gm-Message-State: AOAM531xHMVVHGMj3xrHUuLLPfKa+c3XWEIptklbIRtDAdLD4jTu1zLo xpTk104bZ6eqiBH8AzyPTw8YKQ== X-Google-Smtp-Source: ABdhPJzk9qpnI9yUmwOaxqU5ALMvmfWfI4N5ixuLoClwbRn/ueEUAkkLlMA8Jw8WKqsKsrA7069JbA== X-Received: by 2002:a05:6402:34e:: with SMTP id r14mr14619562edw.351.1591098383894; Tue, 02 Jun 2020 04:46:23 -0700 (PDT) Received: from myrica ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id h17sm1047269edw.92.2020.06.02.04.46.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 04:46:22 -0700 (PDT) Date: Tue, 2 Jun 2020 13:46:11 +0200 From: Jean-Philippe Brucker To: Shameerali Kolothum Thodi Cc: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "will@kernel.org" , "fenghua.yu@intel.com" , "jgg@ziepe.ca" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "hch@infradead.org" , "linux-mm@kvack.org" , "iommu@lists.linux-foundation.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "linux-arm-kernel@lists.infradead.org" Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Message-ID: <20200602114611.GB1029680@myrica> References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> X-Rspamd-Queue-Id: BAAAC1A4A5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote: > Hi Jean, > > > -----Original Message----- > > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > > On Behalf Of Jean-Philippe Brucker > > Sent: 02 June 2020 10:39 > > To: Shameerali Kolothum Thodi > > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org; > > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org; > > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org; > > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > > zhangfei.gao@linaro.org; robin.murphy@arm.com; > > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > > platform devices > > > > Hi Shameer, > > > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > > wrote: > > > > /* IRQ and event handlers */ > > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > > +*evt) { > > > > + int ret; > > > > + u32 perm = 0; > > > > + struct arm_smmu_master *master; > > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > > + struct iommu_fault_event fault_evt = { }; > > > > + struct iommu_fault *flt = &fault_evt.fault; > > > > + > > > > + /* Stage-2 is always pinned at the moment */ > > > > + if (evt[1] & EVTQ_1_S2) > > > > + return -EFAULT; > > > > + > > > > + master = arm_smmu_find_master(smmu, sid); > > > > + if (!master) > > > > + return -EINVAL; > > > > + > > > > + if (evt[1] & EVTQ_1_READ) > > > > + perm |= IOMMU_FAULT_PERM_READ; > > > > + else > > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > > + > > > > + if (evt[1] & EVTQ_1_EXEC) > > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > > + > > > > + if (evt[1] & EVTQ_1_PRIV) > > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > > + > > > > + if (evt[1] & EVTQ_1_STALL) { > > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > > + flt->prm = (struct iommu_fault_page_request) { > > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > > + .perm = perm, > > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > > + }; > > > > + > > > > > > > + if (ssid_valid) > > > > + flt->prm.flags |= > > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > > > Do we need to set this for STALL mode only support? I had an issue > > > with this being set on a vSVA POC based on our D06 zip device(which is > > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > > and stag only. > > > > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall > > event and RESUME command in your prototype? Are you getting issues with > > the host driver or the guest driver? > > The issue is on the host side iommu_page_response(). The flow is something like > below. > > Stall: Host:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > vfio_pci_iommu_dev_fault_handler() > > Stall: Qemu:- > > vfio_dma_fault_notifier_handler() > inject_faults() > smmuv3_inject_faults() > > Stall: Guest:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > iommu_queue_iopf > ... > iopf_handle_group() > iopf_handle_single() > handle_mm_fault() > iopf_complete() > iommu_page_response() > arm_smmu_page_response() > arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) > > Resume: Qemu:- > > smmuv3_cmdq_consume(SMMU_CMD_RESUME) > smmuv3_notify_page_resp() > vfio:ioctl(page_response) --> struct iommu_page_response is filled > with only version, grpid and code. > > Resume: Host:- > ioctl(page_response) > iommu_page_response() --> fails as the pending req has PASID_VALID flag > set and it checks for a match. I believe the fix needs to be here. It's also wrong for PRI since not all PCIe endpoint require a PASID in the page response. Could you try the attached patch? Thanks, Jean > arm_smmu_page_response() > > Hope the above is clear. > > > We do need to forward the SSV flag all the way to the guest driver, so the guest > > can find the faulting address space using the SSID. Once the guest handled the > > fault, then we don't send the SSID back to the host as part of the RESUME > > command. > > True, the guest requires SSV flag to handle the page fault. But, as shown in the > flow above, the issue is on the host side iommu_page_response() where it > searches for a matching pending req based on pasid. Not sure we can bypass > that and call arm_smmu_page_response() directly but then have to delete the > pending req from the list as well. > > Please let me know if there is a better way to handle the host side page > response. > > Thanks, > Shameer > > > Thanks, > > Jean > > > > > Hence, it is difficult for > > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > > can see that this flag is being checked in iopf_handle_single() (patch > > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let > > me know your thoughts. > > > > > > Thanks, > > > Shameer > > > > > > 1. > > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > > 5897e4becfa378d15 > > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-iommu-Allow-page-responses-without-PASID.patch" >From 9baf5b9894d4e4be05e665d80fd0ebac8b621aa4 Mon Sep 17 00:00:00 2001 From: Jean-Philippe Brucker Date: Tue, 2 Jun 2020 13:13:27 +0200 Subject: [PATCH] iommu: Allow page responses without PASID Some PCIe devices do not expect a PASID value in PRI Page Responses. If the "PRG Response PASID Required" bit in the PRI capability is zero, then the OS should not set the PASID field. Similarly on Arm SMMU, responses to stall events do not have a PASID. Currently iommu_page_response() checks that the PASID in the page response corresponds to the one in the page request without first checking the "PASID valid" bit. A page response coming from a guest OS does not necessarily have a PASID, if the passed-through device does not require one. Allow page responses without PASID. The page request corresponding to a page response is identified by device and by Page Response Group Index (or stall tag). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e61a9fc65b7e4..e481fdfafb77c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1296,7 +1296,8 @@ int iommu_page_response(struct device *dev, */ list_for_each_entry(evt, ¶m->fault_param->faults, list) { prm = &evt->fault.prm; - pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID + && msg->flags & IOMMU_PAGE_RESP_PASID_VALID; if ((pasid_valid && prm->pasid != msg->pasid) || prm->grpid != msg->grpid) -- 2.26.2 --3MwIy2ne0vdjdPXF-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53923C433E2 for ; Tue, 2 Jun 2020 11:46:31 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 16349206A2 for ; Tue, 2 Jun 2020 11:46:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="yijCmqf0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16349206A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9C2B385AF1; Tue, 2 Jun 2020 11:46:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6cnhzoCRK_We; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1577A8584A; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id ED0FAC0178; Tue, 2 Jun 2020 11:46:28 +0000 (UTC) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 799F1C016E for ; Tue, 2 Jun 2020 11:46:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 60D0085876 for ; Tue, 2 Jun 2020 11:46:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8orwD3Zz_840 for ; Tue, 2 Jun 2020 11:46:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7ED5E8584A for ; Tue, 2 Jun 2020 11:46:25 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id l1so8973676ede.11 for ; Tue, 02 Jun 2020 04:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=yijCmqf0VG0ntw+uiPMpdUH4LMWIvqHsCVmInI53fK9mX51SGYwD72IA6DhJmlD97a UNj+wr1xk+MpFkSEXLQv8eoO3cAwX2PJr2eu2gMt+Wdvj5/sAsHXbV5SzeGPc9ipSP0Q iHfDaFS6bvC9IQ/hqK24QfP+vlNw2yFbtAIcg3kNx4x1y/EZKku2MJwvmhK6aMevkQFs 8lcMkEWWV5PLBUJ0P4IRbkLYfbsmxy3mAD17T5a69W7H5qUaRUYgujxCLI7dNY/j2CSu DluWkc/YGeWTqDa51l8CLmHNfPGrr46903qlPJBuohlTXpBNuzAlqELce9wbsiG2I2+U /T8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=lxsOdxneMb+0bGX+iIK3dczqqTYDLY8A6NwZb1AUm2W7TEkDz5nr5c/JOgCirlbFy1 Qj1lr+e5g6JeIPkUTCcOcDSgtOBxcNSxfkxQlewDItbF7FdUObnmPz2zQWyfhytURGMx +EzIN8ohYLmt+dM2sVKJhPuK6qG9ccOiEVhowL+fFe57BNsC4vxkzSDefxqvYreYhFTr zryBLeK4EneC1SQAA7kPD8OOAiHWg8QLKanPmhf6AByJRhGckfvPFMAOGohR3ICxxnjL hGx2MVfzNis5jCstF8EYCBYns7LRCCdjRZR9dSqNFF9TLrp9OGQhcIt+msZa2CUSk1NC bY5g== X-Gm-Message-State: AOAM532v+gt/Hn2ijASbtj30FjYsmdktBBvlyGh03Gou1290copnnjJZ 0MJEUquBRPDNLW5g9m193HRJCA== X-Google-Smtp-Source: ABdhPJzk9qpnI9yUmwOaxqU5ALMvmfWfI4N5ixuLoClwbRn/ueEUAkkLlMA8Jw8WKqsKsrA7069JbA== X-Received: by 2002:a05:6402:34e:: with SMTP id r14mr14619562edw.351.1591098383894; Tue, 02 Jun 2020 04:46:23 -0700 (PDT) Received: from myrica ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id h17sm1047269edw.92.2020.06.02.04.46.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 04:46:22 -0700 (PDT) Date: Tue, 2 Jun 2020 13:46:11 +0200 From: Jean-Philippe Brucker To: Shameerali Kolothum Thodi Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Message-ID: <20200602114611.GB1029680@myrica> References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> Cc: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "fenghua.yu@intel.com" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "hch@infradead.org" , "jgg@ziepe.ca" , "iommu@lists.linux-foundation.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "will@kernel.org" , "linux-mm@kvack.org" , "linux-arm-kernel@lists.infradead.org" X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote: > Hi Jean, > > > -----Original Message----- > > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > > On Behalf Of Jean-Philippe Brucker > > Sent: 02 June 2020 10:39 > > To: Shameerali Kolothum Thodi > > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org; > > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org; > > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org; > > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > > zhangfei.gao@linaro.org; robin.murphy@arm.com; > > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > > platform devices > > > > Hi Shameer, > > > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > > wrote: > > > > /* IRQ and event handlers */ > > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > > +*evt) { > > > > + int ret; > > > > + u32 perm = 0; > > > > + struct arm_smmu_master *master; > > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > > + struct iommu_fault_event fault_evt = { }; > > > > + struct iommu_fault *flt = &fault_evt.fault; > > > > + > > > > + /* Stage-2 is always pinned at the moment */ > > > > + if (evt[1] & EVTQ_1_S2) > > > > + return -EFAULT; > > > > + > > > > + master = arm_smmu_find_master(smmu, sid); > > > > + if (!master) > > > > + return -EINVAL; > > > > + > > > > + if (evt[1] & EVTQ_1_READ) > > > > + perm |= IOMMU_FAULT_PERM_READ; > > > > + else > > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > > + > > > > + if (evt[1] & EVTQ_1_EXEC) > > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > > + > > > > + if (evt[1] & EVTQ_1_PRIV) > > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > > + > > > > + if (evt[1] & EVTQ_1_STALL) { > > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > > + flt->prm = (struct iommu_fault_page_request) { > > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > > + .perm = perm, > > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > > + }; > > > > + > > > > > > > + if (ssid_valid) > > > > + flt->prm.flags |= > > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > > > Do we need to set this for STALL mode only support? I had an issue > > > with this being set on a vSVA POC based on our D06 zip device(which is > > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > > and stag only. > > > > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall > > event and RESUME command in your prototype? Are you getting issues with > > the host driver or the guest driver? > > The issue is on the host side iommu_page_response(). The flow is something like > below. > > Stall: Host:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > vfio_pci_iommu_dev_fault_handler() > > Stall: Qemu:- > > vfio_dma_fault_notifier_handler() > inject_faults() > smmuv3_inject_faults() > > Stall: Guest:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > iommu_queue_iopf > ... > iopf_handle_group() > iopf_handle_single() > handle_mm_fault() > iopf_complete() > iommu_page_response() > arm_smmu_page_response() > arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) > > Resume: Qemu:- > > smmuv3_cmdq_consume(SMMU_CMD_RESUME) > smmuv3_notify_page_resp() > vfio:ioctl(page_response) --> struct iommu_page_response is filled > with only version, grpid and code. > > Resume: Host:- > ioctl(page_response) > iommu_page_response() --> fails as the pending req has PASID_VALID flag > set and it checks for a match. I believe the fix needs to be here. It's also wrong for PRI since not all PCIe endpoint require a PASID in the page response. Could you try the attached patch? Thanks, Jean > arm_smmu_page_response() > > Hope the above is clear. > > > We do need to forward the SSV flag all the way to the guest driver, so the guest > > can find the faulting address space using the SSID. Once the guest handled the > > fault, then we don't send the SSID back to the host as part of the RESUME > > command. > > True, the guest requires SSV flag to handle the page fault. But, as shown in the > flow above, the issue is on the host side iommu_page_response() where it > searches for a matching pending req based on pasid. Not sure we can bypass > that and call arm_smmu_page_response() directly but then have to delete the > pending req from the list as well. > > Please let me know if there is a better way to handle the host side page > response. > > Thanks, > Shameer > > > Thanks, > > Jean > > > > > Hence, it is difficult for > > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > > can see that this flag is being checked in iopf_handle_single() (patch > > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let > > me know your thoughts. > > > > > > Thanks, > > > Shameer > > > > > > 1. > > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > > 5897e4becfa378d15 > > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-iommu-Allow-page-responses-without-PASID.patch" >From 9baf5b9894d4e4be05e665d80fd0ebac8b621aa4 Mon Sep 17 00:00:00 2001 From: Jean-Philippe Brucker Date: Tue, 2 Jun 2020 13:13:27 +0200 Subject: [PATCH] iommu: Allow page responses without PASID Some PCIe devices do not expect a PASID value in PRI Page Responses. If the "PRG Response PASID Required" bit in the PRI capability is zero, then the OS should not set the PASID field. Similarly on Arm SMMU, responses to stall events do not have a PASID. Currently iommu_page_response() checks that the PASID in the page response corresponds to the one in the page request without first checking the "PASID valid" bit. A page response coming from a guest OS does not necessarily have a PASID, if the passed-through device does not require one. Allow page responses without PASID. The page request corresponding to a page response is identified by device and by Page Response Group Index (or stall tag). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e61a9fc65b7e4..e481fdfafb77c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1296,7 +1296,8 @@ int iommu_page_response(struct device *dev, */ list_for_each_entry(evt, ¶m->fault_param->faults, list) { prm = &evt->fault.prm; - pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID + && msg->flags & IOMMU_PAGE_RESP_PASID_VALID; if ((pasid_valid && prm->pasid != msg->pasid) || prm->grpid != msg->grpid) -- 2.26.2 --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu --3MwIy2ne0vdjdPXF-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EF9EC433E3 for ; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 74D2C207ED for ; Tue, 2 Jun 2020 11:46:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="KnZ9j2RU"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="yijCmqf0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74D2C207ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To: Content-Type:MIME-Version:References:Message-ID:Subject:To:From:Date:Reply-To :Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=00VDNtZtuFeWgyRTmCVuwQbYpbsgMcPBV9ONAlVLybw=; b=KnZ9j2RUCMG+SWxXRBbj7UG+Q Gt8RkLS9Z1hIQhzFvhKcq5eXcNjShkM/9Y6Ux0p99D7Hi8mw6oVe9/rmuZ9ahCZ2Nj6pJjpp5sUPm uMl/VMuZHxuo2mPoZnqUsr8SH2oLSU8RE1t5BbwRh/SXEVcKXVmqFbF7LQVDlG5kJtsAx3M2XtuOg dSl9+BAvYUO0YpcGsRbeg82rDOg8w7IMVp0ve1idtItjrI0qr5QY3XlcYH7rJsndNJ5DwcmSR2Xu/ Ba+7ZVPTwCadUbwkW7QTmNBpFDXJzL4+LTSxtulwNmaD9ZcNAQQe+oOw+TacW1BNc597ErRJ3x8Fq KPx6RFLFA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jg5NE-0002Nq-VZ; Tue, 02 Jun 2020 11:46:28 +0000 Received: from mail-ed1-x544.google.com ([2a00:1450:4864:20::544]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jg5NB-0002Mg-Hv for linux-arm-kernel@lists.infradead.org; Tue, 02 Jun 2020 11:46:27 +0000 Received: by mail-ed1-x544.google.com with SMTP id g1so8943935edv.6 for ; Tue, 02 Jun 2020 04:46:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=yijCmqf0VG0ntw+uiPMpdUH4LMWIvqHsCVmInI53fK9mX51SGYwD72IA6DhJmlD97a UNj+wr1xk+MpFkSEXLQv8eoO3cAwX2PJr2eu2gMt+Wdvj5/sAsHXbV5SzeGPc9ipSP0Q iHfDaFS6bvC9IQ/hqK24QfP+vlNw2yFbtAIcg3kNx4x1y/EZKku2MJwvmhK6aMevkQFs 8lcMkEWWV5PLBUJ0P4IRbkLYfbsmxy3mAD17T5a69W7H5qUaRUYgujxCLI7dNY/j2CSu DluWkc/YGeWTqDa51l8CLmHNfPGrr46903qlPJBuohlTXpBNuzAlqELce9wbsiG2I2+U /T8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=ics4wOjFIjzNBxAIlnIKnUmEnmyw0rmLzT7c6ne7pSI=; b=D/BAdZjbpsy7LRAqeMUZ182Z78DL2lT4mGhuuzzqeqUPOIYl5/pxRCze+pPKPCj6Xp 4eyn0WK+KwD7UmgLFGRVGXSpJ7FL8YsMwAdEebrkkVKCyR+O6rvsQzXTD2MBeZNsTcPm n5EMIywz26n5R9+HT1LXhEZiwnI+6Lo1s+ZIxPYaU/+19TbC+zJdi0gt0AQs2lQ6qrPm vRk2XzGbZLWk8WOrd3QiYF/M2o0dWHcj4TETuKC7qhLzt7L/BKq4UWBRz3WhrM+GpT0s b2T+W5wc6ANDMoLBvUZpbZDiEwrYRQaBw38V+WjZ9CBQIvVpgkCn4wNXQFDMkXiEhJAX l/PA== X-Gm-Message-State: AOAM533kNYQRnYJX/OihtsWsX6SBCUf8fy4AfOWsuFzI3hngMzwhaU+9 6SUNAaPjQ0urj9xNdRljsrW5Xg== X-Google-Smtp-Source: ABdhPJzk9qpnI9yUmwOaxqU5ALMvmfWfI4N5ixuLoClwbRn/ueEUAkkLlMA8Jw8WKqsKsrA7069JbA== X-Received: by 2002:a05:6402:34e:: with SMTP id r14mr14619562edw.351.1591098383894; Tue, 02 Jun 2020 04:46:23 -0700 (PDT) Received: from myrica ([2001:171b:226e:c200:c43b:ef78:d083:b355]) by smtp.gmail.com with ESMTPSA id h17sm1047269edw.92.2020.06.02.04.46.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jun 2020 04:46:22 -0700 (PDT) Date: Tue, 2 Jun 2020 13:46:11 +0200 From: Jean-Philippe Brucker To: Shameerali Kolothum Thodi Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for platform devices Message-ID: <20200602114611.GB1029680@myrica> References: <20200519175502.2504091-1-jean-philippe@linaro.org> <20200519175502.2504091-22-jean-philippe@linaro.org> <4741b6c45d1a43b69041ecb5ce0be0d5@huawei.com> <20200602093836.GA1029680@myrica> <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="3MwIy2ne0vdjdPXF" Content-Disposition: inline In-Reply-To: <1517c4d97b5849e6b6d32e7d7ed35289@huawei.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200602_044625_623951_34E8D5F4 X-CRM114-Status: GOOD ( 39.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "devicetree@vger.kernel.org" , "kevin.tian@intel.com" , "fenghua.yu@intel.com" , "linux-pci@vger.kernel.org" , "felix.kuehling@amd.com" , "robin.murphy@arm.com" , "christian.koenig@amd.com" , "hch@infradead.org" , "jgg@ziepe.ca" , "iommu@lists.linux-foundation.org" , "catalin.marinas@arm.com" , "zhangfei.gao@linaro.org" , "will@kernel.org" , "linux-mm@kvack.org" , "linux-arm-kernel@lists.infradead.org" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Jun 02, 2020 at 10:31:29AM +0000, Shameerali Kolothum Thodi wrote: > Hi Jean, > > > -----Original Message----- > > From: linux-arm-kernel [mailto:linux-arm-kernel-bounces@lists.infradead.org] > > On Behalf Of Jean-Philippe Brucker > > Sent: 02 June 2020 10:39 > > To: Shameerali Kolothum Thodi > > Cc: devicetree@vger.kernel.org; kevin.tian@intel.com; will@kernel.org; > > fenghua.yu@intel.com; jgg@ziepe.ca; linux-pci@vger.kernel.org; > > felix.kuehling@amd.com; hch@infradead.org; linux-mm@kvack.org; > > iommu@lists.linux-foundation.org; catalin.marinas@arm.com; > > zhangfei.gao@linaro.org; robin.murphy@arm.com; > > christian.koenig@amd.com; linux-arm-kernel@lists.infradead.org > > Subject: Re: [PATCH v7 21/24] iommu/arm-smmu-v3: Add stall support for > > platform devices > > > > Hi Shameer, > > > > On Mon, Jun 01, 2020 at 12:42:15PM +0000, Shameerali Kolothum Thodi > > wrote: > > > > /* IRQ and event handlers */ > > > > +static int arm_smmu_handle_evt(struct arm_smmu_device *smmu, u64 > > > > +*evt) { > > > > + int ret; > > > > + u32 perm = 0; > > > > + struct arm_smmu_master *master; > > > > + bool ssid_valid = evt[0] & EVTQ_0_SSV; > > > > + u8 type = FIELD_GET(EVTQ_0_ID, evt[0]); > > > > + u32 sid = FIELD_GET(EVTQ_0_SID, evt[0]); > > > > + struct iommu_fault_event fault_evt = { }; > > > > + struct iommu_fault *flt = &fault_evt.fault; > > > > + > > > > + /* Stage-2 is always pinned at the moment */ > > > > + if (evt[1] & EVTQ_1_S2) > > > > + return -EFAULT; > > > > + > > > > + master = arm_smmu_find_master(smmu, sid); > > > > + if (!master) > > > > + return -EINVAL; > > > > + > > > > + if (evt[1] & EVTQ_1_READ) > > > > + perm |= IOMMU_FAULT_PERM_READ; > > > > + else > > > > + perm |= IOMMU_FAULT_PERM_WRITE; > > > > + > > > > + if (evt[1] & EVTQ_1_EXEC) > > > > + perm |= IOMMU_FAULT_PERM_EXEC; > > > > + > > > > + if (evt[1] & EVTQ_1_PRIV) > > > > + perm |= IOMMU_FAULT_PERM_PRIV; > > > > + > > > > + if (evt[1] & EVTQ_1_STALL) { > > > > + flt->type = IOMMU_FAULT_PAGE_REQ; > > > > + flt->prm = (struct iommu_fault_page_request) { > > > > + .flags = IOMMU_FAULT_PAGE_REQUEST_LAST_PAGE, > > > > + .pasid = FIELD_GET(EVTQ_0_SSID, evt[0]), > > > > + .grpid = FIELD_GET(EVTQ_1_STAG, evt[1]), > > > > + .perm = perm, > > > > + .addr = FIELD_GET(EVTQ_2_ADDR, evt[2]), > > > > + }; > > > > + > > > > > > > + if (ssid_valid) > > > > + flt->prm.flags |= > > IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; > > > > > > Do we need to set this for STALL mode only support? I had an issue > > > with this being set on a vSVA POC based on our D06 zip device(which is > > > a "fake " pci dev that supports STALL mode but no PRI). The issue is, > > > CMDQ_OP_RESUME doesn't have any ssid or SSV params and works on sid > > and stag only. > > > > I don't understand the problem, arm_smmu_page_response() doesn't set SSID > > or SSV when sending a CMDQ_OP_RESUME. Could you detail the flow of a stall > > event and RESUME command in your prototype? Are you getting issues with > > the host driver or the guest driver? > > The issue is on the host side iommu_page_response(). The flow is something like > below. > > Stall: Host:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > vfio_pci_iommu_dev_fault_handler() > > Stall: Qemu:- > > vfio_dma_fault_notifier_handler() > inject_faults() > smmuv3_inject_faults() > > Stall: Guest:- > > arm_smmu_handle_evt() > iommu_report_device_fault() > iommu_queue_iopf > ... > iopf_handle_group() > iopf_handle_single() > handle_mm_fault() > iopf_complete() > iommu_page_response() > arm_smmu_page_response() > arm_smmu_cmdq_issue_cmd(CMDQ_OP_RESUME) > > Resume: Qemu:- > > smmuv3_cmdq_consume(SMMU_CMD_RESUME) > smmuv3_notify_page_resp() > vfio:ioctl(page_response) --> struct iommu_page_response is filled > with only version, grpid and code. > > Resume: Host:- > ioctl(page_response) > iommu_page_response() --> fails as the pending req has PASID_VALID flag > set and it checks for a match. I believe the fix needs to be here. It's also wrong for PRI since not all PCIe endpoint require a PASID in the page response. Could you try the attached patch? Thanks, Jean > arm_smmu_page_response() > > Hope the above is clear. > > > We do need to forward the SSV flag all the way to the guest driver, so the guest > > can find the faulting address space using the SSID. Once the guest handled the > > fault, then we don't send the SSID back to the host as part of the RESUME > > command. > > True, the guest requires SSV flag to handle the page fault. But, as shown in the > flow above, the issue is on the host side iommu_page_response() where it > searches for a matching pending req based on pasid. Not sure we can bypass > that and call arm_smmu_page_response() directly but then have to delete the > pending req from the list as well. > > Please let me know if there is a better way to handle the host side page > response. > > Thanks, > Shameer > > > Thanks, > > Jean > > > > > Hence, it is difficult for > > > Qemu SMMUv3 to populate this fields while preparing a page response. I > > > can see that this flag is being checked in iopf_handle_single() (patch > > > 04/24) as well. For POC, I used a temp fix[1] to work around this. Please let > > me know your thoughts. > > > > > > Thanks, > > > Shameer > > > > > > 1. > > > https://github.com/hisilicon/kernel-dev/commit/99ff96146e924055f38d97a > > > 5897e4becfa378d15 > > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-iommu-Allow-page-responses-without-PASID.patch" >From 9baf5b9894d4e4be05e665d80fd0ebac8b621aa4 Mon Sep 17 00:00:00 2001 From: Jean-Philippe Brucker Date: Tue, 2 Jun 2020 13:13:27 +0200 Subject: [PATCH] iommu: Allow page responses without PASID Some PCIe devices do not expect a PASID value in PRI Page Responses. If the "PRG Response PASID Required" bit in the PRI capability is zero, then the OS should not set the PASID field. Similarly on Arm SMMU, responses to stall events do not have a PASID. Currently iommu_page_response() checks that the PASID in the page response corresponds to the one in the page request without first checking the "PASID valid" bit. A page response coming from a guest OS does not necessarily have a PASID, if the passed-through device does not require one. Allow page responses without PASID. The page request corresponding to a page response is identified by device and by Page Response Group Index (or stall tag). Signed-off-by: Jean-Philippe Brucker --- drivers/iommu/iommu.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index e61a9fc65b7e4..e481fdfafb77c 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1296,7 +1296,8 @@ int iommu_page_response(struct device *dev, */ list_for_each_entry(evt, ¶m->fault_param->faults, list) { prm = &evt->fault.prm; - pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID; + pasid_valid = prm->flags & IOMMU_FAULT_PAGE_REQUEST_PASID_VALID + && msg->flags & IOMMU_PAGE_RESP_PASID_VALID; if ((pasid_valid && prm->pasid != msg->pasid) || prm->grpid != msg->grpid) -- 2.26.2 --3MwIy2ne0vdjdPXF Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel --3MwIy2ne0vdjdPXF--