From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDAC7C433F5 for ; Wed, 29 Sep 2021 02:26:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A28F9613DB for ; Wed, 29 Sep 2021 02:26:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243782AbhI2C1j (ORCPT ); Tue, 28 Sep 2021 22:27:39 -0400 Received: from mga17.intel.com ([192.55.52.151]:64380 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243226AbhI2C1i (ORCPT ); Tue, 28 Sep 2021 22:27:38 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10121"; a="204995620" X-IronPort-AV: E=Sophos;i="5.85,331,1624345200"; d="scan'208";a="204995620" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2021 19:25:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,331,1624345200"; d="scan'208";a="476495153" Received: from allen-box.sh.intel.com (HELO [10.239.159.118]) ([10.239.159.118]) by orsmga007.jf.intel.com with ESMTP; 28 Sep 2021 19:25:45 -0700 Cc: baolu.lu@linux.intel.com, "Tian, Kevin" , "Liu, Yi L" , "alex.williamson@redhat.com" , "hch@lst.de" , "jasowang@redhat.com" , "joro@8bytes.org" , "jean-philippe@linaro.org" , "parav@mellanox.com" , "lkml@metux.net" , "pbonzini@redhat.com" , "lushenming@huawei.com" , "eric.auger@redhat.com" , "corbet@lwn.net" , "Raj, Ashok" , "yi.l.liu@linux.intel.com" , "Tian, Jun J" , "Wu, Hao" , "Jiang, Dave" , "jacob.jun.pan@linux.intel.com" , "kwankhede@nvidia.com" , "robin.murphy@arm.com" , "kvm@vger.kernel.org" , "iommu@lists.linux-foundation.org" , "dwmw2@infradead.org" , "linux-kernel@vger.kernel.org" , "david@gibson.dropbear.id.au" , "nicolinc@nvidia.com" Subject: Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces To: Jason Gunthorpe References: <20210919063848.1476776-1-yi.l.liu@intel.com> <20210919063848.1476776-7-yi.l.liu@intel.com> <20210921170943.GS327412@nvidia.com> <20210922123931.GI327412@nvidia.com> <20210927150928.GA1517957@nvidia.com> <20210928115751.GK964074@nvidia.com> <9a314095-3db9-30fc-2ed9-4e46d385036d@linux.intel.com> <20210928140712.GL964074@nvidia.com> From: Lu Baolu Message-ID: <4ba3294b-1628-0522-17ff-8aa38ed5a615@linux.intel.com> Date: Wed, 29 Sep 2021 10:22:12 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <20210928140712.GL964074@nvidia.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 9/28/21 10:07 PM, Jason Gunthorpe wrote: > On Tue, Sep 28, 2021 at 09:35:05PM +0800, Lu Baolu wrote: >> Another issue is, when putting a device into user-dma mode, all devices >> belonging to the same iommu group shouldn't be bound with a kernel-dma >> driver. Kevin's prototype checks this by READ_ONCE(dev->driver). This is >> not lock safe as discussed below, >> >> https://lore.kernel.org/linux-iommu/20210927130935.GZ964074@nvidia.com/ >> >> Any guidance on this? > > Something like this? > > > int iommu_set_device_dma_owner(struct device *dev, enum device_dma_owner mode, > struct file *user_owner) > { > struct iommu_group *group = group_from_dev(dev); > > spin_lock(&iommu_group->dma_owner_lock); > switch (mode) { > case DMA_OWNER_KERNEL: > if (iommu_group->dma_users[DMA_OWNER_USERSPACE]) > return -EBUSY; > break; > case DMA_OWNER_SHARED: > break; > case DMA_OWNER_USERSPACE: > if (iommu_group->dma_users[DMA_OWNER_KERNEL]) > return -EBUSY; > if (iommu_group->dma_owner_file != user_owner) { > if (iommu_group->dma_users[DMA_OWNER_USERSPACE]) > return -EPERM; > get_file(user_owner); > iommu_group->dma_owner_file = user_owner; > } > break; > default: > spin_unlock(&iommu_group->dma_owner_lock); > return -EINVAL; > } > iommu_group->dma_users[mode]++; > spin_unlock(&iommu_group->dma_owner_lock); > return 0; > } > > int iommu_release_device_dma_owner(struct device *dev, > enum device_dma_owner mode) > { > struct iommu_group *group = group_from_dev(dev); > > spin_lock(&iommu_group->dma_owner_lock); > if (WARN_ON(!iommu_group->dma_users[mode])) > goto err_unlock; > if (!iommu_group->dma_users[mode]--) { > if (mode == DMA_OWNER_USERSPACE) { > fput(iommu_group->dma_owner_file); > iommu_group->dma_owner_file = NULL; > } > } > err_unlock: > spin_unlock(&iommu_group->dma_owner_lock); > } > > > Where, the driver core does before probe: > > iommu_set_device_dma_owner(dev, DMA_OWNER_KERNEL, NULL) > > pci_stub/etc does in their probe func: > > iommu_set_device_dma_owner(dev, DMA_OWNER_SHARED, NULL) > > And vfio/iommfd does when a struct vfio_device FD is attached: > > iommu_set_device_dma_owner(dev, DMA_OWNER_USERSPACE, group_file/iommu_file) Really good design. It also helps alleviating some pains elsewhere in the iommu core. Just a nit comment, we also need DMA_OWNER_NONE which will be set when the driver core unbinds the driver from the device. > > Jason > Best regards, baolu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4867CC433F5 for ; Wed, 29 Sep 2021 02:25:57 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E208B613E6 for ; Wed, 29 Sep 2021 02:25:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E208B613E6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AFDDE80F57; Wed, 29 Sep 2021 02:25:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Vb8wOxXqLuV; Wed, 29 Sep 2021 02:25:55 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 819178174C; Wed, 29 Sep 2021 02:25:55 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 54FDCC0011; Wed, 29 Sep 2021 02:25:55 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1B171C000D for ; Wed, 29 Sep 2021 02:25:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 02E62415DF for ; Wed, 29 Sep 2021 02:25:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qv0YF6IsjR44 for ; Wed, 29 Sep 2021 02:25:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by smtp4.osuosl.org (Postfix) with ESMTPS id 3987F415CF for ; Wed, 29 Sep 2021 02:25:53 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10121"; a="285860300" X-IronPort-AV: E=Sophos;i="5.85,331,1624345200"; d="scan'208";a="285860300" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Sep 2021 19:25:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,331,1624345200"; d="scan'208";a="476495153" Received: from allen-box.sh.intel.com (HELO [10.239.159.118]) ([10.239.159.118]) by orsmga007.jf.intel.com with ESMTP; 28 Sep 2021 19:25:45 -0700 Subject: Re: [RFC 06/20] iommu: Add iommu_device_init[exit]_user_dma interfaces To: Jason Gunthorpe References: <20210919063848.1476776-1-yi.l.liu@intel.com> <20210919063848.1476776-7-yi.l.liu@intel.com> <20210921170943.GS327412@nvidia.com> <20210922123931.GI327412@nvidia.com> <20210927150928.GA1517957@nvidia.com> <20210928115751.GK964074@nvidia.com> <9a314095-3db9-30fc-2ed9-4e46d385036d@linux.intel.com> <20210928140712.GL964074@nvidia.com> From: Lu Baolu Message-ID: <4ba3294b-1628-0522-17ff-8aa38ed5a615@linux.intel.com> Date: Wed, 29 Sep 2021 10:22:12 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <20210928140712.GL964074@nvidia.com> Content-Language: en-US Cc: "kvm@vger.kernel.org" , "jasowang@redhat.com" , "kwankhede@nvidia.com" , "hch@lst.de" , "jean-philippe@linaro.org" , "Jiang, Dave" , "Raj, Ashok" , "corbet@lwn.net" , "Tian, Kevin" , "parav@mellanox.com" , "alex.williamson@redhat.com" , "lkml@metux.net" , "david@gibson.dropbear.id.au" , "dwmw2@infradead.org" , "Tian, Jun J" , "linux-kernel@vger.kernel.org" , "lushenming@huawei.com" , "iommu@lists.linux-foundation.org" , "pbonzini@redhat.com" , "robin.murphy@arm.com" X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On 9/28/21 10:07 PM, Jason Gunthorpe wrote: > On Tue, Sep 28, 2021 at 09:35:05PM +0800, Lu Baolu wrote: >> Another issue is, when putting a device into user-dma mode, all devices >> belonging to the same iommu group shouldn't be bound with a kernel-dma >> driver. Kevin's prototype checks this by READ_ONCE(dev->driver). This is >> not lock safe as discussed below, >> >> https://lore.kernel.org/linux-iommu/20210927130935.GZ964074@nvidia.com/ >> >> Any guidance on this? > > Something like this? > > > int iommu_set_device_dma_owner(struct device *dev, enum device_dma_owner mode, > struct file *user_owner) > { > struct iommu_group *group = group_from_dev(dev); > > spin_lock(&iommu_group->dma_owner_lock); > switch (mode) { > case DMA_OWNER_KERNEL: > if (iommu_group->dma_users[DMA_OWNER_USERSPACE]) > return -EBUSY; > break; > case DMA_OWNER_SHARED: > break; > case DMA_OWNER_USERSPACE: > if (iommu_group->dma_users[DMA_OWNER_KERNEL]) > return -EBUSY; > if (iommu_group->dma_owner_file != user_owner) { > if (iommu_group->dma_users[DMA_OWNER_USERSPACE]) > return -EPERM; > get_file(user_owner); > iommu_group->dma_owner_file = user_owner; > } > break; > default: > spin_unlock(&iommu_group->dma_owner_lock); > return -EINVAL; > } > iommu_group->dma_users[mode]++; > spin_unlock(&iommu_group->dma_owner_lock); > return 0; > } > > int iommu_release_device_dma_owner(struct device *dev, > enum device_dma_owner mode) > { > struct iommu_group *group = group_from_dev(dev); > > spin_lock(&iommu_group->dma_owner_lock); > if (WARN_ON(!iommu_group->dma_users[mode])) > goto err_unlock; > if (!iommu_group->dma_users[mode]--) { > if (mode == DMA_OWNER_USERSPACE) { > fput(iommu_group->dma_owner_file); > iommu_group->dma_owner_file = NULL; > } > } > err_unlock: > spin_unlock(&iommu_group->dma_owner_lock); > } > > > Where, the driver core does before probe: > > iommu_set_device_dma_owner(dev, DMA_OWNER_KERNEL, NULL) > > pci_stub/etc does in their probe func: > > iommu_set_device_dma_owner(dev, DMA_OWNER_SHARED, NULL) > > And vfio/iommfd does when a struct vfio_device FD is attached: > > iommu_set_device_dma_owner(dev, DMA_OWNER_USERSPACE, group_file/iommu_file) Really good design. It also helps alleviating some pains elsewhere in the iommu core. Just a nit comment, we also need DMA_OWNER_NONE which will be set when the driver core unbinds the driver from the device. > > Jason > Best regards, baolu _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu