From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19518C04EB8 for ; Tue, 4 Dec 2018 06:23:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CAA4F208A3 for ; Tue, 4 Dec 2018 06:23:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CAA4F208A3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726175AbeLDGXL (ORCPT ); Tue, 4 Dec 2018 01:23:11 -0500 Received: from mga17.intel.com ([192.55.52.151]:8029 "EHLO mga17.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725983AbeLDGXK (ORCPT ); Tue, 4 Dec 2018 01:23:10 -0500 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Dec 2018 22:23:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,312,1539673200"; d="scan'208";a="115843305" Received: from allen-box.sh.intel.com (HELO [10.239.161.122]) ([10.239.161.122]) by FMSMGA003.fm.intel.com with ESMTP; 03 Dec 2018 22:23:06 -0800 Cc: baolu.lu@linux.intel.com, kevin.tian@intel.com, ashok.raj@intel.com, tiwei.bie@intel.com, Jean-Philippe Brucker , sanjay.k.kumar@intel.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, yi.y.sun@intel.com, jacob.jun.pan@intel.com, kvm@vger.kernel.org Subject: Re: [PATCH v4 0/8] vfio/mdev: IOMMU aware mediated device To: Xu Zaibo , Joerg Roedel , David Woodhouse , Alex Williamson , Kirti Wankhede References: <20181105073408.21815-1-baolu.lu@linux.intel.com> <5C05F87D.4050206@huawei.com> From: Lu Baolu Message-ID: <9a1d0b6f-7363-495d-2f4b-5a1693fb3fd9@linux.intel.com> Date: Tue, 4 Dec 2018 14:20:02 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5C05F87D.4050206@huawei.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On 12/4/18 11:46 AM, Xu Zaibo wrote: > Hi, > > Is this solution trying to support general user space processes who are > directly working on devices? Yes, it is. > > Thanks, > Zaibo Best regards, Lu Baolu > > . > > On 2018/11/5 15:34, Lu Baolu wrote: >> Hi, >> >> The Mediate Device is a framework for fine-grained physical device >> sharing across the isolated domains. Currently the mdev framework >> is designed to be independent of the platform IOMMU support. As the >> result, the DMA isolation relies on the mdev parent device in a >> vendor specific way. >> >> There are several cases where a mediated device could be protected >> and isolated by the platform IOMMU. For example, Intel vt-d rev3.0 >> [1] introduces a new translation mode called 'scalable mode', which >> enables PASID-granular translations. The vt-d scalable mode is the >> key ingredient for Scalable I/O Virtualization [2] [3] which allows >> sharing a device in minimal possible granularity (ADI - Assignable >> Device Interface). >> >> A mediated device backed by an ADI could be protected and isolated >> by the IOMMU since 1) the parent device supports tagging an unique >> PASID to all DMA traffic out of the mediated device; and 2) the DMA >> translation unit (IOMMU) supports the PASID granular translation. >> We can apply IOMMU protection and isolation to this kind of devices >> just as what we are doing with an assignable PCI device. >> >> In order to distinguish the IOMMU-capable mediated devices from those >> which still need to rely on parent devices, this patch set adds two >> new members in struct mdev_device. >> >> * iommu_device >>    - This, if set, indicates that the mediated device could >>      be fully isolated and protected by IOMMU via attaching >>      an iommu domain to this device. If empty, it indicates >>      using vendor defined isolation. >> >> * iommu_domain >>    - This is a place holder for an iommu domain. A domain >>      could be store here for later use once it has been >>      attached to the iommu_device of this mdev. >> >> Below helpers are added to set and get above iommu device >> and iommu domain pointers in mdev core implementation. >> >> * mdev_set/get_iommu_device(dev, iommu_device) >>    - Set or get the iommu device which represents this mdev >>      in IOMMU's device scope. Drivers don't need to set the >>      iommu device if it uses vendor defined isolation. >> >> * mdev_set/get_iommu_domain(domain) >>    - A iommu domain which has been attached to the iommu >>      device in order to protect and isolate the mediated >>      device will be kept in the mdev data structure and >>      could be retrieved later. >> >> The mdev parent device driver could opt-in that the mdev could be >> fully isolated and protected by the IOMMU when the mdev is being >> created by invoking mdev_set_iommu_device() in its @create(). >> >> In the vfio_iommu_type1_attach_group(), a domain allocated through >> iommu_domain_alloc() will be attached to the mdev iommu device if >> an iommu device has been set. Otherwise, the dummy external domain >> will be used and all the DMA isolation and protection are routed to >> parent driver as the result. >> >> On IOMMU side, a basic requirement is allowing to attach multiple >> domains to a PCI device if the device advertises the capability >> and the IOMMU hardware supports finer granularity translations than >> the normal PCI Source ID based translation. >> >> As the result, a PCI device could work in two modes: normal mode >> and auxiliary mode. In the normal mode, a pci device could be >> isolated in the Source ID granularity; the pci device itself could >> be assigned to a user application by attaching a single domain >> to it. In the auxiliary mode, a pci device could be isolated in >> finer granularity, hence subsets of the device could be assigned >> to different user level application by attaching a different domain >> to each subset. >> >> The device driver is able to switch between above two modes with >> below interfaces: >> >> * iommu_get_dev_attr(dev, IOMMU_DEV_ATTR_AUXD_CAPABILITY) >>    - Represents the ability of supporting multiple domains >>      per device. >> >> * iommu_set_dev_attr(dev, IOMMU_DEV_ATTR_AUXD_ENABLE) >>    - Enable the multiple domains capability for the device >>      referenced by @dev. >> >> * iommu_set_dev_attr(dev, IOMMU_DEV_ATTR_AUXD_DISABLE) >>    - Disable the multiple domains capability for the device >>      referenced by @dev. >> >> * iommu_domain_get_attr(domain, DOMAIN_ATTR_AUXD_ID) >>    - Return ID used for finer-granularity DMA translation. >> >> * iommu_attach_device_aux(domain, dev) >>    - Attach a domain to the device in the auxiliary mode. >> >> * iommu_detach_device_aux(domain, dev) >>    - Detach the aux domain from device. >> >> In order for the ease of discussion, sometimes we call "a domain in >> auxiliary mode' or simply 'an auxiliary domain' when a domain is >> attached to a device for finer granularity translations. But we need >> to keep in mind that this doesn't mean there is a differnt domain >> type. A same domain could be bound to a device for Source ID based >> translation, and bound to another device for finer granularity >> translation at the same time. >> >> This patch series extends both IOMMU and vfio components to support >> mdev device passing through when it could be isolated and protected >> by the IOMMU units. The first part of this series (PATCH 1/08~5/08) >> adds the interfaces and implementation of the multiple domains per >> device. The second part (PATCH 6/08~8/08) adds the iommu device >> attribute to each mdev, determines isolation type according to the >> existence of an iommu device when attaching group in vfio type1 iommu >> module, and attaches the domain to iommu aware mediated devices. >> >> This patch series depends on a patch set posted here [4] for discussion >> which added scalable mode support in Intel IOMMU driver. >> >> References: >> [1] >> https://software.intel.com/en-us/download/intel-virtualization-technology-for-directed-io-architecture-specification >> >> [2] >> https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification >> >> [3] https://schd.ws/hosted_files/lc32018/00/LC3-SIOV-final.pdf >> [4] https://lkml.org/lkml/2018/11/5/136 >> >> Best regards, >> Lu Baolu >> >> Change log: >>    v3->v4: >>    - Use aux domain specific interfaces for domain attach and detach. >>    - Rebase all patches to 4.20-rc1. >> >>    v2->v3: >>    - Remove domain type enum and use a pointer on mdev_device instead. >>    - Add a generic interface for getting/setting per device iommu >>      attributions. And use it for query aux domain capability, enable >>      aux domain and disable aux domain purpose. >>    - Reuse iommu_domain_get_attr() to retrieve the id in a aux domain. >>    - We discussed the impact of the default domain implementation >>      on reusing iommu_at(de)tach_device() interfaces. We agreed >>      that reusing iommu_at(de)tach_device() interfaces is the right >>      direction and we could tweak the code to remove the impact. >>      https://www.spinics.net/lists/kvm/msg175285.html >>    - Removed the RFC tag since no objections received. >>    - This patch has been submitted separately. >>      https://www.spinics.net/lists/kvm/msg173936.html >> >>    v1->v2: >>    - Rewrite the patches with the concept of auxiliary domains. >> >> Lu Baolu (8): >>    iommu: Add APIs for multiple domains per device >>    iommu/vt-d: Add multiple domains per device query >>    iommu/vt-d: Enable/disable multiple domains per device >>    iommu/vt-d: Attach/detach domains in auxiliary mode >>    iommu/vt-d: Return ID associated with an auxiliary domain >>    vfio/mdev: Add iommu place holders in mdev_device >>    vfio/type1: Add domain at(de)taching group helpers >>    vfio/type1: Handle different mdev isolation type >> >>   drivers/iommu/intel-iommu.c      | 315 ++++++++++++++++++++++++++++--- >>   drivers/iommu/iommu.c            |  52 +++++ >>   drivers/vfio/mdev/mdev_core.c    |  36 ++++ >>   drivers/vfio/mdev/mdev_private.h |   2 + >>   drivers/vfio/vfio_iommu_type1.c  | 162 ++++++++++++++-- >>   include/linux/intel-iommu.h      |  11 ++ >>   include/linux/iommu.h            |  52 +++++ >>   include/linux/mdev.h             |  23 +++ >>   8 files changed, 618 insertions(+), 35 deletions(-) >> > > >