From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B0A6C07E96 for ; Thu, 8 Jul 2021 13:38:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2FC6A616ED for ; Thu, 8 Jul 2021 13:38:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231849AbhGHNlV (ORCPT ); Thu, 8 Jul 2021 09:41:21 -0400 Received: from mga09.intel.com ([134.134.136.24]:2879 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231817AbhGHNlU (ORCPT ); Thu, 8 Jul 2021 09:41:20 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10038"; a="209467995" X-IronPort-AV: E=Sophos;i="5.84,222,1620716400"; d="scan'208";a="209467995" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2021 06:38:38 -0700 X-IronPort-AV: E=Sophos;i="5.84,222,1620716400"; d="scan'208";a="487606704" Received: from blu2-mobl3.ccr.corp.intel.com (HELO [10.254.214.103]) ([10.254.214.103]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2021 06:38:36 -0700 Cc: baolu.lu@linux.intel.com, Sergey Senozhatsky , iommu@lists.linux-foundation.org, David Stevens , Christoph Hellwig , linux-kernel@vger.kernel.org To: David Stevens , Joerg Roedel , Will Deacon References: <20210707075505.2896824-1-stevensd@google.com> From: Lu Baolu Subject: Re: [PATCH 0/4] Add dynamic iommu backed bounce buffers Message-ID: Date: Thu, 8 Jul 2021 21:38:34 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210707075505.2896824-1-stevensd@google.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi David, I like this idea. Thanks for proposing this. On 2021/7/7 15:55, David Stevens wrote: > Add support for per-domain dynamic pools of iommu bounce buffers to the > dma-iommu API. This allows iommu mappings to be reused while still > maintaining strict iommu protection. Allocating buffers dynamically > instead of using swiotlb carveouts makes per-domain pools more amenable > on systems with large numbers of devices or where devices are unknown. Have you ever considered leveraging the per-device swiotlb memory pool added by below series? https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/ > > When enabled, all non-direct streaming mappings below a configurable > size will go through bounce buffers. Note that this means drivers which > don't properly use the DMA API (e.g. i915) cannot use an iommu when this > feature is enabled. However, all drivers which work with swiotlb=force > should work. If so, why not making it more scalable by adding a callback into vendor iommu drivers? The vendor iommu drivers have enough information to tell whether the bounce buffer is feasible for a specific domain. > > Bounce buffers serve as an optimization in situations where interactions > with the iommu are very costly. For example, virtio-iommu operations in The simulated IOMMU does the same thing. It's also an optimization for bare metal in cases where the strict mode of cache invalidation is used. CPU moving data is faster than IOMMU cache invalidation if the buffer is small. Best regards, baolu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D3CEC07E96 for ; Thu, 8 Jul 2021 13:38:45 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E2AA1616E8 for ; Thu, 8 Jul 2021 13:38:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2AA1616E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A5CB240259; Thu, 8 Jul 2021 13:38:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oUba-WGdin0Z; Thu, 8 Jul 2021 13:38:42 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 5AC5740213; Thu, 8 Jul 2021 13:38:42 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 35E63C0010; Thu, 8 Jul 2021 13:38:42 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id A3A94C000E for ; Thu, 8 Jul 2021 13:38:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 8665840226 for ; Thu, 8 Jul 2021 13:38:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7KSHtyEPvLf for ; Thu, 8 Jul 2021 13:38:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by smtp2.osuosl.org (Postfix) with ESMTPS id 4AEFA40144 for ; Thu, 8 Jul 2021 13:38:39 +0000 (UTC) X-IronPort-AV: E=McAfee;i="6200,9189,10038"; a="206490430" X-IronPort-AV: E=Sophos;i="5.84,222,1620716400"; d="scan'208";a="206490430" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2021 06:38:38 -0700 X-IronPort-AV: E=Sophos;i="5.84,222,1620716400"; d="scan'208";a="487606704" Received: from blu2-mobl3.ccr.corp.intel.com (HELO [10.254.214.103]) ([10.254.214.103]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Jul 2021 06:38:36 -0700 To: David Stevens , Joerg Roedel , Will Deacon References: <20210707075505.2896824-1-stevensd@google.com> From: Lu Baolu Subject: Re: [PATCH 0/4] Add dynamic iommu backed bounce buffers Message-ID: Date: Thu, 8 Jul 2021 21:38:34 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210707075505.2896824-1-stevensd@google.com> Content-Language: en-US Cc: David Stevens , linux-kernel@vger.kernel.org, Sergey Senozhatsky , iommu@lists.linux-foundation.org, Christoph Hellwig X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" Hi David, I like this idea. Thanks for proposing this. On 2021/7/7 15:55, David Stevens wrote: > Add support for per-domain dynamic pools of iommu bounce buffers to the > dma-iommu API. This allows iommu mappings to be reused while still > maintaining strict iommu protection. Allocating buffers dynamically > instead of using swiotlb carveouts makes per-domain pools more amenable > on systems with large numbers of devices or where devices are unknown. Have you ever considered leveraging the per-device swiotlb memory pool added by below series? https://lore.kernel.org/linux-iommu/20210625123004.GA3170@willie-the-truck/ > > When enabled, all non-direct streaming mappings below a configurable > size will go through bounce buffers. Note that this means drivers which > don't properly use the DMA API (e.g. i915) cannot use an iommu when this > feature is enabled. However, all drivers which work with swiotlb=force > should work. If so, why not making it more scalable by adding a callback into vendor iommu drivers? The vendor iommu drivers have enough information to tell whether the bounce buffer is feasible for a specific domain. > > Bounce buffers serve as an optimization in situations where interactions > with the iommu are very costly. For example, virtio-iommu operations in The simulated IOMMU does the same thing. It's also an optimization for bare metal in cases where the strict mode of cache invalidation is used. CPU moving data is faster than IOMMU cache invalidation if the buffer is small. Best regards, baolu _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu