From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AE95C433EF for ; Thu, 9 Sep 2021 19:54:04 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DD9EE610E9 for ; Thu, 9 Sep 2021 19:54:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DD9EE610E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.183526.331748 (Exim 4.92) (envelope-from ) id 1mOQ7H-0002in-VQ; Thu, 09 Sep 2021 19:53:47 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 183526.331748; Thu, 09 Sep 2021 19:53:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mOQ7H-0002ig-SS; Thu, 09 Sep 2021 19:53:47 +0000 Received: by outflank-mailman (input) for mailman id 183526; Thu, 09 Sep 2021 19:53:46 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mOQ7G-0002ia-1C for xen-devel@lists.xenproject.org; Thu, 09 Sep 2021 19:53:46 +0000 Received: from mail.kernel.org (unknown [198.145.29.99]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 958e9087-1d69-43f8-8d51-8d40c6756bae; Thu, 09 Sep 2021 19:53:44 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id AC7EF610A3; Thu, 9 Sep 2021 19:53:43 +0000 (UTC) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 958e9087-1d69-43f8-8d51-8d40c6756bae DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1631217224; bh=iiysE0Z9f0QioDhx3Nak9Gu1u/jJsXfut2cpl96zhto=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=QtGumlrpMDEsD6CmW948+MWJIbKUOVpd3l8CyaKYosA91NwdFjtYvelg1AXAmhLD2 lFrfATKiF1die7ZJmOyfLHQe/Rx6GyXo2xoHGhAgyNf5GEK7CtCo3HCgobQ5kGKQ+Y FvaUfc6UMbjv+5MKn9CI5giHGg4M0IJMfTi3YN3Nog5vCHt5RPTJuDQvvVbn39d5Jy ELVYmjxpcWP+51lQHasiiavC4P/i0pn1cwNE1S1ZN3rQgqPuoGvDKRooMobFwB9JNh vm6HVQ2lF9TvEHjAs3yRwiAllTp6t9xrRt3NvsA0DCuJ2deoU2kGEGeKMRrVRwHvOj ucTAQKSx3wVHg== Date: Thu, 9 Sep 2021 12:53:43 -0700 (PDT) From: Stefano Stabellini X-X-Sender: sstabellini@sstabellini-ThinkPad-T480s To: Roman Skakun cc: Julien Grall , "xen-devel@lists.xenproject.org" , Bertrand Marquis , Andrii Anisov , Roman Skakun , Oleksandr Tyshchenko , Stefano Stabellini Subject: Re: Disable IOMMU in Dom0 In-Reply-To: Message-ID: References: <179e4b7f-38d9-991d-3f99-64a74559986d@xen.org> User-Agent: Alpine 2.21 (DEB 202 2017-01-01) MIME-Version: 1.0 Content-Type: multipart/mixed; BOUNDARY="8323329-1048009336-1631216039=:10523" Content-ID: This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323329-1048009336-1631216039=:10523 Content-Type: text/plain; CHARSET=UTF-8 Content-Transfer-Encoding: 8BIT Content-ID: I am fine with adding this functionality only to device tree initially. It is certainly true that if a DMA-capable device is behind an IOMMU, then we can skip swiotlb-xen for foreign pages address transactions. Those addresses will be translated just fine thanks to the IOMMU. Skipping swiotlb-xen could provide a non-negligible performance improvement, which is good. However, you should be aware that your proposal should not be needed for correctness or to make devices with a 4GB DMA mask work. Thanks to commit 04085ec1a "xen/arm: fix gnttab_need_iommu_mapping" foreign pages are added to the Dom0 p2m when mapped to Dom0. swiotlb-xen translates foreign pages gfn to mfn then uses the mfn to program the device DMA. The DMA transaction will be accepted by the IOMMU thanks to the 1:1 GFN<->MFN entry added to Dom0 at the time of mapping. If the mfn is >4GB and the device requires an address <4GB, then the dma_capable check at the beginning of xen_swiotlb_map_page fails and the DMA transaction gets bounced on a swiotlb-xen buffer lower than 4GB. Am I missing anything? On Thu, 9 Sep 2021, Roman Skakun wrote: > Hi Julien, > Thanks for the clarification! > > I aim towards to prepare implementation for upstream to disable SWIOTLB for IOMMU-protected devices in Dom0. > To provide this functionality need to add additional binding for each protected device in device-tree. > After this step, I will also prepare the patch to make ensure that ballooning code prepares all allocations below 4GB. > > We are going to prepare this functionality only for device-tree based system configurations. > We don't have resources to support ACPI configuration. > > Would you be ok with upstreaming only device-tree configuration? > > Cheers, > Roman > > ___________________________________________________________________________________________________________________________________________ > From: Julien Grall > Sent: Wednesday, September 1, 2021 1:22 PM > To: Roman Skakun ; Stefano Stabellini > Cc: xen-devel@lists.xenproject.org ; Bertrand Marquis ; Andrii Anisov > ; Roman Skakun ; Oleksandr Tyshchenko > Subject: Re: Disable IOMMU in Dom0   > Hi Roman > > On 01/09/2021 10:59, Roman Skakun wrote: > >> If you have a setup  where Dom0 is not 1:1 mapped (which is not currently > >> possible with upstream  Xen but it is possible with cache coloring) and > >> uses the IOMMU to make  device DMA work like regular DomUs, then passing > >> XENFEAT_not_direct_mapped  to Linux would make it work. Without > >> XENFEAT_not_direct_mapped,  Linux would try to use swiotlb-xen which has > >> code that relies on  Linux being 1:1 mapped to work properly. > > > > I'm using 1:1 Dom0. > > According to your patch series, xen-swiotlb fops will be applied for all > > devices > > because XENFEAT_direct_mapped is active, as shown here: > >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4i > bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com] > > ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com]> > > > > I agreed, that xen-swiotlb should work correctly, but in my case, I > > retrieved MFN here: > >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ib > LDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com] > > bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]> > > which is greater than 32bit and xen-swiotlb tries to use bounce buffer > > as expected. > > It can lead to decrease a performance because I have a long buffer ~4MB. > > > > I thought, that we can disable swiotlb fops for devices which are > > controlled by IOMMU. > > Yes you can disable swiotlb for devices which are controlled by the > IOMMU. But this will not make your problem disappear, it simply hides > until it bites you in a more subttle way. > >  From what you wrote, you have a 32-bit DMA capable. So you always need > to have an address below 4GB. For foreign mapping, there is no guarantee > the Guest Physical Address will actually be below 4GB. > > Today, the ballooning code only ask Linux to steal *a* RAM page for > mapping the foreign page. This may or may not be below 4GB depending on > how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB). > > But that's the current behavior. One of your work colleague is looking > at avoid to steal RAM page to avoid exhausting the memory. So foreign > mapping may end up to be a lot higher in memory. > > IOW, you will need to be able to bounce the DMA buffer for your device. > If you want to avoid bouncing, the proper way would be to rework the > ballonning code so pages are allocated below 4GB. > > Cheers, > > -- > Julien Grall > > --8323329-1048009336-1631216039=:10523--