From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BECBEC432BE for ; Wed, 1 Sep 2021 10:22:33 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7CDB860F6F for ; Wed, 1 Sep 2021 10:22:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7CDB860F6F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=xen.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.176276.320762 (Exim 4.92) (envelope-from ) id 1mLNNg-0002ri-QU; Wed, 01 Sep 2021 10:22:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 176276.320762; Wed, 01 Sep 2021 10:22:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mLNNg-0002rG-NE; Wed, 01 Sep 2021 10:22:08 +0000 Received: by outflank-mailman (input) for mailman id 176276; Wed, 01 Sep 2021 10:22:07 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mLNNf-0002r8-NB for xen-devel@lists.xenproject.org; Wed, 01 Sep 2021 10:22:07 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mLNNd-0007zX-Co; Wed, 01 Sep 2021 10:22:05 +0000 Received: from [54.239.6.187] (helo=a483e7b01a66.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mLNNd-00061t-6E; Wed, 01 Sep 2021 10:22:05 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject; bh=J5xaLmys0DZR5B87YG9hJdaXxyVbpOBZsGEg/stJC60=; b=aWti9DinL7kqLdmR6/NApgKp90 akIy4rqjEAA6OrkcRX7JHprBmlesV4muG2yJ6hbRBQ4aPKMbNZ7AKxSpd2UwAApU1egBJd8Y62MF7 J7BDU2zgRpPJrppbGcUIofyclo98wzOG/T+zB5Wu6LiJIMeU67wEaW1SoLPXX/UkgFj0=; Subject: Re: Disable IOMMU in Dom0 To: Roman Skakun , Stefano Stabellini Cc: "xen-devel@lists.xenproject.org" , Bertrand Marquis , Andrii Anisov , Roman Skakun , Oleksandr Tyshchenko References: From: Julien Grall Message-ID: <179e4b7f-38d9-991d-3f99-64a74559986d@xen.org> Date: Wed, 1 Sep 2021 11:22:03 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit Hi Roman On 01/09/2021 10:59, Roman Skakun wrote: >> If you have a setup where Dom0 is not 1:1 mapped (which is not currently >> possible with upstream Xen but it is possible with cache coloring) and >> uses the IOMMU to make device DMA work like regular DomUs, then passing >> XENFEAT_not_direct_mapped to Linux would make it work. Without >> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has >> code that relies on Linux being 1:1 mapped to work properly. > > I'm using 1:1 Dom0. > According to your patch series, xen-swiotlb fops will be applied for all > devices > because XENFEAT_direct_mapped is active, as shown here: > https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c#L56 > > > I agreed, that xen-swiotlb should work correctly, but in my case, I > retrieved MFN here: > https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c#L366 > > which is greater than 32bit and xen-swiotlb tries to use bounce buffer > as expected. > It can lead to decrease a performance because I have a long buffer ~4MB. > > I thought, that we can disable swiotlb fops for devices which are > controlled by IOMMU. Yes you can disable swiotlb for devices which are controlled by the IOMMU. But this will not make your problem disappear, it simply hides until it bites you in a more subttle way. From what you wrote, you have a 32-bit DMA capable. So you always need to have an address below 4GB. For foreign mapping, there is no guarantee the Guest Physical Address will actually be below 4GB. Today, the ballooning code only ask Linux to steal *a* RAM page for mapping the foreign page. This may or may not be below 4GB depending on how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB). But that's the current behavior. One of your work colleague is looking at avoid to steal RAM page to avoid exhausting the memory. So foreign mapping may end up to be a lot higher in memory. IOW, you will need to be able to bounce the DMA buffer for your device. If you want to avoid bouncing, the proper way would be to rework the ballonning code so pages are allocated below 4GB. Cheers, -- Julien Grall