From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15578C4338F for ; Tue, 10 Aug 2021 16:17:11 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AEE2C60ED8 for ; Tue, 10 Aug 2021 16:17:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AEE2C60ED8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=xen.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.165423.302286 (Exim 4.92) (envelope-from ) id 1mDUQw-00005v-Ig; Tue, 10 Aug 2021 16:16:54 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 165423.302286; Tue, 10 Aug 2021 16:16:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUQw-00005n-Fh; Tue, 10 Aug 2021 16:16:54 +0000 Received: by outflank-mailman (input) for mailman id 165423; Tue, 10 Aug 2021 16:16:52 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUQu-00005f-D9 for xen-devel@lists.xenproject.org; Tue, 10 Aug 2021 16:16:52 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mDUQt-0001LV-Ap; Tue, 10 Aug 2021 16:16:51 +0000 Received: from [54.239.6.184] (helo=a483e7b01a66.ant.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1mDUQt-0002fr-4A; Tue, 10 Aug 2021 16:16:51 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject; bh=C6sFonpPeCIvhGKFsRDlTl8vzERYo0lkY63jvdTn7Ns=; b=Cv3UqnJ413Kd1/RAVmS6l7BKUf 3kvO9bkX/nZJSXZ/BDuEcICjymVc0/o0QbQleSzrv1PotSR26CjyrYVjrRT3CREUaHwAxSlcJ554b Mn836NVxWjz5nFr78Mj9YXr19JZztrJm4vsvJ5vElnD4aqKlSbQJzLfFYxEkA4ghBM6A=; Subject: Re: Disable swiotlb for Dom0 To: Roman Skakun , "sstabellini@kernel.org" Cc: Bertrand Marquis , Andrii Anisov , Volodymyr Babchuk , Oleksandr Tyshchenko , Oleksandr Andrushchenko , "xen-devel@lists.xenproject.org" , Roman Skakun , Jan Beulich References: From: Julien Grall Message-ID: <060b5741-922c-115c-7e8c-97d8aa5f46f4@xen.org> Date: Tue, 10 Aug 2021 17:16:48 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit On 10/08/2021 16:38, Roman Skakun wrote: > Hi, Stefano! Hi, > I have observed your patch here: > https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/ > > And I collided with the same issue, when Dom0 device trying to use > swiotlb fops for devices which are controlled by IOMMU. The issue Stefano reported was when the dom0 is not direct mapped. However... > > Prerequisites: > https://github.com/xen-project/xen/tree/stable-4.15 ... if you are really using 4.15, then the domain will always be direct mapped. So I think this is a different one. > https://github.com/torvalds/linux/tree/v5.10 Any reason to not use the stable branch for 5.10? I don't know whether your issue will be fixed there, but the stable branch usually contains a lot of bug fixes (including security one). So it is a good idea to use it over the first release of a kernel version. > Issue caused in xen_swiotlb_map_page(): > ``` > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, > xen_phys_to_dma(phys): 64b1d0000 > ``` I can't seem to find this printk in Linux 5.10. Did you add it yourself? > > There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0 > swiotlb couldn't proceed to this address and throws the log: > > ``` > [   99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots) > ``` This line suggests that the SWIOTLB tried to bounce the DMA buffer. In general, the use of the bounce buffer should be rare. So I would suggest to find out why this is used. Looking at the code, this suggests that one of the following check is false: /* * If the address happens to be in the device's DMA window, * we can safely return the device addr and not worry about bounce * buffering it. */ if (dma_capable(dev, dev_addr, size, true) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && swiotlb_force != SWIOTLB_FORCE) goto done; > > Temporary, I resolved this issue by disabling swiotlb for dom0 at all > because sure that all devices goes through IOMMU, but this mention can > be true only for me. > > But, I think of a more reliable way is to declare a special IOMMU > property in xen dts for each device. If the device controlled by IOMMU > not need to set swiotlb fops in arch_setup_dma_ops. > What do you think about it? Let me start with that I agree we should disable swiotlb when we know the device is protected. However, from what you describe, it sounds like the same issue would appear if the IOMMU was disabled. Therefore, I think we should first find out why Linux wants to bounce the DMA buffer. Does your device have any DMA restriction? Cheers, -- Julien Grall