From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57147C4338F for ; Fri, 13 Aug 2021 16:00:33 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EDDBB60EE0 for ; Fri, 13 Aug 2021 16:00:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EDDBB60EE0 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=epam.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.166840.304551 (Exim 4.92) (envelope-from ) id 1mEZbK-0005cJ-PU; Fri, 13 Aug 2021 16:00:06 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 166840.304551; Fri, 13 Aug 2021 16:00:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mEZbK-0005cC-Lv; Fri, 13 Aug 2021 16:00:06 +0000 Received: by outflank-mailman (input) for mailman id 166840; Fri, 13 Aug 2021 16:00:04 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mEZbI-0005IV-9d for xen-devel@lists.xenproject.org; Fri, 13 Aug 2021 16:00:04 +0000 Received: from mx0a-0039f301.pphosted.com (unknown [148.163.133.242]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 844aaf7a-fc4f-11eb-a2ec-12813bfff9fa; Fri, 13 Aug 2021 16:00:02 +0000 (UTC) Received: from pps.filterd (m0174676.ppops.net [127.0.0.1]) by mx0a-0039f301.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17DFtW99011679; Fri, 13 Aug 2021 15:59:59 GMT Received: from eur03-am5-obe.outbound.protection.outlook.com (mail-am5eur03lp2059.outbound.protection.outlook.com [104.47.8.59]) by mx0a-0039f301.pphosted.com with ESMTP id 3adrurrweq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 13 Aug 2021 15:59:58 +0000 Received: from AM7PR03MB6593.eurprd03.prod.outlook.com (2603:10a6:20b:1b4::10) by AM6PR03MB4054.eurprd03.prod.outlook.com (2603:10a6:20b:16::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4394.21; Fri, 13 Aug 2021 15:59:54 +0000 Received: from AM7PR03MB6593.eurprd03.prod.outlook.com ([fe80::ec7b:2795:206b:9411]) by AM7PR03MB6593.eurprd03.prod.outlook.com ([fe80::ec7b:2795:206b:9411%3]) with mapi id 15.20.4415.019; Fri, 13 Aug 2021 15:59:54 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 844aaf7a-fc4f-11eb-a2ec-12813bfff9fa ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VicMO0YRfpMYBEavh5Dzk6sbQNOHoXjFGgv21Ps9keHjxjZHmyXMz4bgY34Scok0SNrtv3BrUN8sS1g+Dzb9qf9kLaVKLOfpduVenHTXUBw86x7g3P3KwVKkvs3gPXv9wEPIoVgIo+TCdxUCp1iLuy9kx2+ufgTAJcol/0sNaCkxhlZ/0urizSosheFWZghNx1v67kr4UqWyXcKwSWi5jdfzhcXKFlzmTSARGqHj3FgK+YPRpryhU49Q6xobyjc5a0T94ZdkvO04fuErqDnxerEHVdmKBhysrZdqljTufshyQ7WU8yh2kHrDJgP3NQhzjNHfnv4l4OL2m2n+oUQXzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZregRLNf01nj4+DF+xNM1YazSprrtjcYbgSgxKQ5sBc=; b=fCQxRyeRHM93qKI7fLYwaDQnnwNaBJb0OW3IoG57XGIuew92Di1u7fyJcqqstc8+R6VyRZ7sjQZTZ1mjwmowiFyzXXNSfzDnGAlkJZpEe6q5k3878EtDww/xC0wv4cCCvtntVPOxoATxZfsfF1NRnn1K2S4xMwr4YmjTu9MkS5jUts0v0D1bt4/o1fYP/6AWdrMmkTdy9lDbx0Z15hMZi4DPBEj+J7tH17TQZ5zssk8qkz8Q14GgWHyvQFOt/VGdqCfoMnNQ4VF97fzRdaMOKj1QhS5uHnp6k2dmuSwqu6dFwJ+uThSnTrrdPTV1gke6L+X2Tm/T2mpgWl2/HBFq9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZregRLNf01nj4+DF+xNM1YazSprrtjcYbgSgxKQ5sBc=; b=EEGjOUutYkNpFcWqOg9+YETncOPj3y8FcDcggCIEy+kpPqszMS4+tEclumBjnH5iIIZe02JnkLckBGrEQ/c21hoWiZSDYdhqt6UWgkMeMF5db3YFJSshLGqBQjnsdD3OrPKrpKiscIxUnlRxjgBytp5QWUWZ7XhiSVYWDylHoJRF8S6NXmvV+sC7LY6VIjQZYF9i6BtBn3kV7GFVXPjCtY19ndvmjJXmBziXygOTkKrKnUhE/ZEpGaKqsXkZpRoir40L/JvUQgKaIpJ6ZGgH7nEWzGPLACH16RI7HKXlv8UPWwj6G/su0nB/a88QKykrz/LHLA5RUKvv7/3g/72p1g== From: Roman Skakun To: Julien Grall , "sstabellini@kernel.org" CC: Bertrand Marquis , Andrii Anisov , Volodymyr Babchuk , Oleksandr Tyshchenko , Oleksandr Andrushchenko , "xen-devel@lists.xenproject.org" , Roman Skakun , Jan Beulich , Roman Skakun Subject: Re: Disable swiotlb for Dom0 Thread-Topic: Disable swiotlb for Dom0 Thread-Index: AQHXjfwJK3kYIT7XAUWdmOu1AWm+tKts6kUAgAD/OfqAADJaAIAAJhCIgAArBQCAAqGMMYAAN/uAgABIigY= Date: Fri, 13 Aug 2021 15:59:53 +0000 Message-ID: References: <060b5741-922c-115c-7e8c-97d8aa5f46f4@xen.org> <691e31db-c79b-9196-53e1-cbbdc9bd3a54@xen.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: xen.org; dkim=none (message not signed) header.d=none;xen.org; dmarc=none action=none header.from=epam.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f840a54e-1a64-4898-0d53-08d95e7363f3 x-ms-traffictypediagnostic: AM6PR03MB4054: x-ms-exchange-transport-forked: True x-ld-processed: b41b72d0-4e9f-4c26-8a69-f949f367c91d,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IgeWdp5qmnaP770/ebnXzG+FdZLxdyIY5tLCItoaqXyDSkCT37w2P4dxXA7Axvk3MOQ87HqvZn5Dj2IpbIIlqNOPQOiXspFAxUGNYpLN0XwAzyYA7uh9t3gGEuelJEu9LTdo90grMUGvap6cTuIehU+EhUCkENmLbktmlmhoil1n6kZwHCpdbvKLS3UXM3ekGxL4kIxzAanVJKIX3nXWdN8nL/Y9WDUoiRegNAQjehXUdioKhz3F3miYjUj0OdhQfKAm+q/d+YWckEwNtpwO5eRksYAwF+3xWcb5lxJiZk1kouro+xb7vI2g1GAyo7sJVhl/mf/7b8gf26EOmvkRCkI2B4iJAEmNObUmZl9+G3ww6jmdRgPNa/4pbLBor3uDEk1iRA+A9ndOxC4F4tyf1csZaGGb1umOvNOohwIwqMA0s5pbdC+7MjFNFn21iRSHRbpeJ72wSexbxIQGAjcS/NQqMmOsV6InabzjK3yiyLBJzew3ijOPrYcpf4d+WOkqiuPapG+6yEqZIBQCaYgdU30bQDr65b8RKyFxzRb2tQAzv1tFchVLUXKiIY3qBKtqepeV3llA57fwbvtpwjrg9sQ2ZrFhCMtqZAGHh6z4tgUZeX1TywAF54SFOlSq34EQcfYh9Fm6nK4UaYHW/efG+1Zrp7ZkON2UT0gd0K47M+Np7M9JH5wBelnoQvUlfYYpFLP+2KD0FMvCOmckuiHnbBDlDY4XqXBIqGl+mCRfPEnTvvT63Wyk+HFGGzQyVHaSREDd3X9vKACWPn7MGJoleTpKP6MIn2RRdG2dvDIZk3I= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM7PR03MB6593.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(396003)(366004)(39860400002)(136003)(346002)(376002)(26005)(55016002)(122000001)(9686003)(86362001)(2906002)(38100700002)(186003)(83380400001)(6506007)(8676002)(19627235002)(53546011)(71200400001)(107886003)(4326008)(7696005)(38070700005)(19627405001)(8936002)(21615005)(166002)(478600001)(5660300002)(33656002)(966005)(52536014)(76116006)(110136005)(66556008)(66946007)(66476007)(64756008)(66446008)(316002)(91956017)(54906003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6lijdyGlRmZpQ7TagL1Ni7ao6wLYtkmpkFvKO8zHNzO699hG7AiyMhcqKrfg?= =?us-ascii?Q?9JRT0TkQccCgTzPHYCyEQjiGp+ZwK7NpOd2gT1v+JccwFes6+/yFhnj8Nzdf?= =?us-ascii?Q?qUuJr2eFNMLjndb2lt22CTxh7noKV3KXZPJQ2aXb4cbNWfDzdyNO0t98fvlj?= =?us-ascii?Q?EuvkDzy/lKLr6DxO32IQ2etZ2OsX8KF5Pw8oMs2AVwQLuEWiogQ+BNxv2l6A?= =?us-ascii?Q?lxQIKkY3KX1YJ0iyrgaYs6HQWRy9wrRebM4ArJaj+vIZPoFPdINFG2M9MaHI?= =?us-ascii?Q?yjazdbkqe/CGCST2ynwmzzCigGPnyotiW/52JWGj0CyqsTATApZVgcQdWiM6?= =?us-ascii?Q?v+Hmpt/oaBRZmNzUjloM05OXuNgCVWW3YTw5zWpnGENoz0st9XbGNeu39niX?= =?us-ascii?Q?YIiXFGHZwbsPkIk49QJM45v6lwHNY9kR89aLxnGHehNEXH7eEqL6uu1vXT5N?= =?us-ascii?Q?UQv6B40vu9hbJb7ME11jWJomuMirsg/y8ItuJWbqzwzTBOm0/dPEA2Dlbdmr?= =?us-ascii?Q?ZYQLUpdZ1FO60zxNJpQ/hyjOJairEdKtB7ZNh6Sq+a4X8szZKE7HB6SVq96w?= =?us-ascii?Q?Hcdo4TUMBVDKgBr9onfr6E9hXdz/pAO2PAR8f0ESZtRNHMZGqBCTP4Z/doQ9?= =?us-ascii?Q?PE5cmawVyn2tcDJa9ngqWXCBkP5MewWYDzeq5StTI2UBPRu37ErDOz5AHDHj?= =?us-ascii?Q?6zJ0UJ2+fYu4xJMA4mO0fzjri53KRAWnJxg0lg/ix4SCNMcHwc2y221wPLY2?= =?us-ascii?Q?/wJg7gL0m/jb6zvIamxiyDiQcqlrvwinoARvn+YA2B/nLRq5pUZg5joYixUH?= =?us-ascii?Q?NNjo/W22VkfEc+fn/Xzz/tukvDHh0A1Ap8rt9qMcYEjp/TEYTRmDanFOgIiY?= =?us-ascii?Q?wabkMKlLqZmjigvlXBDi24WegCQ+8eQCFjAGLJwY6SQIQo+YozfaaCarks1w?= =?us-ascii?Q?vhHFeSMWAQHOG+CbWnG7m+eUQKS8ekHMBeSN1xL/DPaReXrwAhRhrspPxBRM?= =?us-ascii?Q?B5iYWhf6yMHwOqmF8RxVdE7JV4kFM38R8SW82/JFN5iCQnxHD53fl8Ius3CN?= =?us-ascii?Q?W9t/Y4uVgYXDrglWXGFUbJZBY+n/AmrvTfBTWdQRiMNGuOMwPXBhnKP3JB+S?= =?us-ascii?Q?eznCAuoUCBgZi3yoOKbz4kT0s2zBIq9dAewuctgr08nSPENezUZTg+T9XhaZ?= =?us-ascii?Q?PM/oJUiHbqXgyc8xR4mkSLhSzikYc3hkH5md79IDXUmepcHmK+U7XT1W1Q7j?= =?us-ascii?Q?MyY4SLFQVIJ2sm+8jectert1oHxyWVg5pZYPZIuHQCIUlyWsfjPtdobcXHlp?= =?us-ascii?Q?4/w=3D?= Content-Type: multipart/alternative; boundary="_000_AM7PR03MB65932EEF162F76F482BF4F8C85FA9AM7PR03MB6593eurp_" MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6593.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f840a54e-1a64-4898-0d53-08d95e7363f3 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Aug 2021 15:59:53.9424 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: jXeMVWFksVAbqbjanqOEpijHjvFbAkz2+c0PnEu6HFor79mhsTtbzasqJqB30qSRfe9ainJmvAQEppnga1aF6w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR03MB4054 X-Proofpoint-GUID: AAxinX7iljIulfRLy981o-0OH18_uD6C X-Proofpoint-ORIG-GUID: AAxinX7iljIulfRLy981o-0OH18_uD6C X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391,18.0.790 definitions=2021-08-13_05:2021-08-13,2021-08-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 mlxlogscore=999 spamscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 adultscore=0 clxscore=1015 suspectscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108130095 --_000_AM7PR03MB65932EEF162F76F482BF4F8C85FA9AM7PR03MB6593eurp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable >> But I think, we cannot use64af97000 address in the swiotlb_bounce() >> directly. >> >I am not sure to understand what you mean. Can you clarify? I thought, that the address 64af97000 can be used directly here: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572 and I was confused about it. After some diggings I realized that this code is not reachable because DMA_ATTR_SKIP_CPU_SYNC is active. >> Swiotlb did not fit requested slots because the maximum slot size equals >> IO_TLB_SEGSIZE=3D128 by default. > >> Ok. So it sounds like your problem is the have a DMA buffer that is too >> large for the default swiotlb. Did you try to bump the value from the >> command line? Yes, you are right. I checked the implementation more detailed and figured = out that swiotlb tries to find a contiguous segment that is a big enough, as shown h= ere: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528, But the maximum TLB segment equals 128K by default in accordance with https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25 This means that the max size of TLB segment equals: 128 * 2048 =3D 256K After this, I tried to increase the TLB segment size like this: diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index fbdc65782195..85f32865bb6c 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -22,7 +22,7 @@ enum swiotlb_force { * must be a power of 2. What is the appropriate value ? * The complexity of {map,unmap}_single is linearly dependent on this valu= e. */ -#define IO_TLB_SEGSIZE 128 +#define IO_TLB_SEGSIZE 2048 /* * log of the size of each IO TLB slab. The number of slabs is command li= ne and the problem is gone. Also, I found the article https://www.xilinx.com/support/answers/72694.html= , where I believe the same issue was mentioned. Thank you so much for your time and help! ________________________________ From: Julien Grall Sent: Friday, August 13, 2021 1:51 PM To: Roman Skakun ; sstabellini@kernel.org Cc: Bertrand Marquis ; Andrii Anisov ; Volodymyr Babchuk ; Oleksandr Tys= hchenko ; Oleksandr Andrushchenko ; xen-devel@lists.xenproject.org ; Roman Skakun ; Jan Beulich Subject: Re: Disable swiotlb for Dom0 On 13/08/2021 10:38, Roman Skakun wrote: > Hi Julien, Hi Roman, >> So 0xb6000000 is most likely the GFN used to mapped the grant from the d= omU. >> >> swiotlb-xen on Arm will convert it to the MFN because it is not aware >> whether the device is behind an IOMMU. > > If I'm understand right, it seems like that swiotlb-xen is not ready to > work properly in case > when we retrieved MFN instead of proper GFN mapped to Dom0 memory. > Maybe you know some ideas to overcome this condition? swiotlb-xen work as intended. You have a DMA buffer at an address that your device cannot deal with. So it will try to bounce it. > >> As the address is too high to be handled by the device, swiotlb will tr= y >> to bounce it. I think it is correct to bounce the page but I am not sur= e >> why it can't. What the size of the DMA transaction? > > The DMA map size is 3686400 bytes. So that's a 3MB buffer. I am slightly confused because in an earlier message you wrote that the memory is coming from the guest. How did you map it? > > I've added several logs to swiotlb map_single() and see: > [ 151.298455] swiotlb_tbl_map_single() origin_addr: > 64af97000, needed: 708, > avail: 7fc0, stride: 2, index: 4160 I am not sure how to read the logs... Are the number in hexadecimal or decimal? It might be useful if you post the diff of your changes. [...] > Swiotlb did not fit requested slots because the maximum slot size equals > IO_TLB_SEGSIZE=3D128 by default. Ok. So it sounds like your problem is the have a DMA buffer that is too large for the default swiotlb. Did you try to bump the value from the command line? > But I think, we cannot use64af97000 address in the swiotlb_bounce() > directly. I am not sure to understand what you mean. Can you clarify? Cheers, [...] -- Julien Grall --_000_AM7PR03MB65932EEF162F76F482BF4F8C85FA9AM7PR03MB6593eurp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

>> But I think, we cannot use64af97000 address in the swiotlb_= bounce()
>> directly.
>>
>I am not sure to understand what you mean. Can you clarify?

I thought, that the address 64af97000 can be used directly here:=
https://elixir.bootlin.com/linux/v5.10/source/= kernel/dma/swiotlb.c#L572
and I was confused about it.
After some diggings I realized that this cod= e is not reachable because 
DMA_ATTR_SKIP_CPU_SYNC is active.

>> Swiotlb did not fit requested slots because the maximum slot size equals
>> IO_TLB_SEGSIZE=3D128 by default.
>=
>> Ok. So it sounds like your problem is the have a DMA buffer that is too=
>> large for the default swiotlb. Did you try to bump the value from the
>> command line?

Yes, you are right. I checked the implement= ation more detailed and figured out that
swiotlb tries to find a contiguous segment that is a big enough, as show= n here:
https://e= lixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528,

But the maximum TLB segment equals= 128K by default in accordance with
https://elixir.bootlin.com/linux/v5.10/source/inclu= de/linux/swiotlb.h#L25

This means that the max size of TLB segment equals:
128 * 2048 =3D 256K
 
After this, I tried to increase the TLB segment size like this:

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index fbdc65782195..85f32865bb6c 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -22,7 +22,7 @@ enum swiotlb_force {
  * must be a power of 2.  What is the = appropriate value ?
  * The complexity of {map,unmap}_single is = linearly dependent on this value.
  */
-#define IO_TLB_SEGSIZE 128
+#define IO_TLB_SEGSIZE 2048
 
 /*
  * log of the size of each IO TLB slab.  The number of slabs is = command line

and the problem is gone.

Also, I found the article https://www.xilinx.com/support/answe= rs/72694.html, 
where I believe the same issue was mentioned.

Thank you so much for your time and help!

From: Julien Grall <juli= en@xen.org>
Sent: Friday, August 13, 2021 1:51 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.o= rg <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov= <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@ep= am.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Ole= ksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Rom= an Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>= ;
Subject: Re: Disable swiotlb for Dom0
 


On 13/08/2021 10:38, Roman Skakun wrote:
> Hi Julien,

Hi Roman,

>> So 0xb6000000 is most likely the GFN used to mapped the grant from= the domU.
>>
>> swiotlb-xen on Arm will convert it to the MFN because it is not aw= are
>> whether the device is behind an IOMMU.
>
> If I'm understand right, it seems like that swiotlb-xen is not ready t= o
> work properly in case
> when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
> Maybe you know some ideas to overcome this condition?

swiotlb-xen work as intended. You have a DMA buffer at an address that
your device cannot deal with. So it will try to bounce it.

>
>>  As the address is too high to be handled by the device, swio= tlb will try
>>  to bounce it. I think it is correct to bounce the page but I= am not sure
>>  why it can't. What the size of the DMA transaction?
>
> The DMA map size is 3686400 bytes.

So that's a 3MB buffer. I am slightly confused because in an earlier
message you wrote that the memory is coming from the guest. How did you map it?

>
> I've added several logs to swiotlb map_single() and see:
> [  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_ad= dr:
> 64af97000, needed: 708,
> avail: 7fc0, stride: 2, index: 4160

I am not sure how to read the logs... Are the number in hexadecimal or
decimal? It might be useful if you post the diff of your changes.

[...]

> Swiotlb did not fit requested slots because the maximum slot size equa= ls
> IO_TLB_SEGSIZE=3D128 by default.

Ok. So it sounds like your problem is the have a DMA buffer that is too large for the default swiotlb. Did you try to bump the value from the
command line?

> But I think, we cannot use64af97000 address in the swiotlb_bounce= ()
> directly.

I am not sure to understand what you mean. Can you clarify?

Cheers,

[...]

--
Julien Grall
--_000_AM7PR03MB65932EEF162F76F482BF4F8C85FA9AM7PR03MB6593eurp_--