From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by smtp.lore.kernel.org (Postfix) with ESMTP id F31AEC678D5 for ; Tue, 7 Mar 2023 16:07:50 +0000 (UTC) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0EF54427E9; Tue, 7 Mar 2023 17:07:50 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2082.outbound.protection.outlook.com [40.107.244.82]) by mails.dpdk.org (Postfix) with ESMTP id 4DC06410EF for ; Tue, 7 Mar 2023 17:07:48 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OrmT+e6LDwzoOfgY9McIKhsZyirS/8/murx87zq4ID+XdZA8mI5FF0wxFDO4JL+uQARNeruK0koHL0KIGDUvrA6Ij0LfxzIy2lkUbAxxtENFKMw8I4fj699wRvedweB7iKngEhAzdfjHK+U4V/6nooalIizom7gw3Fc3WBM3YW0rECFLQU1kQ56D353S9t3skMxe+3/AXirOi5WYhEghRA47HmKHQ7/ubrlxwf76RrBX+My5yA2iMZm9bHImv/MV072ugwQKuppJnxwlcz6aUaI9kktntHfX2RIYrvloUi9mYy/204clmlIATYhigKwny48S3jVnOOIbY1B2NRjRvw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gSaAtmLHzNbRSDjGTeSP2z8L05jyZw9PiXrI8zRZlas=; b=dCud7z+oaSGjMK35IXuidRO9IB/JJkV8k4kw0oUG2fFEEtn4sBOX/JLtX8hvTASGlGK44rNeq7yODZWI617o2ve83NJXnIQsCWBm4I/HTYis2LsQZnGBbd0tH/jmgf5Q44uFzHNHAVmHhixf2xLBO8puHmoWfONsGIDkZpMdvxtsZ09pPmBRc8Mv5MXRIRQCIBlW9/mMvMlI8e00NB0u0r7u0Ycrnm7YenLVIn0qVtm27SAjucVmJXBzlsCI+7qBteVsx58R2W8J3yeB7ze0/h8ny8bpEV5LNvZC7CZzGFC4EDA13nmLNzUBCLaX/X/BKYuTlFW7s9jnISHwW/+iwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gSaAtmLHzNbRSDjGTeSP2z8L05jyZw9PiXrI8zRZlas=; b=kUw6bYR20m2ITec7FBouVC/LSgNBQdqaH/5HtFsC9vtTs8buZOTCHpmpoNr9vfVDu0q8i9sjM8VzptLsG69qQdYfyojuf+3Qjeq6yJOvjYeFFxrI8ceHM8C5yeFLxAUpDvccD2p5YMT3t5vTogVwIx3yKteoytIm+Oprkw5cQktAer17SKf35kHWlLTxhh/IFqWy8M9tYE95AKXWugEyPVrOkHwLwhotAWxtB5xOeJH90qVvSXuBXgENphHDfaMu0xBOwp4iLfh+xp9Rt7psbf8Bmb2kvx8t1hy3nKIgeA0WIpV3HKMvN2WsIjNfIGXf/AzbLkV5mMhs1Z/VKzf2/w== Received: from DM6PR12MB3753.namprd12.prod.outlook.com (2603:10b6:5:1c7::18) by SN7PR12MB7420.namprd12.prod.outlook.com (2603:10b6:806:2a7::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.28; Tue, 7 Mar 2023 16:07:42 +0000 Received: from DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::2a72:9d50:8f0d:a925]) by DM6PR12MB3753.namprd12.prod.outlook.com ([fe80::2a72:9d50:8f0d:a925%5]) with mapi id 15.20.6156.029; Tue, 7 Mar 2023 16:07:42 +0000 From: Slava Ovsiienko To: Honnappa Nagarahalli , "dev@dpdk.org" , Ruifeng Wang , Matan Azrad , Shahaf Shuler CC: nd , Matan Azrad , "stable@dpdk.org" , nd Subject: RE: [PATCH v2] net/mlx5: use just sufficient barrier for Arm platforms Thread-Topic: [PATCH v2] net/mlx5: use just sufficient barrier for Arm platforms Thread-Index: AQHYvKszimfJB26pA0CYyNNBmH2tOq3y/CQAgADz1wCAS76WAICw8Nwg Date: Tue, 7 Mar 2023 16:07:42 +0000 Message-ID: References: <20210606164948.35997-1-honnappa.nagarahalli@arm.com> <20220830200038.1694160-1-honnappa.nagarahalli@arm.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR12MB3753:EE_|SN7PR12MB7420:EE_ x-ms-office365-filtering-correlation-id: 4b28bedb-8fd3-4b11-8b33-08db1f26150d x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TzNj8SRUfoGjR4LgmXXd0gvYqReSIVWMdVEZBEtdPgjBqYayfJgqfF5kpWi4djFpsjcxXVkSk2whb6yBHNn0hYIOJEN+z7voceWt8EXbzlwTIfj2twVu1UjE9KHa8CKbyb1TV+GR6VIEk0S6lVi0JF1qWJn+KAPlpQywlsWkeqtQMtBlPjit44/xxbzNEbprYUXQ6n0flHoFSEOWnOiyk85dXAkVisXoxU94DxmQxo9cJSS7Na1ekiu+j7nmQbPPMACYLfrAVM6omnX8glJdqqApEIF5WE3S53KbTBe7Y3RhpExMKjVmWRRCApO4T/1pt8j3jEUdkyNOLZp2nkfHbeyZuIW+DooTTzZUVQ1RsjAHYwgzbUgLQmAYLWbQYG1AFeX330+VenAU3CRrZbtJmqd0+BeEpr88WwsdGlVNXrSYoo1+kWJY+QPZzYNNDOYkzYwXab6KPhaMrKEj2QCaOqJuCta1eC4HfrSzaVDEtaiZezper2se1y03sFvnVEiKBvjfWWbHFgLeL3ys/JV09C6Q1Hcmr+oCilgplwVdfboUtH5nFNUGVnFBcbWpTe1UHM/6ROWJ01tEP8TsTJxB8QTQK8Ky8lNpRzGD18ttTVsBHlkfMJwKb5w5uXEFV6F8x2+09nvCFiBPuguxmxeIokEIjh4hA6UCz7Sy27xVCXxByQzUhoMuqROPOaNFeEgYXW61dmqRZbF9Z7CTXIAYpkFcMdI/D8e5Asg+EbmEVboNEv82bf0+RlvFn/317rItqeI6r4fWYnmLfvwSOSR6oxjpdSB7rYq3Wj6p6V10+0A= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR12MB3753.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(366004)(346002)(376002)(396003)(39860400002)(136003)(451199018)(38070700005)(86362001)(122000001)(38100700002)(33656002)(4326008)(7696005)(2906002)(6506007)(5660300002)(76116006)(8936002)(66446008)(64756008)(66946007)(66476007)(8676002)(66556008)(41300700001)(52536014)(9686003)(26005)(53546011)(186003)(83380400001)(316002)(478600001)(54906003)(55016003)(966005)(6636002)(71200400001)(110136005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?koi8-r?Q?DUDe1cfeGmVxXx5IRYyEtC0CS++dW1MUKhkBpsBZVPeQRG5lkIjQm90gc80Q55?= =?koi8-r?Q?/AHlIkmCg+FTLP03CfHrm9IRvSmvKvZiTgV0PytIB2w3PldkQ2Z87YZWFxys89?= =?koi8-r?Q?aQTfWjKlliaVR0yMVSaz9b985eWqbVVtPsZSG6bpzY8Tq7HQddfGnXFtPI5JwK?= =?koi8-r?Q?j8bcKkBqaVezjppVIV+ynyz+0ZtfWJXgbyBLqJNeisGgAylB9Kj2V9hIaB+Xtr?= =?koi8-r?Q?GASJZ7pA1ILxQYIa4C7nakSbJJICmxXj0CXygX7XVthuifEo+ceYfia/81Et8x?= =?koi8-r?Q?WZbEtTvEm9dBoNkxbBq8uYYJBSJCSoKv39I+fSFPn5dSD++RFaE9WbWYW1vk6E?= =?koi8-r?Q?k8+d9y3K07SFyVkKWsP2tP539sPqujonuMapSMtWquV2nJWqcqZ6KnJQ4zr7bN?= =?koi8-r?Q?yv0FIGT2KKND9xcb+vv4DCNl/wP5fqxs/7uNYIlIRjBalF+gHdJY2QvZ1y/Fgo?= =?koi8-r?Q?vKvpTwjSpoCvfFm/BGctsO0+ea+Dgq4IL+WV67srhTuI6yBSwz0byGkUGXJzwo?= =?koi8-r?Q?wNPqWFW19AXsCdrFCjcdZVAM6JGpsNd58dljHz7SgHBr8YsaU9CjCDX7N8KpdO?= =?koi8-r?Q?QkCc6LITcx7BHP+vU3RihZX9e37rRba7Xqba/TVZZJclxrkhJPB7lf2wwA+LMk?= =?koi8-r?Q?wbhK99OSWJCEUFqr1N5FKG5IvIMHEC8jrn9qGcz0tWPoVwycfw8xuAOlTBi7y8?= =?koi8-r?Q?o5F3WbUat1leX2fKUHnWFENLUY8/Cx8ZirV8C0f3iW0XnZjSfgEkZJnR11XLUm?= =?koi8-r?Q?eFVGu8h5vVTXQcf1QXKPXu/AUV3l7CDP3bzAzhDTDr7TMeANgNRl3+ENQ0QzFF?= =?koi8-r?Q?v9GgFxW/mHD39S3/uyymbb7W6Ob4e8UdkX7RvfmjHzGAzT/EX+zpd0XTs7dZ7R?= =?koi8-r?Q?WwrYPd/iJkl4u/dwdbHZkoDfOuHdHA5eYju+mK6WOHHomxAsfHjROqK/m1g6y+?= =?koi8-r?Q?wc1hgOtEsHvr9WxyH9PC3kANHnj5JQSZAJkbeA18pGZTmiSDiH/tiqYHPgv0qa?= =?koi8-r?Q?Mb6JZDHEGfQmKcykRFvugXUfh1ncyZgC676QdkhSPqw2jNOcFwukxJWG1rpp2D?= =?koi8-r?Q?OQE4zbWXv8M2Vk4qeHUT3bM1BwhlgGmxgQDW4vrng0L0kE/IgNKdFQ9FJjCkgf?= =?koi8-r?Q?shxkvfBtSsTF5g9o4PD111CIsFP0/w3ASiW514cqqrfrOQafpqIuREuPJpmFLT?= =?koi8-r?Q?d9oXKlZx0X/hTugqpkA6KILHdD/XcnII1hp6Nb9qxTkvr842ICEOA0M50yD7Ec?= =?koi8-r?Q?OnVIlbD5CgnW1i94XNh1GTPFrwpiWTIuznq5wKvL1tBaR+ePk5eqx0w/WWtdYp?= =?koi8-r?Q?M8VS35vjRzfdzfoy2cp7GqvvyTKwBy8zE5TpOk5+/YjFv8ime7x8rEIBWIXJR/?= =?koi8-r?Q?4zCwfKrZn7LKEyq6E5it/k9CsHHYX4FBYBLyd6IcWtJYv00XLIe8Ja94H/BH7f?= =?koi8-r?Q?zt/J+hsq+9cFYCu4KQNktIS6jDOb2LCvkqamzDuJ7Yxcb/Ha0NvkGXsXLfNlVo?= =?koi8-r?Q?ph7DdfXlyWFIzdPaQ2ZQW6fHU2Y+AWFx7vHu9roTw72HZrXCvM?= Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3753.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4b28bedb-8fd3-4b11-8b33-08db1f26150d X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2023 16:07:42.5172 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: pfGotRLi3eeq1iRoOGHyF67h9jEjijfrnpmiB/TxDjrCqGDIbChVGi5AACvUTH7i7wUeYc+rwEycqfAuU4DUug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7420 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi, Honnappa I'm sorry for delay - I had to play with patch, to compile this one with as= sembly listings and check what code is actually generated on x86/ARM. On x86 there is no difference at all (with and w/o patch), so no objection = from my side. On ARM we have: w/o patch: dmb oshld with patch: dmb ishld What is the purpose of the barrier - to not allow the CQE read access reord= ering. On x86, "compiler barrier" is quite enough (due to strong load/store orderi= ng). On ARM load/store might be reordered, AFAIU. CQE resides in the host memory and can be directly written by the NIC via P= CIe. (In my understanding it can cause core cache(s) invalidations).=20 I have a question - should we consider this as outer sharable domain? Is it safe to have a barrier for inner domain only in our case? We have updated the cqe_check() routine, sorry for this. Could you, please, update the patch and send v3? With best regards, Slava > -----Original Message----- > From: Honnappa Nagarahalli > Sent: =D7=D4=CF=D2=CE=C9=CB, 15 =CE=CF=D1=C2=D2=D1 2022 =C7. 03:46 > To: Slava Ovsiienko ; dev@dpdk.org; Ruifeng Wang > ; Matan Azrad ; Shahaf Shuler > > Cc: nd ; Matan Azrad ; stable@dpdk.org; > nd > Subject: RE: [PATCH v2] net/mlx5: use just sufficient barrier for Arm pla= tforms >=20 > >=20 > > > > > > > > Hi, Honnappa > > Hi Slava, thanks for the feedback. > > > > > > > > We discussed the barrier here: > > > http://patches.dpdk.org/project/dpdk/patch/20210606164948.35997-1- > > > honnappa.nagarahalli@arm.com/ > > Yes, I have changed the patch according to the discussion. i.e. > > barrier is needed, but different (inner sharable domain) barrier is req= uired. > > > > > > > > (BTW, it is good practice to keep the reference to previous patch > > > versions below Commit Message of the next ones). > > > > > > This barrier is not about compiler ordering, it is about external HW > > > agent memory action completions. > > > So, I'm not sure the rte_atomic_thread_fence() is safe for x86 - > > > patch impacts > > > x86 as well. > > The earlier barrier 'rte_io_rmb()', resolves to a compiler barrier on x= 86 [1]. > > The rte_atomic_thread_fence(__ATOMIC_ACQUIRE) on x86 also acts as a > > compiler barrier. So, there is no change for x86. > > > > > > [1] > > https://github.com/DPDK/dpdk/blob/main/lib/eal/x86/include/rte_atomic. > > h# > > L80 > Hi Slava, any more comments on this? >=20 > > > > > > > > With best regards, > > > Slava > > > > > > > -----Original Message----- > > > > From: Honnappa Nagarahalli > > > > Sent: Tuesday, August 30, 2022 23:01 > > > > To: dev@dpdk.org; honnappa.nagarahalli@arm.com; > > > ruifeng.wang@arm.com; > > > > Matan Azrad ; Shahaf Shuler > > ; > > > > Slava Ovsiienko > > > > Cc: nd@arm.com; Matan Azrad ; stable@dpdk.org > > > > Subject: [PATCH v2] net/mlx5: use just sufficient barrier for Arm > > > > platforms > > > > > > > > cqe->op_own indicates if the CQE is owned by the NIC. The rest of > > > > the fields in CQE should be read only after op_own is read. On Arm > > > > platforms using "dmb ishld" is sufficient to enforce this. > > > > > > > > Fixes: 88c0733535d6 ("net/mlx5: extend Rx completion with error > > > > handling") > > > > Cc: matan@mellanox.com > > > > Cc: stable@dpdk.org > > > > > > > > Signed-off-by: Honnappa Nagarahalli > > > > Reviewed-by: Ruifeng Wang > > > > --- > > > > drivers/common/mlx5/mlx5_common.h | 6 +++++- > > > > 1 file changed, 5 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/drivers/common/mlx5/mlx5_common.h > > > > b/drivers/common/mlx5/mlx5_common.h > > > > index 5028a05b49..ac2e85b15f 100644 > > > > --- a/drivers/common/mlx5/mlx5_common.h > > > > +++ b/drivers/common/mlx5/mlx5_common.h > > > > @@ -195,7 +195,11 @@ check_cqe(volatile struct mlx5_cqe *cqe, > > > > const uint16_t cqes_n, > > > > > > > > if (unlikely((op_owner !=3D (!!(idx))) || (op_code =3D=3D > > > > MLX5_CQE_INVALID))) > > > > return MLX5_CQE_STATUS_HW_OWN; > > > > - rte_io_rmb(); > > > > + /* Prevent speculative reading of other fields in CQE until > > > > + * CQE is valid. > > > > + */ > > > > + rte_atomic_thread_fence(__ATOMIC_ACQUIRE); > > > > + > > > > if (unlikely(op_code =3D=3D MLX5_CQE_RESP_ERR || > > > > op_code =3D=3D MLX5_CQE_REQ_ERR)) > > > > return MLX5_CQE_STATUS_ERR; > > > > -- > > > > 2.17.1