From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12F86ECAAD3 for ; Thu, 15 Sep 2022 01:52:29 +0000 (UTC) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C84B44021E; Thu, 15 Sep 2022 03:52:28 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 7C96F4021D for ; Thu, 15 Sep 2022 03:52:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1663206746; x=1694742746; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=H8nlUkxP+as6yQx3Yk3sMgdWWowffMTXJkeUejhivqc=; b=nM9cOs25j+iHFn/rqQIJrQ4ZSDDlVy/ZI77JjxE0WHXf4WA7h22aRoOT tcIwABWKPt2CYZZHmFZ61kOXaLI7t25/8ZGaBc9dCNhAzJK4Y5ARw5nxZ NPPU1Q0DCZ+SrJURZs/eFn1feSUudEjkQufNjdFzjoF+l4IcGMmzero3/ 7sY76uuj19bhpsDbCX5fe0S5biicvIyBd3uc3ZJbaFbS6Rs6r9bVRX2fO M7hmi4G4jLAg4I+Ibo+r3p68hsQOS5KfeUohaIcLQd+i1sN+0YvFy1Xdz CmMYSZIF0d9dPlg/YTptlw5p1I9WS2vSv+Q2T7Wp2rFmzIIvid9IIpqAE Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10470"; a="384880458" X-IronPort-AV: E=Sophos;i="5.93,316,1654585200"; d="scan'208";a="384880458" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Sep 2022 18:52:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.93,316,1654585200"; d="scan'208";a="568233974" Received: from fmsmsx602.amr.corp.intel.com ([10.18.126.82]) by orsmga003.jf.intel.com with ESMTP; 14 Sep 2022 18:52:16 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 14 Sep 2022 18:52:16 -0700 Received: from fmsmsx609.amr.corp.intel.com (10.18.126.89) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Wed, 14 Sep 2022 18:52:15 -0700 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx609.amr.corp.intel.com (10.18.126.89) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Wed, 14 Sep 2022 18:52:15 -0700 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (104.47.58.169) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Wed, 14 Sep 2022 18:52:15 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LrX0PrXRxhs96PgTHbqyC+jYVhxbBE26RnAFwmQYQmOttcf83n7iq/ObZ9B01Gmz1208qnQTYX1WC3LaFlMZxVLEsgvYqBt4GWI0RGYufy3E+c65MJ82FVK0dsZ73yUm6TjnjrpXpWVUL4zM7XXt+Ish4YJu0/a7RjKnrWyHKgc4N8bgaxD5A4yOSwHKZ4jlvMPsF8nD6Lm37nQcX+sqbMXuEZ5T37lbydeO/ZtLmiXrnoYCArzjR/KhmcDC745osYKuOlkf6j2bUMptDnnxXfvgwRU1U66OMfKL0+siWMsSBfGs4cy8bzNGaHgo9W9w2hl62BloAJK9a0zJbqC7MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PMzRrtG3o61SGDN0v7btfrewKVs/x/w9vXVZrZWOgbo=; b=ApCJLdGE8s4yrAsx56IY7GBAVncqrXtoKEsNU1atlpjeiUOB7FdcjWj1n1hZmbhBfrzr+VcUWBR1B7qy0oKt/O7u+znN7fQCNUbpDeZHELIjS+XbyZwJjioqbK3RKXnCBaylSeHWtUWrhseN9KYVKszJ1PEVVeTHcVqNZ6HfP814XfEaSSU2mj4OuWsOL0CrhqfwIFXLhFaqb4NnJllZxbyJ1abOVmWYM2KtDk8z1alcXbZeiLfdmp821TNr5nKjShctZFCXTGnicajH6MLmjpuRDRx91/kCO41+JS9AkrYlk2719K5mUNs0DeFUxmknOIaf9wbfT1x1l+07YBHMxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from SJ0PR11MB5022.namprd11.prod.outlook.com (2603:10b6:a03:2d7::17) by DM6PR11MB4515.namprd11.prod.outlook.com (2603:10b6:5:2a9::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5632.15; Thu, 15 Sep 2022 01:52:13 +0000 Received: from SJ0PR11MB5022.namprd11.prod.outlook.com ([fe80::e128:9d7c:7aa9:f366]) by SJ0PR11MB5022.namprd11.prod.outlook.com ([fe80::e128:9d7c:7aa9:f366%4]) with mapi id 15.20.5612.022; Thu, 15 Sep 2022 01:52:13 +0000 From: "Yan, Zhirun" To: Pavan Nikhilesh Bhagavatula , "dev@dpdk.org" , Jerin Jacob Kollanukkaran , "Kiran Kumar Kokkilagadda" CC: "Liang, Cunming" , "Wang, Haiyue" Subject: RE: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core dispatch Thread-Topic: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core dispatch Thread-Index: AQHYwyg1or/PAbheDEad8fb0FDXI5a3VAPCAgAmtRyA= Date: Thu, 15 Sep 2022 01:52:13 +0000 Message-ID: References: <20220908020959.1675953-1-zhirun.yan@intel.com> <20220908020959.1675953-5-zhirun.yan@intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.6.500.17 dlp-reaction: no-action authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SJ0PR11MB5022:EE_|DM6PR11MB4515:EE_ x-ms-office365-filtering-correlation-id: 26a769ff-a369-441e-6188-08da96bce8ea x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: TLVHG5vMe/lC4QUetrEPKnumD3KFocxBsZTEGOQAw0Q9zjmqsnfBRLRL3MB8+YzlwUUzc2Gl2WYCqiTVR2PMnWxfiSe7ej5ZvcBsKEQrYGMgpa0h1elwVnrVUbZJnm34Tyqu+Q1EAsZOm16D/X3JixSuJ6UuJwzx0aECL3t9KyVn+8/NGma8WXAh0S/YURHQpxpHtQBdL71eUBJeLtT1AfMhmOH3jAH13XYgAQG+bNgO5wFnoASWaC5BVZyDaXDWzaeIKHFUu6R8btIhg9ZCZ8T7UG/2pRPDBpBMEbYyTsYbBj0HRC08Se/LBCBRtPrFiARfPJtOtd9PzmXRoZAbfiJU06XTfn8HcpDCi7FyniB/bqweU+EG2S5LzFw1yRMTakyiNrEJFVB5QOSNAG6UjEdOvihSNRWcCt51PaB113CCPqrEX4F53faSq4OR/udiZ494DIvhRfvn8nWDeq9gBJj+NLC46AiJjZ1cQfmC/pUrAxAwbifb7Bj+B9EWqJzNxjRcyJ9c28pgiZ+It5FqX9GeWG8EPHpvBnbofMmrJr4Flrgru3cHpipLbHaFHpI8d8PCDC2yjvgg+2ya4ECUiNUA28hHJHhCR0sSFUbODfYTrTgHa5wgu8s1ay6FJBWZTx4AvvPo2wuYs3VJa3ovL+W61qR6/SWy0UKt1kZxlEcWjigWPGWok9MMi4mXLri/Os9l5gtQTHe/KqH6PDWF9bRTF2e4/+Q+is705w0T8PUVN50RlWdIDlpjY8YLYLt0KV078SNSubQTzbODzSCdKhE+ljqD3Vz/4Q2t1vzkAdBaZB7KKIgV7iqXv93IJvEc x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SJ0PR11MB5022.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(366004)(346002)(136003)(396003)(39860400002)(376002)(451199015)(9686003)(53546011)(55016003)(316002)(26005)(41300700001)(7696005)(107886003)(33656002)(6506007)(110136005)(54906003)(478600001)(71200400001)(38100700002)(122000001)(82960400001)(38070700005)(186003)(83380400001)(86362001)(5660300002)(52536014)(8936002)(4326008)(8676002)(66476007)(66556008)(76116006)(2906002)(66446008)(64756008)(66946007); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?R3+ihJO6RZ6h/WVt5PqIaB3K6H+hX6pCIkz1vDZRz0FT1uGRHIaxFWJIyF9C?= =?us-ascii?Q?FDMoRVywOLY6b3lUWyKGSnWtvaQCiHTDonVuZUbg2EP7Z4rkmyZI4sHUhIrf?= =?us-ascii?Q?vWOF0pC2taw+KRZmsrtTCfKRdlv3AtNEYzlHrEA+Sf7CEBem8ga5zcPH6iaM?= =?us-ascii?Q?EXXJ1aidKNL9gk/hCmewqplWwjLHtWFL6dFF8DZ7MqdfKN13clOVvW6b075r?= =?us-ascii?Q?ACc9CN/CFSmmEs9E5KmPo6f0oZHO+qlPa+4M41Dk8bmO2/iCi9ghIxBUTpjT?= =?us-ascii?Q?SepIZ6AY/BtAzMBbliTzLWgu7P5WTQnF0jZ59Fip1T3o7k1EkF1CjtebBcWO?= =?us-ascii?Q?aGsqNW2POoQ7638//XJFkYtUoSRuY2OdQgRy1zYUATZHgkQfNScnfFNt3fMc?= =?us-ascii?Q?n75N0e07OpEeu97nprJTGPrZt7ZdqE7NIHdv0uicQ5LaotD//VCr01lhzxC9?= =?us-ascii?Q?rbzJMxEJXIPtIXrTz6bon1nuIcopeFNSukTbqX0+s8RCl5qsGzy6sNeFQlZW?= =?us-ascii?Q?wUlSX9CcPbpiPUyr23s5MRH0GWAaThwrl9tU3i2DTVZWHhmsb0upaJw5uUOy?= =?us-ascii?Q?XOhHfR4UtW5XoBV3VVrcXYv6DlhHGiN+xiO+Tu3XPxwcBHGlGKBwMdZ2yJfh?= =?us-ascii?Q?j3+ZwmKC0k0rKQQhnt1V8tZUIdYcPqWUrtEiQ3dBi/dcHf3a1nR2Ys9SDs/P?= =?us-ascii?Q?hyOtyfdIFrSla4QPUoUZeND9XuLuK6pSuo7SggW617VFqgeV985/fhwSzVLb?= =?us-ascii?Q?OIOP6TjmoubzH0a5halbnhZ0T1aSV3V1n9GOvNdNnU70y8TWOApqJ6yZYlN5?= =?us-ascii?Q?dxye1bPDhmNyZyA5RAkJZFhDJgN/G7Nj5IhXAPf7PvRmCSSE07jfhWg2B3JI?= =?us-ascii?Q?ZduNHopfknwerm9glTZq9+ItpXZVBz0mhj+Xiw844bwRSLfvnsOSUdf8JVc3?= =?us-ascii?Q?hbmHizOTeibWC8wfqIPAvCHH2q6g9Uf7AksfcV4uUNxSenDUjUTc8fI1cm1n?= =?us-ascii?Q?tOfv2AaX61FmTyDl4lhOnr4y8SrjP5I7q0Q4z3lwjwgE4xZkOj4LvD8KA7sw?= =?us-ascii?Q?vqH4nXx1aBYdULTW8YkvBsk2C5PfePwVsrmmhdgILCEw7Z0KTuaTx+8YygNF?= =?us-ascii?Q?rQ9zHOK8JDWEmTb82y3gr9sxVD32M8lxE5W+I+b9JJrs0mMyuxVFfyMofblf?= =?us-ascii?Q?TXSx/wc/u1rSHzdDwBeHO5OmQs7JNAOjnqv7w7x4uGRKUU43q1g3Tf2LipCQ?= =?us-ascii?Q?MiuDRsE0wAcFb88VS+ODRuNamtjmhJP2ULvZ4Tj3MQp4bXu2T1qhBE4ZryfL?= =?us-ascii?Q?0zki1sjvRFAqXIT1ygkuj2vODYHc8GuqN8ywkjnd35iFiflTO1K4TpzR7TOZ?= =?us-ascii?Q?9X/OdoCBS2EHhs63NVpOMtU3QZSfdb7QYzyX6B16oSH4rl/kdeWLOfstL7t9?= =?us-ascii?Q?7LEfDjGz0UFSfLCUgYz2MZCSgcWv24x7XcI42PEiHsG91O4aA/hCDYEaQVUA?= =?us-ascii?Q?JYS1Op43fxgbFl0UsEo/xUrRj+VS5zavevdIP2POnQQNNZct+tkLNzR9Dp2C?= =?us-ascii?Q?6ZI/MUvjE52MybrlJgeDOciqPExPsAVDLwgGtQNH?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SJ0PR11MB5022.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 26a769ff-a369-441e-6188-08da96bce8ea X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Sep 2022 01:52:13.1570 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: k/iE9ywUVYTV2iZLbq4w3PGzEMphpUEhhSg5BXhCSDA0/vJ80ZD/eSjw0/T+U6gjF92FEVklyoLZcV0j/zjFtA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4515 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org > -----Original Message----- > From: Pavan Nikhilesh Bhagavatula > Sent: Thursday, September 8, 2022 1:27 PM > To: Yan, Zhirun ; dev@dpdk.org; Jerin Jacob > Kollanukkaran ; Kiran Kumar Kokkilagadda > > Cc: Liang, Cunming ; Wang, Haiyue > > Subject: RE: [EXT] [RFC, v1 4/6] graph: enhance graph walk by cross-core > dispatch >=20 > > This patch enhance the task scheduler mechanism to enable dispatching > > tasks to another worker cores. Currently, there is only a local work > > queue for one graph to walk. We introduce a scheduler worker queue in > > each worker core for dispatching tasks. It will perform the walk on > > scheduler work queue first, then handle the local work queue. > > > > Signed-off-by: Haiyue Wang > > Signed-off-by: Cunming Liang > > Signed-off-by: Zhirun Yan > > --- > > lib/graph/graph.c | 6 ++++++ > > lib/graph/rte_graph_worker.h | 11 +++++++++++ > > 2 files changed, 17 insertions(+) > > > > diff --git a/lib/graph/graph.c b/lib/graph/graph.c index > > b4eb18175a..49ea2b3fbb 100644 > > --- a/lib/graph/graph.c > > +++ b/lib/graph/graph.c > > @@ -368,6 +368,8 @@ rte_graph_destroy(rte_graph_t id) > > while (graph !=3D NULL) { > > tmp =3D STAILQ_NEXT(graph, next); > > if (graph->id =3D=3D id) { > > + /* Destroy the schedule work queue if has */ > > + graph_sched_wq_destroy(graph); > > /* Call fini() of the all the nodes in the graph */ > > graph_node_fini(graph); > > /* Destroy graph fast path memory */ @@ -470,6 +472,10 > @@ > > graph_clone(struct graph *parent_graph, const char *name, > > if (graph_node_init(graph)) > > goto graph_mem_destroy; > > > > + /* Create the graph schedule work queue */ > > + if (graph_sched_wq_create(graph, parent_graph)) > > + goto graph_mem_destroy; > > + > > /* All good, Lets add the graph to the list */ > > graph_id++; > > STAILQ_INSERT_TAIL(&graph_list, graph, next); diff --git > > a/lib/graph/rte_graph_worker.h b/lib/graph/rte_graph_worker.h index > > faf3f31ddc..e98697d880 100644 > > --- a/lib/graph/rte_graph_worker.h > > +++ b/lib/graph/rte_graph_worker.h > > @@ -177,6 +177,7 @@ static inline void rte_graph_walk(struct > > rte_graph *graph) { > > const rte_graph_off_t *cir_start =3D graph->cir_start; > > + const unsigned int lcore_id =3D graph->lcore_id; > > const rte_node_t mask =3D graph->cir_mask; > > uint32_t head =3D graph->head; > > struct rte_node *node; > > @@ -184,6 +185,9 @@ rte_graph_walk(struct rte_graph *graph) > > uint16_t rc; > > void **objs; > > > > + if (graph->wq !=3D NULL) > > + __rte_graph_sched_wq_process(graph); > > + >=20 >=20 > We should introduce a flags field in rte_graph_param which can be used by > the application to define whether a graph should support multi-core > dispatch. >=20 Yes, I will add a flags field in next version. > Then we can make `__rte_graph_sched_wq_process` as node 0 during graph > creation so that it will be always called at the start of graph processin= g > followed by calling rest of the nodes. > This will remove unnecessary branches in fastpath. >=20 Thanks for your comments and sorry for my late reply. Yes, we can make `__rte_graph_sched_wq_process` as node 0 with dispatch fla= gs. But I am not sure whether we need to register a new node here. It means we = should change the graph topo, and it will be an isolated node with no links. > > /* > > * Walk on the source node(s) ((cir_start - head) -> cir_start) and > > then > > * on the pending streams (cir_start -> (cir_start + mask) -> > > cir_start) @@ -205,6 +209,12 @@ rte_graph_walk(struct rte_graph > *graph) > > objs =3D node->objs; > > rte_prefetch0(objs); > > > > + /* Schedule the node until all task/objs are done */ > > + if (node->lcore_id !=3D RTE_MAX_LCORE && (int32_t)head > 0 > > && > > + lcore_id !=3D node->lcore_id && graph->rq !=3D NULL && > > + __rte_graph_sched_node_enqueue(node, graph->rq)) > > + goto next; > > + > > if (rte_graph_has_stats_feature()) { > > start =3D rte_rdtsc(); > > rc =3D node->process(graph, node, objs, node->idx); @@ -215,6 > +225,7 > > @@ rte_graph_walk(struct rte_graph *graph) > > node->process(graph, node, objs, node->idx); > > } > > node->idx =3D 0; > > + next: > > head =3D likely((int32_t)head > 0) ? head & mask : head; > > } > > graph->tail =3D 0; > > -- > > 2.25.1