From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA53DC48BC3 for ; Tue, 20 Feb 2024 11:39:42 +0000 (UTC) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D56DF402A7; Tue, 20 Feb 2024 12:39:41 +0100 (CET) Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) by mails.dpdk.org (Postfix) with ESMTP id A9C8040289 for ; Tue, 20 Feb 2024 12:39:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708429180; x=1739965180; h=date:from:to:cc:subject:message-id:references: content-transfer-encoding:in-reply-to:mime-version; bh=nap6mYBC0bWiCT7XoYer+SLKNcrfiVj0V/Mv0txSpsk=; b=TtAxZDlxXX0LviMGtXtxRa5rn6r61x1ayOYGWc4wQqyZ7E0VHH7zpORT Jm7jUhOyeVFwiyDCx6fVQ3+eegTJNXxOk9yftmP+1/iIeToGgh6ohMuAA vG9K8mdi/U6nxNyhjxwYTwl2D3ss7UuRNRpVkwhyqipIsu5IZ49u9AkLx 5Bl5nv3qZgHhmCzSi2HplYXPNCiN+FX30htc0/nccu4vR7MId4IP/S2fR 1ktigV6guP50zFJWLqMarFZ3wjGDSJxNB4MX31VxhH5ZYgSEciRXh00tB KTLsYTV+dXc7YX8Fl+dtlAeHhLavQOe30LbC2dxcyg0lHq0MYeKWjf0lN A==; X-IronPort-AV: E=McAfee;i="6600,9927,10989"; a="6344424" X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="6344424" Received: from fmviesa005.fm.intel.com ([10.60.135.145]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Feb 2024 03:39:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="9374234" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by fmviesa005.fm.intel.com with ESMTP/TLS/AES256-GCM-SHA384; 20 Feb 2024 03:39:37 -0800 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 20 Feb 2024 03:39:37 -0800 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Tue, 20 Feb 2024 03:39:36 -0800 Received: from ORSEDG602.ED.cps.intel.com (10.7.248.7) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35 via Frontend Transport; Tue, 20 Feb 2024 03:39:36 -0800 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (104.47.51.40) by edgegateway.intel.com (134.134.137.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.35; Tue, 20 Feb 2024 03:39:36 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OWjSozI8RT37kZZSNlgiwgP7D5xXg5UFmB+Fij+W669F2YMYt70JRx3FGe6QhVlf80rw5WpWpmwJpElrP70PM7sqUJhgxr6Z3pB4fGSBMSQ94Ah2EjGTBjoT53vuSnxEJySospNdAT6WG9ul3zvQa4qIISmtcH3lBznK4+ZR/Pg47+5yjGTctA4ccwMCyjIxpNKckw3P60D9es3FVk/sbLECR0tNpKKs0Y7LV5z4tercoS8Q3owv8lXPNUnZTzOWqD29LVwy/FMyjpdkdyRLz1YW0TuUVOBC8KzHaAt122BfpKJsfKFo89V7kOIi/rRkXsi6zasrFZ3q964czukfEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0z4A1BT/kKDB7lC2U6NYxZSWGLOkEV/YgClG8hlDG90=; b=ZTzuKn+2Rt4SPz+Vb0q0G1WGqpb78RZhj3vLm42bHv7hJnu/ustpmbNA6D7Kqf88g2T4er8MAAwLoXWnAw/82lh+1gdrmT4NIk5plRAuc48fgqP9bN0cqeYRa4Fo1ei6qrL+ruU2HRhKeHJ6DlQ5cxq2mTytooxXntP+bEumUI3wxjX+3WXUx/fPvYI4qbmKzb4vaIO6tx52tZa+lw7EhG2dNXVkh5TjFFDvweWdZaFOOV6p1FJ3QlBeulfA40SpxfR6OYhbODjtvAH7qdrk/6EvVdgVdMy4/BkTPk8T9Pk0HlW+0nuMoBn4WyHfsSGPyvTURleGIqkBlf34WbQSVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) by CH3PR11MB8517.namprd11.prod.outlook.com (2603:10b6:610:1ad::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.39; Tue, 20 Feb 2024 11:39:34 +0000 Received: from DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e]) by DS0PR11MB7309.namprd11.prod.outlook.com ([fe80::d10:3009:a8d3:1d2e%7]) with mapi id 15.20.7292.036; Tue, 20 Feb 2024 11:39:34 +0000 Date: Tue, 20 Feb 2024 11:39:27 +0000 From: Bruce Richardson To: Mattias =?iso-8859-1?Q?R=F6nnblom?= CC: Mattias =?iso-8859-1?Q?R=F6nnblom?= , , Morten =?iso-8859-1?Q?Br=F8rup?= , Stephen Hemminger Subject: Re: [RFC v3 1/6] eal: add static per-lcore memory allocation facility Message-ID: References: <20240219094036.485727-2-mattias.ronnblom@ericsson.com> <20240220084908.488252-1-mattias.ronnblom@ericsson.com> <20240220084908.488252-2-mattias.ronnblom@ericsson.com> <68c8f01c-d404-4b63-adca-13b560c95df8@lysator.liu.se> Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <68c8f01c-d404-4b63-adca-13b560c95df8@lysator.liu.se> X-ClientProxiedBy: DB6PR0301CA0071.eurprd03.prod.outlook.com (2603:10a6:6:30::18) To DS0PR11MB7309.namprd11.prod.outlook.com (2603:10b6:8:13e::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS0PR11MB7309:EE_|CH3PR11MB8517:EE_ X-MS-Office365-Filtering-Correlation-Id: 75616324-2229-4f12-5129-08dc32089c71 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: H9DlfdjFbeQmNQ1Luv36MfcXBWvQpFCQaro1FM0hFz4v7KlL7EP04eI4iQPGEZI1oQ0FAOzAjj+2hTwIGzo1tWm0PRzBQ6EiO1gONBqrb1fC873XA2iOX1WH4lm9Ippar6prtu9Leg0qzIUuVOcK7QfVP9W/0DsfHvvdTfFh911l3Hi4mYqdkvZJwJ2zSvwCh8TlenXYi3deXZ9mff2wTOTH6wHkKNyc2qjdu0OS6lP4YxNJ0JQLxfHpcMUz60UaTGTJvUs9YEwv53BM2wg136B5fuxv0RuQPnhbAbstpMa/CJE6whEvySffgkkq+4RzCaO4ehCCOUE+wi86BC3OOsb6IKCoyWPR4GbpkgEDcNol7ZO1EeY+hcgY0F3/mCBfBHrw6S+u0r5VnhiS03lfcwE6EVgG/ZU9AWQXREhp1M/msnfKqWYOEm/xcasqM05SPG2EIrO7BaUlcDCzojh552Fn15JpUzr1rC1MiOLpt3CHu1JZbs2pZt3IWLrEmy/j0RDAugA0Cs/EiXk3pJA1KoVVbp8NwvsaP+ucYSQL/sg= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB7309.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?iso-8859-1?Q?8Ov+FDPQkbDzbhspn1yZR5VjIMMbv8kfz1mEQZWb3oNvfEXqpmZkmWVnwO?= =?iso-8859-1?Q?MmY5fPIgLLvv//nb28zgzmXr/dmQu9bJuU1o58ItdHRGKR0crQtDMHXHVT?= =?iso-8859-1?Q?jQ28WsSgRrxyiw5KEI3Or6KclFY1Pd0+xprv0V9vdX1inI+kLc24ZV9iKi?= =?iso-8859-1?Q?2rKzpt4CVzuxt+pXgKVV35FNtvenUATkeW2Z88eVGcRFGo4Ld5umb5UAFo?= =?iso-8859-1?Q?EQxanvBur6ejh29X6SfslQc9rmZxhORNrmUMp+SxBXym+zxDMBamD+yyhZ?= =?iso-8859-1?Q?Qbz1Qi+gySxAgkn0Kuj2yiphRDvN9Nz26lmuCjtaTQkR+V9TFvr7OxIBxY?= =?iso-8859-1?Q?jprovPs9qe/OEmFTaF7Mfj/jUIVsu9jUvkcsAoyIK4ut4Cn+yu1Uo0d6Fx?= =?iso-8859-1?Q?5Yrg/Qo3W6r5yxHcLL7GtUwD6BTm5jXJj9LwRD1jc01vsdwY/b+gEmTSSx?= =?iso-8859-1?Q?4+Bmu1cdZKniZ2DJgIaAOtLkRXKZ3OtasX+4GkOhmoQT997a8Pj4+8WTXw?= =?iso-8859-1?Q?f3HC1chN1dC/8OQcpDHSNdxpXkKLrN9Yo/zd6uUi9opPKvObSea5EYYB2w?= =?iso-8859-1?Q?x35uH+XwxKAJGrjHWFOIfgOFFJIq2VCd1UPMiL/EwU6w17hug4qfBGfU2h?= =?iso-8859-1?Q?gvV79lytBtWkMzRairJhCDvcHR6HKkY/+A8GLkj2SiVSW4DPy8UpOLLnAe?= =?iso-8859-1?Q?CxJckkLGwSsyW2vHO1IDqrqBCrw+/lZ+HAAe4P9Tf1lwuQdgwF9As6Nsre?= =?iso-8859-1?Q?ribc1UOOBFAs5PoKuEr9btNV9vAjk129tZaesCaVfv+M8sqPK6gNt5ovsG?= =?iso-8859-1?Q?9qLELk1qUvn3iXHtmTFFyVZOL3EWZY3x4nn72Qn1bgYyU5ea+k+5+BvKd9?= =?iso-8859-1?Q?bZVnfxV3U0qmIDCS9cnxlqahWtAbwqKx3LxuH4kiOTK01JkVPnZsNDEj2M?= =?iso-8859-1?Q?YKNMWjq1h3BR7skFglIPFGk62eWmIeMGkMsJvSpzJCpzSVBOzLQeL4SsDT?= =?iso-8859-1?Q?fVBJuOrb7H9QE0eJgRXFtQY4wMQPItForS9CzGtKcaIG7FqjQxQ5toDost?= =?iso-8859-1?Q?I5lhF0gGAvyh09p/AMO9RfAICnEGA8HsPr/Hxq+qVHpz52WbP+w6HZVMLA?= =?iso-8859-1?Q?9TV2N85V/ddwSeDHfQO78chT+Cby6Lh/QW7CCaKt2LxD8u/HeLJas1+dPA?= =?iso-8859-1?Q?qHwKr6ZeVuOfwLYDlgyFjKzr1d3qQ9N3he/tDkVaC+4CCeKrhD9PEYUEGG?= =?iso-8859-1?Q?nVgcecUYNuQhS+k9GyhKOWk4et81XTcMgiQ71TSJXACNVcZJqL+Ej7FAgG?= =?iso-8859-1?Q?/EONI5Cb0k9l7P3/r+EviM0KT1dgBTX/m6Kp89rU499Io0Q6zr7Sh+VGWn?= =?iso-8859-1?Q?MRkjI8rQ06sRFGWTqbGmysujQPFEZxSGN4yT0HiWYJaOEMxuiNfK0xhyih?= =?iso-8859-1?Q?rFoZtNnR3AVI65XIAFWlc39chogL/XC4yZ3uXb21mGbb+ehzBbzxIVQO0H?= =?iso-8859-1?Q?6sJrGRfhK0wvv4cvhd9/zDNEzQ7TUm65yeRo69IzyDZbQqPQPQrvUzbHgo?= =?iso-8859-1?Q?NBn6ohat/B27hPlIB1P9MqAHyosYzld+bvxn9mBT3RS31uVTRQY+4o3QHm?= =?iso-8859-1?Q?Fp60w06A1prf10w2gLxRbUWx16abJd1M/ezesHTEM/XTGieuSKVN52gw?= =?iso-8859-1?Q?=3D=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 75616324-2229-4f12-5129-08dc32089c71 X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB7309.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Feb 2024 11:39:34.7419 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hoM6vXOcq2yZE0srhnDh9jJAC2G4G5EPUv3fvPIeZO0MNYUGc9aIDIglKQtjJvV+3+zrELAwY/y6DEZC3aTUYdCeJxzV4XEUqZkFxldwXxU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR11MB8517 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Tue, Feb 20, 2024 at 11:47:14AM +0100, Mattias Rönnblom wrote: > On 2024-02-20 10:11, Bruce Richardson wrote: > > On Tue, Feb 20, 2024 at 09:49:03AM +0100, Mattias Rönnblom wrote: > > > Introduce DPDK per-lcore id variables, or lcore variables for short. > > > > > > An lcore variable has one value for every current and future lcore > > > id-equipped thread. > > > > > > The primary use case is for statically allocating > > > small chunks of often-used data, which is related logically, but where > > > there are performance benefits to reap from having updates being local > > > to an lcore. > > > > > > Lcore variables are similar to thread-local storage (TLS, e.g., C11 > > > _Thread_local), but decoupling the values' life time with that of the > > > threads. > > > +/* > > > + * Avoid using offset zero, since it would result in a NULL-value > > > + * "handle" (offset) pointer, which in principle and per the API > > > + * definition shouldn't be an issue, but may confuse some tools and > > > + * users. > > > + */ > > > +#define INITIAL_OFFSET 1 > > > + > > > +char rte_lcore_var[RTE_MAX_LCORE][RTE_MAX_LCORE_VAR] __rte_cache_aligned; > > > + > > > > While I like the idea of improved handling for per-core variables, my main > > concern with this set is this definition here, which adds yet another > > dependency on the compile-time defined RTE_MAX_LCORE value. > > > > lcore variables replaces one RTE_MAX_LCORE-dependent pattern with another. > > You could even argue the dependency on RTE_MAX_LCORE is reduced with lcore > variables, if you look at where/in how many places in the code base this > macro is being used. Centralizing per-lcore data management may also provide > some opportunity in the future for extending the API to cope with some more > dynamic RTE_MAX_LCORE variant. Not without ABI breakage of course, but we > are not ever going to change anything related to RTE_MAX_LCORE without > breaking the ABI, since this constant is everywhere, including compiled into > the application itself. > Yep, that is true if it's widely used. > > I believe we already have an issue with this #define where it's impossible > > to come up with a single value that works for all, or nearly all cases. The > > current default is still 128, yet DPDK needs to support systems where the > > number of cores is well into the hundreds, requiring workarounds of core > > mappings or customized builds of DPDK. Upping the value fixes those issues > > at the cost to memory footprint explosion for smaller systems. > > > > I agree this is an issue. > > RTE_MAX_LCORE also need to be sized to accommodate not only all cores used, > but the sum of all EAL threads and registered non-EAL threads. > > So, there is no reliable way to discover what RTE_MAX_LCORE is on a > particular piece of hardware, since the actual number of lcore ids needed is > up to the application. > > Why is the default set so low? Linux has MAX_CPUS, which serves the same > purpose, which is set to 4096 by default, if I recall correctly. Shouldn't > we at least be able to increase it to 256? The default is so low because of the mempool caches. These are an array of buffer pointers with 512 (IIRC) entries per core up to RTE_MAX_LCORE. > > > I'm therefore nervous about putting more dependencies on this value, when I > > feel we should be moving away from its use, to allow more runtime > > configurability of cores. > > > > What more specifically do you have in mind? > I don't think having a dynamically scaling RTE_MAX_LCORE is feasible, but what I would like to see is a runtime specified value. For example, you could run DPDK with EAL parameter "--max-lcores=1024" for large systems or "--max-lcores=32" for small ones. That would then be used at init-time to scale all internal datastructures appropriately. /Bruce