From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4110FC433DB for ; Wed, 17 Feb 2021 13:27:12 +0000 (UTC) Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by mail.kernel.org (Postfix) with ESMTP id 8F98964E63 for ; Wed, 17 Feb 2021 13:27:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F98964E63 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=dev-bounces@dpdk.org Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 61F7240690; Wed, 17 Feb 2021 14:27:10 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id 06B724067A for ; Wed, 17 Feb 2021 14:27:07 +0100 (CET) IronPort-SDR: LShHniH3jfHnNzf4xFtmoF3RsVxyudDZwwXmpMP7fGYwSv11TFsUFKMaFRDfHTqayT3qb6QdFf mFD+f1SXPPXw== X-IronPort-AV: E=McAfee;i="6000,8403,9897"; a="247258897" X-IronPort-AV: E=Sophos;i="5.81,184,1610438400"; d="scan'208";a="247258897" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2021 05:26:54 -0800 IronPort-SDR: V0HywQbKAYV7ZyX+Bet7n0+So2A5JVvcA82X0CwdcyxBsaQ5WYgcDCUiSnyB1obUXyoWoU08IS fexEGZgxEL6A== X-IronPort-AV: E=Sophos;i="5.81,184,1610438400"; d="scan'208";a="385137530" Received: from bricha3-mobl.ger.corp.intel.com ([10.252.4.45]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-SHA; 17 Feb 2021 05:26:53 -0800 Date: Wed, 17 Feb 2021 13:26:49 +0000 From: Bruce Richardson To: "Van Haaren, Harry" Cc: "Burakov, Anatoly" , "dev@dpdk.org" Message-ID: <20210217132649.GA1171@bricha3-MOBL.ger.corp.intel.com> References: <20210216094300.27889-1-bruce.richardson@intel.com> <313c223f-bf1c-9307-75f8-0a0c1da7fd21@intel.com> <20210216104652.GB136@bricha3-MOBL.ger.corp.intel.com> <42706d4c-f8de-55c5-1161-b1e54c77599e@intel.com> <20210216173057.GE136@bricha3-MOBL.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no-affinitization X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Wed, Feb 17, 2021 at 12:14:36PM +0000, Van Haaren, Harry wrote: > > -----Original Message----- > > From: Burakov, Anatoly > > Sent: Wednesday, February 17, 2021 12:09 PM > > To: Van Haaren, Harry ; Richardson, Bruce > > > > Cc: dev@dpdk.org > > Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no- > > affinitization > > > > On 16-Feb-21 5:44 PM, Van Haaren, Harry wrote: > > >> -----Original Message----- > > >> From: Bruce Richardson > > >> Sent: Tuesday, February 16, 2021 5:31 PM > > >> To: Van Haaren, Harry > > >> Cc: Burakov, Anatoly ; dev@dpdk.org > > >> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no- > > >> affinitization > > >> > > >> On Tue, Feb 16, 2021 at 05:22:25PM +0000, Van Haaren, Harry wrote: > > >>>> -----Original Message----- > > >>>> From: Burakov, Anatoly > > >>>> Sent: Tuesday, February 16, 2021 10:53 AM > > >>>> To: Richardson, Bruce ; Van Haaren, Harry > > >>>> > > >>>> Cc: dev@dpdk.org > > >>>> Subject: Re: [dpdk-dev] [PATCH] eal: support using 0 as coremask for no- > > >>>> affinitization > > >>>> > > >>>> On 16-Feb-21 10:46 AM, Bruce Richardson wrote: > > >>>>> On Tue, Feb 16, 2021 at 10:36:13AM +0000, Burakov, Anatoly wrote: > > >>>>>> On 16-Feb-21 9:43 AM, Bruce Richardson wrote: > > >>>>>>> Allow the user to specify that they don't want any core pinning from > > >> DPDK > > >>>>>>> by passing in the coremask of 0. > > >>>>>>> --- > > >>>>>> > > >>>>>> I haven't checked what happens yet, but down the line we also set > > affinity > > >>>>>> for service cores as well as interrupt thread. what would be the > > semantics > > >>>>>> of those in this particular case? do we want the same ability for service > > >>>>>> cores (i.e. pick a non-affinitized core)? And where does interrupt thread > > >>>>>> affinitize in this case (presumably, nowhere too)? > > >>>>>> > > >>>>> I have not checked the service core setup, because a) I forgot about them > > >>>>> and b) I'm not sure how their affinity rules work with respect to the main > > >>>>> lcore mask. On the other hand I did check out that the lcore mask for all > > >>>>> non-pinned threads, or control threads, is the full set of bits as > > >>>>> expected. > > >>>>> > > >>>>> /Bruce > > >>>>> > > >>>> > > >>>> +Harry, > > >>>> > > >>>> I believe service core mask must not overlap with lcore masks, so > > >>>> presumably using 0 as lcore mask would make it so that any service core > > >>>> mask will be valid (which is presumably what we want?). > > >>> > > >>> Services cores -S list or -s *must* overlap with the RTE lcores, EAL > > >>> then"steals" the service cores from the application lcores, code that > > >> implements here: > > >>> http://git.dpdk.org/dpdk- > > >> stable/tree/lib/librte_eal/common/eal_common_options.c?h=20.11#n657 > > >>> > > >>>> Should service cores also have a "just pick a core" parameter? > > >>> > > >>> I'm not sure, depends on what the bigger goal is here. > > >>> Assuming we're enabling this for ROLE_RTE threads, then > > >>> it would seem to me that ROLE_SERVICE and control threads > > >>> would require similar treatment? > > >>> > > >> Control threads are affinitised to all cores not in the coremask, which > > >> means in this case that they can run anywhere on the system the OS chooses. > > > > > > Ah ok, fair enough yes. > > > > > >> In case of service cores, it would seem that using service cores with an > > >> empty coremask is just not compatible. I would assume that this > > >> incompatibility already exists when one has a coremask with only one core > > >> already in it. > > > > > > Yes, correct, it would leave zero lcores for ROLE_RTE, meaning no lcores for the > > application. > > > A possible solution would be to special case a zero service core mask and apply > > the same > > > treatment as ROLE_RTE coremask? > > > > > > Others likely have better ideas - I don't have time to follow DPDK > > threading/pinning topic > > > closely at the moment. > > > > > > > I don't think it's a good idea to disallow service cores functionality > > in this case, but i don't have a way to solve this, other than > > implementing similar 0x0 coremask for service cores and assume it always > > means "one core affinitized to wherever the OS feels like it". After > > all, with lcore mask 0x0 we assume user wants one single core only, so > > following that, one single service core is a valid extrapolation IMO. > > OK with me - seems reasonable. > > > Perhaps specifying the number of l/s cores when using 0x0 would be > > interesting, but IMO unless there's ask for it, i'd rather not > > overcomplicate things and go with similar semantics for service cores, > > and just allow a 0x0 coremask that means only one unaffinitized service > > core will be created. > > > > Thoughts? > > Agree with keeping-it-simple if possible, and agree that unaffinitized with > a single service-core with a 0x0 mask makes sense. > > Most important to me is to maintain backward compatibility with existing > usage of -S and -s, but this shouldn't break anything? (Famous last words..) > Not sure I entirely follow all of this. Is the suggestion just to extend -s processing to allow "0" as coremask too? That would be independent then of any core masks passed in for -c/-l options, right? As well as working well with this patch, it would also solve the issue of using single core with a coremask of e.g. 0x1 too, I think. Is my understanding correct? /Bruce