From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66932C4320A for ; Fri, 20 Aug 2021 20:23:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 469F060E9B for ; Fri, 20 Aug 2021 20:23:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238917AbhHTUYU (ORCPT ); Fri, 20 Aug 2021 16:24:20 -0400 Received: from mga03.intel.com ([134.134.136.65]:2267 "EHLO mga03.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230266AbhHTUYT (ORCPT ); Fri, 20 Aug 2021 16:24:19 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10082"; a="216870886" X-IronPort-AV: E=Sophos;i="5.84,338,1620716400"; d="scan'208";a="216870886" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Aug 2021 13:23:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.84,338,1620716400"; d="scan'208";a="490775322" Received: from orsmsx605.amr.corp.intel.com ([10.22.229.18]) by fmsmga008.fm.intel.com with ESMTP; 20 Aug 2021 13:23:39 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX605.amr.corp.intel.com (10.22.229.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Fri, 20 Aug 2021 13:23:39 -0700 Received: from orsmsx601.amr.corp.intel.com (10.22.229.14) by ORSMSX610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.10; Fri, 20 Aug 2021 13:23:38 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx601.amr.corp.intel.com (10.22.229.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.10 via Frontend Transport; Fri, 20 Aug 2021 13:23:38 -0700 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.102) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.10; Fri, 20 Aug 2021 13:23:10 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BkXyKivrVM2YWjiqMaBn1hvzKSGlZyiJ3KTm2+xD6iM13eWxZCADTq54Nht/QY7cLLL978J15bA49X3/WYlTp4rnDUac+WbskZEp6Jc9Xyf4oXAt4roB7Bj/KoZJdkjLxGoyQQrOzWOVNrIE4B54cgNmFTBLyJhxLC4Jis01SG/03NrxR2+jdypMdROyHflwHT2iLZTklE0W2r0d5Tt8DMdaLTv0aj//LPL8x6GzgXN923Ev7vsC5FzBXe2yXqhAesYtKXiu8AneivXSnptwXLaDfJwaDmcUoOB1wxVv6IsvZjc7M//d3535UOYimAN+2NgKvcN7BVEB1kzN1To4lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ejQ8NFjgbRlplZ6lyu6hA0zZ30xiolwpD9l3CE+1+5I=; b=RDTC1QIoz0ojlnzljE8cOqGsRPZg4FJ5L0ovj8GqoFBAWWLnAn/g2RurP0PTrhrlvZOgFufw7UaU1fAsA7xKduI7RQi0FBQtFw0UCB/stCoHSa716nFS3ugJiW5Z1cGjkLJcCTpHHz/FIRoNxZUo0NUX4ILUYj7uNouVovGhrNxromM/NA0OqDnkmMXvuRaS5wLpDRImPp+dKPHLSBcrz9ldzueC2viMre+xKVsZWhvkOpIbZZ+zI7Yk5SPj6K70REABkkUXL/LtARunQKpubiYF7zWT9PtQVZrB4Sq8CHflf6k4F43dqCJuyfXourhT/3FJmogRFdZpm+l0VQNCAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ejQ8NFjgbRlplZ6lyu6hA0zZ30xiolwpD9l3CE+1+5I=; b=jnBrSezitYvGnxvuyLhr6tqvxH/QBRzok1iTktJvRDywLEUmO0AFadHP3bCno8enLiOGQd/39pZF+yd3wydj4kI6mZu6wG8wPVGaF5pDgWk8mOffGhwP9wDkHDg7z75AMJXZj+sLZdq4pWYwopHkT++dkIoR2LdNhLhFoeX8Nic= Received: from CO1PR11MB5089.namprd11.prod.outlook.com (2603:10b6:303:9b::16) by MW3PR11MB4635.namprd11.prod.outlook.com (2603:10b6:303:2c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Fri, 20 Aug 2021 20:23:09 +0000 Received: from CO1PR11MB5089.namprd11.prod.outlook.com ([fe80::adef:da48:ea32:5960]) by CO1PR11MB5089.namprd11.prod.outlook.com ([fe80::adef:da48:ea32:5960%3]) with mapi id 15.20.4436.021; Fri, 20 Aug 2021 20:23:09 +0000 From: "Keller, Jacob E" To: Leon Romanovsky CC: Jakub Kicinski , "David S . Miller" , Guangbin Huang , Jiri Pirko , "linux-kernel@vger.kernel.org" , "netdev@vger.kernel.org" , Salil Mehta , "Shannon Nelson" , Yisen Zhuang , "Yufeng Mo" Subject: RE: [PATCH net-next 3/6] devlink: Count struct devlink consumers Thread-Topic: [PATCH net-next 3/6] devlink: Count struct devlink consumers Thread-Index: AQHXkPLeq0W5yofhY0KH8k2ic9V3+Kt2SjGAgAABsoCAAAO0AIAAWqrwgAJFW4CAAKBNQIAC1qCAgAB5r8A= Date: Fri, 20 Aug 2021 20:23:09 +0000 Message-ID: References: <20210816084741.1dd1c415@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <20210816090700.313a54ba@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: kernel.org; dkim=none (message not signed) header.d=none;kernel.org; dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 45a69f5e-e8ca-4de0-8b75-08d964185381 x-ms-traffictypediagnostic: MW3PR11MB4635: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5236; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E27WrkeEseuibFTFflEjS02XiBK+Dy0k1aLxGJOJ8LtosBStuT3hDqM/iz9Q5gCfUlMnzFPqHXUHT0ECz1wIuwbD2GHq8THHnM3VkRA+NmhA1VTeXIVNEw07PEm2e+3kDCeSmSLB5M1tGfP59w+GKeYEKHNY3WAF1azy1vyAcfuYpxXKip+XkUPOxGHziTeML868nSiKHe6SOqm91QsvW+j8q2SocLuF+YQ6BZvicloat4eBqqw5RTYHjpqW1f6LlYvkxMa6ndGhz4gcatNiWamhvKWdSTML8ezH1on9OeaIBmv/baDeHlMWTdCfaE7TFLxpdnu5UyKl05ztabkkbV2719DZqXS4pGxp8GxKtBGS7grxg5j3yV+kfkyMeJgUxOMjQ/RNB7bCE2MXzHuhIzaVdmFl32Bsk5WDwx0z963urgPV3dhQqCHEzRW3+Nb+VKF2HYv85PiRxf9VDQ55SB0uEhAuOOPk21FdnDgyFT0dvKOywZN9g/yViEEbl6CEvNZ4Q9ZdZ9kB2mGgJTsaksq7abvGpFKzy59nRFAz4Iryf88B7tQNPY63ENc2Oi4W6pQPtjEN8sOodD1OY+Mmj+8Mwd8RbA+ci5+XT+nEScJPoRo339vOB6g57KJoFlVCNkXxOfsooZ4rjkH/nWGDIhkius9w252vAB/ucfW4CxGofAzHssVbRg4y7sdzhLYH6GSLBJjKY88cMjYdZOUa+Q== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO1PR11MB5089.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(39860400002)(136003)(346002)(396003)(376002)(33656002)(53546011)(2906002)(86362001)(76116006)(66556008)(186003)(66446008)(64756008)(66476007)(71200400001)(66946007)(38070700005)(55016002)(26005)(83380400001)(316002)(478600001)(8676002)(6916009)(8936002)(54906003)(5660300002)(122000001)(38100700002)(7696005)(52536014)(4326008)(6506007)(9686003)(7416002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?oGmpgvsFlY4P7z74a+ica9nqHdQWao9m3BMwBypJVpfzwz5OLBju/GNJodVg?= =?us-ascii?Q?wYLpVKDWbKekkEKe6chWwNjAK5nSSmaku3D6pRbPXfMPLoo7wYRGSiolRr2a?= =?us-ascii?Q?QBIbTBOWnnS/45tbI7CtIPGN4+tMrlX5JPlOmaV2GEMO0tHf5cZTvF95HAex?= =?us-ascii?Q?WH/A5qjMdENZHVmpY9IAymFtnfTSi/EQtZU6SnHlaTuUK/Yz5rkzII2yDVm+?= =?us-ascii?Q?U0dmSpr5EYErJaQWUtmuNfeTzYIc/BvmNaAXZuqewfIZ98kzIEEVXMVcMOLy?= =?us-ascii?Q?9eWlpyoF1ccLTV0slevKeIHC1a7Z7y7Nw0ocUKqTieGs2KR3GC5eizNGiRpm?= =?us-ascii?Q?jT8r0QJckEITrn7WZRjLKlee9hdhJGW3kZOJSPE523VvvfSwMY0R61MorV6E?= =?us-ascii?Q?1XwPHB5hmuY50NYPHAP/SqPz3EpTeFIiNLA9By1HW5vpMZfag3Dkk3Pdz1c/?= =?us-ascii?Q?3dYhzvirLcSqvWqvGO8sdubcdoqFIEZOnkf98lKCZ74mZFNF6Mma48FjKI8o?= =?us-ascii?Q?3a8iozCGajr2UnKcOqHX7f+cToVMb7xm6G3g4qCfP8/K9u6AcN1L4C9KFPyb?= =?us-ascii?Q?WEal0SOxZs3VyWoLmzdIq7eRb+F5tzFwQ1wr5Yy0pqnMDbZPcYISl7ZFmQi1?= =?us-ascii?Q?pIp1NbwOmKx+r10QG/v0feJRx4mYnROaPKHYQhJfOsotl1qCpN9qnO/qHdfq?= =?us-ascii?Q?u39eKJJJZ68VFhnEXO7HYkyjvnxe7vOJ+KMqyDn/dWAryhexVitHEb57cq5Z?= =?us-ascii?Q?U1RjOjtTSYdt/iX9+7sVEx3pox0fVVn9TpfhcSXWMAnDcDlz/RXmLINciuLV?= =?us-ascii?Q?NrVikRc2XO1YTq6T0kb42gvabUDCiAntlaHQQlU/rae3g184FkuriJo+yFWE?= =?us-ascii?Q?KUHbFgVnSemVg7TiVwz214aLwjeG7AcwOP7I6m7hFHrJ4lSLcqHxK6fL4Pzy?= =?us-ascii?Q?fLinBw/vss8xvs/BeP5mOPBM6lyYi2ALsQfcMCkBl3d3u+KeRWtPuoYleq12?= =?us-ascii?Q?XKCgv3/eC+2umQq8OhFEQeZwcCxmhCNZDUKvWU6LHcmFjgAyHVkFCDYW7hzR?= =?us-ascii?Q?NBPjJcSshMYKl1JvfCfqM20Lfvr7ByL1Y2h0mIiYt3jomQjvtNXrQM7pyhVY?= =?us-ascii?Q?soh6ZZDDBNQqJcK7nRUCmHYBp2a9z5rbnmhxRwQ1H1ckyBjik0MLZ/KPGprx?= =?us-ascii?Q?wMShDvj+XoDqZdkelEVWRDuS4XvKppHIKpNEOCfjtNPELdguBE14fxjcnN3k?= =?us-ascii?Q?nkKGwYt0IVj5eyeC9LOwqS1YkuFQx8OrYzDAvSIhiZVbtqaOuU3gbGixlhmo?= =?us-ascii?Q?b5w=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB5089.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45a69f5e-e8ca-4de0-8b75-08d964185381 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Aug 2021 20:23:09.1213 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: w/QhspQJMdckrMBcnIyI74ArYo+voXEwLgG6bZAuqhrSUG+bgF8chwS3VrCvouSaPQVx2NGI6MAoA4feg7TBz55WtvLAWtdGAy072aP3g94= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4635 X-OriginatorOrg: intel.com Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > -----Original Message----- > From: Leon Romanovsky > Sent: Friday, August 20, 2021 6:07 AM > To: Keller, Jacob E > Cc: Jakub Kicinski ; David S . Miller ; > Guangbin Huang ; Jiri Pirko ; > linux-kernel@vger.kernel.org; netdev@vger.kernel.org; Salil Mehta > ; Shannon Nelson ; Yisen > Zhuang ; Yufeng Mo > Subject: Re: [PATCH net-next 3/6] devlink: Count struct devlink consumers >=20 > On Wed, Aug 18, 2021 at 05:50:11PM +0000, Keller, Jacob E wrote: > > > > > > > -----Original Message----- > > > From: Leon Romanovsky > > > Sent: Wednesday, August 18, 2021 1:12 AM > > > To: Keller, Jacob E > > > Cc: Jakub Kicinski ; David S . Miller > ; > > > Guangbin Huang ; Jiri Pirko > ; > > > linux-kernel@vger.kernel.org; netdev@vger.kernel.org; Salil Mehta > > > ; Shannon Nelson ; Yisen > > > Zhuang ; Yufeng Mo > > > Subject: Re: [PATCH net-next 3/6] devlink: Count struct devlink consu= mers > > > > > > On Mon, Aug 16, 2021 at 09:32:17PM +0000, Keller, Jacob E wrote: > > > > > > > > > > > > > -----Original Message----- > > > > > From: Jakub Kicinski > > > > > Sent: Monday, August 16, 2021 9:07 AM > > > > > To: Leon Romanovsky > > > > > Cc: David S . Miller ; Guangbin Huang > > > > > ; Keller, Jacob E > ; > > > Jiri > > > > > Pirko ; linux-kernel@vger.kernel.org; > > > netdev@vger.kernel.org; > > > > > Salil Mehta ; Shannon Nelson > > > > > ; Yisen Zhuang ; > Yufeng > > > > > Mo > > > > > Subject: Re: [PATCH net-next 3/6] devlink: Count struct devlink c= onsumers > > > > > > > > > > On Mon, 16 Aug 2021 18:53:45 +0300 Leon Romanovsky wrote: > > > > > > On Mon, Aug 16, 2021 at 08:47:41AM -0700, Jakub Kicinski wrote: > > > > > > > On Sat, 14 Aug 2021 12:57:28 +0300 Leon Romanovsky wrote: > > > > > > > > From: Leon Romanovsky > > > > > > > > > > > > > > > > The struct devlink itself is protected by internal lock and= doesn't > > > > > > > > need global lock during operation. That global lock is used= to protect > > > > > > > > addition/removal new devlink instances from the global list= in use by > > > > > > > > all devlink consumers in the system. > > > > > > > > > > > > > > > > The future conversion of linked list to be xarray will allo= w us to > > > > > > > > actually delete that lock, but first we need to count all s= truct devlink > > > > > > > > users. > > > > > > > > > > > > > > Not a problem with this set but to state the obvious the glob= al devlink > > > > > > > lock also protects from concurrent execution of all the ops w= hich don't > > > > > > > take the instance lock (DEVLINK_NL_FLAG_NO_LOCK). You most li= kely > > > know > > > > > > > this but I thought I'd comment on an off chance it helps. > > > > > > > > > > > > The end goal will be something like that: > > > > > > 1. Delete devlink lock > > > > > > 2. Rely on xa_lock() while grabbing devlink instance (past > devlink_try_get) > > > > > > 3. Convert devlink->lock to be read/write lock to make sure tha= t we can > run > > > > > > get query in parallel. > > > > > > 4. Open devlink netlink to parallel ops, ".parallel_ops =3D tru= e". > > > > > > > > > > IIUC that'd mean setting eswitch mode would hold write lock on > > > > > the dl instance. What locks does e.g. registering a dl port take > > > > > then? > > > > > > > > Also that I think we have some cases where we want to allow the dri= ver to > > > allocate new devlink objects in response to adding a port, but still = want to > block > > > other global operations from running? > > > > > > I don't see the flow where operations on devlink_A should block devli= nk_B. > > > Only in such flows we will need global lock like we have now - devlin= k->lock. > > > In all other flows, write lock of devlink instance will protect from > > > parallel execution. > > > > > > Thanks > > > > > > But how do we handle what is essentially recursion? >=20 > Let's wait till implementation, I promise it will be covered :). >=20 Sure. It's certainly easier to talk about a proposed implementation once we= have it. > > > > If we add a port on the devlink A: > > > > userspace sends PORT_ADD for devlink A > > driver responds by creating a port > > adding a port causes driver to add a region, or other devlink object > > > > In the current design, if I understand correctly, we hold the global lo= ck but > *not* the instance lock. We can't hold the instance lock while adding por= t > without breaking a bunch of drivers that add many devlink objects in resp= onse to > port creation.. because they'll deadlock when going to add the sub object= s. > > > > But if we don't hold the global lock, then in theory another userspace = program > could attempt to do something inbetween PORT_ADD starting and finishing > which might not be desirable. (Remember, we had to drop the instance loc= k > otherwise drivers get stuck when trying to add many subobjects) >=20 > You just surfaced my main issue with the current devlink > implementation - the purpose of devlink_lock. Over the years devlink > code lost clear separation between user space flows and kernel flows. >=20 > Thanks >=20 Yep. It's definitely complex. > > > > Thanks, > > Jake