From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D72A2C43381 for ; Mon, 18 Mar 2019 15:43:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9795620854 for ; Mon, 18 Mar 2019 15:43:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="MzEJrq4+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727196AbfCRPn2 (ORCPT ); Mon, 18 Mar 2019 11:43:28 -0400 Received: from mail-eopbgr20046.outbound.protection.outlook.com ([40.107.2.46]:18566 "EHLO EUR02-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726808AbfCRPn2 (ORCPT ); Mon, 18 Mar 2019 11:43:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fZvqrOPd3yU20vte6wtA6XLA7jx9bJcTGIxXXs/XDWU=; b=MzEJrq4+iHY5ihytiTNRHle2OScicsgkHqp2WH0+2FTspfE3I25rKVA/9xEAFLagQpLfCJQonGS97Z1mEBNPEB00wCvBbLuXoc81Jcwighe5AFWxU425o7cZ7Dqxe1NapCmlVgPd6FOhzxYcTaJzPVfb5nxj4qySEEYl+42IrYY= Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com (10.169.135.8) by VI1PR0501MB2286.eurprd05.prod.outlook.com (10.169.135.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.14; Mon, 18 Mar 2019 15:43:21 +0000 Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59]) by VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59%6]) with mapi id 15.20.1709.015; Mon, 18 Mar 2019 15:43:21 +0000 From: Parav Pandit To: Jakub Kicinski CC: Jiri Pirko , "Samudrala, Sridhar" , "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Index: AQHU0FlVGn07mv/5S0qtfXL/6zV0naX4F3YAgACpvYCAAl2OgIABFocAgACw24CAAGdAAIABP+yAgABd4gCAAQniAIABHgoAgADJ0ICAAEdZgIAECm0AgAEiPwCAAMbcgIAAc58AgACZ0oCAAKqSgIAAAXSAgAAJR4CAAPajAIAA82GAgAAGytCAABI3gIAAHF7AgAAPYgCAAAD0MIAAPSgAgACeBbCAAE9/AIAACj8AgAAU6MCAADbuAIAEB+WQ Date: Mon, 18 Mar 2019 15:43:20 +0000 Message-ID: References: <20190313095555.0f4f92ca@cakuba.attlocal.net> <20190314073840.GA3034@nanopsycho> <20190314150945.031d1b08@cakuba.netronome.com> <20190314163915.24fd2481@cakuba.netronome.com> <4436da3d-4b99-f792-8e77-695d5958794d@intel.com> <20190315200814.GD2305@nanopsycho> <20190315134454.581f47ca@cakuba.netronome.com> <20190315181620.341bd02c@cakuba.netronome.com> In-Reply-To: <20190315181620.341bd02c@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [208.176.44.194] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: cb54c875-ccf2-427e-1eeb-08d6abb872f5 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020);SRVR:VI1PR0501MB2286; x-ms-traffictypediagnostic: VI1PR0501MB2286: x-microsoft-antispam-prvs: x-forefront-prvs: 098076C36C x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(136003)(396003)(39860400002)(346002)(376002)(366004)(189003)(199004)(52054003)(13464003)(6436002)(93886005)(74316002)(478600001)(52536014)(14454004)(25786009)(66066001)(186003)(7736002)(106356001)(26005)(86362001)(55016002)(5660300002)(105586002)(102836004)(6506007)(53546011)(2906002)(97736004)(9686003)(53936002)(4326008)(8676002)(229853002)(3846002)(11346002)(81166006)(446003)(486006)(561944003)(68736007)(81156014)(316002)(71190400001)(6246003)(7696005)(71200400001)(76176011)(6116002)(54906003)(99286004)(256004)(6916009)(305945005)(8936002)(476003)(33656002);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2286;H:VI1PR0501MB2271.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Broun8GHJureNJRV0j/jVdSvTFasoZKFZ7QiOMXmQr3aSD0kLh9wQPhm/OGeMv+zgK2Irv1Si4hi7oFrqk+6ITWQ19xZyiNeoOqWTYYxZ4TUbUaKHHWbTRU32DAvhPZ6bKYNUV5FYtkxp3BdS6zQ/q+q7WW7edUZaa6k1FO/W7GZm7spysSsbFUUH5gQJw3sLJTwFyR2D+J+Dmlpc3q/JNYnXL0vOamNx/8V8M+t00+cLWh60IeXo+JJq7a34FD0hNS2SMGmW6jjYXtNZ+GjNCp1/C8GixPgkYF+qTIMZYiDrobPwW5TgAz51usVklPvEVDyxWkb0f+1s3AF02tRxdxxzZ0l8JG69WCFtqytXVc4kG23Xm9xsSgp05AJ0MZx1ZvvjEyfFNVvodISDaCxUkhMnRWzXT3TwaSLrKs3W9k= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: cb54c875-ccf2-427e-1eeb-08d6abb872f5 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2019 15:43:21.0022 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2286 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > -----Original Message----- > From: Jakub Kicinski > Sent: Friday, March 15, 2019 8:16 PM > To: Parav Pandit > Cc: Jiri Pirko ; Samudrala, Sridhar > ; davem@davemloft.net; > netdev@vger.kernel.org; oss-drivers@netronome.com > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink P= CI > ports >=20 > On Fri, 15 Mar 2019 22:12:13 +0000, Parav Pandit wrote: > > > On Fri, 15 Mar 2019 21:08:14 +0100, Jiri Pirko wrote: > > > > >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects > > > > >> for each port - host facing ports and switch facing ports. This > > > > >> is in addition to the netdevs that are created today. > > > > > > To be clear I'm not in favour of the dual-object proposal. > > > > > > > >I am not proposing any different. > > > > >I am proposing only two changes. > > > > >1. control hostport params via referring hostport (not via > > > > >indirect > > > > >peer) > > > > > > > > Not really possible. If you passthrough VF into VM, the hostport > > > > goes along with it. > > > > > > > > >2. flavour should not be vf/pf, flavour should be hostport, switch= port. > > > > >Because switch is flat and agnostic of pf/vf/mdev. > > > > > > > > Not sure. It's good to have this kind of visibility. > > > > > > Yes, this subthread honestly makes me go from 60% sure to 95% sure > > > we shouldn't do the dual object thing :( Seems like Parav is > > > already confused by it and suggests host port can exist without > > > switch port :( > > > > > I am almost sure that I am not confused. > > I am clear that hostports should be configured by devlink instance > > which has the capability to program it. >=20 > Right now a devlink port is something that the datapath of an ASIC can > address. All flavours we have presently are basically various MACs - phy= sical > (front panel ports), DSA - for ASIC interconnects on a multi-ASIC board, = CPU - > for connecting to a MAC of a NIC. >=20 Devlink port implementation in commit doesn't say that it is for ASIC datap= ath or limited to ASIC datapath id. It is not right to say that 'whole datapath' object should be represented w= ith just single object 'port'. Datapath involves various stages in ASIC each does different processing. These datapath objects are interconnected, i.e. hostport is connected to sw= itchport. Commit [1] says devlink port is physical port. However we already have 3 fl= avours of port. > Jiri's flavour proposal was strictly extending the same logic to SR-IOV. = Each > object addressable within the datapath gets a port. > The datapath's ID can be used as port_index. >=20 And as I said, it is already restrictive. Port is a port, it can be labeled for vf/pf, but flavour is not really vf/p= f. Also label applies more on the hostport side vs switchport. > I just reimplemented his patches here and added the subports which I thin= k > he wasn't aware of as they are a quirk of old NFP ASICs. >=20 > Having 3 objects for the same datapath ID is a significant departure from= the > existing devlink port semantics. >=20 It is really not same datapath ID. Because if that is the case, we should be programming mac address on the re= p-netdev itself. But we are not doing that because rep-netdev represents only 'eswitch port'= . > > When hostport is in VF, that VF usually won't have privilege to > > program it and won't have visibility to eswitch either. >=20 > If VM has no visibility into the eswitch and no permission to configure > things, what use does the object serve? >=20 To view device properties, health, RO registers, more importantly its port = details. Yonatan is working on grouping these devlink ports and those are control th= rough devlink APIs. Jiri is actively internally reviewing those patches since last 3+ weeks, no= t finished yet. So this visibility is needed anyway. > > Why would you like to start with restrictive model of peer view only? >=20 > "Restrictive model" is one way of putting it. I'd rather say that we are= not > adding objects which: > (a) do not adhere to current semantics; > (b) have no distinct function. >=20 hostport certainly has distinct function than switchport. i.e. to program host side parameters. (eth.mac, rdma.port_guid and more in = future). > We can make the "add MAC address" command not use the word peer: >=20 > devlink port addr_pool add pci/0000:05:00.0/10003 type eth > 00:11:22:33:44:55 devlink port addr_pool del pci/0000:05:00.0/10003 type > eth 00:11:22:33:44:55 >=20 > if the "peer" doesn't sit right. >=20 > > Hostports exist for infiniband HCA without switchport. > > We should be able to manage hostport objects without creating fake > eswitch sw object. >=20 > It sounds like the RDMA subsystem is lacking a model to represent all its > objects, but that's RDMA's problem to solve.. >=20 devlink framework is not limited to Ethernet, it operates on bus/device not= ion. So for Ethernet vendors program mac address. For rdma vendor programs port_guid (which is equivalent of mac address). devlink also publishes rdma device info today. net/core/devlink.c has very well established IB device info exposed via dev= link_nl_port_fill() for more than 3 years now in commit [2]. It is not fair to say create, solve it somewhere else. > In netdev world we have netdevs for ports which a used for bulk of the > configuration, most importantly - forwarding. >=20 > > Jakub, > > Can you please point to some example other than veth-pair where you > > configure host param (such as mac address) through a switch? >=20 > Existing "legacy" SR-IOV NDOs. >=20 That is perfect example of programming hostport parameters, without a eswit= ch.. At high level, I was looking where you open switch GUI/cli or something equ= ivalent that program's host's mac address.. So far we don't have such equivalent good example yet.. > > An existing example will help me to map it to devlink eswitch proposal. > > If we go peer programming route, what are your thoughts on how should > > we program infiniband hostports which doesn't have peer ports? >=20 > Again, you may be trying to fix RDMA's lack of control objects, which may= be > better fixed elsewhere.. devlink port is link agnostic control object. [1] bfcd3a46617209454cfc0947ab093e37fd1e84ef [2] commit id bfcd3a466