From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31150C43381 for ; Wed, 20 Mar 2019 23:40:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C64CB218C3 for ; Wed, 20 Mar 2019 23:40:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="la0M2iRz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727573AbfCTXkN (ORCPT ); Wed, 20 Mar 2019 19:40:13 -0400 Received: from mail-eopbgr80047.outbound.protection.outlook.com ([40.107.8.47]:19732 "EHLO EUR04-VI1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726115AbfCTXkM (ORCPT ); Wed, 20 Mar 2019 19:40:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qyXnZCCDJH4mzJhGcBNsCie71bpC3xSBBZT9dHAhpko=; b=la0M2iRzi8vDiGihqTKQ6UdOTOuao6ct9PXIE3lpRJts36j1CxdEJzcKKsnT7SowyMjGc85Bg+nDYH48RPGWWGUAslFQa2m1nzRX4FUazA8XgaLYSw+v/V7QSiW3Z4sHAv2SriDTwTHEqhDXCy9o4hKMTmERrnMkO02yZvj76UU= Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com (10.169.135.8) by VI1PR0501MB2303.eurprd05.prod.outlook.com (10.169.135.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.14; Wed, 20 Mar 2019 23:39:26 +0000 Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59]) by VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59%6]) with mapi id 15.20.1709.015; Wed, 20 Mar 2019 23:39:26 +0000 From: Parav Pandit To: Jakub Kicinski CC: Jiri Pirko , "Samudrala, Sridhar" , "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Index: AQHU0FlVGn07mv/5S0qtfXL/6zV0naX4F3YAgACpvYCAAl2OgIABFocAgACw24CAAGdAAIABP+yAgABd4gCAAQniAIABHgoAgADJ0ICAAEdZgIAECm0AgAEiPwCAAMbcgIAAc58AgACZ0oCAAKqSgIAAAXSAgAAJR4CAAPajAIAA82GAgAAGytCAABI3gIAAHF7AgAAPYgCAAAD0MIAAPSgAgACeBbCAAE9/AIAAG6zwgAQYzoCAADlP0IAACGDQgAA3/QCAAADJ4IAABaWAgAAGk/CAABKjgIAACNIggALgFpCAACkTgIAAA8Lg Date: Wed, 20 Mar 2019 23:39:26 +0000 Message-ID: References: <20190314150945.031d1b08@cakuba.netronome.com> <20190315200814.GD2305@nanopsycho> <20190318122105.GH2270@nanopsycho> <20190318123634.6e90c043@cakuba.netronome.com> <20190318125935.580c8fbe@cakuba.netronome.com> <20190318142949.36f18af4@cakuba.netronome.com> <20190320132257.372369d7@cakuba.netronome.com> In-Reply-To: <20190320132257.372369d7@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [208.176.44.194] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1283c16f-2046-40c7-532f-08d6ad8d4a36 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:VI1PR0501MB2303; x-ms-traffictypediagnostic: VI1PR0501MB2303: x-ms-exchange-purlcount: 1 x-microsoft-antispam-prvs: x-forefront-prvs: 098291215C x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(376002)(39860400002)(396003)(346002)(136003)(366004)(13464003)(199004)(189003)(25786009)(478600001)(97736004)(81156014)(81166006)(3846002)(33656002)(8676002)(2906002)(6116002)(486006)(476003)(4326008)(966005)(446003)(14454004)(11346002)(186003)(102836004)(93886005)(26005)(6916009)(8936002)(68736007)(53936002)(7696005)(76176011)(6246003)(6306002)(229853002)(99286004)(9686003)(6436002)(55016002)(105586002)(53546011)(6506007)(54906003)(106356001)(316002)(52536014)(305945005)(74316002)(5024004)(256004)(86362001)(5660300002)(7736002)(66066001)(71190400001)(71200400001);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2303;H:VI1PR0501MB2271.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: b2PW4PJpNXueR3hjsYhySQhLSQ1rKfIrY6yx73GXt21xf+4ca7R1dL+La1pYMsrNCERB8UyKaEBvKFBHzpgipYJrBz663U5RUSBrmZOT+eJC7JS7lbM27JCb5gWldHN2YVCIrnhfgtuRHUzrUuRmVXZgjNcIi+xNlfWUDDYUMackvkzuZyov6hsmQ+Td5VSe/NvXKx3pR6XqCqNH0XRcIeV8dT7roOR76uzWMtowwpR4BHOy+AW8UR/ijKWZqhLreWlojRFZm4vNwoqC2Oqlyd3dfLhT+uPaskcdQ0kh5yE9a5IxXIUGbk2/SzoXM/28fdBnTPi9CmLij7BuxBF8tyfOxLEXwaVqbF1MYD3N3Bx+ZW70mV12D/56etsAZ+5gtyYjKRpoTwzf2EE3dF4Isu6KJQl6z4d2t0VR0CabMSs= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1283c16f-2046-40c7-532f-08d6ad8d4a36 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Mar 2019 23:39:26.4100 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2303 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > -----Original Message----- > From: Jakub Kicinski > Sent: Wednesday, March 20, 2019 3:23 PM > To: Parav Pandit > Cc: Jiri Pirko ; Samudrala, Sridhar > ; davem@davemloft.net; > netdev@vger.kernel.org; oss-drivers@netronome.com > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink P= CI > ports >=20 > On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote: > > Hi Jiri, Jakub, Samudrala Sridhar, > > > > > > > And physical port in include/uapi/linux/devlink.h also > > > > > > > describe that. > > > > > > > > > > > > By "that" you must mean that the physical is a user facing port= . > > > > > > > > > > Can you please describe the difference between 'PF port' and > > > > > 'physical port of include/uapi/linux/devlink.h'? I must have > > > > > missed this crisp definition in discussion between you and Jiri. > > > > > I am in meantime checking the thread. > > > > > > > > Perhaps start with the cover letter which includes an ASCII drawing= ? > > > > > > > > Using Mellanox nomenclature - PF port is a "representor" for the > > > > PF which may be on another Host (SmartNIC or multihost). It's > > > > pretty much the same thing as a VF port/"representor". > > > > > > > Yes. We are aligned here. :-) > > > I see your point, where in multi-host scenario, a physical port may > > > be 1, but PF ports are 4, because of 4 PFs for 4 hosts. > > > (just an example of 4 hosts with their own mac address sharing 1 > > > physical port). > > > > > > When there is no multihost and one to one mapping between a PF and > > > physical links, there is some overlap between PF port and physical > > > port attributes. > > > I believe, such overlap is fine as long as we have unique indices for= the > ports. > > > > > > So I am ok to have flavours as physical/cpu/dsa/pf/vf/mdev/switchport= . > > > (last 4 as new port flavours). > > > > > > > Physical port is the hole on the panel of the adapter where cable g= oes. > > > > So my take away from above discussion are: > > 1. Following new port flavours should be added > pci_pf/pci_vf/mdev/switchport. > > a. Switchport indicates port on the eswitch. Normally this port has rep= - > netdev attached to it. >=20 > I don't understand the "switchport". Surely physical ports are also atta= ched > to the eswitch? =20 Yes, physical port is attached to eswitch too. switchport is the one which is not directly visible to the user on the phys= ical panel. Jiri captured that in the diagram visible here. [1] https://www.mail-archive.com/netdev@vger.kernel.org/msg289477.html > And one of the main purpose of adding the pci_pf/pci_vf > flavours was to generate phys_port_name for the port netdevs. >=20 Yes, this is surely needed due to current direction of placement of pf/vf i= nfo in rep-netdev phys_port_name. So this information is available from the past example in thread. I copied here and extended some parts of it. Below example is for one PCI function that has two physical ports, one VF, = and one mdev. It enumerates 6 devlink ports. VF 1 is physical port 1 of PF 0. mdev uuidX's port 0 is connected to PF 0, port 0. $ devlink port show pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id 001= 54d130d2f pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id 001= 54d130d2f pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport switc= h_id 00154d130d2f peer pci/0000:05:00.0/1 pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour switchpo= rt switch_id 00154d130d2f peer mdev/uuidX/0 pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1 mdev/uuidX/0 eth netdev flavour mdev_ctrl Here eswitch side creates phys_port_name from the peer port's information. Something like, struct devlink_peer_info { struct devlink *dev; struct devlink_port *port; }; struct devlink_port { [...] existing fields; struct devlink_peer_info peer_info; }; Depending on peer port flavour, it can construct appropriate phys_port_name= ndo_ops return value. I changed port flavour name to vf_ctrl and mdev_ctrl to make it more expli= cit for clarity. Let's discuss what is missing in this model.. or part that's incorrect... > Please don't use the term representor if possible. Representor for most > developers describes the way the netdev is implemented in the driver, so = for > Mellanox and Netronome different ports will be representors and non- > representors. That's why I prefer port netdev (attached to eswitch, has > switch_id) and host netdev (PF/VF netdev, vNIC, VSI, etc). >=20 Frankly I don't see much difference in both the proposals. No matter which way we go, peer_info is needed anyway for vf/pf/mdev. Only exception is introduction of explicit host_port_side object (instead o= f indirect peer way). This gives clear visibility to users and addresses rdma non_sriov case too. > > b. host side port flavours are pci_pf/pci_vf/mdev which may be > > connected to switchport >=20 > See above, pci_pf/pci_vf are needed for phys_port_name generation. >=20 > > 2. host side port flavours are not limited to Ethernet, as it is for de= vlink's > port instance. > > > > 3. Each port is continue to be accessed using unique port index. > > > > 4. host side ports and switchport are control objects. > > a. switch side ports reside where current eswitch object of devlink > > instance reside b. for a given VF/PF/mdev such host side ports may be > > in hypervisor or VM or both depending on the privilege > > > > 5. eth.mac_address, rdma.port_guid can be programmed at host port > > flavours by extending as $ devlink port param set... > > (similar to devlink dev param set) >=20 > You can keep restating that's your position, but I have *not* conceded to > that. >=20 I understand. Lets discuss any short comings of it (if any) due to which sw= itch side flavours to be defined via indirect peer programming. > > 6. more host port params can be added in future when user need arise > > > > 7. rep-netdev continue to be eswitch (switchport) representor at the sw= itch > side. > > a. Hence rep-netdev cannot be used for programming host port's > parameters. > > > > 8. eswitch devlink instance knows when VF/PF/mdev's switchport are > created/removed. > > Hence, those will be created/deleted by eswitch. > > Similarly for host port flavours too. > > > > Does it look fine? Did I miss something? > > We would like to progress on incremental patches for item-4 and any > > prep work needed to reach to item-4.