From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5C36C43381 for ; Sat, 23 Mar 2019 00:40:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7445F2192B for ; Sat, 23 Mar 2019 00:40:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="ywkhFtns" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727512AbfCWAk3 (ORCPT ); Fri, 22 Mar 2019 20:40:29 -0400 Received: from mail-eopbgr60085.outbound.protection.outlook.com ([40.107.6.85]:46174 "EHLO EUR04-DB3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726589AbfCWAk2 (ORCPT ); Fri, 22 Mar 2019 20:40:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LbNkxMDHyF04050ilJut7neD/VZ1WSGa1mIvGBP4TGI=; b=ywkhFtnsCKGJS8Z4ev+IUYxG8Z05u1RI1pGvPrZ0BqzERwdHNN1OTyyVMlHJLFXGNFLuM1NRR7aDbZnp/Jvo7C6LENSMCEMLmxjIer677wQ38D/Dpcf1UNtdMSn7oSw9w8lm3Mfy09p26E8sxfWtPwNbPcQnDOr/fcuZQkdKbsY= Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com (10.169.135.8) by VI1PR0501MB2575.eurprd05.prod.outlook.com (10.168.137.11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.14; Sat, 23 Mar 2019 00:40:19 +0000 Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59]) by VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59%6]) with mapi id 15.20.1730.017; Sat, 23 Mar 2019 00:40:19 +0000 From: Parav Pandit To: Jiri Pirko CC: Jakub Kicinski , "Samudrala, Sridhar" , "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Index: AQHU0FlVGn07mv/5S0qtfXL/6zV0naX4F3YAgACpvYCAAl2OgIABFocAgACw24CAAGdAAIABP+yAgABd4gCAAQniAIABHgoAgADJ0ICAAEdZgIAECm0AgAEiPwCAAMbcgIAAc58AgACZ0oCAAKqSgIAAAXSAgAAJR4CAAPajAIAA82GAgAAGytCAABI3gIAAHF7AgAAPYgCAAAD0MIAAPSgAgACeBbCAAE9/AIAAG6zwgAQYzoCAADlP0IAACGDQgAA3/QCAAADJ4IAABaWAgAAGk/CAABKjgIAACNIggALgFpCAACkTgIAA1dqAgABiQ4CAABVTAIAACRgwgAAJvwCAAANVYIABTm2AgAC50VA= Date: Sat, 23 Mar 2019 00:40:19 +0000 Message-ID: References: <20190318142949.36f18af4@cakuba.netronome.com> <20190320132257.372369d7@cakuba.netronome.com> <20190321090821.GB2087@nanopsycho> <20190321161622.GS2087@nanopsycho> <20190321172348.GX2087@nanopsycho> <20190322133241.GA2211@nanopsycho> In-Reply-To: <20190322133241.GA2211@nanopsycho> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [2605:6000:ec80:6500:6130:5c73:d296:d351] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 20b4f24f-26c2-45f6-af6a-08d6af28204c x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:VI1PR0501MB2575; x-ms-traffictypediagnostic: VI1PR0501MB2575: x-microsoft-antispam-prvs: x-forefront-prvs: 0985DA2459 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(376002)(136003)(396003)(39860400002)(366004)(346002)(189003)(199004)(13464003)(52536014)(5660300002)(316002)(229853002)(93886005)(478600001)(6246003)(53936002)(71190400001)(86362001)(71200400001)(25786009)(54906003)(55016002)(9686003)(97736004)(6436002)(33656002)(4326008)(6916009)(8676002)(81166006)(81156014)(476003)(7736002)(106356001)(305945005)(8936002)(14454004)(6116002)(105586002)(68736007)(46003)(7696005)(76176011)(5024004)(6506007)(53546011)(186003)(99286004)(102836004)(2906002)(486006)(446003)(11346002)(74316002)(256004);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2575;H:VI1PR0501MB2271.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Zz4tiRjh0s4I4WhDKA/ctyt4j1TR6ByShGQJVk0eY2wkB2pf1EtWbRU1xRFCkxmk/aqqJvLkbQB711gv3W4WVXRVepTGxjvMVM7W71OPfhebJdmKkaoL76WlxcfRf+t3ckhLAKeqPwyI0O0mRqmPz5jysgwcX5VzI5kVXGjOU1VNhIoVjLEoAmKV4KBUFzpE7+ZvKTXA5qUyKxsz4rBaVW8glnjpwvK5bi0NaT0GNUYaW8zbIQZpKkWuQ/37p0xgZbNQBCDA8P4GCIxboHlvlD19Hq48hPDsHyzXv7coPWGG//gcJT1W9KhAOmpPdqXIG/PoMWUvQ8Xe9SGoauuKBRgk1YtCbchJV+B3CIfgGqCKMUGTkiKrIAzUSVkoWGyy3JUarztZHvWmUfbX7U8I21h9jWWMPiHATlTQaSlwMxI= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 20b4f24f-26c2-45f6-af6a-08d6af28204c X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Mar 2019 00:40:19.3291 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2575 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > -----Original Message----- > From: Jiri Pirko > Sent: Friday, March 22, 2019 8:33 AM > To: Parav Pandit > Cc: Jakub Kicinski ; Samudrala, Sridhar > ; davem@davemloft.net; > netdev@vger.kernel.org; oss-drivers@netronome.com > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink P= CI > ports >=20 > Thu, Mar 21, 2019 at 06:42:55PM CET, parav@mellanox.com wrote: > > > > > >> -----Original Message----- > >> From: Jiri Pirko > >> Sent: Thursday, March 21, 2019 12:24 PM > >> To: Parav Pandit > >> Cc: Jakub Kicinski ; Samudrala, Sridhar > >> ; davem@davemloft.net; > >> netdev@vger.kernel.org; oss-drivers@netronome.com > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > >> devlink PCI ports > >> > >> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote: > >> > > >> > > >> >> -----Original Message----- > >> >> From: Jiri Pirko > >> >> Sent: Thursday, March 21, 2019 11:16 AM > >> >> To: Parav Pandit > >> >> Cc: Jakub Kicinski ; Samudrala, > >> >> Sridhar ; davem@davemloft.net; > >> >> netdev@vger.kernel.org; oss-drivers@netronome.com > >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > >> >> devlink PCI ports > >> >> > >> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote: > >> >> >Hi Jiri, > >> >> > > >> >> >> -----Original Message----- > >> >> >> From: Jiri Pirko > >> >> >> Sent: Thursday, March 21, 2019 4:08 AM > >> >> >> To: Jakub Kicinski > >> >> >> Cc: Parav Pandit ; Samudrala, Sridhar > >> >> >> ; davem@davemloft.net; > >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com > >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > >> >> >> devlink PCI ports > >> >> >> > >> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET, > >> >> >> jakub.kicinski@netronome.com > >> >> >> wrote: > >> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote: > >> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar, > >> >> >> >> > > > > > And physical port in include/uapi/linux/devlink.h > >> >> >> >> > > > > > also describe that. > >> >> >> >> > > > > > >> >> >> >> > > > > By "that" you must mean that the physical is a user > >> >> >> >> > > > > facing > >> port. > >> >> >> >> > > > > >> >> >> >> > > > Can you please describe the difference between 'PF port= ' > >> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'? I > >> >> >> >> > > > must have missed this crisp definition in discussion > >> >> >> >> > > > between you and Jiri. I am in meantime checking the > thread. > >> >> >> >> > > > >> >> >> >> > > Perhaps start with the cover letter which includes an > >> >> >> >> > > ASCII > >> >> drawing? > >> >> >> >> > > > >> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor" > >> >> >> >> > > for the PF which may be on another Host (SmartNIC or > >> multihost). > >> >> >> >> > > It's pretty much the same thing as a VF port/"representor= ". > >> >> >> >> > > > >> >> >> >> > Yes. We are aligned here. :-) I see your point, where in > >> >> >> >> > multi-host scenario, a physical port may be 1, but PF > >> >> >> >> > ports are 4, because of 4 PFs for 4 hosts. > >> >> >> >> > (just an example of 4 hosts with their own mac address > >> >> >> >> > sharing 1 physical port). > >> >> >> >> > > >> >> >> >> > When there is no multihost and one to one mapping between > >> >> >> >> > a PF and physical links, there is some overlap between PF > >> >> >> >> > port and physical port attributes. > >> >> >> >> > I believe, such overlap is fine as long as we have unique > >> >> >> >> > indices for the > >> >> >> ports. > >> >> >> >> > > >> >> >> >> > So I am ok to have flavours as > >> >> physical/cpu/dsa/pf/vf/mdev/switchport. > >> >> >> >> > (last 4 as new port flavours). > >> >> >> >> > > >> >> >> >> > > Physical port is the hole on the panel of the adapter > >> >> >> >> > > where cable > >> >> >> goes. > >> >> >> >> > >> >> >> >> So my take away from above discussion are: > >> >> >> >> 1. Following new port flavours should be added > >> >> >> pci_pf/pci_vf/mdev/switchport. > >> >> >> >> a. Switchport indicates port on the eswitch. Normally this > >> >> >> >> port has rep- > >> >> >> netdev attached to it. > >> >> >> > > >> >> >> >I don't understand the "switchport". Surely physical ports > >> >> >> >are also attached to the eswitch? And one of the main purpose > >> >> >> >of adding the pci_pf/pci_vf flavours was to generate > >> >> >> >phys_port_name for the port netdevs. > >> >> >> > > >> >> >> >Please don't use the term representor if possible. > >> >> >> >Representor for most developers describes the way the netdev > >> >> >> >is implemented in the driver, so for Mellanox and Netronome > >> >> >> >different ports will be representors and non-representors. > >> >> >> >That's why I prefer port netdev (attached to eswitch, has > >> >> >> >switch_id) and host netdev (PF/VF netdev, vNIC, VSI, etc). > >> >> >> > > >> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may > >> >> >> >> be connected to switchport > >> >> >> > > >> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name > generation. > >> >> >> > >> >> >> Yep, that makes sense. > >> >> >> > >> >> >> > >> >> >> > > >> >> >> >> 2. host side port flavours are not limited to Ethernet, as > >> >> >> >> it is for devlink's > >> >> >> port instance. > >> >> >> >> > >> >> >> >> 3. Each port is continue to be accessed using unique port ind= ex. > >> >> >> >> > >> >> >> >> 4. host side ports and switchport are control objects. > >> >> >> >> a. switch side ports reside where current eswitch object of > >> >> >> >> devlink instance reside b. for a given VF/PF/mdev such host > >> >> >> >> side ports may be in hypervisor or VM or both depending on > >> >> >> >> the privilege > >> >> >> >> > >> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host > >> >> >> >> port flavours by extending as $ devlink port param set... > >> >> >> >> (similar to devlink dev param set) > >> >> >> > > >> >> >> >You can keep restating that's your position, but I have *not* > >> >> >> >conceded to that. > >> >> >> > >> >> >> I'm also not convinced that host dummy ports are good idea to > >> >> >> hold > >> >> these. > >> >> >> > >> >> >> > >> >> >I didn't understand what do you mean my dummy port. > >> >> > >> >> It's a port for a VF host port which is not actually in the host > >> >> but in the > >> vm. > >> >> Very confusing. > >> >> > >> >It is the vf_ctrl flavour. I don't see it any different than rep-netd= ev. > >> >rep-netdev is not that confusing to us that represent eswitch vport. > >> >Why vf_ctrl flavour port that represents otherside of the pipe as > >> >you have > >> shown in example? > >> >Why it that confusing? > >> > >> Because sometimes it is there only once (PF), sometimes twice (VF) - > >> and one of these is kind-of zombie. > >> > >I gave the example in email that contains description yesterday. > >You didn't respond to it. > >So repeating here. > >Can you please point what looks like zombie below? > > > >$ devlink port show > >pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id > >00154d130d2f > >pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id > >00154d130d2f > >pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport > >switch_id 00154d130d2f peer pci/0000:05:00.0/1 > >pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour > >switchport switch_id 00154d130d2f peer mdev/uuidX/0 > > > >pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1 >=20 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one. > You are missing an actual VF instance. > VF instance is in VM. It is not visible here in Hypervisor. But if you pref= er to see it, It looks like below. pci/0000:05:01.0/0 eth netdev eth0 flavour vf =20 > >mdev/uuidX/0 eth netdev flavour mdev_ctrl > Why "ctrl"? > I suffixed it with ctrl to indicate you that it is used for control functio= nality. Again, I described in previous email to Jakub' response in lot detail. =20 > > > >> > > >> > > >> >> >Can you explain what is wrong in programming host port params > >> >> >using > >> >> host_port object? > >> >> >Few questions are unanswered in my past 2 or 3 emails. > >> >> >Can you please go through them? > >> >> >Can you point to some example switch API where you program host > >> >> >params > >> >> at switch? > >> >> > > >> >> >> > > >> >> >> >> 6. more host port params can be added in future when user > >> >> >> >> need arise > >> >> >> >> > >> >> >> >> 7. rep-netdev continue to be eswitch (switchport) > >> >> >> >> representor at the > >> >> >> switch side. > >> >> >> >> a. Hence rep-netdev cannot be used for programming host > >> >> >> >> port's > >> >> >> parameters. > >> >> >> >> > >> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's > >> >> >> >> switchport are > >> >> >> created/removed. > >> >> >> >> Hence, those will be created/deleted by eswitch. > >> >> >> >> Similarly for host port flavours too. > >> >> >> >> > >> >> >> >> Does it look fine? Did I miss something? > >> >> >> >> We would like to progress on incremental patches for item-4 > >> >> >> >> and any prep work needed to reach to item-4.