From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97A98C43381 for ; Mon, 18 Mar 2019 16:22:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 50C022087E for ; Mon, 18 Mar 2019 16:22:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="OuikONqp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727656AbfCRQWk (ORCPT ); Mon, 18 Mar 2019 12:22:40 -0400 Received: from mail-eopbgr70089.outbound.protection.outlook.com ([40.107.7.89]:13822 "EHLO EUR04-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726719AbfCRQWk (ORCPT ); Mon, 18 Mar 2019 12:22:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=99/qWVrv8OTuLd5al5/7gNfBZLiLOierXF+TNzopkCw=; b=OuikONqpafruBC8ewK9K1SxsmliXvSML7+zBUbN43OCRphMtqH/01Pigr6FrEgyiHaHnnV0pNWj5piIWHM3p6vzMZuAk1gV9tLfCRrI63FrjxyMqHicOEBDx19GkI0iodmJ6OGGz0BJC+w95BqMapZ8/9gJeLvkhWuzGTs+ZUdU= Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com (10.169.135.8) by VI1PR0501MB2432.eurprd05.prod.outlook.com (10.168.136.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.14; Mon, 18 Mar 2019 16:22:33 +0000 Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59]) by VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59%6]) with mapi id 15.20.1709.015; Mon, 18 Mar 2019 16:22:33 +0000 From: Parav Pandit To: Parav Pandit , Jiri Pirko CC: "Samudrala, Sridhar" , Jakub Kicinski , "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Index: AQHU0FlVGn07mv/5S0qtfXL/6zV0naX4F3YAgACpvYCAAl2OgIABFocAgACw24CAAGdAAIABP+yAgABd4gCAAQniAIABHgoAgADJ0ICAAEdZgIAECm0AgAEiPwCAAMbcgIAAc58AgACZ0oCAAKqSgIAAAXSAgAAJR4CAAPajAIAA82GAgAAGytCAABI3gIAAHF7AgAAPYgCAAAD0MIAAPSgAgACeBbCAAE9/AIAAG6zwgAQYzoCAADlP0IAACGDQ Date: Mon, 18 Mar 2019 16:22:33 +0000 Message-ID: References: <20190314150945.031d1b08@cakuba.netronome.com> <20190314163915.24fd2481@cakuba.netronome.com> <4436da3d-4b99-f792-8e77-695d5958794d@intel.com> <20190315200814.GD2305@nanopsycho> <20190318122105.GH2270@nanopsycho> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [208.176.44.194] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0ea94908-89ec-4b67-390d-08d6abbded5c x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:VI1PR0501MB2432; x-ms-traffictypediagnostic: VI1PR0501MB2432: x-microsoft-antispam-prvs: x-forefront-prvs: 098076C36C x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(136003)(346002)(39860400002)(396003)(376002)(366004)(13464003)(189003)(199004)(316002)(97736004)(110136005)(81156014)(6246003)(71190400001)(6116002)(54906003)(76176011)(99286004)(71200400001)(7696005)(68736007)(81166006)(4326008)(8676002)(229853002)(561944003)(486006)(446003)(53936002)(11346002)(3846002)(476003)(8936002)(33656002)(256004)(305945005)(14444005)(105586002)(66066001)(25786009)(186003)(93156006)(7736002)(52536014)(2940100002)(102836004)(478600001)(6436002)(74316002)(14454004)(86362001)(53546011)(2906002)(6506007)(106356001)(26005)(93886005)(9686003)(5660300002)(55016002);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2432;H:VI1PR0501MB2271.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: QKsOa9qDvP3gRhN5jENXMOZh1e2Wiouq8GqAsTJCwkURNgK2XeYZEsZPdSWbJcpAOZL5kfkc4S6hsOU4NDfhBadhD07E3jUYHjkuZijCUD7mvF+z+tDguTmHpxj0iSf00hSDdKp9OKKlaJOEJs5Jnr5QcV3IVduI18lLEQn8J3eArQSBI3ZBXCNY1mqOlzqkiQiLFTi+KW5tyOarnk2gYUs/UC0FMlmUWlgxOrU7qfBybLfvI1w8f+iYPkAyamyttog8/74AScOh44uZ70OwBWxfKhi/TRCVXW+mGNuEvqckrQy9UJe0xt8iFmJbprXHXttunAsUchCmBT9gDbKAr9Jr3w31b/JZfVHQFt1hXm3zc5crhk5M0caW6eqzgIIXQkH28RwEGNZIfbqo4v/DZIcVBj/zUpyLFcPZfNDArdI= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ea94908-89ec-4b67-390d-08d6abbded5c X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Mar 2019 16:22:33.7545 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2432 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org > -----Original Message----- > From: netdev-owner@vger.kernel.org On > Behalf Of Parav Pandit > Sent: Monday, March 18, 2019 10:57 AM > To: Jiri Pirko > Cc: Samudrala, Sridhar ; Jakub Kicinski > ; davem@davemloft.net; > netdev@vger.kernel.org; oss-drivers@netronome.com > Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink P= CI > ports >=20 >=20 >=20 > > -----Original Message----- > > From: Jiri Pirko > > Sent: Monday, March 18, 2019 7:21 AM > > To: Parav Pandit > > Cc: Samudrala, Sridhar ; Jakub Kicinski > > ; davem@davemloft.net; > > netdev@vger.kernel.org; oss-drivers@netronome.com > > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > > devlink PCI ports > > > > Fri, Mar 15, 2019 at 10:59:33PM CET, parav@mellanox.com wrote: > > > > > > > > >> -----Original Message----- > > >> From: Jiri Pirko > > >> Sent: Friday, March 15, 2019 3:08 PM > > >> To: Parav Pandit > > >> Cc: Samudrala, Sridhar ; Jakub > > >> Kicinski ; davem@davemloft.net; > > >> netdev@vger.kernel.org; oss-drivers@netronome.com > > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > > >> devlink PCI ports > > >> > > >> Fri, Mar 15, 2019 at 04:32:24PM CET, parav@mellanox.com wrote: > > >> > > > >> > > > >> >> -----Original Message----- > > >> >> From: Samudrala, Sridhar > > >> >> Sent: Friday, March 15, 2019 12:58 AM > > >> >> To: Parav Pandit ; Jakub Kicinski > > >> >> > > >> >> Cc: Jiri Pirko ; davem@davemloft.net; > > >> >> netdev@vger.kernel.org; oss-drivers@netronome.com > > >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > > >> >> devlink PCI ports > > >> >> > > >> >> > > >> >> On 3/14/2019 7:40 PM, Parav Pandit wrote: > > >> >> > > > >> >> > > > >> >> >> -----Original Message----- > > >> >> >> From: Samudrala, Sridhar > > >> >> >> Sent: Thursday, March 14, 2019 9:16 PM > > >> >> >> To: Parav Pandit ; Jakub Kicinski > > >> >> >> > > >> >> >> Cc: Jiri Pirko ; davem@davemloft.net; > > >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com > > >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports > > >> >> >> on devlink PCI ports > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> On 3/14/2019 6:28 PM, Parav Pandit wrote: > > >> >> >>> > > >> >> >>> > > >> >> >>>> -----Original Message----- > > >> >> >>>> From: Jakub Kicinski > > >> >> >>>> Sent: Thursday, March 14, 2019 6:39 PM > > >> >> >>>> To: Parav Pandit > > >> >> >>>> Cc: Jiri Pirko ; davem@davemloft.net; > > >> >> >>>> netdev@vger.kernel.org; oss-drivers@netronome.com > > >> >> >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow > > >> >> >>>> subports on devlink PCI ports > > >> >> >>>> > > >> >> >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote: > > >> >> >>>>>>> Then instances of flavour pci_vf are going to appear in > > >> >> >>>>>>> the same devlink instance. Those are the switch ports: > > >> >> >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 > > >> >> >>>>>>> flavour pci_vf pf 0 vf 0 > > >> >> >>>>>>> switch_id 00154d130d2f peer > > >> >> >>>>>>> pci/0000:05:10.1/0 > > >> >> >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0 > > >> >> >>>>>>> flavour pci_vf pf 0 vf 0 subpor= t 1 > > >> >> >>>>>>> switch_id 00154d130d2f peer > > >> >> >>>>>>> pci/0000:05:10.1/1 > > >> >> >>>>>>> > > >> >> >>>>>>> With that, peers are going to appear too, and those are > > >> >> >>>>>>> the actual VF/VF > > >> >> >>>>>>> subport: > > >> >> >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_ho= st > > >> >> >>>>>>> peer pci/0000:05:00.0/10002 > > >> >> >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_ho= st > > >> >> >>>>>>> peer pci/0000:05:00.0/10003 > > >> >> >>>>>>> > > >> >> >>>>>>> Later you can push this VF along with all subports to VM. > > >> >> >>>>>>> So in VM, you are going to see the VF like this: > > >> >> >>>>>>> $ devlink dev > > >> >> >>>>>>> pci/0000:00:08.0 > > >> >> >>>>>>> $ devlink port > > >> >> >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour > > >> >> >>>>>>> pci_vf_host > > >> >> >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour > > >> >> >>>>>>> pci_vf_host > > >> >> >>>>>>> > > >> >> >>>>>>> And back to your question of how are they connected in > > eswitch. > > >> >> >>>>>>> That is totally up to the original user John who did the > > creation. > > >> >> >>>>>>> He is in charge of the eswitch on baremetal, he would > > >> >> >>>>>>> configure the forwarding however he likes. > > >> >> >>>>>> > > >> >> >>>>>> Ack, so I think you're saying VM has to communicate to > > >> >> >>>>>> the cloud environment to have this provisioned using some > > >> >> >>>>>> service API, not a kernel API. That's what I wanted to > confirm. > > >> >> >>>>>> > > >> >> >>>>>> I don't see any benefit to having the "host ports" under > > >> >> >>>>>> devlink, as such I think it's a matter of preference. > > >> >> >>>>> > > >> >> >>>>> We need 'host ports' to configure parameters of this host > > >> >> >>>>> port which is not exposed by the rep-netdev. > > >> >> >>>>> Such as mac address. > > >> >> >>>> > > >> >> >>>> Please look at the quote of what Jiri wrote above - the > > >> >> >>>> host port gets passed to the VM, you can't use it as a > > >> >> >>>> handle to set the > > >> MAC. > > >> >> >>>> > > >> >> >>>> The way to set the MAC remains: > > >> >> >>>> > > >> >> >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr > > >> >> >>>> 00:11:22:33:44:55 > > >> >> >>>> > > >> >> >>> Even though it can be done, I think this is wrong model to > > >> >> >>> program > > >> >> >> hostport mac address using eswitch port. > > >> >> >>> All devlink objects are control objects, so what is passed > > >> >> >>> to VM is what is > > >> >> >> represented by devlink. > > >> >> >>> VF in the VM will anyway create its devlink object. > > >> >> >>> What is wrong in programming hostport? > > >> >> >>> It gives a very clear view to users of topology and objects. > > >> >> >> > > >> >> >> The VF or any subport MAC address should be configured by the > > >> >> >> orchestration layer that is running on the hypervisor and > > >> >> >> when a VF is assigned to a VF, the host port is not visible > > >> >> >> to the > > hypervisor. > > >> >> > What prevents creation of hostport due to which is not visible= ? > > >> >> > Hostport is control port to program host side of parameters. > > >> >> > It should be created when user wants to program the parameters. > > >> >> > > > >> >> > Model is really straight forward. > > >> >> > Program host port params using hostport object. > > >> >> > Program switchport params using rep-netdev. > > >> >> > > >> >> IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for > > >> >> each port - host facing ports and switch facing ports. This is > > >> >> in addition to the netdevs that are created today. > > >> >> > > >> >I am not proposing any different. > > >> >I am proposing only two changes. > > >> >1. control hostport params via referring hostport (not via > > >> >indirect > > >> >peer) > > >> > > >> Not really possible. If you passthrough VF into VM, the hostport > > >> goes along with it. > > >> > > >No. > > >I am sorry in showing the enumeration which is the source of confusion= . > > > > > >Below is the right enumeration. > > > > > >When VF is enumerated initially in the host, where eswitch devlink > > >instance > > is located. > > >Below enumeration is seen. > > > > > >First two entries shows the link between hostport and switchport. > > >$ devlink port show > > >pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id > > >00154d130d2f peer pci/0000:05:00.0/1 > > > > > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f > > >peer pci/0000:05:00.0/10002 > > > > Hostport should not have switch_id. > > > > > > > >pci/0000:05:10.1/0 eth netdev flavour hostport This entry won't be > > >seen if VF auto probing is disabled. Because than VF is not enumerated= . > > > > > >As a user, I will be programming the mac address of hostport for a VF. > > >pci/0000:05:00.0/1 eth netdev flavour hostport switch_id 00154d130d2f > > >peer pci/0000:05:00.0/10002 > > > > Hmm, so you are going to have 2 hostports for VF: > > 1) pci/0000:05:10.1/0 > > real one, that is going to go to VM - with a separate pci address > > and devlink instance. >=20 > Yep. This is the one where Yonatan's port grouping APIs work on. >=20 > > 2) pci/0000:05:00.0/1 > > dummy one, which is not really a hostport, as there is no netdev > > created for it. It only models the other side of cable, which is awa= y > > in VM. > > > Right. This is the control object which typically hypervisor programs. >=20 > > > > > > > > >> > > >> >2. flavour should not be vf/pf, flavour should be hostport, switchp= ort. > > >> >Because switch is flat and agnostic of pf/vf/mdev. > > >> > > >> Not sure. It's good to have this kind of visibility. > > >> > > >port can have label/attribute indicating that this belong to VF-1 or > > >mdev as > > long as you are agreeing to have mdev attribute on host port. > > >(and not ask for abstracting it, because mdev is well defined kernel > object). > > > > Why mdev cannot be another flavour? > > >=20 > hostport is of type pf/vf/mdev connected to some switchport. >=20 > So proposal is to have, > port flavour =3D hostport/switchport > port type/label =3D pf/vf/mdev >=20 Instead of having two attributes per port, how about having, port flavour=3D physical/cpu/dsa/pf/vf/mdev/switchport. physical and pf has some overlapping definitions.