From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E3ADC43381 for ; Thu, 14 Mar 2019 22:35:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DE04521874 for ; Thu, 14 Mar 2019 22:35:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="XIUMAg39" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728419AbfCNWfr (ORCPT ); Thu, 14 Mar 2019 18:35:47 -0400 Received: from mail-eopbgr140070.outbound.protection.outlook.com ([40.107.14.70]:23459 "EHLO EUR01-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728392AbfCNWfm (ORCPT ); Thu, 14 Mar 2019 18:35:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yhhFphh37YPnjS6KOY7X8OmChSrvBkaQsG9sMnNgsuE=; b=XIUMAg397MAvi0Y1gMT7oK92GWx18STPgD8HOMAFqP8mQpmxOXKLqWD5iLhHp1AT/9e7S2eD9cM6KnocaE9wKMiy05tqOIRYut2tojIi4JrT3qNqg2tj0Zzey+mcyiwU01AoBDW++btvIS2alPmBIYMcBPP2kZ5BdvcRmySIaiI= Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com (10.169.135.8) by VI1PR0501MB2336.eurprd05.prod.outlook.com (10.169.135.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1709.14; Thu, 14 Mar 2019 22:35:36 +0000 Received: from VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59]) by VI1PR0501MB2271.eurprd05.prod.outlook.com ([fe80::a0b8:7ed8:d657:2f59%6]) with mapi id 15.20.1709.011; Thu, 14 Mar 2019 22:35:36 +0000 From: Parav Pandit To: Jakub Kicinski , Jiri Pirko CC: "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: RE: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Thread-Index: AQHU0FlVGn07mv/5S0qtfXL/6zV0naX4F3YAgACpvYCAAl2OgIABFocAgACw24CAAGdAAIABP+yAgABd4gCAAQniAIABHgoAgADJ0ICAAEdZgIAECm0AgAEiPwCAAMbcgIAAc58AgACZ0oCAAKqSgIAAAXSAgAAJR4CAAPajAIAA82GAgAAGytA= Date: Thu, 14 Mar 2019 22:35:36 +0000 Message-ID: References: <20190308145421.GA2888@nanopsycho.orion> <20190308110943.2ee42bc0@cakuba.hsd1.ca.comcast.net> <20190311085204.GA2194@nanopsycho> <20190311191054.36b801d6@cakuba.hsd1.ca.comcast.net> <20190312140239.GA2455@nanopsycho> <20190312135628.5250135b@cakuba.hsd1.ca.comcast.net> <20190313060701.GB2384@nanopsycho.orion> <20190313091731.76129ece@cakuba.attlocal.net> <20190313162243.GB2270@nanopsycho> <20190313095555.0f4f92ca@cakuba.attlocal.net> <20190314073840.GA3034@nanopsycho> <20190314150945.031d1b08@cakuba.netronome.com> In-Reply-To: <20190314150945.031d1b08@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=parav@mellanox.com; x-originating-ip: [208.176.44.194] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b89a67d6-3050-4de3-f936-08d6a8cd60d9 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:VI1PR0501MB2336; x-ms-traffictypediagnostic: VI1PR0501MB2336: x-microsoft-antispam-prvs: x-forefront-prvs: 09760A0505 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(39860400002)(136003)(366004)(346002)(376002)(396003)(199004)(189003)(13464003)(3846002)(54906003)(110136005)(6246003)(6116002)(316002)(71190400001)(71200400001)(4326008)(99286004)(93886005)(186003)(81166006)(8676002)(305945005)(478600001)(9686003)(33656002)(7736002)(81156014)(2906002)(53936002)(229853002)(52536014)(5660300002)(6436002)(14454004)(68736007)(25786009)(476003)(86362001)(486006)(6506007)(55016002)(74316002)(11346002)(8936002)(256004)(102836004)(14444005)(446003)(106356001)(105586002)(53546011)(76176011)(97736004)(26005)(66066001)(7696005);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2336;H:VI1PR0501MB2271.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: 2PyZXNszqbZhKL7NUeY49GydoihxZOcH5Vgi3NFpBSVd713RowjR00uSQ9ezPQG4tX6p6TuFLBR1xjF3Zxv3VWB+p8SBEsGeCtuSl0UMIQjGNXle9dVr2q7L2tn5e/cBheHPd76Mu70qw5JCewSpTqDhxE0fSmG6uRXoW93Txj56O5VygtlP6XPoPFSuPTPahlWB/lGz5mk1DNq7/pniQ1/DQujRpMJ0qii1UcCjVt+WpX9ZzbAolBiv+8JO4sgA40VkWeZpmqKRYFbgcOApkuAzg1AYTo3ODUly7vrCAirBmiPOvLspw4UqaPzDLxzGi6oRqcTntZrSe1oGNWdW1hHvdRJect/8PVxmkxP+3oBfb3d8bVe0lhOxd5rWCHpyNubeWrQeP0F/eubACjqYqvzmAsNcI3DAJ/9GaFuMDno= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: b89a67d6-3050-4de3-f936-08d6a8cd60d9 X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Mar 2019 22:35:36.4502 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2336 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi Jakub, > -----Original Message----- > From: netdev-owner@vger.kernel.org On > Behalf Of Jakub Kicinski > Sent: Thursday, March 14, 2019 5:10 PM > To: Jiri Pirko > Cc: davem@davemloft.net; netdev@vger.kernel.org; oss- > drivers@netronome.com > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink P= CI > ports >=20 > On Thu, 14 Mar 2019 08:38:40 +0100, Jiri Pirko wrote: > > Wed, Mar 13, 2019 at 05:55:55PM CET, jakub.kicinski@netronome.com > wrote: > > >On Wed, 13 Mar 2019 17:22:43 +0100, Jiri Pirko wrote: > > >> Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com > wrote: > > >> >On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote: > > >> >> Tue, Mar 12, 2019 at 09:56:28PM CET, > jakub.kicinski@netronome.com wrote: > > >> >> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote: > > >> >> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote: > > >> >> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote: > > >> >> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote: > > >> >> >> >> >If the switchport is in the hypervisor then only the hyper= visor > can > > >> >> >> >> >control switching/forwarding, correct? > > >> >> >> >> > > >> >> >> >> Correct. > > >> >> >> >> > > >> >> >> >> >The primary use case for partitioning within a VM (of a VF= ) > would be > > >> >> >> >> >containers (and DPDK)? > > >> >> >> >> > > >> >> >> >> Makes sense. > > >> >> >> >> > > >> >> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably = easy > to grasp. > > >> >> >> >> >I'm trying to get a sense of is how would we control an SR= -IOV > > >> >> >> >> >environment as a whole. > > >> >> >> >> > > >> >> >> >> You mean orchestration? > > >> >> >> > > > >> >> >> >Right, orchestration. > > >> >> >> > > > >> >> >> >To be clear on where I'm going with this - if we want to > > >> >> >> >allow VFs to partition themselves then they have to control w= hat > is effectively > > >> >> >> >a "nested" switch. A per-VF set of rules which would the get > > >> >> >> > > >> >> >> Wait. If you allow to make VF subports (I believe that is > > >> >> >> what you ment by VFs partition themselves), that does not mean > they will have a > > >> >> >> separate nested switch. They would still belong under the same > one. > > >> >> > > > >> >> >But that existing switch is administered by the hypervisor, how > would > > >> >> >the VF owners install forwarding rules in a switch they don't > control? > > >> >> > > >> >> They won't. > > >> > > > >> >Argh. So how is forwarding configured if there are no rules? Are > > >> >you going to assume its switching on MACs? We're supposed to > > >> >offload software constructs. If its a software port it needs to > > >> >be explicitly switched. If it's not explicitly switched - we alrea= dy have > macvlan > > >> >offload. > > >> > > >> Wait a second. You configure the switch. And for that, you have the > > >> switchports (representors). What we are talking about are VF (VF > > >> subport) host legs. Am I missing something? > > > > > >Hm :) So when VM gets a new port, how is it connected? Are we > > >assuming all ports of a VM are plugged into one big L2 switch? > > >The use case for those sub ports is a little murky, sorry about the > > >endless confusion :) > > > > Np. When user John (on baremetal, or whenever the devlink instance > > with switch port is) creates VF of VF subport by: > > $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 or by: > > $ devlink dev port add pci/0000:05:00.0 flavour pci_vf pf 0 vf 0 > > > > Then instances of flavour pci_vf are going to appear in the same > > devlink instance. Those are the switch ports: > > pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 > > flavour pci_vf pf 0 vf 0 > > switch_id 00154d130d2f peer pci/0000:05:10.1/0 > > pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0 > > flavour pci_vf pf 0 vf 0 subport 1 > > switch_id 00154d130d2f peer pci/0000:05:10.1/1 > > > > With that, peers are going to appear too, and those are the actual > > VF/VF > > subport: > > pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host > > peer pci/0000:05:00.0/10002 > > pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host > > peer pci/0000:05:00.0/10003 > > > > Later you can push this VF along with all subports to VM. So in VM, > > you are going to see the VF like this: > > $ devlink dev > > pci/0000:00:08.0 > > $ devlink port > > pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host > > pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host > > > > And back to your question of how are they connected in eswitch. > > That is totally up to the original user John who did the creation. > > He is in charge of the eswitch on baremetal, he would configure the > > forwarding however he likes. >=20 > Ack, so I think you're saying VM has to communicate to the cloud > environment to have this provisioned using some service API, not a kernel > API. That's what I wanted to confirm. >=20 > I don't see any benefit to having the "host ports" under devlink, as such= I > think it's a matter of preference.=20 We need 'host ports' to configure parameters of this=20 host port which is not exposed by the rep-netdev. Such as mac address. > I'll try to describe the two options to > Netronome's FAEs and see which one they find more intuitive. >=20 > Makes sense?