From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DA27C43381 for ; Fri, 22 Mar 2019 13:43:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 17096218E2 for ; Fri, 22 Mar 2019 13:43:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=resnulli-us.20150623.gappssmtp.com header.i=@resnulli-us.20150623.gappssmtp.com header.b="FaD5n+T4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728136AbfCVNnZ (ORCPT ); Fri, 22 Mar 2019 09:43:25 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:35344 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728010AbfCVNnY (ORCPT ); Fri, 22 Mar 2019 09:43:24 -0400 Received: by mail-wm1-f66.google.com with SMTP id y197so2247666wmd.0 for ; Fri, 22 Mar 2019 06:43:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=+/Kx9l88yheknJE9BPOXNVnAB2BLdH40gPRKubHb2iA=; b=FaD5n+T4vVlSBEeneBZECZsXKC/hmw1GIt1tQOa/E6AyVzBLPYrng/lyHYJI6fIqIg CJC/8bmV43IXRw+JYfSR3oCNPX7p6k7ye7zg7mtA8hk7U8WxAixJGNdZNX3Nczsnep3K 79PRIXDOiVDEaOLvTaw1npnrxKIPPZRuaLkAaeKoQNp15Xauqsaq9cHd7sxP8uSAcFnV CeTQV0tHwXiwCELlcecIUkPaFx3MZRJI0wrpgMdzNBLhIwefgucSj3xdp15dMqQgsvvp oVLhAC5MCcULBh/R5c71anL2yle/XI9/7nqy7Yr/aCrn0VB5zNkmAsdcl7g4VubsijvN 9jVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=+/Kx9l88yheknJE9BPOXNVnAB2BLdH40gPRKubHb2iA=; b=L/4uLq5dOh5HzAqeEI+zTQddANmw5RkX7jQ7d8IsuLlkABWmekde1kEewuYOFlFdZF 0VzknlrMQeIP3ew9+V4/U3PyJYqhOBwcVG/AniZI4YPBKuxXoo2XN1CxxRRsa4bWZHsY QBPNyCvYoxaBS1tYvNBqZtdaR52edV5b2WyDajuJo0ttcfKG7VvEgpsgljcel2RyRjRZ 9vVP5JvqhdB1xC51J+WImdrurVzmR38jHSgWktPEV0KrPukNatIpI1CpGlxXNmr61Ll7 bOFnHoSiKdOZW5SCQdyZj1xpfsK2TiGVjiNNdyEvFVQ9DMpo0i6cSI0rLjJdeKLt8YVf TJ1A== X-Gm-Message-State: APjAAAVq/MwbepBo0gpWfrllnkTczvKq7NcpItCU2l69bAxgdGP6xn6c gWBSaIOx1Ow0OKSoHrpGxneiyw== X-Google-Smtp-Source: APXvYqyRfRbOQd4O4mn3re/FxB2qIO7Mlov95+4wv4+FEetvanQyZtKMUuxxYksXOB1qzBmLgay7fg== X-Received: by 2002:a1c:b403:: with SMTP id d3mr3067061wmf.85.1553262202694; Fri, 22 Mar 2019 06:43:22 -0700 (PDT) Received: from localhost ([195.39.71.253]) by smtp.gmail.com with ESMTPSA id h14sm11889184wre.90.2019.03.22.06.43.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 22 Mar 2019 06:43:21 -0700 (PDT) Date: Fri, 22 Mar 2019 14:32:41 +0100 From: Jiri Pirko To: Parav Pandit Cc: Jakub Kicinski , "Samudrala, Sridhar" , "davem@davemloft.net" , "netdev@vger.kernel.org" , "oss-drivers@netronome.com" Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Message-ID: <20190322133241.GA2211@nanopsycho> References: <20190318142949.36f18af4@cakuba.netronome.com> <20190320132257.372369d7@cakuba.netronome.com> <20190321090821.GB2087@nanopsycho> <20190321161622.GS2087@nanopsycho> <20190321172348.GX2087@nanopsycho> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Thu, Mar 21, 2019 at 06:42:55PM CET, parav@mellanox.com wrote: > > >> -----Original Message----- >> From: Jiri Pirko >> Sent: Thursday, March 21, 2019 12:24 PM >> To: Parav Pandit >> Cc: Jakub Kicinski ; Samudrala, Sridhar >> ; davem@davemloft.net; >> netdev@vger.kernel.org; oss-drivers@netronome.com >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI >> ports >> >> Thu, Mar 21, 2019 at 05:50:37PM CET, parav@mellanox.com wrote: >> > >> > >> >> -----Original Message----- >> >> From: Jiri Pirko >> >> Sent: Thursday, March 21, 2019 11:16 AM >> >> To: Parav Pandit >> >> Cc: Jakub Kicinski ; Samudrala, Sridhar >> >> ; davem@davemloft.net; >> >> netdev@vger.kernel.org; oss-drivers@netronome.com >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on >> >> devlink PCI ports >> >> >> >> Thu, Mar 21, 2019 at 04:03:58PM CET, parav@mellanox.com wrote: >> >> >Hi Jiri, >> >> > >> >> >> -----Original Message----- >> >> >> From: Jiri Pirko >> >> >> Sent: Thursday, March 21, 2019 4:08 AM >> >> >> To: Jakub Kicinski >> >> >> Cc: Parav Pandit ; Samudrala, Sridhar >> >> >> ; davem@davemloft.net; >> >> >> netdev@vger.kernel.org; oss-drivers@netronome.com >> >> >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on >> >> >> devlink PCI ports >> >> >> >> >> >> Wed, Mar 20, 2019 at 09:22:57PM CET, jakub.kicinski@netronome.com >> >> >> wrote: >> >> >> >On Wed, 20 Mar 2019 18:24:15 +0000, Parav Pandit wrote: >> >> >> >> Hi Jiri, Jakub, Samudrala Sridhar, >> >> >> >> > > > > > And physical port in include/uapi/linux/devlink.h >> >> >> >> > > > > > also describe that. >> >> >> >> > > > > >> >> >> >> > > > > By "that" you must mean that the physical is a user facing >> port. >> >> >> >> > > > >> >> >> >> > > > Can you please describe the difference between 'PF port' >> >> >> >> > > > and 'physical port of include/uapi/linux/devlink.h'? I >> >> >> >> > > > must have missed this crisp definition in discussion >> >> >> >> > > > between you and Jiri. I am in meantime checking the thread. >> >> >> >> > > >> >> >> >> > > Perhaps start with the cover letter which includes an ASCII >> >> drawing? >> >> >> >> > > >> >> >> >> > > Using Mellanox nomenclature - PF port is a "representor" >> >> >> >> > > for the PF which may be on another Host (SmartNIC or >> multihost). >> >> >> >> > > It's pretty much the same thing as a VF port/"representor". >> >> >> >> > > >> >> >> >> > Yes. We are aligned here. :-) I see your point, where in >> >> >> >> > multi-host scenario, a physical port may be 1, but PF ports >> >> >> >> > are 4, because of 4 PFs for 4 hosts. >> >> >> >> > (just an example of 4 hosts with their own mac address >> >> >> >> > sharing 1 physical port). >> >> >> >> > >> >> >> >> > When there is no multihost and one to one mapping between a >> >> >> >> > PF and physical links, there is some overlap between PF port >> >> >> >> > and physical port attributes. >> >> >> >> > I believe, such overlap is fine as long as we have unique >> >> >> >> > indices for the >> >> >> ports. >> >> >> >> > >> >> >> >> > So I am ok to have flavours as >> >> physical/cpu/dsa/pf/vf/mdev/switchport. >> >> >> >> > (last 4 as new port flavours). >> >> >> >> > >> >> >> >> > > Physical port is the hole on the panel of the adapter where >> >> >> >> > > cable >> >> >> goes. >> >> >> >> >> >> >> >> So my take away from above discussion are: >> >> >> >> 1. Following new port flavours should be added >> >> >> pci_pf/pci_vf/mdev/switchport. >> >> >> >> a. Switchport indicates port on the eswitch. Normally this port >> >> >> >> has rep- >> >> >> netdev attached to it. >> >> >> > >> >> >> >I don't understand the "switchport". Surely physical ports are >> >> >> >also attached to the eswitch? And one of the main purpose of >> >> >> >adding the pci_pf/pci_vf flavours was to generate phys_port_name >> >> >> >for the port netdevs. >> >> >> > >> >> >> >Please don't use the term representor if possible. Representor >> >> >> >for most developers describes the way the netdev is implemented >> >> >> >in the driver, so for Mellanox and Netronome different ports will >> >> >> >be representors and non-representors. That's why I prefer port >> >> >> >netdev (attached to eswitch, has switch_id) and host netdev >> >> >> >(PF/VF netdev, vNIC, VSI, etc). >> >> >> > >> >> >> >> b. host side port flavours are pci_pf/pci_vf/mdev which may be >> >> >> >> connected to switchport >> >> >> > >> >> >> >See above, pci_pf/pci_vf are needed for phys_port_name generation. >> >> >> >> >> >> Yep, that makes sense. >> >> >> >> >> >> >> >> >> > >> >> >> >> 2. host side port flavours are not limited to Ethernet, as it >> >> >> >> is for devlink's >> >> >> port instance. >> >> >> >> >> >> >> >> 3. Each port is continue to be accessed using unique port index. >> >> >> >> >> >> >> >> 4. host side ports and switchport are control objects. >> >> >> >> a. switch side ports reside where current eswitch object of >> >> >> >> devlink instance reside b. for a given VF/PF/mdev such host >> >> >> >> side ports may be in hypervisor or VM or both depending on the >> >> >> >> privilege >> >> >> >> >> >> >> >> 5. eth.mac_address, rdma.port_guid can be programmed at host >> >> >> >> port flavours by extending as $ devlink port param set... >> >> >> >> (similar to devlink dev param set) >> >> >> > >> >> >> >You can keep restating that's your position, but I have *not* >> >> >> >conceded to that. >> >> >> >> >> >> I'm also not convinced that host dummy ports are good idea to hold >> >> these. >> >> >> >> >> >> >> >> >I didn't understand what do you mean my dummy port. >> >> >> >> It's a port for a VF host port which is not actually in the host but in the >> vm. >> >> Very confusing. >> >> >> >It is the vf_ctrl flavour. I don't see it any different than rep-netdev. >> >rep-netdev is not that confusing to us that represent eswitch vport. >> >Why vf_ctrl flavour port that represents otherside of the pipe as you have >> shown in example? >> >Why it that confusing? >> >> Because sometimes it is there only once (PF), sometimes twice (VF) - and one >> of these is kind-of zombie. >> >I gave the example in email that contains description yesterday. >You didn't respond to it. >So repeating here. >Can you please point what looks like zombie below? > >$ devlink port show >pci/0000:05:00.0/0 eth netdev repndev_pf0_p0 flavour physical switch_id 00154d130d2f >pci/0000:05:00.0/1 eth netdev repndev_pf0_p1 flavour physical switch_id 00154d130d2f >pci/0000:05:00.0/10001 eth netdev repndev_pf0_vf_1 flavour switchport switch_id 00154d130d2f peer pci/0000:05:00.0/1 >pci/0000:05:00.0/10002 eth netdev repndev_pf0_p0_mdev_8000 flavour switchport switch_id 00154d130d2f peer mdev/uuidX/0 > >pci/0000:05:00.0/1 eth netdev flavour vf_ctrl vf 1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this one. You are missing an actual VF instance. >mdev/uuidX/0 eth netdev flavour mdev_ctrl Why "ctrl"? > >> > >> > >> >> >Can you explain what is wrong in programming host port params using >> >> host_port object? >> >> >Few questions are unanswered in my past 2 or 3 emails. >> >> >Can you please go through them? >> >> >Can you point to some example switch API where you program host >> >> >params >> >> at switch? >> >> > >> >> >> > >> >> >> >> 6. more host port params can be added in future when user need >> >> >> >> arise >> >> >> >> >> >> >> >> 7. rep-netdev continue to be eswitch (switchport) representor >> >> >> >> at the >> >> >> switch side. >> >> >> >> a. Hence rep-netdev cannot be used for programming host port's >> >> >> parameters. >> >> >> >> >> >> >> >> 8. eswitch devlink instance knows when VF/PF/mdev's switchport >> >> >> >> are >> >> >> created/removed. >> >> >> >> Hence, those will be created/deleted by eswitch. >> >> >> >> Similarly for host port flavours too. >> >> >> >> >> >> >> >> Does it look fine? Did I miss something? >> >> >> >> We would like to progress on incremental patches for item-4 and >> >> >> >> any prep work needed to reach to item-4.