From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0E27C43381 for ; Tue, 5 Mar 2019 11:17:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 84D9D20848 for ; Tue, 5 Mar 2019 11:17:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=resnulli-us.20150623.gappssmtp.com header.i=@resnulli-us.20150623.gappssmtp.com header.b="aFW9Z7RV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727722AbfCELRI (ORCPT ); Tue, 5 Mar 2019 06:17:08 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:52545 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727424AbfCELRI (ORCPT ); Tue, 5 Mar 2019 06:17:08 -0500 Received: by mail-wm1-f67.google.com with SMTP id f65so2153952wma.2 for ; Tue, 05 Mar 2019 03:17:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=resnulli-us.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=wNu1Mf22wkapadVrIs5R1Cl95iQk0k0JFLj+mUSVP4E=; b=aFW9Z7RVKNwCQQNS2B94GkLNg829tHHrfVZvS6JN2f0K2p5A88FwqlvVsF0uLIsgWm kmxgLakW9CKkflmDAzgbLk5uYJBVAzCOE2Hr//frtlUX1lg1AAh5i11kOb6tVIvfkrIK tcHh5+IkpKUC5wwvpckSIjk75RXikpb10HlbPfVW4GBPCMBYZNQtwWg8gYN5cIpM/EBb j2v2UA86LdI75BCBSjXeNZ7LYwIZ7rjJoa+IbhdFts90//MB9FSZzch+fLGfqvEtqHAN fbR7g9Qi82Mco45eDENw438UfkT9fHmGxFgLXL5BmHfYJpw3+Ymcc3WAcWMHDRO/N1r1 Y/sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=wNu1Mf22wkapadVrIs5R1Cl95iQk0k0JFLj+mUSVP4E=; b=FFtwa3kvYZvC++2QkbcIY6VpXcbe6j9gj/rKdqsUqsIGamsyClG5Ej6n18I6GUSpE5 0hJDTqtiboOEESwaD6nN66H9NSgVuplWfkUKXoWaNpjY38AyXcMV0qW+zvSGf0FtXi/3 LW0jHwP1dICZKum9K3DOex85XT8sIQoLfy7xWVGosDzx7nbod84ryd1zSepZVCz+qAmB h/cFMtFjGbot7+CItIhY9hLa18Q8QiOjGPT/EaLFKr5DmTB5d4eyFMZPrde4iwczekXi 5v+/vn48h1VEuw6krujvXqnL3L1l+zVSdPusKc4EUU/xrzCrVolG4fSCByjtgj8wplDI 9lVg== X-Gm-Message-State: APjAAAXAVeO68Kk4/RkpXw7WOacI0rsTZ/xmHlxAuF83hrawTXKCAifI AumOfbisqGTVkIHUPP6yjJNjxg== X-Google-Smtp-Source: APXvYqx6dzbKrxDcb8EEfs+ugoPO98Q80jVHH+zDcwptj1QvzY7KN4tLIcrg53CtFWsjFYN6BtqZiw== X-Received: by 2002:a1c:c010:: with SMTP id q16mr2493159wmf.134.1551784626120; Tue, 05 Mar 2019 03:17:06 -0800 (PST) Received: from localhost (ip-89-177-134-16.net.upcbroadband.cz. [89.177.134.16]) by smtp.gmail.com with ESMTPSA id o127sm13110463wmo.20.2019.03.05.03.17.05 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 05 Mar 2019 03:17:05 -0800 (PST) Date: Tue, 5 Mar 2019 12:07:18 +0100 From: Jiri Pirko To: Jakub Kicinski Cc: davem@davemloft.net, netdev@vger.kernel.org, oss-drivers@netronome.com Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Message-ID: <20190305110718.GD2314@nanopsycho> References: <20190301180453.17778-1-jakub.kicinski@netronome.com> <20190301180453.17778-5-jakub.kicinski@netronome.com> <20190302094116.GQ2314@nanopsycho> <20190302114847.733759a1@cakuba.netronome.com> <20190304111902.GX2314@nanopsycho> <20190304164007.7cef8af9@cakuba.netronome.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190304164007.7cef8af9@cakuba.netronome.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Tue, Mar 05, 2019 at 01:40:07AM CET, jakub.kicinski@netronome.com wrote: >On Mon, 4 Mar 2019 12:19:02 +0100, Jiri Pirko wrote: >> Sat, Mar 02, 2019 at 08:48:47PM CET, jakub.kicinski@netronome.com wrote: >> >On Sat, 2 Mar 2019 10:41:16 +0100, Jiri Pirko wrote: >> >> Fri, Mar 01, 2019 at 07:04:50PM CET, jakub.kicinski@netronome.com wrote: >> >> >PCI endpoint corresponds to a PCI device, but such device >> >> >can have one more more logical device ports associated with it. >> >> >We need a way to distinguish those. Add a PCI subport in the >> >> >dumps and print the info in phys_port_name appropriately. >> >> > >> >> >This is not equivalent to port splitting, there is no split >> >> >group. It's just a way of representing multiple netdevs on >> >> >a single PCI function. >> >> > >> >> >Note that the quality of being multiport pertains only to >> >> >the PCI function itself. A PF having multiple netdevs does >> >> >not mean that its VFs will also have multiple, or that VFs >> >> >are associated with any particular port of a multiport VF. >> >> > >> >> >Example (bus 05 device has subports, bus 82 has only one port per >> >> >function): >> >> > >> >> >$ devlink port >> >> >pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical >> >> >pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 >> >> >pci/0000:05:00.0/4: type eth netdev enp5s0np1 flavour physical >> >> >pci/0000:05:00.0/11000: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 >> >> >> >> So these subport devlink ports are eswitch ports for subports, right? >> >> >> >> Please see the following drawing: >> >> >> >> +---+ +---+ +---+ >> >> pfsub| 5 | vf| 6 | | 7 |pfsub >> >> +-+-+ +-+-+ +-+-+ >> >> physical link <---------+ | | | >> >> | | | | >> >> | | | | >> >> | | | | >> >> +-+-+ +-+-+ +-+-+ +-+-+ >> >> | 1 | | 2 | | 3 | | 4 | >> >> +--+---+------+---+------+---+------+---+--+ >> >> | physical pfsub vf pfsub | >> >> | port port port port | >> >> | | >> >> | eswitch | >> >> | | >> >> | | >> >> +------------------------------------------+ >> >> >> >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 flavour physical switch_id 00154d130d2f >> >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 flavour pci_pf pf 0 subport 0 switch_id 00154d130d2f >> >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 flavour pci_vf pf 0 vf 0 switch_id 00154d130d2f >> >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 flavour pci_pf pf 0 subport 1 switch_id 00154d130d2f >> >> >> >> This is basically what you have and I think we are in sync with that. >> >> But what about 5,6,7? Should they have devlink port instances too? >> >> >> >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 0 >> >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 flavour ???? pf 0 vf 0 >> >> 7) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? flavour ???? pf 0 subport 1 >> >> >> >> These are the "peers". >> >> I think that there could be flavours "pci_pf" and "pci_vf". Then the >> >> "representors" (switch ports) could have flavours "pci_pf_port" and >> >> "pci_vf_port" or something like that. User can see right away >> >> that is not "PF" of "VF" but rather something "on the other end". >> >> Note there is no "switch_id" for these devlink ports that tells the user >> >> these devlink ports are not part of any switch. >> >> What do you think? >> > >> >Hmmm.. Hm. Hm. >> > >> >To me its neat if the devlink instance matches an ASIC. I think it's >> >kind of clear for people to understand what it stands for then. So if >> >we wanted to do the above we'd have to make the switch_id the first >> >class identifier for devlink instances, rather than the bus? But then >> >VF instances don't have a switch ID so that doesn't work... >> > >> >I need to think about it. >> > >> >It's also kind of strange that we have to add the noun *port* to the >> >flavour of... a port... So I would prefer not to have those showing up >> >as ports. Can we invent a new command (say "partition"?) that'd take >> >the bus info where the partition is to be spawned? >> >> Devlink does not supposed to be only there for switches. From the >> beginning the design was to handle cases where the netdev/ib_dev is not >> the correct handle. Not only in case you have multiple instances (ports) >> for one ASIC, but also in case you have only one. Example use case is >> port-type-change (eth->ib,ib->eth). >> >> I chose word "port" as the parent devlink instance is "dev" and if you >> partition the ASIC you basically got "ports", each of a different flavour. >> >> And as you said, devlink instance matches one ASIC. Therefore the >> devlink instance should contain all bits there are part of that ASIC, >> not only switch/eswitch ports. That would be very limitting. > >I could read this as us being in full agreement, but I'm not sure.. >I think we agree that all objects of an ASIC should be under one >devlink instance, the question remains whether both ends of the pipe >for PCI devices (subdevs or not) should appear under ports or does the >"far end" (from ASICs perspective)/"host end" get its own category. Yep. Please see the suggestion about "flavour host" I did in other reply in this thread.