From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 380F3C43381 for ; Wed, 6 Mar 2019 17:56:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0213F20661 for ; Wed, 6 Mar 2019 17:56:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="sDQV80e7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730591AbfCFR4s (ORCPT ); Wed, 6 Mar 2019 12:56:48 -0500 Received: from mail-qt1-f177.google.com ([209.85.160.177]:42388 "EHLO mail-qt1-f177.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726973AbfCFR4s (ORCPT ); Wed, 6 Mar 2019 12:56:48 -0500 Received: by mail-qt1-f177.google.com with SMTP id u7so13840482qtg.9 for ; Wed, 06 Mar 2019 09:56:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :organization:mime-version:content-transfer-encoding; bh=TttLhP8BYlXTTkhiCxSIce/P0u/I7b963Rq79uhEaB8=; b=sDQV80e7dABwDlCnDA7XAJupsNmVnqgXJRdOCKrt0/TDZOLFjgYosw6Gg9zhogu/f6 //EfFe4HZs2ol2fE1aEeOLi1szda/Z9L1aKw4LGytSnTVU3N/dKz2XMg8nv4t6mnIFFp oTZy2qmh4RnWtLog2blGr/syRCDrJWjBrpnAIIdZh7reju5r5C/H8Vrl9ba0Vw2whSEj lYuUFrW+x0Ws2I6i6thmTcQlGXrctjbf/U/PUOPnH7yZd16ycW+4y6A/kRjgmY/L0WF4 Zsdr2k/s403prYZVqUFWD2zRXrjZhvUvRHgJRl1bjs+4W5zILEaaj9VL23yunsVNfZ0K HLKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:organization:mime-version:content-transfer-encoding; bh=TttLhP8BYlXTTkhiCxSIce/P0u/I7b963Rq79uhEaB8=; b=eDVdL9ehlh1wAek+6Vv/XISvgn7Gec6CceNt9f6zUm2IlY5sCPZ0zISXc1EN76Hlao MuVkhQEn3R/am6u8zmEWu4B/lKG/8y0w07u8cDvvm9WMaKBkEzCWnyEEMi/3uc2JYMoT PzQNlzowpvMXedx8eza9LWIdR5whrLuELjGwSEf8ZSMRfNycSdehvkBixbBjdcobZ6fr LXGuEb8e6ZaJOrOTbFaBKv5VjqhC8LNyQAvzh/cMHs+JDcbvIlWJcv2KRdLDoF4hA/gb qQggREog39DRAvZOVL6aXQtWJWVMRTK2MBqtQebx/evsG4kcSwUwOQORzoieaMm6druO 4nSQ== X-Gm-Message-State: APjAAAWoIR2jetNHwc2MZftScfljitLVHyX8ZpVFry5pNbJ4PKskFLXD DfHMxSXTx2L3WVEPT3FYvzYIHg== X-Google-Smtp-Source: APXvYqxoOLIdLpUiTk9eONJIQCbyj+lhcWbE+3SKez8k2jnopzUEwXQI6myzSj+zarMp5xt/mO9zLQ== X-Received: by 2002:a0c:8186:: with SMTP id 6mr7243374qvd.139.1551895006793; Wed, 06 Mar 2019 09:56:46 -0800 (PST) Received: from cakuba.hsd1.ca.comcast.net ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id m12sm1158705qke.17.2019.03.06.09.56.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Mar 2019 09:56:46 -0800 (PST) Date: Wed, 6 Mar 2019 09:56:38 -0800 From: Jakub Kicinski To: Jiri Pirko Cc: davem@davemloft.net, netdev@vger.kernel.org, oss-drivers@netronome.com Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports Message-ID: <20190306095638.7c028bdd@cakuba.hsd1.ca.comcast.net> In-Reply-To: <20190306122037.GB2819@nanopsycho> References: <20190301180453.17778-1-jakub.kicinski@netronome.com> <20190301180453.17778-5-jakub.kicinski@netronome.com> <20190302094116.GQ2314@nanopsycho> <20190302114847.733759a1@cakuba.netronome.com> <20190304075609.GV2314@nanopsycho> <20190304163302.7e40219e@cakuba.netronome.com> <20190305110601.GC2314@nanopsycho> <20190305091534.36200de6@cakuba.hsd1.ca.comcast.net> <20190306122037.GB2819@nanopsycho> Organization: Netronome Systems, Ltd. MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Wed, 6 Mar 2019 13:20:37 +0100, Jiri Pirko wrote: > Tue, Mar 05, 2019 at 06:15:34PM CET, jakub.kicinski@netronome.com wrote: > >On Tue, 5 Mar 2019 12:06:01 +0100, Jiri Pirko wrote: > >> >> >as ports. Can we invent a new command (say "partition"?) that'd take > >> >> >the bus info where the partition is to be spawned? > >> >> > >> >> Got it. But the question is how different this object would be from the > >> >> existing "port" we have today. > >> > > >> >They'd be where "the other side of a PCI link" is represented, > >> >restricting ports to only ASIC's forwarding plane ports. > >> > >> Basically a "host port", right? It can still be the same port object, > >> only with different flavour and attributes. So we would have: > >> > >> 1) pci/0000:05:00.0/0: type eth netdev enp5s0np0 > >> flavour physical switch_id 00154d130d2f > >> 2) pci/0000:05:00.0/10000: type eth netdev enp5s0npf0s0 > >> flavour pci_pf pf 0 subport 0 > >> switch_id 00154d130d2f > >> peer pci/0000:05:00.0/1 > >> 3) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0vf0 > >> flavour pci_vf pf 0 vf 0 > >> switch_id 00154d130d2f > >> peer pci/0000:05:10.1/0 > >> 4) pci/0000:05:00.0/10001: type eth netdev enp5s0npf0s1 > >> flavour pci_pf pf 0 subport 1 > >> switch_id 00154d130d2f > >> peer pci/0000:05:00.0/2 > >> 5) pci/0000:05:00.0/1: type eth netdev enp5s0f0?? > >> flavour host <---------------- > >> peer pci/0000:05:00.0/10000 > >> 6) pci/0000:05:10.1/0: type eth netdev enp5s10f0 > >> flavour host <---------------- > >> peer pci/0000:05:00.0/10001 > >> 7) pci/0000:05:00.0/2: type eth netdev enp5s0f0?? > >> flavour host <---------------- > >> peer pci/0000:05:00.0/10001 > >> > >> I think it looks quite clear, it gives complete topology view. > > > >Okay, I have some of questions :) > > > >What do we use for port_index? > > That is just a number totally in control of the driver. Driver can > assign it in any way. > > > > >What are the operations one can perform on "host ports"? > > That is a good question. I would start with *none* and extend it upon > needs. > > > > > >If we have PCI parameters, do they get set on the ASIC side of the port > >or the host side of the port? > > Could you give me an example? Let's take msix_vec_per_pf_min as an example. > But I believe that on switch-port side. Ok. > >How do those behave when device is passed to VM? > > In case of VF? VF will have separate devlink instance (separate handle, > probably "aliased" to the PF handle). So it would disappear from > baremetal and appear in VM: > $ devlink dev > pci/0000:00:10.0 > $ devlink dev port > pci/0000:00:10.1/0: type eth netdev enp5s10f0 > flavour host > That's it for the VM. > > There's no linkage (peer, alias) between this and the instances on > baremetal. Ok, I guess this is the main advantage from your perspective? The fact that "host ports" are visible inside a VM? Or do you believe that having both ends of a pipe as ports makes the topology easier to understand? For creating subdevices, I don't think the handle should ever be port. We create new ports on a devlink instance, and configure its forwarding with offloads of well established Linux SW constructs. New devices are not logically associated with other ports (see how in my patches there are 2 "subports" but no main port on that PF - a split not a hierarchy). How we want to model forwarding inside a VM (who configures the underlying switching) remains unclear. > >You have a VF devlink instance there - what ports does it show? > > See above. > > > > > >How do those look when the PF is connected to another host? Do they > >get spawned at all? > > What do you mean by "PF is connected to another host"? Either "SmartNIC": http://www.mellanox.com/products/smartnic/?ls=gppc&lsd=SmartNIC-gen-smartnic&gclid=EAIaIQobChMIxIrGmYju4AIVy5yzCh2SFwQJEAAYASAAEgIui_D_BwE or Multi-host NIC: http://www.mellanox.com/page/multihost > >Will this not be confusing to DSA folks who have a CPU port? > > Why do you think so? Host and CPU sound quite similar, it is unclear how they differ, and why we have a need for both (from user's perspective).