From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B358C43381 for ; Mon, 4 Mar 2019 16:15:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0792020823 for ; Mon, 4 Mar 2019 16:15:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="vQy0V4sS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726862AbfCDQPU (ORCPT ); Mon, 4 Mar 2019 11:15:20 -0500 Received: from mail-eopbgr130070.outbound.protection.outlook.com ([40.107.13.70]:53380 "EHLO EUR01-HE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726082AbfCDQPU (ORCPT ); Mon, 4 Mar 2019 11:15:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SofLqC09jiPa3n32WwA5yeeY5TOy8Z8otxCBiBAdF9Q=; b=vQy0V4sS/I9YWrsXgoFZ0vkAPY7K3vN3iGDJEeQnXj8+hkFjiooTaiqvA5V6QlN+0WmxoW9B3cW/KU26dbhccQ0DkApJ1VqVH1+Q3pPHcfYE7lReQJVHSOZyVTaTQ9dIBGY36egCS5lBIOhvWpEZre83ayCkpG5P8y38Tb1Ef/Q= Received: from DBBPR05MB6570.eurprd05.prod.outlook.com (20.179.44.81) by DBBPR05MB6571.eurprd05.prod.outlook.com (20.179.44.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.20; Mon, 4 Mar 2019 16:15:14 +0000 Received: from DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f]) by DBBPR05MB6570.eurprd05.prod.outlook.com ([fe80::5d59:2e1c:c260:ea6f%2]) with mapi id 15.20.1665.019; Mon, 4 Mar 2019 16:15:14 +0000 From: Jason Gunthorpe To: Jakub Kicinski CC: Jiri Pirko , "davem@davemloft.net" , "oss-drivers@netronome.com" , "netdev@vger.kernel.org" , Parav Pandit Subject: Re: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Topic: [PATCH net-next 4/8] devlink: allow subports on devlink PCI ports Thread-Index: AQHUzplETPTCP2diS0WZgKtanQm+oKXz97IAgAe1/AA= Date: Mon, 4 Mar 2019 16:15:14 +0000 Message-ID: <20190304161510.GO8627@mellanox.com> References: <20190226182436.23811-1-jakub.kicinski@netronome.com> <20190226182436.23811-5-jakub.kicinski@netronome.com> <20190227123753.GB2240@nanopsycho> <20190227103000.6ea6f7c0@cakuba.netronome.com> In-Reply-To: <20190227103000.6ea6f7c0@cakuba.netronome.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: YQXPR0101CA0056.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:14::33) To DBBPR05MB6570.eurprd05.prod.outlook.com (2603:10a6:10:d1::17) authentication-results: spf=none (sender IP is ) smtp.mailfrom=jgg@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [24.137.65.181] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 218e3c7f-9213-4b7e-b74c-08d6a0bc95a6 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020);SRVR:DBBPR05MB6571; x-ms-traffictypediagnostic: DBBPR05MB6571: x-microsoft-exchange-diagnostics: =?us-ascii?Q?1;DBBPR05MB6571;23:rlU6lAEUyReqYyJXm6pjsX/Krkd1PI/fDNUCLrMme?= =?us-ascii?Q?lo+wy/3AEqJQWN2GIwwlZI5XHkP8aa84rg02RM70fbqpmWt9+L4XB5oqbtA2?= =?us-ascii?Q?btgEDfxvI9Z33SNpjcS5SE2VX4mJzAO9LyS5Zl1Ca18kK9YVoXdU5GqUVSzb?= =?us-ascii?Q?mdwbU8kIDJZyt8wefnWm79Ewa4yXGxoUyXWtupe+vPUfYEKIFZcycfke/Vda?= =?us-ascii?Q?HzBlz3kin6+EYOyVSFtFw6WqcpmpwpUP8FOq3NFexfRUYNeWfU2IEFIA6SvU?= =?us-ascii?Q?Pybx7FV8r9vhcisww0JXHqyh3zukWgEIx29n1j2xWnPvjuKFNGnqmFbVLN5p?= =?us-ascii?Q?Niz5gqKVhLOQiBuE/VJCyNEjQRQgq451VFx3qSjjLnyLlkJ65vaBINUgc6Ya?= =?us-ascii?Q?f73uUoJUHbNGOTPa3USApS/3d/o6KXA4uWj9B+RDkplAwDLnNDxQxYqJG8zW?= =?us-ascii?Q?JI9Z3rse42QrMGGq1YziozedBwHx4jmPKU7oF9ve3gXKDdPBMQm82ZVm5S22?= =?us-ascii?Q?wlVhXghYFFCXHOXsiW+8JFZQgBMjr5R7P3CG/Mu3DwxabhSnwMri2NfAV0Gz?= =?us-ascii?Q?XA3Fo/DtqkCFQeAWmALIik29qTm/twDAw1ILKqCxdF5fzujs6est+k8hm3fO?= =?us-ascii?Q?qieymK2Mr46OVZHSesllFius9QRCsue17pxAinZf8V5DYetwAWudrBJBTtFc?= =?us-ascii?Q?WKLRmV44u+es9IVXk3hgOPU4HxeF6x+ANWbx7N1nE/AL/ULXL5IbILnorJr5?= =?us-ascii?Q?tBUz9oji0sTF/8eofawuYrj2HoZZF7rR7WiYvPiBck6INYTB52vIMqTmRg7Z?= =?us-ascii?Q?Nb63Nf4lHIWFDnH5AJo1yighhliFAatLZbJX3jSifRxY8V5puvJk0tCN0Yf0?= =?us-ascii?Q?c1Gb32KHIkX2pFf2cRgnBxeBq7jsD/Ln5wnqjvJfipFyJ6lr1y7hWZGuoRs9?= =?us-ascii?Q?+1J6IwcpItGYSC4MPkSpt48eWhFl3KWKNgia/PP5y9zVnrKpBRYr0XAs4p2j?= =?us-ascii?Q?OpAV2ielxCHs/eNbkeIe2dNda8sWziLAUXB9C+dIyLKzG1XYQiQu1naTGKIT?= =?us-ascii?Q?3+B/eKfyMYX/UFipNYBFisYDIyBY2oNuPvPCD/2LQIrMk3pQjl4vUsaUFcue?= =?us-ascii?Q?NBRbb1nXBhgbp+xb5RNrVBZVnzwILXYWncikmTAFj5BzdusKA3/1aFmPIbrn?= =?us-ascii?Q?yFT/TzV3UzCb3eKdygoXQX2Rrfr8q95dJvep+mpVmxlJe1Im6Yb0jqnO9f5h?= =?us-ascii?Q?zTYwKEGn2aXpUUJ02W8vDTb7ZlBNF2wDHM8HmjuWXZ1wH2RG11+MY8nPHBVh?= =?us-ascii?B?UT09?= x-microsoft-antispam-prvs: x-forefront-prvs: 09669DB681 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(346002)(366004)(136003)(39860400002)(376002)(396003)(199004)(189003)(446003)(316002)(6116002)(6916009)(3846002)(7736002)(476003)(11346002)(2616005)(4326008)(33656002)(68736007)(6486002)(52116002)(6246003)(8936002)(107886003)(93886005)(305945005)(6436002)(8676002)(5660300002)(106356001)(486006)(76176011)(81156014)(81166006)(105586002)(99286004)(25786009)(1076003)(14454004)(6512007)(66066001)(86362001)(229853002)(36756003)(478600001)(71190400001)(71200400001)(53936002)(2906002)(97736004)(186003)(386003)(6506007)(26005)(256004)(102836004)(54906003);DIR:OUT;SFP:1101;SCL:1;SRVR:DBBPR05MB6571;H:DBBPR05MB6570.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: aM//v6THaFnNqIHWncvu2/ugsIE0RrbVDV5xM2fRNYYkABmE/pR27RUH48P9Bi5dqahzg8CSQDvFWPQEQLyaB5pRi4QdnknRoiOADagn6oM/oa2ME/qBgFiNAUc0sv9ACzHIPte0RvNtlGVIEBvwRtbdQjwt86B6ON4Fnf9GDD6GlyjRUiLg8voGCC2KpFhl8jOiqeC5zFz2ZLuEEN9J2tZJ6H0ylazw4OqbFQrODmCQOewLx9nK1/0YFSnZeWMrSYKZs9XN3BIlPgD3x2eFCeXq5aV3q/rZSPIfoFDQiv2OssBg7xJ+ymmsfvmsWc6EH7Rno2JKLWMo8azH2k9pePh6ceSCGhs3kfU4Dn68XDe61yMJSDhfDAvOlHP9x2wZNGnZKA5B+WcdH557UM4vcg4o7SowlNE8yVbFEqXeqn8= Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 218e3c7f-9213-4b7e-b74c-08d6a0bc95a6 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Mar 2019 16:15:14.6842 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR05MB6571 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Wed, Feb 27, 2019 at 10:30:00AM -0800, Jakub Kicinski wrote: > On Wed, 27 Feb 2019 13:37:53 +0100, Jiri Pirko wrote: > > Tue, Feb 26, 2019 at 07:24:32PM CET, jakub.kicinski@netronome.com wrote= : > > >PCI endpoint corresponds to a PCI device, but such device > > >can have one more more logical device ports associated with it. > > >We need a way to distinguish those. Add a PCI subport in the > > >dumps and print the info in phys_port_name appropriately. > > > > > >This is not equivalent to port splitting, there is no split > > >group. It's just a way of representing multiple netdevs on > > >a single PCI function. > > > > > >Note that the quality of being multiport pertains only to > > >the PCI function itself. A PF having multiple netdevs does > > >not mean that its VFs will also have multiple, or that VFs > > >are associated with any particular port of a multiport VF. > >=20 > > We've been discussing the problem of subport (we call it "subfunction" > > or "SF") for some time internally. Turned out, this is probably harder > > task to model. Please prove me wrong. > >=20 > > The nature of VF makes it a logically separate entity. It has a separat= e > > PCI address, it should therefore have a separate devlink instance. > > You can pass it through to VM, then the same devlink instance should be > > created inside the VM and disappear from the host. >=20 > Depends what a devlink instance represents :/ On one hand you may want > to create an instance for a VF to allow it to spawn soft ports, on the > other you may want to group multiple functions together. >=20 > IOW if devlink instance is for an ASIC, there should be one per device > per host. =20 Don't we already have devlink instances for every mlx5 physical port and VF as they are unique PCI functions? > You guys come from the RDMA side of the world, with which I'm less > familiar, and the soft bus + spawning devices seems to be a popular > design there. Could you describe the advantages of that model for=20 > the sake of the netdev-only folks? :) I don't think we do this in RDMA at all yet, or maybe I'm not sure what you are thinking of? The forward looking discussion is mainly to create something like macvlan that can be offloaded, so we have something like a 'rdma offload HW context' for each 'rdma macvlan' .. and that 'rdma offload HW context' has all the same knobs as an offload context for a VF that would normally be tied to a unique PCI BDF (via devlink). But in this case there is no unique BDF. >From another angle this is broadly similar to the scalable IOV stuff, but without placing requirements on the host IOMMU to implement it. Jason