From: Jason Gunthorpe <email@example.com> To: David Ahern <firstname.lastname@example.org> Cc: Parav Pandit <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, Jiri Pirko <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, Leon Romanovsky <email@example.com>, Saeed Mahameed <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, Vu Pham <email@example.com> Subject: Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port Date: Wed, 18 Nov 2020 14:38:30 -0400 [thread overview] Message-ID: <20201118183830.GA917484@nvidia.com> (raw) In-Reply-To: <firstname.lastname@example.org> On Wed, Nov 18, 2020 at 11:03:24AM -0700, David Ahern wrote: > With Connectx-4 Lx for example the netdev can have at most 63 queues What netdev calls a queue is really a "can the device deliver interrupts and packets to a given per-CPU queue" and covers a whole spectrum of smaller limits like RSS scheme, # of available interrupts, ability of the device to create queues, etc. CX4Lx can create a huge number of queues, but hits one of these limits that mean netdev's specific usage can't scale up. Other stuff like RDMA doesn't have the same limits, and has tonnes of queues. What seems to be needed is a resource controller concept like cgroup has for processes. The system is really organized into a tree: physical device mlx5_core / | \ \ (aux bus) netdev rdma vdpa SF etc | (aux bus) mlx5_core / \ (aux bus) netdev vdpa And it does make a lot of sense to start to talk about limits at each tree level. eg the top of the tree may have 128 physical interrupts. With 128 CPU cores that isn't enough interrupts to support all of those things concurrently. So the user may want to configure: - The first level netdev only gets 64, - 3rd level mlx5_core gets 32 - Final level vdpa gets 8 Other stuff has to fight it out with the remaining shared interrupts. In netdev land # of interrupts governs # of queues For RDMA # of interrupts limits the CPU affinities for queues VPDA limits the # of VMs that can use VT-d The same story repeats for other less general resources, mlx5 also has consumption of limited BAR space, and consumption of some limited memory elements. These numbers are much bigger and may not need explicit governing, but the general concept holds. It would be very nice if the limit could be injected when the aux device is created but before the driver is bound. I'm not sure how to manage that though.. I assume other devices will be different, maybe some devices have a limit on the number of total queues, or a limit on the number of VDPA or RDMA devices. Jason
next prev parent reply other threads:[~2020-11-18 18:38 UTC|newest] Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-12 19:24 [PATCH net-next 00/13] Add mlx5 subfunction support Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 01/13] devlink: Prepare code to fill multiple port function attributes Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 02/13] devlink: Introduce PCI SF port flavour and port attribute Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 03/13] devlink: Support add and delete devlink port Parav Pandit 2020-11-18 16:21 ` David Ahern 2020-11-18 17:02 ` Parav Pandit 2020-11-18 18:03 ` David Ahern 2020-11-18 18:38 ` Jason Gunthorpe [this message] 2020-11-18 19:36 ` David Ahern 2020-11-18 20:42 ` Jason Gunthorpe 2020-11-18 19:22 ` Parav Pandit 2020-11-19 0:41 ` Jacob Keller 2020-11-19 1:17 ` David Ahern 2020-11-19 1:56 ` Samudrala, Sridhar 2020-11-19 0:52 ` Jacob Keller 2020-11-12 19:24 ` [PATCH net-next 04/13] devlink: Support get and set state of port function Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 05/13] devlink: Avoid global devlink mutex, use per instance reload lock Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 06/13] devlink: Introduce devlink refcount to reduce scope of global devlink_mutex Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 07/13] net/mlx5: SF, Add auxiliary device support Parav Pandit 2020-12-07 2:48 ` David Ahern 2020-12-07 4:53 ` Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 08/13] net/mlx5: SF, Add auxiliary device driver Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 09/13] net/mlx5: E-switch, Prepare eswitch to handle SF vport Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 10/13] net/mlx5: E-switch, Add eswitch helpers for " Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 11/13] net/mlx5: SF, Add SF configuration hardware commands Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 12/13] net/mlx5: SF, Add port add delete functionality Parav Pandit 2020-11-12 19:24 ` [PATCH net-next 13/13] net/mlx5: SF, Port function state change support Parav Pandit 2020-11-16 22:52 ` [PATCH net-next 00/13] Add mlx5 subfunction support Jakub Kicinski 2020-11-17 0:06 ` Saeed Mahameed 2020-11-17 1:58 ` Jakub Kicinski 2020-11-17 4:08 ` Parav Pandit 2020-11-17 17:11 ` Jakub Kicinski 2020-11-17 18:49 ` Jason Gunthorpe 2020-11-19 2:14 ` Jakub Kicinski 2020-11-19 4:35 ` David Ahern 2020-11-19 5:57 ` Saeed Mahameed 2020-11-20 1:31 ` Jakub Kicinski 2020-11-25 5:33 ` David Ahern 2020-11-25 6:00 ` Parav Pandit 2020-11-25 14:37 ` David Ahern 2020-11-20 1:29 ` Jakub Kicinski 2020-11-20 17:58 ` Alexander Duyck 2020-11-20 19:04 ` Samudrala, Sridhar 2020-11-23 21:51 ` Saeed Mahameed 2020-11-24 7:01 ` Jason Wang 2020-11-24 7:05 ` Jason Wang 2020-11-19 6:12 ` Saeed Mahameed 2020-11-19 8:25 ` Parav Pandit 2020-11-20 1:35 ` Jakub Kicinski 2020-11-20 3:34 ` Parav Pandit 2020-11-17 18:50 ` Parav Pandit 2020-11-19 2:23 ` Jakub Kicinski 2020-11-19 6:22 ` Saeed Mahameed 2020-11-19 14:00 ` Jason Gunthorpe 2020-11-20 3:35 ` Jakub Kicinski 2020-11-20 3:50 ` Parav Pandit 2020-11-20 16:16 ` Jason Gunthorpe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201118183830.GA917484@nvidia.com \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH net-next 03/13] devlink: Support add and delete devlink port' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).