Netdev Archive on lore.kernel.org
 help / color / Atom feed
From: Jason Wang <jasowang@redhat.com>
To: Yongji Xie <xieyongji@bytedance.com>, Parav Pandit <parav@nvidia.com>
Cc: virtualization@lists.linux-foundation.org,
	"Michael S. Tsirkin" <mst@redhat.com>,
	elic@nvidia.com, netdev@vger.kernel.org
Subject: Re: [External] Re: [PATCH 0/7] Introduce vdpa management tool
Date: Mon, 30 Nov 2020 11:36:18 +0800
Message-ID: <182708df-1082-0678-49b2-15d0199f20df@redhat.com> (raw)
In-Reply-To: <CACycT3sYScObb9nN3g7L3cesjE7sCZWxZ5_5R1usGU9ePZEeqA@mail.gmail.com>


On 2020/11/27 下午1:52, Yongji Xie wrote:
> On Fri, Nov 27, 2020 at 11:53 AM Jason Wang <jasowang@redhat.com 
> <mailto:jasowang@redhat.com>> wrote:
>
>
>     On 2020/11/12 下午2:39, Parav Pandit wrote:
>     > This patchset covers user requirements for managing existing
>     vdpa devices,
>     > using a tool and its internal design notes for kernel drivers.
>     >
>     > Background and user requirements:
>     > ----------------------------------
>     > (1) Currently VDPA device is created by driver when driver is
>     loaded.
>     > However, user should have a choice when to create or not create
>     a vdpa device
>     > for the underlying parent device.
>     >
>     > For example, mlx5 PCI VF and subfunction device supports
>     multiple classes of
>     > device such netdev, vdpa, rdma. Howevever it is not required to
>     always created
>     > vdpa device for such device.
>     >
>     > (2) In another use case, a device may support creating one or
>     multiple vdpa
>     > device of same or different class such as net and block.
>     > Creating vdpa devices at driver load time further limits this
>     use case.
>     >
>     > (3) A user should be able to monitor and query vdpa queue level
>     or device level
>     > statistics for a given vdpa device.
>     >
>     > (4) A user should be able to query what class of vdpa devices
>     are supported
>     > by its parent device.
>     >
>     > (5) A user should be able to view supported features and
>     negotiated features
>     > of the vdpa device.
>     >
>     > (6) A user should be able to create a vdpa device in vendor
>     agnostic manner
>     > using single tool.
>     >
>     > Hence, it is required to have a tool through which user can
>     create one or more
>     > vdpa devices from a parent device which addresses above user
>     requirements.
>     >
>     > Example devices:
>     > ----------------
>     >   +-----------+ +-----------+ +---------+ +--------+ +-----------+
>     >   |vdpa dev 0 | |vdpa dev 1 | |rdma dev | |netdev  | |vdpa dev 3 |
>     >   |type=net   | |type=block | |mlx5_0   | |ens3f0  | |type=net   |
>     >   +----+------+ +-----+-----+ +----+----+ +-----+--+ +----+------+
>     >        |              |            |            |    |
>     >        |              |            |            |    |
>     >   +----+-----+        |       +----+----+       | +----+----+
>     >   |  mlx5    +--------+       |mlx5     +-------+ |mlx5     |
>     >   |pci vf 2  |                |pci vf 4 | |pci sf 8 |
>     >   |03:00:2   |                |03:00.4  | |mlx5_sf.8|
>     >   +----+-----+                +----+----+ +----+----+
>     >        |                           |   |
>     >        |                      +----+-----+   |
>     >        +----------------------+mlx5 +----------------+
>     >                               |pci pf 0  |
>     >                               |03:00.0   |
>     >                               +----------+
>     >
>     > vdpa tool:
>     > ----------
>     > vdpa tool is a tool to create, delete vdpa devices from a parent
>     device. It is a
>     > tool that enables user to query statistics, features and may be
>     more attributes
>     > in future.
>     >
>     > vdpa tool command draft:
>     > ------------------------
>     > (a) List parent devices which supports creating vdpa devices.
>     > It also shows which class types supported by this parent device.
>     > In below command example two parent devices support vdpa device
>     creation.
>     > First is PCI VF whose bdf is 03.00:2.
>     > Second is PCI VF whose name is 03:00.4.
>     > Third is PCI SF whose name is mlx5_core.sf.8
>     >
>     > $ vdpa parentdev list
>     > vdpasim
>     >    supported_classes
>     >      net
>     > pci/0000:03.00:3
>     >    supported_classes
>     >      net block
>     > pci/0000:03.00:4
>     >    supported_classes
>     >      net block
>     > auxiliary/mlx5_core.sf.8
>     >    supported_classes
>     >      net
>     >
>     > (b) Now add a vdpa device of networking class and show the device.
>     > $ vdpa dev add parentdev pci/0000:03.00:2 type net name foo0 $
>     vdpa dev show foo0
>     > foo0: parentdev pci/0000:03.00:2 type network parentdev vdpasim
>     vendor_id 0 max_vqs 2 max_vq_size 256
>     >
>     > (c) Show features of a vdpa device
>     > $ vdpa dev features show foo0
>     > supported
>     >    iommu platform
>     >    version 1
>     >
>     > (d) Dump vdpa device statistics
>     > $ vdpa dev stats show foo0
>     > kickdoorbells 10
>     > wqes 100
>     >
>     > (e) Now delete a vdpa device previously created.
>     > $ vdpa dev del foo0
>     >
>     > vdpa tool support in this patchset:
>     > -----------------------------------
>     > vdpa tool is created to create, delete and query vdpa devices.
>     > examples:
>     > Show vdpa parent device that supports creating, deleting vdpa
>     devices.
>     >
>     > $ vdpa parentdev show
>     > vdpasim:
>     >    supported_classes
>     >      net
>     >
>     > $ vdpa parentdev show -jp
>     > {
>     >      "show": {
>     >         "vdpasim": {
>     >            "supported_classes": {
>     >               "net"
>     >          }
>     >      }
>     > }
>     >
>     > Create a vdpa device of type networking named as "foo2" from the
>     parent device vdpasim:
>     >
>     > $ vdpa dev add parentdev vdpasim type net name foo2
>     >
>     > Show the newly created vdpa device by its name:
>     > $ vdpa dev show foo2
>     > foo2: type network parentdev vdpasim vendor_id 0 max_vqs 2
>     max_vq_size 256
>     >
>     > $ vdpa dev show foo2 -jp
>     > {
>     >      "dev": {
>     >          "foo2": {
>     >              "type": "network",
>     >              "parentdev": "vdpasim",
>     >              "vendor_id": 0,
>     >              "max_vqs": 2,
>     >              "max_vq_size": 256
>     >          }
>     >      }
>     > }
>     >
>     > Delete the vdpa device after its use:
>     > $ vdpa dev del foo2
>     >
>     > vdpa tool support by kernel:
>     > ----------------------------
>     > vdpa tool user interface will be supported by existing vdpa
>     kernel framework,
>     > i.e. drivers/vdpa/vdpa.c It services user command through a
>     netlink interface.
>     >
>     > Each parent device registers supported callback operations with
>     vdpa subsystem
>     > through which vdpa device(s) can be managed.
>     >
>     > FAQs:
>     > -----
>     > 1. Where does userspace vdpa tool reside which users can use?
>     > Ans: vdpa tool can possibly reside in iproute2 [1] as it enables
>     user to
>     > create vdpa net devices.
>     >
>     > 2. Why not create and delete vdpa device using sysfs/configfs?
>     > Ans:
>     > (a) A device creation may involve passing one or more attributes.
>     > Passing multiple attributes and returning error code and more
>     verbose
>     > information for invalid attributes cannot be handled by
>     sysfs/configfs.
>     >
>     > (b) netlink framework is rich that enables user space and kernel
>     driver to
>     > provide nested attributes.
>     >
>     > (c) Exposing device specific file under sysfs without net namespace
>     > awareness exposes details to multiple containers. Instead exposing
>     > attributes via a netlink socket secures the communication
>     channel with kernel.
>     >
>     > (d) netlink socket interface enables to run syscaller kernel tests.
>     >
>     > 3. Why not use ioctl() interface?
>     > Ans: ioctl() interface replicates the necessary plumbing which
>     already
>     > exists through netlink socket.
>     >
>     > 4. What happens when one or more user created vdpa devices exist
>     for a
>     > parent PCI VF or SF and such parent device is removed?
>     > Ans: All user created vdpa devices are removed that belong to a
>     parent.
>     >
>     > [1]
>     git://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git
>     <http://git.kernel.org/pub/scm/network/iproute2/iproute2-next.git>
>     >
>     > Next steps:
>     > -----------
>     > (a) Post this patchset and iproute2/vdpa inclusion, remaining
>     two drivers
>     > will be coverted to support vdpa tool instead of creating
>     unmanaged default
>     > device on driver load.
>     > (b) More net specific parameters such as mac, mtu will be added.
>     > (c) Features bits get and set interface will be added.
>
>
>     Adding Yong Ji for sharing some thoughts from the view of
>     userspace vDPA
>     device.
>
>
> Thanks for adding me, Jason!
>
> Now I'm working on a v2 patchset for VDUSE (vDPA Device in Userspace) 
> [1]. This tool is very useful for the vduse device. So I'm considering 
> integrating this into my v2 patchset. But there is one problem:
>
> In this tool, vdpa device config action and enable action are combined 
> into one netlink msg: VDPA_CMD_DEV_NEW. But in vduse case, it needs to 
> be splitted because a chardev should be created and opened by a 
> userspace process before we enable the vdpa device (call 
> vdpa_register_device()).
>
> So I'd like to know whether it's possible (or have some plans) to add 
> two new netlink msgs something like: VDPA_CMD_DEV_ENABLE and 
> VDPA_CMD_DEV_DISABLE to make the config path more flexible.
>

Actually, we've discussed such intermediate step in some early 
discussion. It looks to me VDUSE could be one of the users of this.

Or I wonder whether we can switch to use anonymous inode(fd) for VDUSE 
then fetching it via an VDUSE_GET_DEVICE_FD ioctl?

Thanks


> Thanks,
> Yongji
>
> [1] https://www.spinics.net/lists/linux-mm/msg231576.html


  parent reply index

Thread overview: 79+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-12  6:39 Parav Pandit
2020-11-12  6:39 ` [PATCH 1/7] vdpa: Add missing comment for virtqueue count Parav Pandit
2020-11-12  6:40 ` [PATCH 2/7] vdpa: Use simpler version of ida allocation Parav Pandit
2020-11-12  6:40 ` [PATCH 3/7] vdpa: Extend routine to accept vdpa device name Parav Pandit
2020-11-12  6:40 ` [PATCH 4/7] vdpa: Define vdpa parent device, ops and a netlink interface Parav Pandit
2020-11-12  6:40 ` [PATCH 5/7] vdpa: Enable a user to add and delete a vdpa device Parav Pandit
2020-11-12  6:40 ` [PATCH 6/7] vdpa: Enable user to query vdpa device info Parav Pandit
2020-11-12  6:40 ` [PATCH 7/7] vdpa/vdpa_sim: Enable user to create vdpasim net devices Parav Pandit
2020-11-16  9:41 ` [PATCH 0/7] Introduce vdpa management tool Stefan Hajnoczi
2020-11-17 19:41   ` Parav Pandit
2020-11-16 22:23 ` Jakub Kicinski
2020-11-17 19:51   ` Parav Pandit
2020-12-16  9:13     ` Michael S. Tsirkin
     [not found]       ` <20201216080610.08541f44@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com>
2020-12-16 16:54         ` Parav Pandit
2020-12-16 19:57           ` Michael S. Tsirkin
2020-12-17 12:13             ` Parav Pandit
2020-11-27  3:53 ` Jason Wang
     [not found]   ` <CACycT3sYScObb9nN3g7L3cesjE7sCZWxZ5_5R1usGU9ePZEeqA@mail.gmail.com>
2020-11-30  3:36     ` Jason Wang [this message]
2020-11-30  7:07       ` [External] " Yongji Xie
2020-12-01  6:25         ` Jason Wang
2020-12-01  9:55           ` Yongji Xie
2020-12-01 11:32             ` Parav Pandit
2020-12-01 14:18               ` Yongji Xie
2020-12-01 15:58                 ` Parav Pandit
2020-12-02  3:29                   ` Yongji Xie
2020-12-02  4:53                     ` Parav Pandit
2020-12-02  5:51                       ` Jason Wang
2020-12-02  6:24                         ` Parav Pandit
2020-12-02  7:55                           ` Jason Wang
2020-12-02  9:27                         ` Yongji Xie
2020-12-02  9:21                       ` Yongji Xie
2020-12-02 11:13                         ` Parav Pandit
2020-12-02 13:18                           ` Yongji Xie
2020-12-02  5:48             ` Jason Wang
2020-12-08 22:47   ` David Ahern
2021-01-19  4:21     ` Parav Pandit
2020-12-16  9:16 ` Michael S. Tsirkin
2021-01-04  3:31 ` [PATCH linux-next v2 " Parav Pandit
2021-01-04  3:31   ` [PATCH linux-next v2 1/7] vdpa_sim_net: Make mac address array static Parav Pandit
2021-01-04  7:00     ` Jason Wang
2021-01-04  3:31   ` [PATCH linux-next v2 2/7] vdpa_sim_net: Add module param to disable default vdpa net device Parav Pandit
2021-01-04  3:31   ` [PATCH linux-next v2 3/7] vdpa: Extend routine to accept vdpa device name Parav Pandit
2021-01-04  3:31   ` [PATCH linux-next v2 4/7] vdpa: Define vdpa mgmt device, ops and a netlink interface Parav Pandit
2021-01-04  7:03     ` Jason Wang
2021-01-04  7:24       ` Parav Pandit
2021-01-05  4:10         ` Jason Wang
2021-01-05  6:33           ` Parav Pandit
2021-01-05  8:36             ` Jason Wang
2021-01-04  3:31   ` [PATCH linux-next v2 5/7] vdpa: Enable a user to add and delete a vdpa device Parav Pandit
2021-01-04  3:31   ` [PATCH linux-next v2 6/7] vdpa: Enable user to query vdpa device info Parav Pandit
2021-01-04  3:31   ` [PATCH linux-next v2 7/7] vdpa_sim_net: Add support for user supported devices Parav Pandit
2021-01-04  7:05     ` Jason Wang
2021-01-04  7:21       ` Parav Pandit
2021-01-05  4:06         ` Jason Wang
2021-01-05  6:22           ` Parav Pandit
2021-01-05 10:31 ` [PATCH linux-next v3 0/6] Introduce vdpa management tool Parav Pandit
2021-01-05 10:31   ` [PATCH linux-next v3 1/6] vdpa_sim_net: Make mac address array static Parav Pandit
2021-01-07 13:45     ` Stefano Garzarella
2021-01-05 10:31   ` [PATCH linux-next v3 2/6] vdpa: Extend routine to accept vdpa device name Parav Pandit
2021-01-05 10:32   ` [PATCH linux-next v3 3/6] vdpa: Define vdpa mgmt device, ops and a netlink interface Parav Pandit
2021-01-05 10:32   ` [PATCH linux-next v3 4/6] vdpa: Enable a user to add and delete a vdpa device Parav Pandit
2021-01-05 10:32   ` [PATCH linux-next v3 5/6] vdpa: Enable user to query vdpa device info Parav Pandit
2021-01-05 10:32   ` [PATCH linux-next v3 6/6] vdpa_sim_net: Add support for user supported devices Parav Pandit
2021-01-05 11:48     ` Michael S. Tsirkin
2021-01-05 12:02       ` Parav Pandit
2021-01-05 12:14         ` Michael S. Tsirkin
2021-01-05 12:30           ` Parav Pandit
2021-01-05 13:23             ` Michael S. Tsirkin
2021-01-07  3:48               ` Parav Pandit
2021-01-12  4:14                 ` Parav Pandit
2021-01-14  4:17                 ` Jason Wang
2021-01-14  7:58                   ` Parav Pandit
2021-01-15  5:38                     ` Jason Wang
2021-01-15  6:27                       ` Parav Pandit
2021-01-19 11:09                         ` Jason Wang
2021-01-20  3:21                           ` Parav Pandit
2021-01-20  3:46                             ` Parav Pandit
2021-01-18 18:03                       ` Parav Pandit
2021-01-20  7:53                         ` Michael S. Tsirkin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=182708df-1082-0678-49b2-15d0199f20df@redhat.com \
    --to=jasowang@redhat.com \
    --cc=elic@nvidia.com \
    --cc=mst@redhat.com \
    --cc=netdev@vger.kernel.org \
    --cc=parav@nvidia.com \
    --cc=virtualization@lists.linux-foundation.org \
    --cc=xieyongji@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Netdev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/netdev/0 netdev/git/0.git
	git clone --mirror https://lore.kernel.org/netdev/1 netdev/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netdev netdev/ https://lore.kernel.org/netdev \
		netdev@vger.kernel.org
	public-inbox-index netdev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.netdev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git