From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 669C9C433DB for ; Sun, 10 Jan 2021 10:05:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 34094238E7 for ; Sun, 10 Jan 2021 10:05:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726250AbhAJKEr (ORCPT ); Sun, 10 Jan 2021 05:04:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725907AbhAJKEq (ORCPT ); Sun, 10 Jan 2021 05:04:46 -0500 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B445AC061786 for ; Sun, 10 Jan 2021 02:04:04 -0800 (PST) Received: by mail-ej1-x632.google.com with SMTP id lt17so20555592ejb.3 for ; Sun, 10 Jan 2021 02:04:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ntHLah0q+SvwKSjeDWR+MVFLLttqY1N89uUPbqPMpiA=; b=ZGwx+oiiGqvNDB+EaNh7KH6pXXDJEN6Q1ZH/kdatn05h3TURc1jdAQPJK2Hd1dt3y9 oX9qanMfO/pXUBOO5CoyFgf6lRFCoVfxQ3mFPJHZnlvUdkDjlt0GIj245r6osobCwOlt rV7pJn8gsCpRiAAnmMLALjrtwokfqMai7FP927XnZ5UzVr16916CyigDjon5TNvYTytQ cn9l6tUn9hlp6zO8JVbySRHAx75MxIw0hNpWAiOdFrdHTMSmNqGtIFyqUALoWPUJdkEf eeYy6crslJzT0GNwXexlvaM5HjhSavw0e4FhTYttSXZE+kdUSJspWEvV5UqgUW+OpJNe OdgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ntHLah0q+SvwKSjeDWR+MVFLLttqY1N89uUPbqPMpiA=; b=MZDSSAZ/MDTaZvloNhbwwKEa6ecm3rur45PqJbtlrefaMMGR9dy1b4ce9dZl8ib1uG cvbN9R/yPEz/STWHFtRJDt7J4U61M0hWtAfoIAMbkLrpl/oxkzJ5khPmaQF+3CEs2d1A K82dSPNHPSPiyEbJGbDw9UGxEkTreC0NYKs70RsWmhn0x83aUDuMamQ4Y8Z8G9Tu7O+y zhdhNJqdkh9Dq6xJRx9xOsd5vMxoYuzPBjDga6J40hG4Deps8KBthFnRx0k/bENGoZaE gH5/TEJqCQBIl3YZPkXoXqDrJnT2wKqFwGVNenfQGkwI61JFrR7HGZIk7m7HDjNSQ+uN Z7zg== X-Gm-Message-State: AOAM531gTplmjAMcGNnRZ6ec9JHj4vT02MSkTOPEEPMym8GEMUTUzMVQ dwyw+sj1uy7poXo8HK+kxIAjTUxVWukV7tssSm91 X-Google-Smtp-Source: ABdhPJxxW/yAqeYuRuPessHdacKwHtFahI0Eg6d90J6rPXonweRKhqtq6PPHcOf9f2M6pT+y8/HK7s5rhlXTDDVOoNY= X-Received: by 2002:a17:907:1629:: with SMTP id hb41mr7211587ejc.197.1610273043284; Sun, 10 Jan 2021 02:04:03 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <20201222145221.711-7-xieyongji@bytedance.com> In-Reply-To: From: Yongji Xie Date: Sun, 10 Jan 2021 18:03:52 +0800 Message-ID: Subject: Re: Re: [RFC v2 06/13] vduse: Introduce VDUSE - vDPA Device in Userspace To: Bob Liu Cc: "Michael S. Tsirkin" , Jason Wang , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Fri, Jan 8, 2021 at 9:32 PM Bob Liu wrote: > > On 12/22/20 10:52 PM, Xie Yongji wrote: > > This VDUSE driver enables implementing vDPA devices in userspace. > > Both control path and data path of vDPA devices will be able to > > be handled in userspace. > > > > In the control path, the VDUSE driver will make use of message > > mechnism to forward the config operation from vdpa bus driver > > to userspace. Userspace can use read()/write() to receive/reply > > those control messages. > > > > In the data path, the VDUSE driver implements a MMU-based on-chip > > IOMMU driver which supports mapping the kernel dma buffer to a > > userspace iova region dynamically. Userspace can access those > > iova region via mmap(). Besides, the eventfd mechanism is used to > > trigger interrupt callbacks and receive virtqueue kicks in userspace > > > > Now we only support virtio-vdpa bus driver with this patch applied. > > > > Signed-off-by: Xie Yongji > > --- > > Documentation/driver-api/vduse.rst | 74 ++ > > Documentation/userspace-api/ioctl/ioctl-number.rst | 1 + > > drivers/vdpa/Kconfig | 8 + > > drivers/vdpa/Makefile | 1 + > > drivers/vdpa/vdpa_user/Makefile | 5 + > > drivers/vdpa/vdpa_user/eventfd.c | 221 ++++ > > drivers/vdpa/vdpa_user/eventfd.h | 48 + > > drivers/vdpa/vdpa_user/iova_domain.c | 442 ++++++++ > > drivers/vdpa/vdpa_user/iova_domain.h | 93 ++ > > drivers/vdpa/vdpa_user/vduse.h | 59 ++ > > drivers/vdpa/vdpa_user/vduse_dev.c | 1121 ++++++++++++++++++++ > > include/uapi/linux/vdpa.h | 1 + > > include/uapi/linux/vduse.h | 99 ++ > > 13 files changed, 2173 insertions(+) > > create mode 100644 Documentation/driver-api/vduse.rst > > create mode 100644 drivers/vdpa/vdpa_user/Makefile > > create mode 100644 drivers/vdpa/vdpa_user/eventfd.c > > create mode 100644 drivers/vdpa/vdpa_user/eventfd.h > > create mode 100644 drivers/vdpa/vdpa_user/iova_domain.c > > create mode 100644 drivers/vdpa/vdpa_user/iova_domain.h > > create mode 100644 drivers/vdpa/vdpa_user/vduse.h > > create mode 100644 drivers/vdpa/vdpa_user/vduse_dev.c > > create mode 100644 include/uapi/linux/vduse.h > > > > diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-api/vduse.rst > > new file mode 100644 > > index 000000000000..da9b3040f20a > > --- /dev/null > > +++ b/Documentation/driver-api/vduse.rst > > @@ -0,0 +1,74 @@ > > +================================== > > +VDUSE - "vDPA Device in Userspace" > > +================================== > > + > > +vDPA (virtio data path acceleration) device is a device that uses a > > +datapath which complies with the virtio specifications with vendor > > +specific control path. vDPA devices can be both physically located on > > +the hardware or emulated by software. VDUSE is a framework that makes it > > +possible to implement software-emulated vDPA devices in userspace. > > + > > Could you explain a bit more why need a VDUSE framework? This can be used to implement a userspace I/O (such as storage, network and so on) solution (virtio-based) for both container and VM. > Software emulated vDPA devices is more likely used by debugging only when > don't have real hardware. I think software emulated vDPA devices should be also useful in other cases, just like FUSE does. > Do you think do the emulation in kernel space is not enough? > Doing the emulation in userspace should be more flexible. Thanks, Yongji From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CAE3C433E0 for ; Sun, 10 Jan 2021 10:04:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DAB6423108 for ; Sun, 10 Jan 2021 10:04:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DAB6423108 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F3CF16B00AC; Sun, 10 Jan 2021 05:04:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F12746B009E; Sun, 10 Jan 2021 05:04:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3F5E8D0017; Sun, 10 Jan 2021 05:04:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0050.hostedemail.com [216.40.44.50]) by kanga.kvack.org (Postfix) with ESMTP id BF33F6B009E for ; Sun, 10 Jan 2021 05:04:05 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7B1332C18 for ; Sun, 10 Jan 2021 10:04:05 +0000 (UTC) X-FDA: 77689429650.20.feet90_300b6db27503 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 5B606180C07A3 for ; Sun, 10 Jan 2021 10:04:05 +0000 (UTC) X-HE-Tag: feet90_300b6db27503 X-Filterd-Recvd-Size: 6956 Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Sun, 10 Jan 2021 10:04:04 +0000 (UTC) Received: by mail-ej1-f53.google.com with SMTP id 6so20494816ejz.5 for ; Sun, 10 Jan 2021 02:04:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ntHLah0q+SvwKSjeDWR+MVFLLttqY1N89uUPbqPMpiA=; b=ZGwx+oiiGqvNDB+EaNh7KH6pXXDJEN6Q1ZH/kdatn05h3TURc1jdAQPJK2Hd1dt3y9 oX9qanMfO/pXUBOO5CoyFgf6lRFCoVfxQ3mFPJHZnlvUdkDjlt0GIj245r6osobCwOlt rV7pJn8gsCpRiAAnmMLALjrtwokfqMai7FP927XnZ5UzVr16916CyigDjon5TNvYTytQ cn9l6tUn9hlp6zO8JVbySRHAx75MxIw0hNpWAiOdFrdHTMSmNqGtIFyqUALoWPUJdkEf eeYy6crslJzT0GNwXexlvaM5HjhSavw0e4FhTYttSXZE+kdUSJspWEvV5UqgUW+OpJNe OdgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ntHLah0q+SvwKSjeDWR+MVFLLttqY1N89uUPbqPMpiA=; b=Gf1qEh7BZvdOSHGgEVa+3yV4codknsh30ghMPlSJaEmlRx6nUw+ocSAoeaVeS3gAzX 6M4snuBsEL8Uva5bwrZ8c6Zso7bBjfB4tkrNt3JoAm06o0MV0UXRU8qqn3/3RquD1k+B IOViuHm39Fr/nm8dhPyKCM3i0OAzXhl8oPPD/fW9imCgj8GN+az7tdXnSYYLA7e2DZDU davV1aPBuEkzE0EiggnGGfaATQ98MplzyQsyt4MHZZDDrJ4SM+XzyVNS8xfyQaEEtMI0 1xTXJ37FC1HRYmd6MjN5s4SDt6kiba34dhw6+hQqKIZkiXcG9ZYzcFQxiMTSYKGtRn+3 13lg== X-Gm-Message-State: AOAM532xsH32eOHFEtSAtbj0vjPnkbcOeXpgQqkbKgzc3WrUnWLPOOpE ioNOhfDI1Jd609PcWos5y0FtRiHbNz7QHgRj39DZ X-Google-Smtp-Source: ABdhPJxxW/yAqeYuRuPessHdacKwHtFahI0Eg6d90J6rPXonweRKhqtq6PPHcOf9f2M6pT+y8/HK7s5rhlXTDDVOoNY= X-Received: by 2002:a17:907:1629:: with SMTP id hb41mr7211587ejc.197.1610273043284; Sun, 10 Jan 2021 02:04:03 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <20201222145221.711-7-xieyongji@bytedance.com> In-Reply-To: From: Yongji Xie Date: Sun, 10 Jan 2021 18:03:52 +0800 Message-ID: Subject: Re: Re: [RFC v2 06/13] vduse: Introduce VDUSE - vDPA Device in Userspace To: Bob Liu Cc: "Michael S. Tsirkin" , Jason Wang , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 8, 2021 at 9:32 PM Bob Liu wrote: > > On 12/22/20 10:52 PM, Xie Yongji wrote: > > This VDUSE driver enables implementing vDPA devices in userspace. > > Both control path and data path of vDPA devices will be able to > > be handled in userspace. > > > > In the control path, the VDUSE driver will make use of message > > mechnism to forward the config operation from vdpa bus driver > > to userspace. Userspace can use read()/write() to receive/reply > > those control messages. > > > > In the data path, the VDUSE driver implements a MMU-based on-chip > > IOMMU driver which supports mapping the kernel dma buffer to a > > userspace iova region dynamically. Userspace can access those > > iova region via mmap(). Besides, the eventfd mechanism is used to > > trigger interrupt callbacks and receive virtqueue kicks in userspace > > > > Now we only support virtio-vdpa bus driver with this patch applied. > > > > Signed-off-by: Xie Yongji > > --- > > Documentation/driver-api/vduse.rst | 74 ++ > > Documentation/userspace-api/ioctl/ioctl-number.rst | 1 + > > drivers/vdpa/Kconfig | 8 + > > drivers/vdpa/Makefile | 1 + > > drivers/vdpa/vdpa_user/Makefile | 5 + > > drivers/vdpa/vdpa_user/eventfd.c | 221 ++++ > > drivers/vdpa/vdpa_user/eventfd.h | 48 + > > drivers/vdpa/vdpa_user/iova_domain.c | 442 ++++++++ > > drivers/vdpa/vdpa_user/iova_domain.h | 93 ++ > > drivers/vdpa/vdpa_user/vduse.h | 59 ++ > > drivers/vdpa/vdpa_user/vduse_dev.c | 1121 ++++++++++++++++++++ > > include/uapi/linux/vdpa.h | 1 + > > include/uapi/linux/vduse.h | 99 ++ > > 13 files changed, 2173 insertions(+) > > create mode 100644 Documentation/driver-api/vduse.rst > > create mode 100644 drivers/vdpa/vdpa_user/Makefile > > create mode 100644 drivers/vdpa/vdpa_user/eventfd.c > > create mode 100644 drivers/vdpa/vdpa_user/eventfd.h > > create mode 100644 drivers/vdpa/vdpa_user/iova_domain.c > > create mode 100644 drivers/vdpa/vdpa_user/iova_domain.h > > create mode 100644 drivers/vdpa/vdpa_user/vduse.h > > create mode 100644 drivers/vdpa/vdpa_user/vduse_dev.c > > create mode 100644 include/uapi/linux/vduse.h > > > > diff --git a/Documentation/driver-api/vduse.rst b/Documentation/driver-api/vduse.rst > > new file mode 100644 > > index 000000000000..da9b3040f20a > > --- /dev/null > > +++ b/Documentation/driver-api/vduse.rst > > @@ -0,0 +1,74 @@ > > +================================== > > +VDUSE - "vDPA Device in Userspace" > > +================================== > > + > > +vDPA (virtio data path acceleration) device is a device that uses a > > +datapath which complies with the virtio specifications with vendor > > +specific control path. vDPA devices can be both physically located on > > +the hardware or emulated by software. VDUSE is a framework that makes it > > +possible to implement software-emulated vDPA devices in userspace. > > + > > Could you explain a bit more why need a VDUSE framework? This can be used to implement a userspace I/O (such as storage, network and so on) solution (virtio-based) for both container and VM. > Software emulated vDPA devices is more likely used by debugging only when > don't have real hardware. I think software emulated vDPA devices should be also useful in other cases, just like FUSE does. > Do you think do the emulation in kernel space is not enough? > Doing the emulation in userspace should be more flexible. Thanks, Yongji