From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F254C433ED for ; Wed, 14 Apr 2021 07:55:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 16F66601FE for ; Wed, 14 Apr 2021 07:55:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349204AbhDNHza (ORCPT ); Wed, 14 Apr 2021 03:55:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231713AbhDNHzY (ORCPT ); Wed, 14 Apr 2021 03:55:24 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99B16C061756 for ; Wed, 14 Apr 2021 00:55:03 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id e7so22480587edu.10 for ; Wed, 14 Apr 2021 00:55:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=fsahu37Y9/zL9oMkhbhBb4dx/NliRzgzOHM8isOmQ2c=; b=z2bybWOetPXB3jsL3na6wHIjbhGwOFP4FlYyvjARRQ1zuln9Lg+pL/kY6otUTEnO/g lmz71dHcX7tKWEiSE99JfBpy0QPoIkirIF0HR4SxhOuk6U5ekM+uU2utMNHwOx3cCbcq bCu1H5WIWPAi2qO/R+8sNYMXhnOVoGZsu+injmNkkTjSwg2SZp+RIgyPkSAgjGaugyob gEGTc8F9Bmf0SfFBYHjbLcueTejwnl3jiJTW2tF1CJ6lHx/ZbegmpfL7n+XHH9NGvEyA QB4Qq51ZR8J5qZqX8ZmsBgnY0lAv6Uis+Ylxce/zXDy/3QHC7GyNvd235IoyuHn9vUiM MXwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=fsahu37Y9/zL9oMkhbhBb4dx/NliRzgzOHM8isOmQ2c=; b=tYtJTCoYbgcquvZ26+0BVn2xJ5DT6JcJpKEwkluhCZAss18HMIbQy8Mun/rP9as8jJ L4deqgOAV59/EC+x8HqirUHkDNIWYD2O4FS3+s2y23wHa42Ov4xlx5qWqtfIlwzAfGuu eI7LBUO69Jzwx9BF6KZgvHoRRUh0Imq44VXLMQukLCWcAgIhFM43j+OpwvGS6nrZ2qj9 zA0WFA0xdPJu2FDRMiijYVgwLLHOsjEdBdK2LiLnvuwS794VG4HmeDsZ0oyRjCaxg05x +OBm0DkMYp6ydWVueFag+9CVFWYutzEzgMlYrZd3lRAoTJIIBp7Of9C/8xRdNznSEN3a Iy9Q== X-Gm-Message-State: AOAM533uqaGQgXBao0uTn7jVqhIYvuhYKjLQ2Ombb7RqcpyBUrFOLzKp 7tO2oWUygW5/0e3PEo7OXLijikjIyyh8NC3yZOJG X-Google-Smtp-Source: ABdhPJymgoRaZS3OHeEHzYqRFfSPikKe+INgr5Ie+JbVLw+hozOq1xXj6dKOG0EeaiMEyL5Q6pe0Af2kIWIy/wVn3cY= X-Received: by 2002:a05:6402:4314:: with SMTP id m20mr38627332edc.5.1618386902328; Wed, 14 Apr 2021 00:55:02 -0700 (PDT) MIME-Version: 1.0 References: <20210331080519.172-1-xieyongji@bytedance.com> <20210414032909-mutt-send-email-mst@kernel.org> In-Reply-To: <20210414032909-mutt-send-email-mst@kernel.org> From: Yongji Xie Date: Wed, 14 Apr 2021 15:54:51 +0800 Message-ID: Subject: Re: Re: [PATCH v6 00/10] Introduce VDUSE - vDPA Device in Userspace To: "Michael S. Tsirkin" Cc: Jason Wang , Stefan Hajnoczi , Stefano Garzarella , Parav Pandit , Christoph Hellwig , Christian Brauner , Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, Jens Axboe , bcrl@kvack.org, Jonathan Corbet , =?UTF-8?Q?Mika_Penttil=C3=A4?= , Dan Carpenter , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-fsdevel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, Apr 14, 2021 at 3:35 PM Michael S. Tsirkin wrote: > > On Wed, Mar 31, 2021 at 04:05:09PM +0800, Xie Yongji wrote: > > This series introduces a framework, which can be used to implement > > vDPA Devices in a userspace program. The work consist of two parts: > > control path forwarding and data path offloading. > > > > In the control path, the VDUSE driver will make use of message > > mechnism to forward the config operation from vdpa bus driver > > to userspace. Userspace can use read()/write() to receive/reply > > those control messages. > > > > In the data path, the core is mapping dma buffer into VDUSE > > daemon's address space, which can be implemented in different ways > > depending on the vdpa bus to which the vDPA device is attached. > > > > In virtio-vdpa case, we implements a MMU-based on-chip IOMMU driver wit= h > > bounce-buffering mechanism to achieve that. And in vhost-vdpa case, the= dma > > buffer is reside in a userspace memory region which can be shared to th= e > > VDUSE userspace processs via transferring the shmfd. > > > > The details and our user case is shown below: > > > > ------------------------ ------------------------- ---------------= ------------------------------- > > | Container | | QEMU(VM) | | = VDUSE daemon | > > | --------- | | ------------------- | | -------------= ------------ ---------------- | > > | |dev/vdx| | | |/dev/vhost-vdpa-x| | | | vDPA device= emulation | | block driver | | > > ------------+----------- -----------+------------ -------------+-= ---------------------+--------- > > | | | = | > > | | | = | > > ------------+---------------------------+----------------------------+-= ---------------------+--------- > > | | block device | | vhost device | | vduse dr= iver | | TCP/IP | | > > | -------+-------- --------+-------- -------+--= ------ -----+---- | > > | | | | = | | > > | ----------+---------- ----------+----------- -------+--= ----- | | > > | | virtio-blk driver | | vhost-vdpa driver | | vdpa dev= ice | | | > > | ----------+---------- ----------+----------- -------+--= ----- | | > > | | virtio bus | | = | | > > | --------+----+----------- | | = | | > > | | | | = | | > > | ----------+---------- | | = | | > > | | virtio-blk device | | | = | | > > | ----------+---------- | | = | | > > | | | | = | | > > | -----------+----------- | | = | | > > | | virtio-vdpa driver | | | = | | > > | -----------+----------- | | = | | > > | | | | = vdpa bus | | > > | -----------+----------------------+---------------------------+--= ---------- | | > > | = ---+--- | > > -----------------------------------------------------------------------= ------------------| NIC |------ > > = ---+--- > > = | > > = ---------+--------- > > = | Remote Storages | > > = ------------------- > > This all looks quite similar to vhost-user-block except that one > does not need any kernel support at all. > > So I am still scratching my head about its advantages over > vhost-user-block. > It plays the same role as vhost-user-block in VM user cases. > > > We make use of it to implement a block device connecting to > > our distributed storage, which can be used both in containers and > > VMs. Thus, we can have an unified technology stack in this two cases. > > Maybe the container part is the answer. How does that stack look? > Yes, it enables containers to reuse virtio software stack. We can have one daemon that provides service to both containers and virtual machines. Thanks=EF=BC=8C Yongji