From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09BDBC433DF for ; Tue, 20 Oct 2020 07:39:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0A9F4223FB for ; Tue, 20 Oct 2020 07:39:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="cHPActzS" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0A9F4223FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C45B06B005C; Tue, 20 Oct 2020 03:39:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BF4706B0062; Tue, 20 Oct 2020 03:39:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE4516B0068; Tue, 20 Oct 2020 03:39:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0223.hostedemail.com [216.40.44.223]) by kanga.kvack.org (Postfix) with ESMTP id 77AF96B005C for ; Tue, 20 Oct 2020 03:39:25 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1F0B8180AD811 for ; Tue, 20 Oct 2020 07:39:25 +0000 (UTC) X-FDA: 77391503490.26.fuel16_59165902723d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 001511804B656 for ; Tue, 20 Oct 2020 07:39:24 +0000 (UTC) X-HE-Tag: fuel16_59165902723d X-Filterd-Recvd-Size: 18487 Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) by imf15.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Oct 2020 07:39:24 +0000 (UTC) Received: by mail-ed1-f66.google.com with SMTP id dg9so781211edb.12 for ; Tue, 20 Oct 2020 00:39:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=tAn6bbjBQHHU7dd3tgEujx6tT9qWrDj/EGxG356oFbM=; b=cHPActzSON+pzIFfJ8DG7WD9+zoFjfGo+48XnuD7l7sRX3MaTil7YW44JXb6iX7CsC uwbnOXMUffQDS6lyHNaH+fk1ZQenbLRfEe3y0iSa4WXjM37PwgPNF2ak5lCufFc1cLt+ d5UJ5tVVBFbWzJ8YYBjY1sVY3wXgda9uTSobX3j99kXKWaNwRwgudCklqlC8TmUZzvAT a+FqWvLsx9QPgmgrL/wlZ3TD0jX3tHKD6FpB2snx+0yj8egPKbtrzM8DzCGsJGcnVqgR 0aoIuKIQkRDO/WtLnyExw70CdwHrYCO4lMMHWMzM2ROBI2O0hju6nX/NqUKHCOw5ruBb mpxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=tAn6bbjBQHHU7dd3tgEujx6tT9qWrDj/EGxG356oFbM=; b=KhVcd9R3Sru0VeMkPpnA2XDllrPmwgw5UlaNBB1CKlYYo92zGT6sKWj5RJrnSKZmEj sR6OHJOvYpm22PjG1gdzK0+YHGQzL8trb5L+TNqOZ3icdtkI/V9QYcelilwzysLH8bdo Gr6KHAcVAYNFJ0YPyUHDZtUO/B5xA8ExBs6kJWQL7DkJ2qD2TnnZeGCe0B3JLIr4g4Sz sajBb7zCXZZ6U9rRlwLLzwrqJqmqsT/BWA6V4AGl0uo0oJAHn8PzT5GL3zxP00fvYPfP fXJW1jybODVupPiEngCpMamMQdHizwhUZlLmsm8AdwIcJFgQrSIc+OQMVzoGkSzwG1i4 fVIA== X-Gm-Message-State: AOAM530N477HAWB03RqWu5vB/o5jPh5GRcg8so1H4WIc8NWeB+N+xBAU mPZIFskwl5fDvoCt1wLWl6IIvg1rP86o1GJoaElc X-Google-Smtp-Source: ABdhPJy+egjEm0Efrrp6YOm47EzNdZ4/Gvk1X7fAWrmtaQDvfDTdOTOd/TD3ont/DMbHzMHCtJFRMP22fJIqo87rAt8= X-Received: by 2002:a05:6402:17e4:: with SMTP id t4mr1478989edy.118.1603179562648; Tue, 20 Oct 2020 00:39:22 -0700 (PDT) MIME-Version: 1.0 References: <20201019145623.671-1-xieyongji@bytedance.com> In-Reply-To: From: Yongji Xie Date: Tue, 20 Oct 2020 15:39:11 +0800 Message-ID: Subject: Re: [External] Re: [RFC 0/4] Introduce VDUSE - vDPA Device in Userspace To: Jason Wang Cc: "Michael S. Tsirkin" , akpm@linux-foundation.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org Content-Type: multipart/alternative; boundary="00000000000057f56605b2155423" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --00000000000057f56605b2155423 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 20, 2020 at 11:20 AM Jason Wang wrote: > > On 2020/10/19 =E4=B8=8B=E5=8D=8810:56, Xie Yongji wrote: > > This series introduces a framework, which can be used to implement > > vDPA Devices in a userspace program. To implement it, the work > > consist of two parts: control path emulating and data path offloading. > > > > In the control path, the VDUSE driver will make use of message > > mechnism to forward the actions (get/set features, get/st status, > > get/set config space and set virtqueue states) from virtio-vdpa > > driver to userspace. Userspace can use read()/write() to > > receive/reply to those control messages. > > > > In the data path, the VDUSE driver implements a MMU-based > > on-chip IOMMU driver which supports both direct mapping and > > indirect mapping with bounce buffer. Then userspace can access > > those iova space via mmap(). Besides, eventfd mechnism is used to > > trigger interrupts and forward virtqueue kicks. > > > This is pretty interesting! > > For vhost-vdpa, it should work, but for virtio-vdpa, I think we should > carefully deal with the IOMMU/DMA ops stuffs. > I notice that neither dma_map nor set_map is implemented in > vduse_vdpa_config_ops, this means you want to let vhost-vDPA to deal > with IOMMU domains stuffs. Any reason for doing that? > > Actually, this series only focus on virtio-vdpa case now. To support vhost-vdpa, as you said, we need to implement dma_map/dma_unmap. But there is a limit that vm's memory can't be anonymous pages which are forbidden in vm_insert_page(). Maybe we need to add some limits on vhost-vdpa? > The reason for the questions are: > > 1) You've implemented a on-chip IOMMU driver but don't expose it to > generic IOMMU layer (or generic IOMMU layer may need some extension to > support this) > 2) We will probably remove the IOMMU domain management in vhost-vDPA, > and move it to the device(parent). > > So if it's possible, please implement either set_map() or > dma_map()/dma_unmap(), this may align with our future goal and may speed > up the development. > > Btw, it would be helpful to give even more details on how the on-chip > IOMMU driver in implemented. > The basic idea is treating MMU (VA->PA) as IOMMU (IOVA->PA). And using vm_insert_page()/zap_page_range() to do address mapping/unmapping. And the address mapping will be done in page fault handler because vm_insert_page() can't be called in atomic_context such as dma_map_ops->map_page(). > > > The details and our user case is shown below: > > > > ------------------------ > ----------------------------------------------------------- > > | APP | | QEMU > | > > | --------- | | -------------------- > -------------------+<-->+------ | > > | |dev/vdx| | | | device emulation | | virtio > dataplane | | BDS | | > > ------------+----------- > -----------+-----------------------+-----------------+----- > > | | | > | > > | | emulating | > offloading | > > > ------------+---------------------------+-----------------------+--------= ---------+------ > > | | block device | | vduse driver | | vdpa device > | | TCP/IP | | > > | -------+-------- --------+-------- > +------+------- -----+---- | > > | | | | | > | | > > | | | | | > | | > > | ----------+---------- ----------+----------- | | > | | > > | | virtio-blk driver | | virtio-vdpa driver | | | > | | > > | ----------+---------- ----------+----------- | | > | | > > | | | | | > | | > > | | ------------------ | > | | > > | ----------------------------------------------------- > ---+--- | > > > -------------------------------------------------------------------------= ----- > | NIC |--- > > > ---+--- > > > | > > > ---------+--------- > > > | Remote Storages | > > > ------------------- > > > The figure is not very clear to me in the following points: > > 1) if the device emulation and virtio dataplane is all implemented in > QEMU, what's the point of doing this? I thought the device should be a > remove process? 2) it would be better to draw a vDPA bus somewhere to help people to > understand the architecture > 3) for the "offloading" I guess it should be done virtio vhost-vDPA, so > it's better to draw a vhost-vDPA block there > > This figure only shows virtio-vdpa case, I will take vhost-vdpa case into consideration in next version. Thanks, Yongji --00000000000057f56605b2155423 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Oct 20, 2020 at 11:20 AM Jaso= n Wang <jasowang@redhat.com&g= t; wrote:

On 2020/10/19 =E4=B8=8B=E5=8D=8810:56, Xie Yongji wrote:
> This series introduces a framework, which can be used to implement
> vDPA Devices in a userspace program. To implement it, the work
> consist of two parts: control path emulating and data path offloading.=
>
> In the control path, the VDUSE driver will make use of message
> mechnism to forward the actions (get/set features, get/st status,
> get/set config space and set virtqueue states) from virtio-vdpa
> driver to userspace. Userspace can use read()/write() to
> receive/reply to those control messages.
>
> In the data path, the VDUSE driver implements a MMU-based
> on-chip IOMMU driver which supports both direct mapping and
> indirect mapping with bounce buffer. Then userspace can access
> those iova space via mmap(). Besides, eventfd mechnism is used to
> trigger interrupts and forward virtqueue kicks.


This is pretty interesting!

For vhost-vdpa, it should work, but for virtio-vdpa, I think we should
carefully deal with the IOMMU/DMA ops stuffs.
=C2=A0
I notice that neither dma_map nor set_map is implemented in
vduse_vdpa_config_ops, this means you want to let vhost-vDPA to deal
with IOMMU domains stuffs.=C2=A0 Any reason for doing that?

=C2=A0
Actually, this series=20 only focus on virtio-vdpa case now. To support vhost-vdpa,=C2=A0 as you sa= id, we need to implement dma_map/dma_unmap. But there is a limit that vm= 9;s memory can't be anonymous pages which are forbidden in vm_insert_pa= ge(). Maybe we need to add some limits on vhost-vdpa?
=C2=A0
The reason for the questions are:

1) You've implemented a on-chip IOMMU driver but don't expose it to=
generic IOMMU layer (or generic IOMMU layer may need some extension to
support this)
2) We will probably remove the IOMMU domain management in vhost-vDPA,
and move it to the device(parent).

So if it's possible, please implement either set_map() or
dma_map()/dma_unmap(), this may align with our future goal and may speed up the development.

Btw, it would be helpful to give even more details on how the on-chip
IOMMU driver in implemented.=C2=A0

The = basic idea is treating MMU (VA->PA) as IOMMU (IOVA->PA). And using vm= _insert_page()/zap_page_range() to do address mapping/unmapping. And the ad= dress mapping will be done in page fault handler because vm_insert_page() c= an't be called in atomic_context such as=C2=A0dma_map_ops->map_page(= ).

>
> The details and our user case is shown below:
>
> ------------------------=C2=A0 =C2=A0 =C2=A0--------------------------= ---------------------------------
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 APP |= =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 QEMU=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0---------=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2= =A0 =C2=A0| --------------------=C2=A0 =C2=A0 -------------------+<-->= ;+------ |
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0|dev/vdx|=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2= =A0 =C2=A0| | device emulation |=C2=A0 =C2=A0 | virtio dataplane |=C2=A0 = =C2=A0 | BDS | |
> ------------+-----------=C2=A0 =C2=A0 =C2=A0-----------+--------------= ---------+-----------------+-----
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0| emulating=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| offloading= =C2=A0 =C2=A0 =C2=A0 |
> ------------+---------------------------+-----------------------+-----= ------------+------
> |=C2=A0 =C2=A0 | block device |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 vduse driver |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 vdpa device |= =C2=A0 =C2=A0 | TCP/IP | |
> |=C2=A0 =C2=A0 -------+--------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0--------+--------=C2=A0 =C2=A0 =C2=A0 =C2=A0 +------+-------=C2=A0 =C2= =A0 =C2=A0-----+---- |
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0|
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0|
> | ----------+----------=C2=A0 =C2=A0 =C2=A0 =C2=A0----------+---------= --=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
> | | virtio-blk driver |=C2=A0 =C2=A0 =C2=A0 =C2=A0| virtio-vdpa driver= |=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
> | ----------+----------=C2=A0 =C2=A0 =C2=A0 =C2=A0----------+---------= --=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0|
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-----= -------------=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0----------------------------= -------------------------=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -= --+---=C2=A0 |
> ----------------------------------------------------------------------= -------- | NIC |---
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0---+--= -
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0---------+---------
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| Remote Storages |
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-------------------


The figure is not very clear to me in the following points:

1) if the device emulation and virtio dataplane is all implemented in
QEMU, what's the point of doing this? I thought the device should be a =
remove process?
2) it would be better to draw a vDPA bus somewhere to help people to
understand the architecture
3) for the "offloading" I guess it should be done virtio vhost-vD= PA, so
it's better to draw a vhost-vDPA block there


This figure only shows virtio-vdpa cas= e, I will take vhost-vdpa case into consideration=C2=A0in next version.

Thanks,
Yongji
--00000000000057f56605b2155423--