From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5488BC433E7 for ; Tue, 20 Oct 2020 08:35:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 679AB2222F for ; Tue, 20 Oct 2020 08:35:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=bytedance-com.20150623.gappssmtp.com header.i=@bytedance-com.20150623.gappssmtp.com header.b="m74Zjga8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 679AB2222F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6835F6B005C; Tue, 20 Oct 2020 04:35:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 60BA96B0062; Tue, 20 Oct 2020 04:35:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AB056B0068; Tue, 20 Oct 2020 04:35:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 0912B6B005C for ; Tue, 20 Oct 2020 04:35:45 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9666F82E4E8B for ; Tue, 20 Oct 2020 08:35:45 +0000 (UTC) X-FDA: 77391645450.20.drain08_5509e422723e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 777F3180C07AB for ; Tue, 20 Oct 2020 08:35:45 +0000 (UTC) X-HE-Tag: drain08_5509e422723e X-Filterd-Recvd-Size: 23092 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 20 Oct 2020 08:35:44 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id dg9so938574edb.12 for ; Tue, 20 Oct 2020 01:35:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6x/uAdbyQOPjCt9JN+2Ip/DQEJ9ut+rXyYw55Ht8xeo=; b=m74Zjga8W+BsaSqiYaAHmYVBN2wHKkptZNNyxWSfNrll27EHPGyNEDwg+9QMf1o91v stEE1YAdM0jGPVz1RhLWtOvMbX3VGM6cJNO41jx7K4wPD6mwd0Jte4GpOCXfvTzwzkcL m/j46X8DqRHKsGv3Xh2jB/HiJmNRJ0Pkd4ayTBM94d/BSHwsehyMQJqQxZFXK/YZ4URw cVYnNjg77IPhWApBELh/WlHaVdW2ysqJfr7VUR0IIXeLTcMjHXyTTSk8WbPCDa2pPaj7 Wj9fhkSLlc9O3D3LdbK9qEtKgqFK36vNpnpZHyYS/ZrIAhETLCkheGfWXO9uWl8+x1Gx ArdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6x/uAdbyQOPjCt9JN+2Ip/DQEJ9ut+rXyYw55Ht8xeo=; b=E0yy4lJJWEOT2zpVsPP29lBv6dIWCeuJIylsEFk57K7aFg7BlcbOLXgfl2Pjg/WSIj 3e/iOCyIFDjHeENbXMAf+kcHOEfGSLUm8bg/D/SUyN1jzoLTQkJ5tr4Wy9eWPUXWkbdc hO315D5l6TNsF+fGiJRAwyJSl4VoXcqDmAtj0FLmIw6+xFPNp/YRh3ZE/FZOfZ1EPbBO vf53Au1xI6z6JodKScXGdY5jExgm+gosLXGDVl9Njc5Iy6JUnbRXVCbD6tBYJDFgEX7f hR2JXB9LXh/cwortzguGTVe7Lneop3XkPKytPgUhP9cb/SCkSd1pXadpCEbcoi3MYPA+ nUTw== X-Gm-Message-State: AOAM53098/oODV41Q5vrXwYHZbR5VKh8on4+1SopJITLptn/jZ1iQgxC C4NhEk75ImtCSbrG6lbu08sIy6r1Ib7esiTJ1fGO X-Google-Smtp-Source: ABdhPJwjsfMUkZz8u94o4YCn/D/FUgucEEzOgA1vggfMkSYtX1gX3r079uSfg8q/QMieAPhHA1KkqPn9PGbHh712opM= X-Received: by 2002:a50:ef0e:: with SMTP id m14mr1671216eds.5.1603182943420; Tue, 20 Oct 2020 01:35:43 -0700 (PDT) MIME-Version: 1.0 References: <20201019145623.671-1-xieyongji@bytedance.com> In-Reply-To: From: Yongji Xie Date: Tue, 20 Oct 2020 16:35:32 +0800 Message-ID: Subject: Re: [External] Re: [RFC 0/4] Introduce VDUSE - vDPA Device in Userspace To: Jason Wang Cc: "Michael S. Tsirkin" , akpm@linux-foundation.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org Content-Type: multipart/alternative; boundary="000000000000da62ad05b2161df7" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --000000000000da62ad05b2161df7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Oct 20, 2020 at 4:01 PM Jason Wang wrote: > > On 2020/10/20 =E4=B8=8B=E5=8D=883:39, Yongji Xie wrote: > > > > > > On Tue, Oct 20, 2020 at 11:20 AM Jason Wang > > wrote: > > > > > > On 2020/10/19 =E4=B8=8B=E5=8D=8810:56, Xie Yongji wrote: > > > This series introduces a framework, which can be used to implemen= t > > > vDPA Devices in a userspace program. To implement it, the work > > > consist of two parts: control path emulating and data path > > offloading. > > > > > > In the control path, the VDUSE driver will make use of message > > > mechnism to forward the actions (get/set features, get/st status, > > > get/set config space and set virtqueue states) from virtio-vdpa > > > driver to userspace. Userspace can use read()/write() to > > > receive/reply to those control messages. > > > > > > In the data path, the VDUSE driver implements a MMU-based > > > on-chip IOMMU driver which supports both direct mapping and > > > indirect mapping with bounce buffer. Then userspace can access > > > those iova space via mmap(). Besides, eventfd mechnism is used to > > > trigger interrupts and forward virtqueue kicks. > > > > > > This is pretty interesting! > > > > For vhost-vdpa, it should work, but for virtio-vdpa, I think we > > should > > carefully deal with the IOMMU/DMA ops stuffs. > > > > > > I notice that neither dma_map nor set_map is implemented in > > vduse_vdpa_config_ops, this means you want to let vhost-vDPA to dea= l > > with IOMMU domains stuffs. Any reason for doing that? > > > > Actually, this series only focus on virtio-vdpa case now. To support > > vhost-vdpa, as you said, we need to implement dma_map/dma_unmap. But > > there is a limit that vm's memory can't be anonymous pages which are > > forbidden in vm_insert_page(). Maybe we need to add some limits on > > vhost-vdpa? > > > I'm not sure I get this, any reason that you want to use > vm_insert_page() to VM's memory. Or do you mean you want to implement > some kind of zero-copy? > If my understanding is right, we will have a QEMU (VM) process and a device emulation process in the vhost-vdpa case, right? When I/O happens, the virtio driver in VM will put the IOVA to vring and device emulation process will get the IOVA from vring. Then the device emulation process will translate the IOVA to its VA to access the dma buffer which resides in VM's memory. That means the device emulation process needs to access VM's memory, so we should use vm_insert_page() to build the page table of the device emulation process. I guess from the software device implemention in user space it only need > to receive IOVA ranges and map them in its own address space. How to map them in its own address space if we don't use vm_insert_page()? > > > The reason for the questions are: > > > > 1) You've implemented a on-chip IOMMU driver but don't expose it to > > generic IOMMU layer (or generic IOMMU layer may need some > > extension to > > support this) > > 2) We will probably remove the IOMMU domain management in vhost-vDP= A, > > and move it to the device(parent). > > > > So if it's possible, please implement either set_map() or > > dma_map()/dma_unmap(), this may align with our future goal and may > > speed > > up the development. > > > > Btw, it would be helpful to give even more details on how the on-ch= ip > > IOMMU driver in implemented. > > > > > > The basic idea is treating MMU (VA->PA) as IOMMU (IOVA->PA). And using > > vm_insert_page()/zap_page_range() to do address mapping/unmapping. And > > the address mapping will be done in page fault handler because > > vm_insert_page() can't be called in atomic_context such > > as dma_map_ops->map_page(). > > > Ok, please add it in the cover letter or patch 2 in the next version. > > > > > > > > > The details and our user case is shown below: > > > > > > ------------------------ > > ----------------------------------------------------------- > > > | APP | | QEMU | > > > | --------- | | -------------------- > > -------------------+<-->+------ | > > > | |dev/vdx| | | | device emulation | | virtio > > dataplane | | BDS | | > > > ------------+----------- > > -----------+-----------------------+-----------------+----- > > > | | | > > | > > > | | emulating | > > offloading | > > > > > > ------------+---------------------------+-----------------------+-------= ----------+------ > > > | | block device | | vduse driver | | vdpa > > device | | TCP/IP | | > > > | -------+-------- --------+-------- > > +------+------- -----+---- | > > > | | | | | > > | | > > > | | | | | > > | | > > > | ----------+---------- ----------+----------- | | > > | | > > > | | virtio-blk driver | | virtio-vdpa driver | | | > > | | > > > | ----------+---------- ----------+----------- | | > > | | > > > | | | | | > > | | > > > | | ------------------ | | | > > > | ----------------------------------------------------- > > ---+--- | > > > > > > ------------------------------------------------------------------------= ------ > > | NIC |--- > > > ---+--- > > > | > > > ---------+--------- > > > | Remote Storages | > > > ------------------- > > > > > > The figure is not very clear to me in the following points: > > > > 1) if the device emulation and virtio dataplane is all implemented = in > > QEMU, what's the point of doing this? I thought the device should > > be a > > remove process? > > > > 2) it would be better to draw a vDPA bus somewhere to help people t= o > > understand the architecture > > 3) for the "offloading" I guess it should be done virtio > > vhost-vDPA, so > > it's better to draw a vhost-vDPA block there > > > > > > This figure only shows virtio-vdpa case, I will take vhost-vdpa case > > into consideration in next version. > > > Please do that, otherwise this proposal is incomplete. > > Sure. Thanks, Yongji --000000000000da62ad05b2161df7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, Oct 20, 2020 at 4:01 PM Jason= Wang <jasowang@redhat.com>= ; wrote:

On 2020/10/20 =E4=B8=8B=E5=8D=883:39, Yongji Xie wrote:
>
>
> On Tue, Oct 20, 2020 at 11:20 AM Jason Wang <jasowang@redhat.com
> <mailto:ja= sowang@redhat.com>> wrote:
>
>
>=C2=A0 =C2=A0 =C2=A0On 2020/10/19 =E4=B8=8B=E5=8D=8810:56, Xie Yongji w= rote:
>=C2=A0 =C2=A0 =C2=A0> This series introduces a framework, which can = be used to implement
>=C2=A0 =C2=A0 =C2=A0> vDPA Devices in a userspace program. To implem= ent it, the work
>=C2=A0 =C2=A0 =C2=A0> consist of two parts: control path emulating a= nd data path
>=C2=A0 =C2=A0 =C2=A0offloading.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> In the control path, the VDUSE driver will mak= e use of message
>=C2=A0 =C2=A0 =C2=A0> mechnism to forward the actions (get/set featu= res, get/st status,
>=C2=A0 =C2=A0 =C2=A0> get/set config space and set virtqueue states)= from virtio-vdpa
>=C2=A0 =C2=A0 =C2=A0> driver to userspace. Userspace can use read()/= write() to
>=C2=A0 =C2=A0 =C2=A0> receive/reply to those control messages.
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> In the data path, the VDUSE driver implements = a MMU-based
>=C2=A0 =C2=A0 =C2=A0> on-chip IOMMU driver which supports both direc= t mapping and
>=C2=A0 =C2=A0 =C2=A0> indirect mapping with bounce buffer. Then user= space can access
>=C2=A0 =C2=A0 =C2=A0> those iova space via mmap(). Besides, eventfd = mechnism is used to
>=C2=A0 =C2=A0 =C2=A0> trigger interrupts and forward virtqueue kicks= .
>
>
>=C2=A0 =C2=A0 =C2=A0This is pretty interesting!
>
>=C2=A0 =C2=A0 =C2=A0For vhost-vdpa, it should work, but for virtio-vdpa= , I think we
>=C2=A0 =C2=A0 =C2=A0should
>=C2=A0 =C2=A0 =C2=A0carefully deal with the IOMMU/DMA ops stuffs.
>
>
>=C2=A0 =C2=A0 =C2=A0I notice that neither dma_map nor set_map is implem= ented in
>=C2=A0 =C2=A0 =C2=A0vduse_vdpa_config_ops, this means you want to let v= host-vDPA to deal
>=C2=A0 =C2=A0 =C2=A0with IOMMU domains stuffs.=C2=A0 Any reason for doi= ng that?
>
> Actually, this series only focus on virtio-vdpa case now. To support <= br> > vhost-vdpa,=C2=A0 as you said, we need to implement dma_map/dma_unmap.= But
> there is a limit that vm's memory can't be anonymous pages whi= ch are
> forbidden in vm_insert_page(). Maybe we need to add some limits on > vhost-vdpa?


I'm not sure I get this, any reason that you want to use
vm_insert_page() to VM's memory. Or do you mean you want to implement <= br> some kind of zero-copy?=C2=A0
=C2=A0

If my understan= ding is right, we will have a QEMU (VM) process and a device emulation proc= ess in the vhost-vdpa case, right? When I/O happens, the virtio driver in V= M will put the IOVA to vring and device emulation process will get the IOVA= from vring. Then the device emulation process will=C2=A0translate the IOVA= to its VA to access the dma buffer which resides in VM's memory. That = means the device emulation process needs to access VM's=C2=A0memory, so= we should use vm_insert_page() to build the page table of the device emulation process.

I guess from the software device implemention in user space it only need to receive IOVA ranges and map them in its own address space.<= div>
How to map them in its own address space if we don't= use vm_insert_page()?

=C2=A0

>=C2=A0 =C2=A0 =C2=A0The reason for the questions are:
>
>=C2=A0 =C2=A0 =C2=A01) You've implemented a on-chip IOMMU driver bu= t don't expose it to
>=C2=A0 =C2=A0 =C2=A0generic IOMMU layer (or generic IOMMU layer may nee= d some
>=C2=A0 =C2=A0 =C2=A0extension to
>=C2=A0 =C2=A0 =C2=A0support this)
>=C2=A0 =C2=A0 =C2=A02) We will probably remove the IOMMU domain managem= ent in vhost-vDPA,
>=C2=A0 =C2=A0 =C2=A0and move it to the device(parent).
>
>=C2=A0 =C2=A0 =C2=A0So if it's possible, please implement either se= t_map() or
>=C2=A0 =C2=A0 =C2=A0dma_map()/dma_unmap(), this may align with our futu= re goal and may
>=C2=A0 =C2=A0 =C2=A0speed
>=C2=A0 =C2=A0 =C2=A0up the development.
>
>=C2=A0 =C2=A0 =C2=A0Btw, it would be helpful to give even more details = on how the on-chip
>=C2=A0 =C2=A0 =C2=A0IOMMU driver in implemented.
>
>
> The basic idea is treating MMU (VA->PA) as IOMMU (IOVA->PA). And= using
> vm_insert_page()/zap_page_range() to do address mapping/unmapping. And=
> the address mapping will be done in page fault handler because
> vm_insert_page() can't be called in atomic_context such
> as=C2=A0dma_map_ops->map_page().


Ok, please add it in the cover letter or patch 2 in the next version.

>
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> The details and our user case is shown below:<= br> >=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0> ------------------------
>=C2=A0 =C2=A0 =C2=A0=C2=A0---------------------------------------------= --------------
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 APP |=C2=A0 =C2=A0 =C2=A0| QEMU=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<= br> >=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0---------=C2=A0 = =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0| --------------------
>=C2=A0 =C2=A0 =C2=A0-------------------+<-->+------ |
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0|dev/vdx|=C2=A0 = =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0| | device emulation | | virtio
>=C2=A0 =C2=A0 =C2=A0dataplane |=C2=A0 =C2=A0 | BDS | |
>=C2=A0 =C2=A0 =C2=A0> ------------+-----------
>=C2=A0 =C2=A0 =C2=A0=C2=A0-----------+-----------------------+---------= --------+-----
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0| emulating =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<= br> >=C2=A0 =C2=A0 =C2=A0offloading=C2=A0 =C2=A0 =C2=A0 |
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0------------+---------------------------+----------= -------------+-----------------+------
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 | block device |=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 vduse driver | =C2=A0 |=C2=A0 vdpa
>=C2=A0 =C2=A0 =C2=A0device |=C2=A0 =C2=A0 | TCP/IP | |
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 -------+--------=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0--------+-------- =C2=A0
>=C2=A0 =C2=A0 =C2=A0+------+-------=C2=A0 =C2=A0 =C2=A0-----+---- |
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0| =C2=A0 |=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0| =C2=A0 |=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> | ----------+----------=C2=A0 =C2=A0 =C2=A0 = =C2=A0----------+----------- =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> | | virtio-blk driver |=C2=A0 =C2=A0 =C2=A0 = =C2=A0| virtio-vdpa driver | =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> | ----------+----------=C2=A0 =C2=A0 =C2=A0 = =C2=A0----------+----------- =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0| =C2=A0 |=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| = =C2=A0------------------=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0|
>=C2=A0 =C2=A0 =C2=A0> | =C2=A0--------------------------------------= --------------- =C2=A0 =C2=A0 =C2=A0 =C2=A0
>=C2=A0 =C2=A0 =C2=A0---+---=C2=A0 |
>=C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0---------------------------------------------------= ---------------------------
>=C2=A0 =C2=A0 =C2=A0| NIC |---
>=C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0---+---
>=C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
>=C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0---------+---------
>=C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0| Remote Storages |
>=C2=A0 =C2=A0 =C2=A0> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0-------------------
>
>
>=C2=A0 =C2=A0 =C2=A0The figure is not very clear to me in the following= points:
>
>=C2=A0 =C2=A0 =C2=A01) if the device emulation and virtio dataplane is = all implemented in
>=C2=A0 =C2=A0 =C2=A0QEMU, what's the point of doing this? I thought= the device should
>=C2=A0 =C2=A0 =C2=A0be a
>=C2=A0 =C2=A0 =C2=A0remove process?
>
>=C2=A0 =C2=A0 =C2=A02) it would be better to draw a vDPA bus somewhere = to help people to
>=C2=A0 =C2=A0 =C2=A0understand the architecture
>=C2=A0 =C2=A0 =C2=A03) for the "offloading" I guess it should= be done virtio
>=C2=A0 =C2=A0 =C2=A0vhost-vDPA, so
>=C2=A0 =C2=A0 =C2=A0it's better to draw a vhost-vDPA block there >
>
> This figure only shows virtio-vdpa case, I will take vhost-vdpa case <= br> > into consideration=C2=A0in next version.


Please do that, otherwise this proposal is incomplete.

=

Sure.

Thanks,
Yongji=
--000000000000da62ad05b2161df7--