From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C302EC433DB for ; Wed, 30 Dec 2020 10:13:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6DD39221FA for ; Wed, 30 Dec 2020 10:13:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726318AbgL3KNe (ORCPT ); Wed, 30 Dec 2020 05:13:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726161AbgL3KNc (ORCPT ); Wed, 30 Dec 2020 05:13:32 -0500 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A018C06179E for ; Wed, 30 Dec 2020 02:12:52 -0800 (PST) Received: by mail-ej1-x62b.google.com with SMTP id b9so21381271ejy.0 for ; Wed, 30 Dec 2020 02:12:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=J5WJPM6JQMx4Jbj6E9atrShfsotzXemKwoPLiDZZeBk=; b=rxuHPmrEKQkHWNReHzNXMMVqRLOCJizhg5zmF8EXSiwuAnfVOYbth3/EzLtp5m31bP xaanEaEdkEsLiYv5Rw8VhMhbaWSk5vQ6QGR8dPzTgWHKGG7jf5EEaoD46EPhgg3kgKiC Sp/iJpU5Lvh0tRAS/oacUHmEZf8ZL1o4KTwZN6hSI7GiJ0NY/YMyfFNt+17zgAV+RezW 0cl4iyu2wLI9FM79Z7ibczdgISZ80aptDAiWswwdM/Lge9mNEWI4yH1JyLl9xVRY0uEa B07wMrk/Vk5tSBA0FQyPUhs64R9GxUaaS6coE2ln9Wz8I5oKwQLjO56qcGvqffzupefC 1NMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=J5WJPM6JQMx4Jbj6E9atrShfsotzXemKwoPLiDZZeBk=; b=qpjf/+pIx/2E6pVEEfSvIe0JErBh8EGhamLz4aUFuFHZSeXsx2qP0fzRp00BXcCytW h2a9ZRbzSIsl1sqYPM0r3XOU4eeYPOok5rn9IUo7nXj894YLLKwDpIRftTIV78Op89Xj 0Y7g98IOko5Abf3Yt+EFFHla3boHKVHkGbbDDw4JnM/ExiZumCFBxHME8w2ssJoBrBp/ b409gUkytgipT5PshN0+8HxOdKqtXdWf/n/E6T4m5x54AANw7JZ8/eaGkGEENBncOHyv pxM4Jvmfzo+JbBn3JiOIQ+zEOhNKEI5TjoqMnGxOvMfwiPRrG0T8MCIjQGfHqgadonAd FE9A== X-Gm-Message-State: AOAM530/GzLCzYehvYt1V2B2gN+9hI81IIP1D2I5erVWaRrzSqj6JLP+ jSwi3z9Pc3RVd6GH5U6WBJv/rq1hvPxlFPBJ34SV X-Google-Smtp-Source: ABdhPJzfcf6oYCVKKo5/Wxp79G1FfJjdqXlE5V4OiBdpmsAmSa6Oyi1m1HZCfASuNteTrZena0AnCK+Qcxz5m86DSM4= X-Received: by 2002:a17:906:878d:: with SMTP id za13mr48518386ejb.395.1609323170757; Wed, 30 Dec 2020 02:12:50 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <0e6faf9c-117a-e23c-8d6d-488d0ec37412@redhat.com> <2b24398c-e6d9-14ec-2c0d-c303d528e377@redhat.com> <1356137727.40748805.1609233068675.JavaMail.zimbra@redhat.com> <3fc6a132-9fc2-c4e2-7fb1-b5a8bfb771fa@redhat.com> In-Reply-To: <3fc6a132-9fc2-c4e2-7fb1-b5a8bfb771fa@redhat.com> From: Yongji Xie Date: Wed, 30 Dec 2020 18:12:40 +0800 Message-ID: Subject: Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message To: Jason Wang Cc: "Michael S. Tsirkin" , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Wed, Dec 30, 2020 at 4:41 PM Jason Wang wrote: > > > On 2020/12/30 =E4=B8=8B=E5=8D=883:09, Yongji Xie wrote: > > On Wed, Dec 30, 2020 at 2:11 PM Jason Wang wrote: > >> > >> On 2020/12/29 =E4=B8=8B=E5=8D=886:26, Yongji Xie wrote: > >>> On Tue, Dec 29, 2020 at 5:11 PM Jason Wang wrot= e: > >>>> > >>>> ----- Original Message ----- > >>>>> On Mon, Dec 28, 2020 at 4:43 PM Jason Wang wr= ote: > >>>>>> On 2020/12/28 =E4=B8=8B=E5=8D=884:14, Yongji Xie wrote: > >>>>>>>> I see. So all the above two questions are because VHOST_IOTLB_IN= VALIDATE > >>>>>>>> is expected to be synchronous. This need to be solved by tweakin= g the > >>>>>>>> current VDUSE API or we can re-visit to go with descriptors rela= ying > >>>>>>>> first. > >>>>>>>> > >>>>>>> Actually all vdpa related operations are synchronous in current > >>>>>>> implementation. The ops.set_map/dma_map/dma_unmap should not retu= rn > >>>>>>> until the VDUSE_UPDATE_IOTLB/VDUSE_INVALIDATE_IOTLB message is re= plied > >>>>>>> by userspace. Could it solve this problem? > >>>>>> I was thinking whether or not we need to generate IOTLB_INVALI= DATE > >>>>>> message to VDUSE during dma_unmap (vduse_dev_unmap_page). > >>>>>> > >>>>>> If we don't, we're probably fine. > >>>>>> > >>>>> It seems not feasible. This message will be also used in the > >>>>> virtio-vdpa case to notify userspace to unmap some pages during > >>>>> consistent dma unmapping. Maybe we can document it to make sure the > >>>>> users can handle the message correctly. > >>>> Just to make sure I understand your point. > >>>> > >>>> Do you mean you plan to notify the unmap of 1) streaming DMA or 2) > >>>> coherent DMA? > >>>> > >>>> For 1) you probably need a workqueue to do that since dma unmap can > >>>> be done in irq or bh context. And if usrspace does't do the unmap, i= t > >>>> can still access the bounce buffer (if you don't zap pte)? > >>>> > >>> I plan to do it in the coherent DMA case. > >> > >> Any reason for treating coherent DMA differently? > >> > > Now the memory of the bounce buffer is allocated page by page in the > > page fault handler. So it can't be used in coherent DMA mapping case > > which needs some memory with contiguous virtual addresses. I can use > > vmalloc() to do allocation for the bounce buffer instead. But it might > > cause some memory waste. Any suggestion? > > > I may miss something. But I don't see a relationship between the > IOTLB_UNMAP and vmalloc(). > In the vmalloc() case, the coherent DMA page will be taken from the memory allocated by vmalloc(). So IOTLB_UNMAP is not needed anymore during coherent DMA unmapping because those vmalloc'ed memory which has been mapped into userspace address space during initialization can be reused. And userspace should not unmap the region until we destroy the device. > > > > >>> It's true that userspace can > >>> access the dma buffer if userspace doesn't do the unmap. But the dma > >>> pages would not be freed and reused unless user space called munmap() > >>> for them. > >> > >> I wonder whether or not we could recycle IOVA in this case to avoid th= e > >> IOTLB_UMAP message. > >> > > We can achieve that if we use vmalloc() to do allocation for the > > bounce buffer which can be used in coherent DMA mapping case. But > > looks like we still have no way to avoid the IOTLB_UMAP message in > > vhost-vdpa case. > > > I think that's fine. For virtio-vdpa, from VDUSE userspace perspective, > it works like a driver that is using SWIOTLB in this case. > OK, will do it in v3. Thanks, Yongji From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D0C3C433E0 for ; Wed, 30 Dec 2020 10:12:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9AB18221F8 for ; Wed, 30 Dec 2020 10:12:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9AB18221F8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 75A3F8D00A5; Wed, 30 Dec 2020 05:12:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A7058D00A4; Wed, 30 Dec 2020 05:12:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5961F8D007F; Wed, 30 Dec 2020 05:12:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0060.hostedemail.com [216.40.44.60]) by kanga.kvack.org (Postfix) with ESMTP id 3C7878D00A5 for ; Wed, 30 Dec 2020 05:12:53 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EF6C334A3 for ; Wed, 30 Dec 2020 10:12:52 +0000 (UTC) X-FDA: 77649534984.13.spark86_3813aae274a4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id D002C18140B60 for ; Wed, 30 Dec 2020 10:12:52 +0000 (UTC) X-HE-Tag: spark86_3813aae274a4 X-Filterd-Recvd-Size: 7587 Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Wed, 30 Dec 2020 10:12:52 +0000 (UTC) Received: by mail-ej1-f52.google.com with SMTP id ga15so21321139ejb.4 for ; Wed, 30 Dec 2020 02:12:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=J5WJPM6JQMx4Jbj6E9atrShfsotzXemKwoPLiDZZeBk=; b=rxuHPmrEKQkHWNReHzNXMMVqRLOCJizhg5zmF8EXSiwuAnfVOYbth3/EzLtp5m31bP xaanEaEdkEsLiYv5Rw8VhMhbaWSk5vQ6QGR8dPzTgWHKGG7jf5EEaoD46EPhgg3kgKiC Sp/iJpU5Lvh0tRAS/oacUHmEZf8ZL1o4KTwZN6hSI7GiJ0NY/YMyfFNt+17zgAV+RezW 0cl4iyu2wLI9FM79Z7ibczdgISZ80aptDAiWswwdM/Lge9mNEWI4yH1JyLl9xVRY0uEa B07wMrk/Vk5tSBA0FQyPUhs64R9GxUaaS6coE2ln9Wz8I5oKwQLjO56qcGvqffzupefC 1NMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=J5WJPM6JQMx4Jbj6E9atrShfsotzXemKwoPLiDZZeBk=; b=VVxRnl1+QiFdk4PUxvYBEULTKel2KWDr8Gaw96Y+zHh5RTxmqg+L61K8otVzFv/V/l MDQHTal9KSLQOHP1nY1XLNoVogdTqxe6rRtYkNGvbhNt1DcWCeSXA/LG940AevcK/C8r BoXveyffuKsBifOACTr+u9P/EmBQn0KVSQOn2wmlK6GivzKRXxjF0z02EZ4NLqc0GbMK W5TrcPwmzrlps8teYY+43oWQDUFmbmRKqu+CV0vlTniwRcQAwt4wNopF3XyIeAMRfP+v nG/7FuvG4YpHM0ZKnOLN8YT8KhOIEyobufy1M8ma/H3Ovc0WlgUM4RK3Tp2GADwxi9dk GOeQ== X-Gm-Message-State: AOAM532Dk/5CmV1FmKsO8xtXnGc54oVDH0sNZgSN5teYK7OXnAaYgOSD pjX+KfCNpB5TGGWFY2aoPkPkt0K91DPjZDMr7XZe X-Google-Smtp-Source: ABdhPJzfcf6oYCVKKo5/Wxp79G1FfJjdqXlE5V4OiBdpmsAmSa6Oyi1m1HZCfASuNteTrZena0AnCK+Qcxz5m86DSM4= X-Received: by 2002:a17:906:878d:: with SMTP id za13mr48518386ejb.395.1609323170757; Wed, 30 Dec 2020 02:12:50 -0800 (PST) MIME-Version: 1.0 References: <20201222145221.711-1-xieyongji@bytedance.com> <0e6faf9c-117a-e23c-8d6d-488d0ec37412@redhat.com> <2b24398c-e6d9-14ec-2c0d-c303d528e377@redhat.com> <1356137727.40748805.1609233068675.JavaMail.zimbra@redhat.com> <3fc6a132-9fc2-c4e2-7fb1-b5a8bfb771fa@redhat.com> In-Reply-To: <3fc6a132-9fc2-c4e2-7fb1-b5a8bfb771fa@redhat.com> From: Yongji Xie Date: Wed, 30 Dec 2020 18:12:40 +0800 Message-ID: Subject: Re: Re: [RFC v2 09/13] vduse: Add support for processing vhost iotlb message To: Jason Wang Cc: "Michael S. Tsirkin" , Stefan Hajnoczi , sgarzare@redhat.com, Parav Pandit , akpm@linux-foundation.org, Randy Dunlap , Matthew Wilcox , viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-aio@kvack.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Dec 30, 2020 at 4:41 PM Jason Wang wrote: > > > On 2020/12/30 =E4=B8=8B=E5=8D=883:09, Yongji Xie wrote: > > On Wed, Dec 30, 2020 at 2:11 PM Jason Wang wrote: > >> > >> On 2020/12/29 =E4=B8=8B=E5=8D=886:26, Yongji Xie wrote: > >>> On Tue, Dec 29, 2020 at 5:11 PM Jason Wang wrot= e: > >>>> > >>>> ----- Original Message ----- > >>>>> On Mon, Dec 28, 2020 at 4:43 PM Jason Wang wr= ote: > >>>>>> On 2020/12/28 =E4=B8=8B=E5=8D=884:14, Yongji Xie wrote: > >>>>>>>> I see. So all the above two questions are because VHOST_IOTLB_IN= VALIDATE > >>>>>>>> is expected to be synchronous. This need to be solved by tweakin= g the > >>>>>>>> current VDUSE API or we can re-visit to go with descriptors rela= ying > >>>>>>>> first. > >>>>>>>> > >>>>>>> Actually all vdpa related operations are synchronous in current > >>>>>>> implementation. The ops.set_map/dma_map/dma_unmap should not retu= rn > >>>>>>> until the VDUSE_UPDATE_IOTLB/VDUSE_INVALIDATE_IOTLB message is re= plied > >>>>>>> by userspace. Could it solve this problem? > >>>>>> I was thinking whether or not we need to generate IOTLB_INVALI= DATE > >>>>>> message to VDUSE during dma_unmap (vduse_dev_unmap_page). > >>>>>> > >>>>>> If we don't, we're probably fine. > >>>>>> > >>>>> It seems not feasible. This message will be also used in the > >>>>> virtio-vdpa case to notify userspace to unmap some pages during > >>>>> consistent dma unmapping. Maybe we can document it to make sure the > >>>>> users can handle the message correctly. > >>>> Just to make sure I understand your point. > >>>> > >>>> Do you mean you plan to notify the unmap of 1) streaming DMA or 2) > >>>> coherent DMA? > >>>> > >>>> For 1) you probably need a workqueue to do that since dma unmap can > >>>> be done in irq or bh context. And if usrspace does't do the unmap, i= t > >>>> can still access the bounce buffer (if you don't zap pte)? > >>>> > >>> I plan to do it in the coherent DMA case. > >> > >> Any reason for treating coherent DMA differently? > >> > > Now the memory of the bounce buffer is allocated page by page in the > > page fault handler. So it can't be used in coherent DMA mapping case > > which needs some memory with contiguous virtual addresses. I can use > > vmalloc() to do allocation for the bounce buffer instead. But it might > > cause some memory waste. Any suggestion? > > > I may miss something. But I don't see a relationship between the > IOTLB_UNMAP and vmalloc(). > In the vmalloc() case, the coherent DMA page will be taken from the memory allocated by vmalloc(). So IOTLB_UNMAP is not needed anymore during coherent DMA unmapping because those vmalloc'ed memory which has been mapped into userspace address space during initialization can be reused. And userspace should not unmap the region until we destroy the device. > > > > >>> It's true that userspace can > >>> access the dma buffer if userspace doesn't do the unmap. But the dma > >>> pages would not be freed and reused unless user space called munmap() > >>> for them. > >> > >> I wonder whether or not we could recycle IOVA in this case to avoid th= e > >> IOTLB_UMAP message. > >> > > We can achieve that if we use vmalloc() to do allocation for the > > bounce buffer which can be used in coherent DMA mapping case. But > > looks like we still have no way to avoid the IOTLB_UMAP message in > > vhost-vdpa case. > > > I think that's fine. For virtio-vdpa, from VDUSE userspace perspective, > it works like a driver that is using SWIOTLB in this case. > OK, will do it in v3. Thanks, Yongji