From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9338BC4338F for ; Tue, 3 Aug 2021 07:30:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77EBA60F8F for ; Tue, 3 Aug 2021 07:30:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234173AbhHCHai (ORCPT ); Tue, 3 Aug 2021 03:30:38 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:33079 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234183AbhHCHag (ORCPT ); Tue, 3 Aug 2021 03:30:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1627975825; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2q5cKVt8HkdvWLVDPoAENZkOV4GLcj8mh5u4LwwqXQg=; b=NhQgdrvVrlJoU9lC1ntcW+R8P/deCNvTEs5OqstbA5nO98NEzEPJVYxqF22l8xyT5bgbPb OAzOBfQHDBSlWXL/rjS+clDm3y5yDXOKffgrOnQkdKHq5YCX4LA2GEBsk6RYrrdpojnaec jCFr++Vx824IT93aBwbqtJADPptlT98= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-409-ypCYa4wqNCu8HeoEbISSDg-1; Tue, 03 Aug 2021 03:30:24 -0400 X-MC-Unique: ypCYa4wqNCu8HeoEbISSDg-1 Received: by mail-pj1-f69.google.com with SMTP id 16-20020a17090a1990b029017582e03c3bso2120549pji.7 for ; Tue, 03 Aug 2021 00:30:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=2q5cKVt8HkdvWLVDPoAENZkOV4GLcj8mh5u4LwwqXQg=; b=TqV/wF+pcbhI9hskcniK5f3N8xXBX1MWGSJ4j4CEu2C8Y5HPtEzcnFBiS55Pyqbnc4 a/e7GsjJ3eDID7jJKxARZYO69l4aYuDIJX8FGikU3C4B5TnsltFV1QN0F1gU6czowKxv fzjqnsc3cnwIm5b2dSOPmyr5jzw0YQcNOorzjNCR749LY0lcKZ31TLrk8FPMmGzhVtfg Ek7QJkSHqKcdgAOvlD11gqEzSg5fA7cQf77ya51/W7w5IN1B21uUqSOCiaJ6lJ/OdTG0 UqYxTmXrK9sBaZmfKDqvfaxxhn4pWe2QbdIMx0cZ5inGsSuk66DramSee935QeF6kOgs 2Oug== X-Gm-Message-State: AOAM531kIbfqu1lqO6JdcWsy0l8n27AU5cAVj6hjj6wBp+4fTvQLtv44 Yz9wOiTTKIwQAg7XnEnSOkmVG8jKNm9Typ9q/3atG65m1Dx2uY6j+WMR97mvY5HvPWI2mq1O1am AKaCpCA7NlHTE8LNPvLovuYH+XHWJY20kbZjRXz3g5gW38F8eLW8SdBh0LwAJm1Q5sEYCS3Zcjq T0 X-Received: by 2002:a17:902:7c87:b029:12c:8f2d:4238 with SMTP id y7-20020a1709027c87b029012c8f2d4238mr2334712pll.50.1627975823548; Tue, 03 Aug 2021 00:30:23 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyxQbskpfqVraXEkI1LzhRrk+KradYU967AMB1od8U2sh7UOY8/HP+aNVn3ptEFADzrVm8jMg== X-Received: by 2002:a17:902:7c87:b029:12c:8f2d:4238 with SMTP id y7-20020a1709027c87b029012c8f2d4238mr2334670pll.50.1627975823183; Tue, 03 Aug 2021 00:30:23 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id u3sm1749726pjn.18.2021.08.03.00.30.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 03 Aug 2021 00:30:22 -0700 (PDT) Subject: Re: [PATCH v10 16/17] vduse: Introduce VDUSE - vDPA Device in Userspace To: Xie Yongji , mst@redhat.com, stefanha@redhat.com, sgarzare@redhat.com, parav@nvidia.com, hch@infradead.org, christian.brauner@canonical.com, rdunlap@infradead.org, willy@infradead.org, viro@zeniv.linux.org.uk, axboe@kernel.dk, bcrl@kvack.org, corbet@lwn.net, mika.penttila@nextfour.com, dan.carpenter@oracle.com, joro@8bytes.org, gregkh@linuxfoundation.org, zhe.he@windriver.com, xiaodong.liu@intel.com, joe@perches.com Cc: songmuchun@bytedance.com, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, kvm@vger.kernel.org, linux-fsdevel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org References: <20210729073503.187-1-xieyongji@bytedance.com> <20210729073503.187-17-xieyongji@bytedance.com> From: Jason Wang Message-ID: Date: Tue, 3 Aug 2021 15:30:13 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: <20210729073503.187-17-xieyongji@bytedance.com> Content-Type: text/plain; charset=gbk; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org ÔÚ 2021/7/29 ÏÂÎç3:35, Xie Yongji дµÀ: > This VDUSE driver enables implementing software-emulated vDPA > devices in userspace. The vDPA device is created by > ioctl(VDUSE_CREATE_DEV) on /dev/vduse/control. Then a char device > interface (/dev/vduse/$NAME) is exported to userspace for device > emulation. > > In order to make the device emulation more secure, the device's > control path is handled in kernel. A message mechnism is introduced > to forward some dataplane related control messages to userspace. > > And in the data path, the DMA buffer will be mapped into userspace > address space through different ways depending on the vDPA bus to > which the vDPA device is attached. In virtio-vdpa case, the MMU-based > software IOTLB is used to achieve that. And in vhost-vdpa case, the > DMA buffer is reside in a userspace memory region which can be shared > to the VDUSE userspace processs via transferring the shmfd. > > For more details on VDUSE design and usage, please see the follow-on > Documentation commit. > > Signed-off-by: Xie Yongji > --- > Documentation/userspace-api/ioctl/ioctl-number.rst | 1 + > drivers/vdpa/Kconfig | 10 + > drivers/vdpa/Makefile | 1 + > drivers/vdpa/vdpa_user/Makefile | 5 + > drivers/vdpa/vdpa_user/vduse_dev.c | 1541 ++++++++++++++++++++ > include/uapi/linux/vduse.h | 220 +++ > 6 files changed, 1778 insertions(+) > create mode 100644 drivers/vdpa/vdpa_user/Makefile > create mode 100644 drivers/vdpa/vdpa_user/vduse_dev.c > create mode 100644 include/uapi/linux/vduse.h > > diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst > index 1409e40e6345..293ca3aef358 100644 > --- a/Documentation/userspace-api/ioctl/ioctl-number.rst > +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst > @@ -300,6 +300,7 @@ Code Seq# Include File Comments > 'z' 10-4F drivers/s390/crypto/zcrypt_api.h conflict! > '|' 00-7F linux/media.h > 0x80 00-1F linux/fb.h > +0x81 00-1F linux/vduse.h > 0x89 00-06 arch/x86/include/asm/sockios.h > 0x89 0B-DF linux/sockios.h > 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range > diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig > index a503c1b2bfd9..6e23bce6433a 100644 > --- a/drivers/vdpa/Kconfig > +++ b/drivers/vdpa/Kconfig > @@ -33,6 +33,16 @@ config VDPA_SIM_BLOCK > vDPA block device simulator which terminates IO request in a > memory buffer. > > +config VDPA_USER > + tristate "VDUSE (vDPA Device in Userspace) support" > + depends on EVENTFD && MMU && HAS_DMA > + select DMA_OPS > + select VHOST_IOTLB > + select IOMMU_IOVA > + help > + With VDUSE it is possible to emulate a vDPA Device > + in a userspace program. > + > config IFCVF > tristate "Intel IFC VF vDPA driver" > depends on PCI_MSI > diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile > index 67fe7f3d6943..f02ebed33f19 100644 > --- a/drivers/vdpa/Makefile > +++ b/drivers/vdpa/Makefile > @@ -1,6 +1,7 @@ > # SPDX-License-Identifier: GPL-2.0 > obj-$(CONFIG_VDPA) += vdpa.o > obj-$(CONFIG_VDPA_SIM) += vdpa_sim/ > +obj-$(CONFIG_VDPA_USER) += vdpa_user/ > obj-$(CONFIG_IFCVF) += ifcvf/ > obj-$(CONFIG_MLX5_VDPA) += mlx5/ > obj-$(CONFIG_VP_VDPA) += virtio_pci/ > diff --git a/drivers/vdpa/vdpa_user/Makefile b/drivers/vdpa/vdpa_user/Makefile > new file mode 100644 > index 000000000000..260e0b26af99 > --- /dev/null > +++ b/drivers/vdpa/vdpa_user/Makefile > @@ -0,0 +1,5 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +vduse-y := vduse_dev.o iova_domain.o > + > +obj-$(CONFIG_VDPA_USER) += vduse.o > diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c > new file mode 100644 > index 000000000000..6addc62e7de6 > --- /dev/null > +++ b/drivers/vdpa/vdpa_user/vduse_dev.c > @@ -0,0 +1,1541 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * VDUSE: vDPA Device in Userspace > + * > + * Copyright (C) 2020-2021 Bytedance Inc. and/or its affiliates. All rights reserved. > + * > + * Author: Xie Yongji > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "iova_domain.h" > + > +#define DRV_AUTHOR "Yongji Xie " > +#define DRV_DESC "vDPA Device in Userspace" > +#define DRV_LICENSE "GPL v2" > + > +#define VDUSE_DEV_MAX (1U << MINORBITS) > +#define VDUSE_BOUNCE_SIZE (64 * 1024 * 1024) > +#define VDUSE_IOVA_SIZE (128 * 1024 * 1024) > +#define VDUSE_REQUEST_TIMEOUT 30 I think we need make this as a module parameter. 0 probably means we need to wait for ever. This can help in the case when the userspace is attached by GDB. If Michael is still not happy, we can find other solution (e.g only offload the datapath). Other looks good. Thanks