From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E75F3C48BE0 for ; Fri, 11 Jun 2021 05:44:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C4C1361009 for ; Fri, 11 Jun 2021 05:44:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231132AbhFKFqJ (ORCPT ); Fri, 11 Jun 2021 01:46:09 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:20789 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230001AbhFKFqI (ORCPT ); Fri, 11 Jun 2021 01:46:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623390250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5nlidzY7vVox2ieeNTmLbeG44JwkLPe3MMhpIq19x6g=; b=LOiTUDx60R9wOz9sP1ZGZtlwFXZen6zjRQvWMitayuXLMNIcTLGlDvaUF3WVODXI8I35q7 iC5GJros2ztPy4IZuHHeOWvIAZPoIshMuJIs4cISIPcMr8QCZIZeoMxJ19vNCCOGDXS0aU rlpnURci5sjJN8MvfCzJQE+fbqDvKZ0= Received: from mail-pg1-f200.google.com (mail-pg1-f200.google.com [209.85.215.200]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-321-YXNFEZA0ON6mm0nD4C4AZQ-1; Fri, 11 Jun 2021 01:44:09 -0400 X-MC-Unique: YXNFEZA0ON6mm0nD4C4AZQ-1 Received: by mail-pg1-f200.google.com with SMTP id a10-20020a65418a0000b029021b78388f57so1085398pgq.15 for ; Thu, 10 Jun 2021 22:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=5nlidzY7vVox2ieeNTmLbeG44JwkLPe3MMhpIq19x6g=; b=Au6v3TGgT9vIRiIPpbgH/ZNCs0pprsoh/5EedVYx6h2Ym0BPFAMRa3fql6y6t3Q6ND 6Z6KH8karyLG5wPP2+NN9HMtEN/HfYQDxcDMG6N81uGDFj9aRmKsO0K0q+pMkUN+peTy kQCakiM//UUCx8fAFWajh6dGePEyKqYWb+FC9r0FzRdA2VI7OyoFRtt/9RzKShPkx7nK lGsIMMAux8SGdSq+neAQvzzAZ21KQpXATDlF4Sffn/kBgKUCmO4QR0EOD/4fw/xJs8gI vF3ZbrWxRZrwNk+z2/3Ccqaq/flnW1ZdB/ijOZ7OhPOQPA/JFP/JaWhuTfka0w8jNdBo Yw/Q== X-Gm-Message-State: AOAM530lPIuYAge2O2cOX342T5NrpfKtKsdn13yAkEDGskggyRR/mnmt vFkXu/1u6YDqt66AiNDhuhZLnNexH+AJAmOfqjyuN2xmLIIz7n6kifZVH78HL6pEQzuRdZAueqS URTmFXcvZ9OVwj7rpKGS0HhcO X-Received: by 2002:a63:6547:: with SMTP id z68mr1935901pgb.341.1623390248323; Thu, 10 Jun 2021 22:44:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKB65M93ciDtRjp4dLL7boO1gINllpJ4guVwZKP5dhsZp4Ljb4TFphg7Htp0BodlrU9JnK0w== X-Received: by 2002:a63:6547:: with SMTP id z68mr1935883pgb.341.1623390248048; Thu, 10 Jun 2021 22:44:08 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id d8sm4085729pfq.198.2021.06.10.22.44.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Jun 2021 22:44:07 -0700 (PDT) Subject: Re: [RFC] /dev/ioasid uAPI proposal To: Jason Gunthorpe Cc: "Tian, Kevin" , Lu Baolu , Liu Yi L , "kvm@vger.kernel.org" , Jonathan Corbet , "iommu@lists.linux-foundation.org" , LKML , "Alex Williamson (alex.williamson@redhat.com)\"\"" , David Woodhouse References: <05d7f790-870d-5551-1ced-86926a0aa1a6@redhat.com> <42a71462-1abc-0404-156c-60a7ee1ad333@redhat.com> <20210601173138.GM1002214@nvidia.com> <20210602172154.GC1002214@nvidia.com> <20210608132039.GG1002214@nvidia.com> <20210610114751.GK1002214@nvidia.com> From: Jason Wang Message-ID: Date: Fri, 11 Jun 2021 13:43:59 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210610114751.GK1002214@nvidia.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 在 2021/6/10 下午7:47, Jason Gunthorpe 写道: > On Thu, Jun 10, 2021 at 10:00:01AM +0800, Jason Wang wrote: >> 在 2021/6/8 下午9:20, Jason Gunthorpe 写道: >>> On Tue, Jun 08, 2021 at 09:10:42AM +0800, Jason Wang wrote: >>> >>>> Well, this sounds like a re-invention of io_uring which has already worked >>>> for multifds. >>> How so? io_uring is about sending work to the kernel, not getting >>> structued events back? >> >> Actually it can. Userspace can poll multiple fds via preparing multiple sqes >> with IORING_OP_ADD flag. > Poll is only a part of what is needed here, the main issue is > transfering the PRI events to userspace quickly. Do we really care e.g at most one more syscall in this case? I think the time spent on demand paging is much more than transferring #PF to userspace. What's more, a well designed vIOMMU capable IOMMU hardware should have the ability to inject such event directly to guest if #PF happens on L1. > >> This means another ring and we need introduce ioctl() to add or remove >> ioasids from the poll. And it still need a kind of fallback like a list if >> the ring is full. > The max size of the ring should be determinable based on the PRI > concurrance of each device and the number of devices sharing the ring This has at least one assumption, #PF event is the only event for the ring, I'm not sure this is the case. Thanks > > In any event, I'm not entirely convinced eliding the PRI user/kernel > copy is the main issue here.. If we want this to be low latency I > think it ends up with some kernel driver component assisting the > vIOMMU emulation and avoiding the round trip to userspace > > Jason > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_RED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F44EC4743D for ; Fri, 11 Jun 2021 05:44:17 +0000 (UTC) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DE2D961374 for ; Fri, 11 Jun 2021 05:44:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE2D961374 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 9E28C83B58; Fri, 11 Jun 2021 05:44:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qCIO7pBcbyJo; Fri, 11 Jun 2021 05:44:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 916CD83B48; Fri, 11 Jun 2021 05:44:15 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 64A1BC000E; Fri, 11 Jun 2021 05:44:15 +0000 (UTC) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 30AC2C000B for ; Fri, 11 Jun 2021 05:44:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0F2E0606A9 for ; Fri, 11 Jun 2021 05:44:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0SXk9Od_2AWz for ; Fri, 11 Jun 2021 05:44:12 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 0478C6066A for ; Fri, 11 Jun 2021 05:44:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623390250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5nlidzY7vVox2ieeNTmLbeG44JwkLPe3MMhpIq19x6g=; b=LOiTUDx60R9wOz9sP1ZGZtlwFXZen6zjRQvWMitayuXLMNIcTLGlDvaUF3WVODXI8I35q7 iC5GJros2ztPy4IZuHHeOWvIAZPoIshMuJIs4cISIPcMr8QCZIZeoMxJ19vNCCOGDXS0aU rlpnURci5sjJN8MvfCzJQE+fbqDvKZ0= Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-462-gBzea7UvMyqBgwJUAzn97Q-1; Fri, 11 Jun 2021 01:44:09 -0400 X-MC-Unique: gBzea7UvMyqBgwJUAzn97Q-1 Received: by mail-pf1-f197.google.com with SMTP id j206-20020a6280d70000b02902e9e02e1654so2616940pfd.6 for ; Thu, 10 Jun 2021 22:44:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=5nlidzY7vVox2ieeNTmLbeG44JwkLPe3MMhpIq19x6g=; b=YrOf2Y2dasRoMObSJvwmpQJjCNZEJBrVo0aW0iiYJ34hohiq75fy8AAE6PdrHyob6B Wm5x/yQZRnaig6LLGYabtfehayWWtSHaXs9Z2HW9MdX5LFoF40A/46Dcb0+4hFt3GRM9 GhOOpjFlMFVREGkcYVZyt6JtGKRG9vDtQ1eES0Hth7hEWOlPNYsgvaN8QwOXG8PWFboo zbTugF2eyfkpfZnhQrS8yfl5DT6Jg9p8zy8pQWirWEMlA4ZQ4sYYa2miDnTX3mZGQxjN FkrofcW+fY4I5AlhMRcHK9Y+z9gs7bvRDsNWVEj3cb98c91JngGU+MG1ZWvwHNvwhOzk Z2Zg== X-Gm-Message-State: AOAM532UEsJPZSmSukqlfJkWf75nLe+OIi7lyNhyuqpZrzq3Av6MSSQV TZcDNxY23DOxL90CfRqsWa+lMInj+VBDAFLqkLsWrbnLeo8EML25Q3oHEU7vnUYgYZv8oS0dz6I s1tgVUYdW+t1tRrygNAuUlRaDGwxtcw== X-Received: by 2002:a63:6547:: with SMTP id z68mr1935897pgb.341.1623390248322; Thu, 10 Jun 2021 22:44:08 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKB65M93ciDtRjp4dLL7boO1gINllpJ4guVwZKP5dhsZp4Ljb4TFphg7Htp0BodlrU9JnK0w== X-Received: by 2002:a63:6547:: with SMTP id z68mr1935883pgb.341.1623390248048; Thu, 10 Jun 2021 22:44:08 -0700 (PDT) Received: from wangxiaodeMacBook-Air.local ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id d8sm4085729pfq.198.2021.06.10.22.44.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Jun 2021 22:44:07 -0700 (PDT) Subject: Re: [RFC] /dev/ioasid uAPI proposal To: Jason Gunthorpe References: <05d7f790-870d-5551-1ced-86926a0aa1a6@redhat.com> <42a71462-1abc-0404-156c-60a7ee1ad333@redhat.com> <20210601173138.GM1002214@nvidia.com> <20210602172154.GC1002214@nvidia.com> <20210608132039.GG1002214@nvidia.com> <20210610114751.GK1002214@nvidia.com> From: Jason Wang Message-ID: Date: Fri, 11 Jun 2021 13:43:59 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210610114751.GK1002214@nvidia.com> Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jasowang@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: "Tian, Kevin" , "Alex Williamson \(alex.williamson@redhat.com\)\"\"" , "kvm@vger.kernel.org" , Jonathan Corbet , LKML , "iommu@lists.linux-foundation.org" , David Woodhouse X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" CuWcqCAyMDIxLzYvMTAg5LiL5Y2INzo0NywgSmFzb24gR3VudGhvcnBlIOWGmemBkzoKPiBPbiBU aHUsIEp1biAxMCwgMjAyMSBhdCAxMDowMDowMUFNICswODAwLCBKYXNvbiBXYW5nIHdyb3RlOgo+ PiDlnKggMjAyMS82Lzgg5LiL5Y2IOToyMCwgSmFzb24gR3VudGhvcnBlIOWGmemBkzoKPj4+IE9u IFR1ZSwgSnVuIDA4LCAyMDIxIGF0IDA5OjEwOjQyQU0gKzA4MDAsIEphc29uIFdhbmcgd3JvdGU6 Cj4+Pgo+Pj4+IFdlbGwsIHRoaXMgc291bmRzIGxpa2UgYSByZS1pbnZlbnRpb24gb2YgaW9fdXJp bmcgd2hpY2ggaGFzIGFscmVhZHkgd29ya2VkCj4+Pj4gZm9yIG11bHRpZmRzLgo+Pj4gSG93IHNv PyBpb191cmluZyBpcyBhYm91dCBzZW5kaW5nIHdvcmsgdG8gdGhlIGtlcm5lbCwgbm90IGdldHRp bmcKPj4+IHN0cnVjdHVlZCBldmVudHMgYmFjaz8KPj4KPj4gQWN0dWFsbHkgaXQgY2FuLiBVc2Vy c3BhY2UgY2FuIHBvbGwgbXVsdGlwbGUgZmRzIHZpYSBwcmVwYXJpbmcgbXVsdGlwbGUgc3Flcwo+ PiB3aXRoIElPUklOR19PUF9BREQgZmxhZy4KPiBQb2xsIGlzIG9ubHkgYSBwYXJ0IG9mIHdoYXQg aXMgbmVlZGVkIGhlcmUsIHRoZSBtYWluIGlzc3VlIGlzCj4gdHJhbnNmZXJpbmcgdGhlIFBSSSBl dmVudHMgdG8gdXNlcnNwYWNlIHF1aWNrbHkuCgoKRG8gd2UgcmVhbGx5IGNhcmUgZS5nIGF0IG1v c3Qgb25lIG1vcmUgc3lzY2FsbCBpbiB0aGlzIGNhc2U/IEkgdGhpbmsgdGhlIAp0aW1lIHNwZW50 IG9uIGRlbWFuZCBwYWdpbmcgaXMgbXVjaCBtb3JlIHRoYW4gdHJhbnNmZXJyaW5nICNQRiB0byAK dXNlcnNwYWNlLiBXaGF0J3MgbW9yZSwgYSB3ZWxsIGRlc2lnbmVkIHZJT01NVSBjYXBhYmxlIElP TU1VIGhhcmR3YXJlIApzaG91bGQgaGF2ZSB0aGUgYWJpbGl0eSB0byBpbmplY3Qgc3VjaCBldmVu dCBkaXJlY3RseSB0byBndWVzdCBpZiAjUEYgCmhhcHBlbnMgb24gTDEuCgoKPgo+PiBUaGlzIG1l YW5zIGFub3RoZXIgcmluZyBhbmQgd2UgbmVlZCBpbnRyb2R1Y2UgaW9jdGwoKSB0byBhZGQgb3Ig cmVtb3ZlCj4+IGlvYXNpZHMgZnJvbSB0aGUgcG9sbC4gQW5kIGl0IHN0aWxsIG5lZWQgYSBraW5k IG9mIGZhbGxiYWNrIGxpa2UgYSBsaXN0IGlmCj4+IHRoZSByaW5nIGlzIGZ1bGwuCj4gVGhlIG1h eCBzaXplIG9mIHRoZSByaW5nIHNob3VsZCBiZSBkZXRlcm1pbmFibGUgYmFzZWQgb24gdGhlIFBS SQo+IGNvbmN1cnJhbmNlIG9mIGVhY2ggZGV2aWNlIGFuZCB0aGUgbnVtYmVyIG9mIGRldmljZXMg c2hhcmluZyB0aGUgcmluZwoKClRoaXMgaGFzIGF0IGxlYXN0IG9uZSBhc3N1bXB0aW9uLCAjUEYg ZXZlbnQgaXMgdGhlIG9ubHkgZXZlbnQgZm9yIHRoZSAKcmluZywgSSdtIG5vdCBzdXJlIHRoaXMg aXMgdGhlIGNhc2UuCgpUaGFua3MKCgo+Cj4gSW4gYW55IGV2ZW50LCBJJ20gbm90IGVudGlyZWx5 IGNvbnZpbmNlZCBlbGlkaW5nIHRoZSBQUkkgdXNlci9rZXJuZWwKPiBjb3B5IGlzIHRoZSBtYWlu IGlzc3VlIGhlcmUuLiBJZiB3ZSB3YW50IHRoaXMgdG8gYmUgbG93IGxhdGVuY3kgSQo+IHRoaW5r IGl0IGVuZHMgdXAgd2l0aCBzb21lIGtlcm5lbCBkcml2ZXIgY29tcG9uZW50IGFzc2lzdGluZyB0 aGUKPiB2SU9NTVUgZW11bGF0aW9uIGFuZCBhdm9pZGluZyB0aGUgcm91bmQgdHJpcCB0byB1c2Vy c3BhY2UKPgo+IEphc29uCj4KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fCmlvbW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51eC1mb3VuZGF0aW9u Lm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby9p b21tdQ==