From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0055EC433EF for ; Tue, 7 Dec 2021 10:27:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235120AbhLGKbT (ORCPT ); Tue, 7 Dec 2021 05:31:19 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28205 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235090AbhLGKbR (ORCPT ); Tue, 7 Dec 2021 05:31:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638872867; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=Qzyxl7F8hJMl4RapqEy8wr2fAlZeeqPiHstGT9HapqfdFgkw2CV4OUlv4oUDlQ0Cp+QKUT VkQQQy6DIDkN8ewsUB01dQvMW/1slQW/fSo3kKPmmzSt+KDWwjZgQCXFuk1EQN+k+ohNoA wycVGXED2KHXN3Hgn3u7lpwUmmTU2BE= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-433-_f8qVsljNTKDh6_W6EsF1g-1; Tue, 07 Dec 2021 05:27:46 -0500 X-MC-Unique: _f8qVsljNTKDh6_W6EsF1g-1 Received: by mail-wm1-f69.google.com with SMTP id r129-20020a1c4487000000b00333629ed22dso1012405wma.6 for ; Tue, 07 Dec 2021 02:27:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=HaaOjo8ZEBCEmI4mEKp8N9SIh0F2Z5955kWkUWwz/1rvbtsU2oktI2griXMv418Mj0 LCkRZmmiR2LHp3dGVgmYMzuBg06QKLt6VXwT8k00RZIC/s50c8sypma30mSQZZzQZcoe DJC1xjFt9JRNbua25ULZxXAWtM65IfE0PPfuGTgCO3I71LnCmEgTy6wrkGCD7NdQSy94 9VbY05O1cjaoSvyVPFDEzSrrkmNos1cN/nMMZeCNLmYGzR3WK+1HT289e+DOJjx9yD+Z TgEcQ03Hrh8T1hJwdyXRA0gdRr9fRP1GqIS2TSnYthToe4sN+UoC3iP2BaQgxseGuFp7 gB5A== X-Gm-Message-State: AOAM530ENbYivmgrGimWyj9H3lmSdSfb3x2tQt21AgldsXvqBMXtPfUe qlu+0mDstoeO/w5p2Sp7Fkhfe3Ej9VtImC5NhIgcGm12u63t9g395qiTKnc+nziVw6G7CwL1L76 9nUFXjtmuKF/pJOFIr6cLTW1c X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871503wrh.80.1638872865020; Tue, 07 Dec 2021 02:27:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJw5SYKxnf/czGLnzekcByHTylqSUw2oDFUY77k108vZsKZFOCs+7tv9nPFdqCwIsIDIRwPszg== X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871467wrh.80.1638872864803; Tue, 07 Dec 2021 02:27:44 -0800 (PST) Received: from ?IPv6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id d15sm18622235wri.50.2021.12.07.02.27.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 07 Dec 2021 02:27:43 -0800 (PST) Reply-To: eric.auger@redhat.com Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part) To: Zhangfei Gao , eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, zhukeqian1@huawei.com Cc: alex.williamson@redhat.com, jacob.jun.pan@linux.intel.com, yi.l.liu@intel.com, kevin.tian@intel.com, ashok.raj@intel.com, maz@kernel.org, peter.maydell@linaro.org, vivek.gautam@arm.com, shameerali.kolothum.thodi@huawei.com, wangxingang5@huawei.com, jiangkunkun@huawei.com, yuzenghui@huawei.com, nicoleotsuka@gmail.com, chenxiang66@hisilicon.com, sumitg@nvidia.com, nicolinc@nvidia.com, vdumpa@nvidia.com, zhangfei.gao@gmail.com, lushenming@huawei.com, vsethi@nvidia.com References: <20211027104428.1059740-1-eric.auger@redhat.com> From: Eric Auger Message-ID: <7763531a-625d-10c6-c35e-2ce41e75f606@redhat.com> Date: Tue, 7 Dec 2021 11:27:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Zhangfei, On 12/3/21 1:27 PM, Zhangfei Gao wrote: > > Hi, Eric > > On 2021/10/27 下午6:44, Eric Auger wrote: >> This series brings the IOMMU part of HW nested paging support >> in the SMMUv3. >> >> The SMMUv3 driver is adapted to support 2 nested stages. >> >> The IOMMU API is extended to convey the guest stage 1 >> configuration and the hook is implemented in the SMMUv3 driver. >> >> This allows the guest to own the stage 1 tables and context >> descriptors (so-called PASID table) while the host owns the >> stage 2 tables and main configuration structures (STE). >> >> This work mainly is provided for test purpose as the upper >> layer integration is under rework and bound to be based on >> /dev/iommu instead of VFIO tunneling. In this version we also get >> rid of the MSI BINDING ioctl, assuming the guest enforces >> flat mapping of host IOVAs used to bind physical MSI doorbells. >> In the current QEMU integration this is achieved by exposing >> RMRs to the guest, using Shameer's series [1]. This approach >> is RFC as the IORT spec is not really meant to do that >> (single mapping flag limitation). >> >> Best Regards >> >> Eric >> >> This series (Host) can be found at: >> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16 >> This includes a rebased VFIO integration (although not meant >> to be upstreamed) >> >> Guest kernel branch can be found at: >> https://github.com/eauger/linux/tree/shameer_rmrr_v7 >> featuring [1] >> >> QEMU integration (still based on VFIO and exposing RMRs) >> can be found at: >> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 >> (use iommu=nested-smmuv3 ARM virt option) >> >> Guest dependency: >> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > Thanks a lot for upgrading these patches. > > I have basically verified these patches on HiSilicon Kunpeng920. > And integrated them to these branches. > https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16 > https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10 > > Though they are provided for test purpose, > > Tested-by: Zhangfei Gao Thank you very much. As you mentioned, until we do not have the /dev/iommu integration this is maintained for testing purpose. The SMMU changes shouldn't be much impacted though. The added value of this respin was to propose an MSI binding solution based on RMRRs which simplify things at kernel level. Thanks! Eric > > Thanks > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D1EC2C433F5 for ; Tue, 7 Dec 2021 10:27:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 49F3140533; Tue, 7 Dec 2021 10:27:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5zWwGrz6BRD6; Tue, 7 Dec 2021 10:27:54 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id D80CF40104; Tue, 7 Dec 2021 10:27:53 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C0EF4C002F; Tue, 7 Dec 2021 10:27:53 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 94637C0012 for ; Tue, 7 Dec 2021 10:27:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 8F65941C62 for ; Tue, 7 Dec 2021 10:27:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Xl2XIOEXKI_y for ; Tue, 7 Dec 2021 10:27:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4982A41C59 for ; Tue, 7 Dec 2021 10:27:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638872869; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=BeY/pB6JBFzY59tmTdZe4AXopZACiSWFnLMm5ByGHxj4l18n9rPvASGok1s/vKma1NmW3+ m4EDR83Dd97tKH3i+szdQWqGGpovYoftjC2L3dpRjA/vCdaDCGVGyp2oKjAEbMryQskZr1 9xjFUrfKHrkwBvhRjL+4f2lXeqYSo54= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-89-ViLELDSbOku16I2uFpxnYw-1; Tue, 07 Dec 2021 05:27:46 -0500 X-MC-Unique: ViLELDSbOku16I2uFpxnYw-1 Received: by mail-wm1-f69.google.com with SMTP id a64-20020a1c7f43000000b003335e5dc26bso7509037wmd.8 for ; Tue, 07 Dec 2021 02:27:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=aJwHbM4KXlcN73db9nuheypbBb6EHoPkuO/0pvc7dl5HdVOekFsygk5Lh79T432kLd M7vesv5nbCuDW5h1O1LBm6R8eERCK1/pd/qfOnmetfXCyr6baChS3rXOnna4Z070nAQ5 1OdfV+VLcnOSZB39nlodaz2zTMSYrL852j0dZrROtSKMWLANF/xtD7zHTjHXi/XzD5kb M086Qr6MCN99ZAYuK4gJKY8mDhKiJduRhgctl9hFhxytE/fDyCGptXZVbJ69tGUXllwi u+bQjk3Z3fxlfZ/1mNsk+K2nCZ46yS9Khb0ucWqC4ux89W9v5pdD7rcctnsgUCqtCeh3 MLag== X-Gm-Message-State: AOAM533Yxttj/BXAm5sPcK3+BVHpseZhHuYsZDUUNwffAae1lcFqF0TD MMMc0bhhE5XPLCIK/vDzpTfn0KmmzPF2d8VB6m7gIFuWjDFM/OKeYRpY6Lf78q9Mn533A9PyGVm htoQG1W38gKbc6p8ZSof6+cCIsxktSA== X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871510wrh.80.1638872865021; Tue, 07 Dec 2021 02:27:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJw5SYKxnf/czGLnzekcByHTylqSUw2oDFUY77k108vZsKZFOCs+7tv9nPFdqCwIsIDIRwPszg== X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871467wrh.80.1638872864803; Tue, 07 Dec 2021 02:27:44 -0800 (PST) Received: from ?IPv6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id d15sm18622235wri.50.2021.12.07.02.27.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 07 Dec 2021 02:27:43 -0800 (PST) Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part) To: Zhangfei Gao , eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, zhukeqian1@huawei.com References: <20211027104428.1059740-1-eric.auger@redhat.com> From: Eric Auger Message-ID: <7763531a-625d-10c6-c35e-2ce41e75f606@redhat.com> Date: Tue, 7 Dec 2021 11:27:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eric.auger@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: peter.maydell@linaro.org, kevin.tian@intel.com, ashok.raj@intel.com, maz@kernel.org, vivek.gautam@arm.com, alex.williamson@redhat.com, vsethi@nvidia.com, lushenming@huawei.com, wangxingang5@huawei.com X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: eric.auger@redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" SGkgWmhhbmdmZWksCgpPbiAxMi8zLzIxIDE6MjcgUE0sIFpoYW5nZmVpIEdhbyB3cm90ZToKPgo+ IEhpLCBFcmljCj4KPiBPbiAyMDIxLzEwLzI3IOS4i+WNiDY6NDQsIEVyaWMgQXVnZXIgd3JvdGU6 Cj4+IFRoaXMgc2VyaWVzIGJyaW5ncyB0aGUgSU9NTVUgcGFydCBvZiBIVyBuZXN0ZWQgcGFnaW5n IHN1cHBvcnQKPj4gaW4gdGhlIFNNTVV2My4KPj4KPj4gVGhlIFNNTVV2MyBkcml2ZXIgaXMgYWRh cHRlZCB0byBzdXBwb3J0IDIgbmVzdGVkIHN0YWdlcy4KPj4KPj4gVGhlIElPTU1VIEFQSSBpcyBl eHRlbmRlZCB0byBjb252ZXkgdGhlIGd1ZXN0IHN0YWdlIDEKPj4gY29uZmlndXJhdGlvbiBhbmQg dGhlIGhvb2sgaXMgaW1wbGVtZW50ZWQgaW4gdGhlIFNNTVV2MyBkcml2ZXIuCj4+Cj4+IFRoaXMg YWxsb3dzIHRoZSBndWVzdCB0byBvd24gdGhlIHN0YWdlIDEgdGFibGVzIGFuZCBjb250ZXh0Cj4+ IGRlc2NyaXB0b3JzIChzby1jYWxsZWQgUEFTSUQgdGFibGUpIHdoaWxlIHRoZSBob3N0IG93bnMg dGhlCj4+IHN0YWdlIDIgdGFibGVzIGFuZCBtYWluIGNvbmZpZ3VyYXRpb24gc3RydWN0dXJlcyAo U1RFKS4KPj4KPj4gVGhpcyB3b3JrIG1haW5seSBpcyBwcm92aWRlZCBmb3IgdGVzdCBwdXJwb3Nl IGFzIHRoZSB1cHBlcgo+PiBsYXllciBpbnRlZ3JhdGlvbiBpcyB1bmRlciByZXdvcmsgYW5kIGJv dW5kIHRvIGJlIGJhc2VkIG9uCj4+IC9kZXYvaW9tbXUgaW5zdGVhZCBvZiBWRklPIHR1bm5lbGlu Zy4gSW4gdGhpcyB2ZXJzaW9uIHdlIGFsc28gZ2V0Cj4+IHJpZCBvZiB0aGUgTVNJIEJJTkRJTkcg aW9jdGwsIGFzc3VtaW5nIHRoZSBndWVzdCBlbmZvcmNlcwo+PiBmbGF0IG1hcHBpbmcgb2YgaG9z dCBJT1ZBcyB1c2VkIHRvIGJpbmQgcGh5c2ljYWwgTVNJIGRvb3JiZWxscy4KPj4gSW4gdGhlIGN1 cnJlbnQgUUVNVSBpbnRlZ3JhdGlvbiB0aGlzIGlzIGFjaGlldmVkIGJ5IGV4cG9zaW5nCj4+IFJN UnMgdG8gdGhlIGd1ZXN0LCB1c2luZyBTaGFtZWVyJ3Mgc2VyaWVzIFsxXS4gVGhpcyBhcHByb2Fj aAo+PiBpcyBSRkMgYXMgdGhlIElPUlQgc3BlYyBpcyBub3QgcmVhbGx5IG1lYW50IHRvIGRvIHRo YXQKPj4gKHNpbmdsZSBtYXBwaW5nIGZsYWcgbGltaXRhdGlvbikuCj4+Cj4+IEJlc3QgUmVnYXJk cwo+Pgo+PiBFcmljCj4+Cj4+IFRoaXMgc2VyaWVzIChIb3N0KSBjYW4gYmUgZm91bmQgYXQ6Cj4+ IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvbGludXgvdHJlZS92NS4xNS1yYzctbmVzdGVkLXYx Ngo+PiBUaGlzIGluY2x1ZGVzIGEgcmViYXNlZCBWRklPIGludGVncmF0aW9uIChhbHRob3VnaCBu b3QgbWVhbnQKPj4gdG8gYmUgdXBzdHJlYW1lZCkKPj4KPj4gR3Vlc3Qga2VybmVsIGJyYW5jaCBj YW4gYmUgZm91bmQgYXQ6Cj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvbGludXgvdHJlZS9z aGFtZWVyX3JtcnJfdjcKPj4gZmVhdHVyaW5nIFsxXQo+Pgo+PiBRRU1VIGludGVncmF0aW9uIChz dGlsbCBiYXNlZCBvbiBWRklPIGFuZCBleHBvc2luZyBSTVJzKQo+PiBjYW4gYmUgZm91bmQgYXQ6 Cj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvcWVtdS90cmVlL3Y2LjEuMC1ybXItdjItbmVz dGVkX3NtbXV2M192MTAKPj4gKHVzZSBpb21tdT1uZXN0ZWQtc21tdXYzIEFSTSB2aXJ0IG9wdGlv bikKPj4KPj4gR3Vlc3QgZGVwZW5kZW5jeToKPj4gWzFdIFtQQVRDSCB2NyAwLzldIEFDUEkvSU9S VDogU3VwcG9ydCBmb3IgSU9SVCBSTVIgbm9kZQo+Cj4gVGhhbmtzIGEgbG90IGZvciB1cGdyYWRp bmcgdGhlc2UgcGF0Y2hlcy4KPgo+IEkgaGF2ZSBiYXNpY2FsbHkgdmVyaWZpZWQgdGhlc2UgcGF0 Y2hlcyBvbiBIaVNpbGljb24gS3VucGVuZzkyMC4KPiBBbmQgaW50ZWdyYXRlZCB0aGVtIHRvIHRo ZXNlIGJyYW5jaGVzLgo+IGh0dHBzOi8vZ2l0aHViLmNvbS9MaW5hcm8vbGludXgta2VybmVsLXVh ZGsvdHJlZS91YWNjZS1kZXZlbC01LjE2Cj4gaHR0cHM6Ly9naXRodWIuY29tL0xpbmFyby9xZW11 L3RyZWUvdjYuMS4wLXJtci12Mi1uZXN0ZWRfc21tdXYzX3YxMAo+Cj4gVGhvdWdoIHRoZXkgYXJl IHByb3ZpZGVkIGZvciB0ZXN0IHB1cnBvc2UsCj4KPiBUZXN0ZWQtYnk6IFpoYW5nZmVpIEdhbyA8 emhhbmdmZWkuZ2FvQGxpbmFyby5vcmc+CgpUaGFuayB5b3UgdmVyeSBtdWNoLiBBcyB5b3UgbWVu dGlvbmVkLCB1bnRpbCB3ZSBkbyBub3QgaGF2ZSB0aGUKL2Rldi9pb21tdSBpbnRlZ3JhdGlvbiB0 aGlzIGlzIG1haW50YWluZWQgZm9yIHRlc3RpbmcgcHVycG9zZS4gVGhlIFNNTVUKY2hhbmdlcyBz aG91bGRuJ3QgYmUgbXVjaCBpbXBhY3RlZCB0aG91Z2guClRoZSBhZGRlZCB2YWx1ZSBvZiB0aGlz IHJlc3BpbiB3YXMgdG8gcHJvcG9zZSBhbiBNU0kgYmluZGluZyBzb2x1dGlvbgpiYXNlZCBvbiBS TVJScyB3aGljaCBzaW1wbGlmeSB0aGluZ3MgYXQga2VybmVsIGxldmVsLgoKVGhhbmtzIQoKRXJp Ywo+Cj4gVGhhbmtzCj4KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCmlvbW11IG1haWxpbmcgbGlzdAppb21tdUBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9y ZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby9pb21t dQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF093C433EF for ; Tue, 7 Dec 2021 10:27:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 43DC640667; Tue, 7 Dec 2021 05:27:55 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 77boIs8He-k4; Tue, 7 Dec 2021 05:27:53 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id B4B4A49F5F; Tue, 7 Dec 2021 05:27:53 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 3917849F5F for ; Tue, 7 Dec 2021 05:27:52 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yibsmfDp4-70 for ; Tue, 7 Dec 2021 05:27:51 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 09F0240667 for ; Tue, 7 Dec 2021 05:27:51 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638872870; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=D0iZdz0y1cfknqbh9GRULsU7V9SHmLbBvwHqPiuKgXunx/F6/1m3BIldJ8J7zq1vV5i1Bo k5LZ7nW07hTvE1SCM15OHlHrrn78rOlGiVABKdo+V0p7AphTvrXMzrXpDiGxo3waUJkWzr ZohXaX/0kM+xaBoO40lF/WWzk4YomZo= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-520-_kOML7uePhqovr-PuqIH7Q-1; Tue, 07 Dec 2021 05:27:46 -0500 X-MC-Unique: _kOML7uePhqovr-PuqIH7Q-1 Received: by mail-wr1-f72.google.com with SMTP id k8-20020a5d5248000000b001763e7c9ce5so2775515wrc.22 for ; Tue, 07 Dec 2021 02:27:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:reply-to:subject:to:cc:references:from :message-id:date:user-agent:mime-version:in-reply-to :content-transfer-encoding:content-language; bh=WuGNH/dyVygpiGOJkCWGmo3DKhkMVbJWC9pFgiPTsyo=; b=YaEdU+1dt5q8KsEdQmVUW61SrNKr3Wni3DI2ZRQh0kaYOzmLYRjRd6UFOYopgwegqm OKZfbR1RoeYLL9aLVGfe5FjOrinSx0n/wEGy4k/kC3y+Nsb6zvI2J6sH7UkeSDHrhHe/ /woyNwdKfA5cQhpNyNRZzDb8+4n/oV//hWwkEfOO8vrPUIouD+240kYaA3Occo+qvIAn Gzw3wSPQQiAUauaZrEZ2jdCK/kNRAiecD/7QrWNgi4Ni+BJrZCXPMjirNR2D+RnbPqcU pKAMdKrTeh7Xze1brACkZXLyXvMb7dJtPPrBx2kVjRogvZnIR0MCLbxRYem2BjAu4k7l 4sOA== X-Gm-Message-State: AOAM533V7+XSgYTBfORmIFhzrQks9TDAqQSWoXilt60JiZpNUo7TtroW n8CX+RAfWaOqDR0DLM+J3m3Cc2z1098PHkPF8gmKZyUyi+TsoHvmQFwsbs/MOsFs4p28pysYvpf va9rMH96jNOY7vVdsAv38/tGN X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871524wrh.80.1638872865076; Tue, 07 Dec 2021 02:27:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJw5SYKxnf/czGLnzekcByHTylqSUw2oDFUY77k108vZsKZFOCs+7tv9nPFdqCwIsIDIRwPszg== X-Received: by 2002:adf:d22a:: with SMTP id k10mr51871467wrh.80.1638872864803; Tue, 07 Dec 2021 02:27:44 -0800 (PST) Received: from ?IPv6:2a01:e0a:59e:9d80:527b:9dff:feef:3874? ([2a01:e0a:59e:9d80:527b:9dff:feef:3874]) by smtp.gmail.com with ESMTPSA id d15sm18622235wri.50.2021.12.07.02.27.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 07 Dec 2021 02:27:43 -0800 (PST) Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part) To: Zhangfei Gao , eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, zhukeqian1@huawei.com References: <20211027104428.1059740-1-eric.auger@redhat.com> From: Eric Auger Message-ID: <7763531a-625d-10c6-c35e-2ce41e75f606@redhat.com> Date: Tue, 7 Dec 2021 11:27:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.10.1 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eric.auger@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Cc: kevin.tian@intel.com, jacob.jun.pan@linux.intel.com, ashok.raj@intel.com, chenxiang66@hisilicon.com, maz@kernel.org, vdumpa@nvidia.com, nicoleotsuka@gmail.com, vivek.gautam@arm.com, alex.williamson@redhat.com, yi.l.liu@intel.com, nicolinc@nvidia.com, vsethi@nvidia.com, sumitg@nvidia.com, lushenming@huawei.com, wangxingang5@huawei.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list Reply-To: eric.auger@redhat.com List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu SGkgWmhhbmdmZWksCgpPbiAxMi8zLzIxIDE6MjcgUE0sIFpoYW5nZmVpIEdhbyB3cm90ZToKPgo+ IEhpLCBFcmljCj4KPiBPbiAyMDIxLzEwLzI3IOS4i+WNiDY6NDQsIEVyaWMgQXVnZXIgd3JvdGU6 Cj4+IFRoaXMgc2VyaWVzIGJyaW5ncyB0aGUgSU9NTVUgcGFydCBvZiBIVyBuZXN0ZWQgcGFnaW5n IHN1cHBvcnQKPj4gaW4gdGhlIFNNTVV2My4KPj4KPj4gVGhlIFNNTVV2MyBkcml2ZXIgaXMgYWRh cHRlZCB0byBzdXBwb3J0IDIgbmVzdGVkIHN0YWdlcy4KPj4KPj4gVGhlIElPTU1VIEFQSSBpcyBl eHRlbmRlZCB0byBjb252ZXkgdGhlIGd1ZXN0IHN0YWdlIDEKPj4gY29uZmlndXJhdGlvbiBhbmQg dGhlIGhvb2sgaXMgaW1wbGVtZW50ZWQgaW4gdGhlIFNNTVV2MyBkcml2ZXIuCj4+Cj4+IFRoaXMg YWxsb3dzIHRoZSBndWVzdCB0byBvd24gdGhlIHN0YWdlIDEgdGFibGVzIGFuZCBjb250ZXh0Cj4+ IGRlc2NyaXB0b3JzIChzby1jYWxsZWQgUEFTSUQgdGFibGUpIHdoaWxlIHRoZSBob3N0IG93bnMg dGhlCj4+IHN0YWdlIDIgdGFibGVzIGFuZCBtYWluIGNvbmZpZ3VyYXRpb24gc3RydWN0dXJlcyAo U1RFKS4KPj4KPj4gVGhpcyB3b3JrIG1haW5seSBpcyBwcm92aWRlZCBmb3IgdGVzdCBwdXJwb3Nl IGFzIHRoZSB1cHBlcgo+PiBsYXllciBpbnRlZ3JhdGlvbiBpcyB1bmRlciByZXdvcmsgYW5kIGJv dW5kIHRvIGJlIGJhc2VkIG9uCj4+IC9kZXYvaW9tbXUgaW5zdGVhZCBvZiBWRklPIHR1bm5lbGlu Zy4gSW4gdGhpcyB2ZXJzaW9uIHdlIGFsc28gZ2V0Cj4+IHJpZCBvZiB0aGUgTVNJIEJJTkRJTkcg aW9jdGwsIGFzc3VtaW5nIHRoZSBndWVzdCBlbmZvcmNlcwo+PiBmbGF0IG1hcHBpbmcgb2YgaG9z dCBJT1ZBcyB1c2VkIHRvIGJpbmQgcGh5c2ljYWwgTVNJIGRvb3JiZWxscy4KPj4gSW4gdGhlIGN1 cnJlbnQgUUVNVSBpbnRlZ3JhdGlvbiB0aGlzIGlzIGFjaGlldmVkIGJ5IGV4cG9zaW5nCj4+IFJN UnMgdG8gdGhlIGd1ZXN0LCB1c2luZyBTaGFtZWVyJ3Mgc2VyaWVzIFsxXS4gVGhpcyBhcHByb2Fj aAo+PiBpcyBSRkMgYXMgdGhlIElPUlQgc3BlYyBpcyBub3QgcmVhbGx5IG1lYW50IHRvIGRvIHRo YXQKPj4gKHNpbmdsZSBtYXBwaW5nIGZsYWcgbGltaXRhdGlvbikuCj4+Cj4+IEJlc3QgUmVnYXJk cwo+Pgo+PiBFcmljCj4+Cj4+IFRoaXMgc2VyaWVzIChIb3N0KSBjYW4gYmUgZm91bmQgYXQ6Cj4+ IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvbGludXgvdHJlZS92NS4xNS1yYzctbmVzdGVkLXYx Ngo+PiBUaGlzIGluY2x1ZGVzIGEgcmViYXNlZCBWRklPIGludGVncmF0aW9uIChhbHRob3VnaCBu b3QgbWVhbnQKPj4gdG8gYmUgdXBzdHJlYW1lZCkKPj4KPj4gR3Vlc3Qga2VybmVsIGJyYW5jaCBj YW4gYmUgZm91bmQgYXQ6Cj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvbGludXgvdHJlZS9z aGFtZWVyX3JtcnJfdjcKPj4gZmVhdHVyaW5nIFsxXQo+Pgo+PiBRRU1VIGludGVncmF0aW9uIChz dGlsbCBiYXNlZCBvbiBWRklPIGFuZCBleHBvc2luZyBSTVJzKQo+PiBjYW4gYmUgZm91bmQgYXQ6 Cj4+IGh0dHBzOi8vZ2l0aHViLmNvbS9lYXVnZXIvcWVtdS90cmVlL3Y2LjEuMC1ybXItdjItbmVz dGVkX3NtbXV2M192MTAKPj4gKHVzZSBpb21tdT1uZXN0ZWQtc21tdXYzIEFSTSB2aXJ0IG9wdGlv bikKPj4KPj4gR3Vlc3QgZGVwZW5kZW5jeToKPj4gWzFdIFtQQVRDSCB2NyAwLzldIEFDUEkvSU9S VDogU3VwcG9ydCBmb3IgSU9SVCBSTVIgbm9kZQo+Cj4gVGhhbmtzIGEgbG90IGZvciB1cGdyYWRp bmcgdGhlc2UgcGF0Y2hlcy4KPgo+IEkgaGF2ZSBiYXNpY2FsbHkgdmVyaWZpZWQgdGhlc2UgcGF0 Y2hlcyBvbiBIaVNpbGljb24gS3VucGVuZzkyMC4KPiBBbmQgaW50ZWdyYXRlZCB0aGVtIHRvIHRo ZXNlIGJyYW5jaGVzLgo+IGh0dHBzOi8vZ2l0aHViLmNvbS9MaW5hcm8vbGludXgta2VybmVsLXVh ZGsvdHJlZS91YWNjZS1kZXZlbC01LjE2Cj4gaHR0cHM6Ly9naXRodWIuY29tL0xpbmFyby9xZW11 L3RyZWUvdjYuMS4wLXJtci12Mi1uZXN0ZWRfc21tdXYzX3YxMAo+Cj4gVGhvdWdoIHRoZXkgYXJl IHByb3ZpZGVkIGZvciB0ZXN0IHB1cnBvc2UsCj4KPiBUZXN0ZWQtYnk6IFpoYW5nZmVpIEdhbyA8 emhhbmdmZWkuZ2FvQGxpbmFyby5vcmc+CgpUaGFuayB5b3UgdmVyeSBtdWNoLiBBcyB5b3UgbWVu dGlvbmVkLCB1bnRpbCB3ZSBkbyBub3QgaGF2ZSB0aGUKL2Rldi9pb21tdSBpbnRlZ3JhdGlvbiB0 aGlzIGlzIG1haW50YWluZWQgZm9yIHRlc3RpbmcgcHVycG9zZS4gVGhlIFNNTVUKY2hhbmdlcyBz aG91bGRuJ3QgYmUgbXVjaCBpbXBhY3RlZCB0aG91Z2guClRoZSBhZGRlZCB2YWx1ZSBvZiB0aGlz IHJlc3BpbiB3YXMgdG8gcHJvcG9zZSBhbiBNU0kgYmluZGluZyBzb2x1dGlvbgpiYXNlZCBvbiBS TVJScyB3aGljaCBzaW1wbGlmeSB0aGluZ3MgYXQga2VybmVsIGxldmVsLgoKVGhhbmtzIQoKRXJp Ywo+Cj4gVGhhbmtzCj4KCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCmt2bWFybSBtYWlsaW5nIGxpc3QKa3ZtYXJtQGxpc3RzLmNzLmNvbHVtYmlhLmVkdQpo dHRwczovL2xpc3RzLmNzLmNvbHVtYmlhLmVkdS9tYWlsbWFuL2xpc3RpbmZvL2t2bWFybQo=