From mboxrd@z Thu Jan 1 00:00:00 1970 From: Auger Eric Subject: Re: [PATCH v3 0/7] Add virtio-iommu driver Date: Tue, 16 Oct 2018 22:31:18 +0200 Message-ID: References: <20181012145917.6840-1-jean-philippe.brucker@arm.com> <7874471b-3711-ca44-d220-5e756ac7db8c@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Jean-Philippe Brucker , "iommu@lists.linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "devicetree@vger.kernel.org" Cc: Mark Rutland , "peter.maydell@linaro.org" , Lorenzo Pieralisi , "tnowicki@caviumnetworks.com" , "mst@redhat.com" , Marc Zyngier , "linux-pci@vger.kernel.org" , Will Deacon , "kvmarm@lists.cs.columbia.edu" , "robh+dt@kernel.org" , Robin Murphy , "joro@8bytes.org" List-Id: devicetree@vger.kernel.org SGkgSmVhbiwKCk9uIDEwLzE2LzE4IDg6NDQgUE0sIEplYW4tUGhpbGlwcGUgQnJ1Y2tlciB3cm90 ZToKPiBPbiAxNi8xMC8yMDE4IDEwOjI1LCBBdWdlciBFcmljIHdyb3RlOgo+PiBIaSBKZWFuLAo+ Pgo+PiBPbiAxMC8xMi8xOCA0OjU5IFBNLCBKZWFuLVBoaWxpcHBlIEJydWNrZXIgd3JvdGU6Cj4+ PiBJbXBsZW1lbnQgdGhlIHZpcnRpby1pb21tdSBkcml2ZXIsIGZvbGxvd2luZyBzcGVjaWZpY2F0 aW9uIHYwLjggWzFdLgo+Pj4gQ2hhbmdlcyBzaW5jZSB2MiBbMl06Cj4+Pgo+Pj4gKiBQYXRjaGVz IDItNCBhbGxvdyB2aXJ0aW8taW9tbXUgdG8gdXNlIHRoZSBQQ0kgdHJhbnNwb3J0LCBzaW5jZSBR RU1VCj4+PiDCoMKgIHdvdWxkIGxpa2UgdG8gcGhhc2Ugb3V0IHRoZSBNTUlPIHRyYW5zcG9ydC4g VGhpcyBwcm9kdWNlcyBhIGNvbXBsZXgKPj4+IMKgwqAgdG9wb2xvZ3kgd2hlcmUgdGhlIHByb2dy YW1taW5nIGludGVyZmFjZSBvZiB0aGUgSU9NTVUgY291bGQgYXBwZWFyCj4+PiDCoMKgIGxvd2Vy IHRoYW4gdGhlIGVuZHBvaW50cyB0aGF0IGl0IHRyYW5zbGF0ZXMuIEl0J3Mgbm90IHVuaGVhcmQg b2YgKGUuZy4KPj4+IMKgwqAgQU1EIElPTU1VKSwgYW5kIHRoZSBndWVzdCBlYXNpbHkgY29wZXMg d2l0aCB0aGlzLgo+Pj4gwqDCoCAKPj4+IMKgwqAgVGhlICJGaXJtd2FyZSBkZXNjcmlwdGlvbiIg c2VjdGlvbiBvZiB0aGUgc3BlY2lmaWNhdGlvbiBoYXMgYmVlbgo+Pj4gwqDCoCB1cGRhdGVkIHdp dGggYWxsIGNvbWJpbmF0aW9ucyBvZiBQQ0ksIE1NSU8gYW5kIERULCBBQ1BJLgo+Pgo+PiBJIGhh dmUgYSBxdWVzdGlvbiB3cnQgdGhlIEZXIHNwZWNpZmljYXRpb24uIFRoZSBJT01NVSBjb25zdW1l cyAxIHNsb3QgaW4KPj4gdGhlIFBDSSBkb21haW4gYW5kIG9uZSBuZWVkcyB0byBsZWF2ZSBhIFJJ RCBob2xlIGluIHRoZSBpb21tdS1tYXAuwqAgSXQKPj4gaXMgbm90IG9idmlvdXMgdG8gbWUgdGhh dCB0aGlzIFJJRCBhbHdheXMgaXMgcHJlZGljdGFibGUgZ2l2ZW4gdGhlIHBjaWUKPj4gZW51bWVy YXRpb24gbWVjaGFuaXNtLiBHZW5lcmFsbHkgd2UgaGF2ZSBhIGNvYXJzZSBncmFpbiBtYXBwaW5n IG9mIFJJRAo+PiBvbnRvIGlvbW11IHBoYW5kbGVzL1NUUkVBTUlEcy4gSGVyZSwgaWYgSSB1bmRl cnN0YW5kIGNvcnJlY3RseSB3ZSBuZWVkCj4+IHRvIHByZWNpc2VseSBpZGVudGlmeSB0aGUgUklE IGdyYW50ZWQgdG8gdGhlIGlvbW11LiBPbiBRRU1VIHRoaXMgbWF5Cj4+IGRlcGVuZCBvbiB0aGUg aW5zdGFudGlhdGlvbiBvcmRlciBvZiB0aGUgdmlydGlvLXBjaSBkZXZpY2UgcmlnaHQ/Cj4gCj4g WWVzLCBhbHRob3VnaCBpdCBzaG91bGQgYWxsIGhhcHBlbiBiZWZvcmUgeW91IGJvb3QgdGhlIGd1 ZXN0LCBzaW5jZQo+IHRoZXJlIGlzIG5vIGhvdHBsdWdnaW5nIGFuIElPTU1VLiBDb3VsZCB5b3Ug cmVzZXJ2ZSBhIFBDSSBzbG90IHVwZnJvbnQKPiBhbmQgdXNlIGl0IGZvciB2aXJ0aW8taW9tbXUg bGF0ZXI/IE9yIGdlbmVyYXRlIHRoZSBpb21tdS1tYXAgYXQgdGhlIHNhbWUKPiB0aW1lIGFzIGdl bmVyYXRpbmcgdGhlIGNoaWxkIG5vZGUgb2YgdGhlIFBDSSBSQz8KCkV2ZW4gd2hlbiBjb2xkLXBs dWdnaW5nIHRoZSBQQ0llIGRldmljZXMgdGhyb3VnaCBxZW11IENMSSwgdGhpcyBkZXBlbmRzCm9u IHRoZSBvcmRlciBvZiB0aGUgcGNpZSBkZXZpY2VzIGluIHRoZSBsaXN0IEkgZ3Vlc3MuIEkgbmVl ZCB0byBmdXJ0aGVyCmV4cGVyaW1lbnQuCj4gCj4+IFNvCj4+IHRoaXMgZG9lcyBub3QgbG9vayB0 cml2aWFsIHRvIGJ1aWxkIHRoaXMgaW5mby4gSXNuJ3QgaXQgcG9zc2libGUgdG8gZG8KPj4gdGhp cyBleGNsdXNpb24gYXQga2VybmVsIGxldmVsIGluc3RlYWQ/Cj4gCj4gU28gaW4gdGhlb3J5IFZJ UlRJT19GX0lPTU1VX1BMQVRGT1JNIGFscmVhZHkgZG9lcyB0aGF0Ogo+IAo+IFZJUlRJT19GX0lP TU1VX1BMQVRGT1JNKDMzKQo+ICAgICBUaGlzIGZlYXR1cmUgaW5kaWNhdGVzIHRoYXQgdGhlIGRl dmljZSBpcyBiZWhpbmQgYW4gSU9NTVUgdGhhdAo+ICAgICB0cmFuc2xhdGVzIGJ1cyBhZGRyZXNz ZXMgZnJvbSB0aGUgZGV2aWNlIGludG8gcGh5c2ljYWwgYWRkcmVzc2VzIGluCj4gICAgIG1lbW9y eS4gSWYgdGhpcyBmZWF0dXJlIGJpdCBpcyBzZXQgdG8gMCwgdGhlbiB0aGUgZGV2aWNlIGVtaXRz Cj4gICAgIHBoeXNpY2FsIGFkZHJlc3NlcyB3aGljaCBhcmUgbm90IHRyYW5zbGF0ZWQgZnVydGhl ciwgZXZlbiB0aG91Z2ggYW4KPiAgICAgSU9NTVUgbWF5IGJlIHByZXNlbnQuCgpUaGlzIHRlbGxz IHRoZSBkcml2ZXIgdG8gdXNlIHRoZSBkbWEgYXBpLCByaWdodD8gRWZmZWN0aXZlbHkgdGhpcwpl eHBsaWNpdGx5IHNheXMgd2hldGhlciB0aGUgZGV2aWNlIGlzIHN1cHBvc2VkIHRvIGJlIHVwZnJv bnQgYW4gSU9NTVUuCj4gCj4gRm9yIGJldHRlciBvciBmb3Igd29yc2UsIHRoZSBndWVzdCBoYXMg dG8gaW1wbGVtZW50IGl0LiBJZiB0aGlzIGZlYXR1cmUKPiBiaXQgaXMgdW5zZXQgZm9yIHZpcnRp by1pb21tdSwgaXQgZG9lcyBETUEgb24gdGhlIHBoeXNpY2FsIGFkZHJlc3MKPiBzcGFjZSwgcmVn YXJkbGVzcyBvZiB3aGF0IHRoZSBzdGF0aWMgdG9wb2xvZ3kgZGVzY3JpcHRpb24gc2F5cy4KPiAK PiBJbiBwcmFjdGljZSBpdCBkb2Vzbid0IHF1aXRlIHdvcmsuIElmIHlvdXIgaW9tbXUtbWFwIGRl c2NyaWJlcyB0aGUgSU9NTVUKPiBhcyB0cmFuc2xhdGluZyBpdHNlbGYsIExpbnV4JyBPRiBjb2Rl IHdpbGwgd2FpdCBmb3IgdGhlIElPTU1VIHRvIGJlCj4gcHJvYmVkIGJlZm9yZSBwcm9iaW5nIHRo ZSBJT01NVS4gV29ya2luZyBhcm91bmQgdGhpcyB3aXRoIGhhY2tzIGlzCj4gcG9zc2libGUsIGJ1 dCBJIGRvbid0IHdhbnQgdG8gaW50cm9kdWNlIG1vcmUgcXVlc3Rpb25hYmxlIGNvZGUgdG8gT0Yg YW5kCj4gZGV2aWNlIHRyZWUgYmluZGluZ3MgaWYgdGhlcmUgaXMgYW55IG90aGVyIHdheS4KSHVt IG9rLiBJIGNhbm5vdCByZWFsbHkgY29tbWVudCBvbiB0aGlzLgoKSSBqdXN0IHdhbnRlZCB0byBy YWlzZSB0aGlzIGNvbmNlcm4gYWJvdXQgUklEIGlkZW50ZmljYXRpb24uCgpUaGFua3MKCkVyaWMK PiAKPiBUaGFua3MsCj4gSmVhbgo+IApfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlvbiBtYWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25A bGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24u b3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVhbGl6YXRpb24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95E87C5ACCC for ; Tue, 16 Oct 2018 20:31:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4585621470 for ; Tue, 16 Oct 2018 20:31:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4585621470 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-pci-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726067AbeJQEXe (ORCPT ); Wed, 17 Oct 2018 00:23:34 -0400 Received: from mx1.redhat.com ([209.132.183.28]:42566 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726048AbeJQEXe (ORCPT ); Wed, 17 Oct 2018 00:23:34 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9C443155DB; Tue, 16 Oct 2018 20:31:26 +0000 (UTC) Received: from [10.36.118.67] (unknown [10.36.118.67]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 197C677F83; Tue, 16 Oct 2018 20:31:19 +0000 (UTC) Subject: Re: [PATCH v3 0/7] Add virtio-iommu driver To: Jean-Philippe Brucker , "iommu@lists.linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "devicetree@vger.kernel.org" Cc: "linux-pci@vger.kernel.org" , "kvmarm@lists.cs.columbia.edu" , "peter.maydell@linaro.org" , "joro@8bytes.org" , "mst@redhat.com" , "jasowang@redhat.com" , "robh+dt@kernel.org" , Mark Rutland , "tnowicki@caviumnetworks.com" , "kevin.tian@intel.com" , Marc Zyngier , Robin Murphy , Will Deacon , Lorenzo Pieralisi References: <20181012145917.6840-1-jean-philippe.brucker@arm.com> <7874471b-3711-ca44-d220-5e756ac7db8c@redhat.com> From: Auger Eric Message-ID: Date: Tue, 16 Oct 2018 22:31:18 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Tue, 16 Oct 2018 20:31:26 +0000 (UTC) Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Hi Jean, On 10/16/18 8:44 PM, Jean-Philippe Brucker wrote: > On 16/10/2018 10:25, Auger Eric wrote: >> Hi Jean, >> >> On 10/12/18 4:59 PM, Jean-Philippe Brucker wrote: >>> Implement the virtio-iommu driver, following specification v0.8 [1]. >>> Changes since v2 [2]: >>> >>> * Patches 2-4 allow virtio-iommu to use the PCI transport, since QEMU >>>    would like to phase out the MMIO transport. This produces a complex >>>    topology where the programming interface of the IOMMU could appear >>>    lower than the endpoints that it translates. It's not unheard of (e.g. >>>    AMD IOMMU), and the guest easily copes with this. >>>    >>>    The "Firmware description" section of the specification has been >>>    updated with all combinations of PCI, MMIO and DT, ACPI. >> >> I have a question wrt the FW specification. The IOMMU consumes 1 slot in >> the PCI domain and one needs to leave a RID hole in the iommu-map.  It >> is not obvious to me that this RID always is predictable given the pcie >> enumeration mechanism. Generally we have a coarse grain mapping of RID >> onto iommu phandles/STREAMIDs. Here, if I understand correctly we need >> to precisely identify the RID granted to the iommu. On QEMU this may >> depend on the instantiation order of the virtio-pci device right? > > Yes, although it should all happen before you boot the guest, since > there is no hotplugging an IOMMU. Could you reserve a PCI slot upfront > and use it for virtio-iommu later? Or generate the iommu-map at the same > time as generating the child node of the PCI RC? Even when cold-plugging the PCIe devices through qemu CLI, this depends on the order of the pcie devices in the list I guess. I need to further experiment. > >> So >> this does not look trivial to build this info. Isn't it possible to do >> this exclusion at kernel level instead? > > So in theory VIRTIO_F_IOMMU_PLATFORM already does that: > > VIRTIO_F_IOMMU_PLATFORM(33) > This feature indicates that the device is behind an IOMMU that > translates bus addresses from the device into physical addresses in > memory. If this feature bit is set to 0, then the device emits > physical addresses which are not translated further, even though an > IOMMU may be present. This tells the driver to use the dma api, right? Effectively this explicitly says whether the device is supposed to be upfront an IOMMU. > > For better or for worse, the guest has to implement it. If this feature > bit is unset for virtio-iommu, it does DMA on the physical address > space, regardless of what the static topology description says. > > In practice it doesn't quite work. If your iommu-map describes the IOMMU > as translating itself, Linux' OF code will wait for the IOMMU to be > probed before probing the IOMMU. Working around this with hacks is > possible, but I don't want to introduce more questionable code to OF and > device tree bindings if there is any other way. Hum ok. I cannot really comment on this. I just wanted to raise this concern about RID identfication. Thanks Eric > > Thanks, > Jean >