From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jean-Philippe Brucker Subject: Re: [PATCH v3 0/7] Add virtio-iommu driver Date: Wed, 17 Oct 2018 12:54:28 +0100 Message-ID: <40dddb20-6248-5bb0-e0ed-48bacd1867a1@arm.com> References: <20181012145917.6840-1-jean-philippe.brucker@arm.com> <7874471b-3711-ca44-d220-5e756ac7db8c@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Auger Eric , "iommu@lists.linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "devicetree@vger.kernel.org" Cc: Mark Rutland , "peter.maydell@linaro.org" , Lorenzo Pieralisi , "tnowicki@caviumnetworks.com" , "mst@redhat.com" , Marc Zyngier , "linux-pci@vger.kernel.org" , Will Deacon , "kvmarm@lists.cs.columbia.edu" , "robh+dt@kernel.org" , Robin Murphy , "joro@8bytes.org" List-Id: devicetree@vger.kernel.org T24gMTYvMTAvMjAxOCAyMTozMSwgQXVnZXIgRXJpYyB3cm90ZToKPiBIaSBKZWFuLAo+IAo+IE9u IDEwLzE2LzE4IDg6NDQgUE0sIEplYW4tUGhpbGlwcGUgQnJ1Y2tlciB3cm90ZToKPj4gT24gMTYv MTAvMjAxOCAxMDoyNSwgQXVnZXIgRXJpYyB3cm90ZToKPj4+IEhpIEplYW4sCj4+Pgo+Pj4gT24g MTAvMTIvMTggNDo1OSBQTSwgSmVhbi1QaGlsaXBwZSBCcnVja2VyIHdyb3RlOgo+Pj4+IEltcGxl bWVudCB0aGUgdmlydGlvLWlvbW11IGRyaXZlciwgZm9sbG93aW5nIHNwZWNpZmljYXRpb24gdjAu OCBbMV0uCj4+Pj4gQ2hhbmdlcyBzaW5jZSB2MiBbMl06Cj4+Pj4KPj4+PiAqIFBhdGNoZXMgMi00 IGFsbG93IHZpcnRpby1pb21tdSB0byB1c2UgdGhlIFBDSSB0cmFuc3BvcnQsIHNpbmNlIFFFTVUK Pj4+PiDCoMKgIHdvdWxkIGxpa2UgdG8gcGhhc2Ugb3V0IHRoZSBNTUlPIHRyYW5zcG9ydC4gVGhp cyBwcm9kdWNlcyBhIGNvbXBsZXgKPj4+PiDCoMKgIHRvcG9sb2d5IHdoZXJlIHRoZSBwcm9ncmFt bWluZyBpbnRlcmZhY2Ugb2YgdGhlIElPTU1VIGNvdWxkIGFwcGVhcgo+Pj4+IMKgwqAgbG93ZXIg dGhhbiB0aGUgZW5kcG9pbnRzIHRoYXQgaXQgdHJhbnNsYXRlcy4gSXQncyBub3QgdW5oZWFyZCBv ZiAoZS5nLgo+Pj4+IMKgwqAgQU1EIElPTU1VKSwgYW5kIHRoZSBndWVzdCBlYXNpbHkgY29wZXMg d2l0aCB0aGlzLgo+Pj4+IMKgwqAgCj4+Pj4gwqDCoCBUaGUgIkZpcm13YXJlIGRlc2NyaXB0aW9u IiBzZWN0aW9uIG9mIHRoZSBzcGVjaWZpY2F0aW9uIGhhcyBiZWVuCj4+Pj4gwqDCoCB1cGRhdGVk IHdpdGggYWxsIGNvbWJpbmF0aW9ucyBvZiBQQ0ksIE1NSU8gYW5kIERULCBBQ1BJLgo+Pj4KPj4+ IEkgaGF2ZSBhIHF1ZXN0aW9uIHdydCB0aGUgRlcgc3BlY2lmaWNhdGlvbi4gVGhlIElPTU1VIGNv bnN1bWVzIDEgc2xvdCBpbgo+Pj4gdGhlIFBDSSBkb21haW4gYW5kIG9uZSBuZWVkcyB0byBsZWF2 ZSBhIFJJRCBob2xlIGluIHRoZSBpb21tdS1tYXAuwqAgSXQKPj4+IGlzIG5vdCBvYnZpb3VzIHRv IG1lIHRoYXQgdGhpcyBSSUQgYWx3YXlzIGlzIHByZWRpY3RhYmxlIGdpdmVuIHRoZSBwY2llCj4+ PiBlbnVtZXJhdGlvbiBtZWNoYW5pc20uIEdlbmVyYWxseSB3ZSBoYXZlIGEgY29hcnNlIGdyYWlu IG1hcHBpbmcgb2YgUklECj4+PiBvbnRvIGlvbW11IHBoYW5kbGVzL1NUUkVBTUlEcy4gSGVyZSwg aWYgSSB1bmRlcnN0YW5kIGNvcnJlY3RseSB3ZSBuZWVkCj4+PiB0byBwcmVjaXNlbHkgaWRlbnRp ZnkgdGhlIFJJRCBncmFudGVkIHRvIHRoZSBpb21tdS4gT24gUUVNVSB0aGlzIG1heQo+Pj4gZGVw ZW5kIG9uIHRoZSBpbnN0YW50aWF0aW9uIG9yZGVyIG9mIHRoZSB2aXJ0aW8tcGNpIGRldmljZSBy aWdodD8KPj4gCj4+IFllcywgYWx0aG91Z2ggaXQgc2hvdWxkIGFsbCBoYXBwZW4gYmVmb3JlIHlv dSBib290IHRoZSBndWVzdCwgc2luY2UKPj4gdGhlcmUgaXMgbm8gaG90cGx1Z2dpbmcgYW4gSU9N TVUuIENvdWxkIHlvdSByZXNlcnZlIGEgUENJIHNsb3QgdXBmcm9udAo+PiBhbmQgdXNlIGl0IGZv ciB2aXJ0aW8taW9tbXUgbGF0ZXI/IE9yIGdlbmVyYXRlIHRoZSBpb21tdS1tYXAgYXQgdGhlIHNh bWUKPj4gdGltZSBhcyBnZW5lcmF0aW5nIHRoZSBjaGlsZCBub2RlIG9mIHRoZSBQQ0kgUkM/Cj4g Cj4gRXZlbiB3aGVuIGNvbGQtcGx1Z2dpbmcgdGhlIFBDSWUgZGV2aWNlcyB0aHJvdWdoIHFlbXUg Q0xJLCB0aGlzIGRlcGVuZHMKPiBvbiB0aGUgb3JkZXIgb2YgdGhlIHBjaWUgZGV2aWNlcyBpbiB0 aGUgbGlzdCBJIGd1ZXNzLiBJIG5lZWQgdG8gZnVydGhlcgo+IGV4cGVyaW1lbnQuCgpQbGVhc2Ug bGV0IG1lIGtub3cgaG93IGl0IGdvZXMuIEkgZ3Vlc3MgdGhlIHByb2JsZW0gd2lsbCBiZSB0aGUg c2FtZSBmb3IKYnVpbGRpbmcgSU9SVCB0YWJsZXM/IFlvdSdyZSBhbHNvIGdvaW5nIHRvIG5lZWQg YSBob2xlIGluIHRoZSBJRAptYXBwaW5ncyBvZiB0aGUgUENJIHJvb3QgY29tcGxleCBub2RlLgoK Pj4+IFNvCj4+PiB0aGlzIGRvZXMgbm90IGxvb2sgdHJpdmlhbCB0byBidWlsZCB0aGlzIGluZm8u IElzbid0IGl0IHBvc3NpYmxlIHRvIGRvCj4+PiB0aGlzIGV4Y2x1c2lvbiBhdCBrZXJuZWwgbGV2 ZWwgaW5zdGVhZD8KPj4gCj4+IFNvIGluIHRoZW9yeSBWSVJUSU9fRl9JT01NVV9QTEFURk9STSBh bHJlYWR5IGRvZXMgdGhhdDoKPj4gCj4+IFZJUlRJT19GX0lPTU1VX1BMQVRGT1JNKDMzKQo+PsKg wqDCoMKgIFRoaXMgZmVhdHVyZSBpbmRpY2F0ZXMgdGhhdCB0aGUgZGV2aWNlIGlzIGJlaGluZCBh biBJT01NVSB0aGF0Cj4+wqDCoMKgwqAgdHJhbnNsYXRlcyBidXMgYWRkcmVzc2VzIGZyb20gdGhl IGRldmljZSBpbnRvIHBoeXNpY2FsIGFkZHJlc3NlcyBpbgo+PsKgwqDCoMKgIG1lbW9yeS4gSWYg dGhpcyBmZWF0dXJlIGJpdCBpcyBzZXQgdG8gMCwgdGhlbiB0aGUgZGV2aWNlIGVtaXRzCj4+wqDC oMKgwqAgcGh5c2ljYWwgYWRkcmVzc2VzIHdoaWNoIGFyZSBub3QgdHJhbnNsYXRlZCBmdXJ0aGVy LCBldmVuIHRob3VnaCBhbgo+PsKgwqDCoMKgIElPTU1VIG1heSBiZSBwcmVzZW50Lgo+IAo+IFRo aXMgdGVsbHMgdGhlIGRyaXZlciB0byB1c2UgdGhlIGRtYSBhcGksIHJpZ2h0PyAKClRoYXQncyBo b3cgTGludXggaW1wbGVtZW50cyB0aGUgYml0LCBpbnN0YWxsIGN1c3RvbSBETUEgb3BzIHdoZW4g dGhlIGJpdAppcyBhYnNlbnQuIEJ1dCBpdCBkb2Vzbid0IHdvcmsgZm9yIGV2ZXJ5b25lIGFuZCBo YXMgY2F1c2VkIGEgbG90IG9mCmRlYmF0ZSAoaHR0cHM6Ly9wYXRjaHdvcmsub3psYWJzLm9yZy9j b3Zlci85NDY3MDgvKQoKPiBFZmZlY3RpdmVseSB0aGlzCj4gZXhwbGljaXRseSBzYXlzIHdoZXRo ZXIgdGhlIGRldmljZSBpcyBzdXBwb3NlZCB0byBiZSB1cGZyb250IGFuIElPTU1VLgoKWWVzLiBJ dCdzIHF1aXRlIHN0cmFuZ2UgaWYgeW91IGNvbnNpZGVyIGhvdHBsdWdnYWJsZSBoYXJkd2FyZSwg c2luY2UKdGhvc2UgZGV2aWNlcyBzaG91bGRuJ3QgZ2V0IHRvIGNob29zZSB3aGV0aGVyIHRoZXkg YXJlIG1hbmFnZWQgYnkgYW4KSU9NTVUuIEZvciB0aGUgSU9NTVUgaXRzZWxmLCBpdCBzaG91bGQg YmUgZmluZQoKPj4gRm9yIGJldHRlciBvciBmb3Igd29yc2UsIHRoZSBndWVzdCBoYXMgdG8gaW1w bGVtZW50IGl0LiBJZiB0aGlzIGZlYXR1cmUKPj4gYml0IGlzIHVuc2V0IGZvciB2aXJ0aW8taW9t bXUsIGl0IGRvZXMgRE1BIG9uIHRoZSBwaHlzaWNhbCBhZGRyZXNzCj4+IHNwYWNlLCByZWdhcmRs ZXNzIG9mIHdoYXQgdGhlIHN0YXRpYyB0b3BvbG9neSBkZXNjcmlwdGlvbiBzYXlzLgo+PiAKPj4g SW4gcHJhY3RpY2UgaXQgZG9lc24ndCBxdWl0ZSB3b3JrLiBJZiB5b3VyIGlvbW11LW1hcCBkZXNj cmliZXMgdGhlIElPTU1VCj4+IGFzIHRyYW5zbGF0aW5nIGl0c2VsZiwgTGludXgnIE9GIGNvZGUg d2lsbCB3YWl0IGZvciB0aGUgSU9NTVUgdG8gYmUKPj4gcHJvYmVkIGJlZm9yZSBwcm9iaW5nIHRo ZSBJT01NVS4gV29ya2luZyBhcm91bmQgdGhpcyB3aXRoIGhhY2tzIGlzCj4+IHBvc3NpYmxlLCBi dXQgSSBkb24ndCB3YW50IHRvIGludHJvZHVjZSBtb3JlIHF1ZXN0aW9uYWJsZSBjb2RlIHRvIE9G IGFuZAo+PiBkZXZpY2UgdHJlZSBiaW5kaW5ncyBpZiB0aGVyZSBpcyBhbnkgb3RoZXIgd2F5Lgo+ IEh1bSBvay4gSSBjYW5ub3QgcmVhbGx5IGNvbW1lbnQgb24gdGhpcy4KPiAKPiBJIGp1c3Qgd2Fu dGVkIHRvIHJhaXNlIHRoaXMgY29uY2VybiBhYm91dCBSSUQgaWRlbnRmaWNhdGlvbi4KCldlIGNh biBhbHdheXMgdHJ5LiBSZWxheGluZyBpb21tdS1tYXAgZnVydGhlciB3b3VsZCBiZSBvbmUgYWRk aXRpb25hbApwYXRjaCB0byBEb2N1bWVudGF0aW9uL2RldmljZXRyZWUvYmluZGluZ3MvcGNpL3Bj aS1pb21tdS50eHQsIGFuZCBvbmUgdG8KZHJpdmVycy9pb21tdS9vZi1pb21tdS5jLiBJJ2QgcmF0 aGVyIG1ha2UgaXQgYSBzZXBhcmF0ZSBSRkMuCgpTaW5jZSB3ZSBuZWVkIGFja3MgZnJvbSBhbiBP RiBtYWludGFpbmVyIGFuZCBJJ2QgYWxzbyBsaWtlIEpvZXJnJ3MKYXBwcm92YWwgZm9yIGFkZGlu ZyBhIG5ldyBkcml2ZXIgdG8gdGhlIElPTU1VIHRyZWUsIEkgdGhpbmsgaXQncyB0b28KbGF0ZSBm b3IgdGhpcyBpdGVyYXRpb24uIEkgd2Fzbid0IGludGVuZGluZyBmb3IgdGhpcyB0byBnbyBpbnRv IDQuMjAsCmp1c3QgaGF2ZSBzb21ldGhpbmcgdG8gZGlzY3VzcyBhdCBLVk0gZm9ydW0gbmV4dCB3 ZWVrLgoKVGhhbmtzLApKZWFuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlzdApWaXJ0dWFsaXphdGlvbkBsaXN0 cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRhdGlvbi5vcmcv bWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91EB6ECDE32 for ; Wed, 17 Oct 2018 11:54:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 58BCD2152A for ; Wed, 17 Oct 2018 11:54:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 58BCD2152A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-pci-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727192AbeJQTuG (ORCPT ); Wed, 17 Oct 2018 15:50:06 -0400 Received: from foss.arm.com ([217.140.101.70]:50494 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727013AbeJQTuG (ORCPT ); Wed, 17 Oct 2018 15:50:06 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CD1B280D; Wed, 17 Oct 2018 04:54:44 -0700 (PDT) Received: from [10.1.196.78] (unknown [10.1.196.78]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 673E63F59C; Wed, 17 Oct 2018 04:54:41 -0700 (PDT) Subject: Re: [PATCH v3 0/7] Add virtio-iommu driver To: Auger Eric , "iommu@lists.linux-foundation.org" , "virtualization@lists.linux-foundation.org" , "devicetree@vger.kernel.org" Cc: "linux-pci@vger.kernel.org" , "kvmarm@lists.cs.columbia.edu" , "peter.maydell@linaro.org" , "joro@8bytes.org" , "mst@redhat.com" , "jasowang@redhat.com" , "robh+dt@kernel.org" , Mark Rutland , "tnowicki@caviumnetworks.com" , "kevin.tian@intel.com" , Marc Zyngier , Robin Murphy , Will Deacon , Lorenzo Pieralisi References: <20181012145917.6840-1-jean-philippe.brucker@arm.com> <7874471b-3711-ca44-d220-5e756ac7db8c@redhat.com> From: Jean-Philippe Brucker Message-ID: <40dddb20-6248-5bb0-e0ed-48bacd1867a1@arm.com> Date: Wed, 17 Oct 2018 12:54:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On 16/10/2018 21:31, Auger Eric wrote: > Hi Jean, > > On 10/16/18 8:44 PM, Jean-Philippe Brucker wrote: >> On 16/10/2018 10:25, Auger Eric wrote: >>> Hi Jean, >>> >>> On 10/12/18 4:59 PM, Jean-Philippe Brucker wrote: >>>> Implement the virtio-iommu driver, following specification v0.8 [1]. >>>> Changes since v2 [2]: >>>> >>>> * Patches 2-4 allow virtio-iommu to use the PCI transport, since QEMU >>>>    would like to phase out the MMIO transport. This produces a complex >>>>    topology where the programming interface of the IOMMU could appear >>>>    lower than the endpoints that it translates. It's not unheard of (e.g. >>>>    AMD IOMMU), and the guest easily copes with this. >>>>    >>>>    The "Firmware description" section of the specification has been >>>>    updated with all combinations of PCI, MMIO and DT, ACPI. >>> >>> I have a question wrt the FW specification. The IOMMU consumes 1 slot in >>> the PCI domain and one needs to leave a RID hole in the iommu-map.  It >>> is not obvious to me that this RID always is predictable given the pcie >>> enumeration mechanism. Generally we have a coarse grain mapping of RID >>> onto iommu phandles/STREAMIDs. Here, if I understand correctly we need >>> to precisely identify the RID granted to the iommu. On QEMU this may >>> depend on the instantiation order of the virtio-pci device right? >> >> Yes, although it should all happen before you boot the guest, since >> there is no hotplugging an IOMMU. Could you reserve a PCI slot upfront >> and use it for virtio-iommu later? Or generate the iommu-map at the same >> time as generating the child node of the PCI RC? > > Even when cold-plugging the PCIe devices through qemu CLI, this depends > on the order of the pcie devices in the list I guess. I need to further > experiment. Please let me know how it goes. I guess the problem will be the same for building IORT tables? You're also going to need a hole in the ID mappings of the PCI root complex node. >>> So >>> this does not look trivial to build this info. Isn't it possible to do >>> this exclusion at kernel level instead? >> >> So in theory VIRTIO_F_IOMMU_PLATFORM already does that: >> >> VIRTIO_F_IOMMU_PLATFORM(33) >>     This feature indicates that the device is behind an IOMMU that >>     translates bus addresses from the device into physical addresses in >>     memory. If this feature bit is set to 0, then the device emits >>     physical addresses which are not translated further, even though an >>     IOMMU may be present. > > This tells the driver to use the dma api, right? That's how Linux implements the bit, install custom DMA ops when the bit is absent. But it doesn't work for everyone and has caused a lot of debate (https://patchwork.ozlabs.org/cover/946708/) > Effectively this > explicitly says whether the device is supposed to be upfront an IOMMU. Yes. It's quite strange if you consider hotpluggable hardware, since those devices shouldn't get to choose whether they are managed by an IOMMU. For the IOMMU itself, it should be fine >> For better or for worse, the guest has to implement it. If this feature >> bit is unset for virtio-iommu, it does DMA on the physical address >> space, regardless of what the static topology description says. >> >> In practice it doesn't quite work. If your iommu-map describes the IOMMU >> as translating itself, Linux' OF code will wait for the IOMMU to be >> probed before probing the IOMMU. Working around this with hacks is >> possible, but I don't want to introduce more questionable code to OF and >> device tree bindings if there is any other way. > Hum ok. I cannot really comment on this. > > I just wanted to raise this concern about RID identfication. We can always try. Relaxing iommu-map further would be one additional patch to Documentation/devicetree/bindings/pci/pci-iommu.txt, and one to drivers/iommu/of-iommu.c. I'd rather make it a separate RFC. Since we need acks from an OF maintainer and I'd also like Joerg's approval for adding a new driver to the IOMMU tree, I think it's too late for this iteration. I wasn't intending for this to go into 4.20, just have something to discuss at KVM forum next week. Thanks, Jean