From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98F0BC46470 for ; Wed, 8 Aug 2018 13:19:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4633D21729 for ; Wed, 8 Aug 2018 13:19:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4633D21729 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.crashing.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727290AbeHHPjN (ORCPT ); Wed, 8 Aug 2018 11:39:13 -0400 Received: from gate.crashing.org ([63.228.1.57]:42490 "EHLO gate.crashing.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726772AbeHHPjM (ORCPT ); Wed, 8 Aug 2018 11:39:12 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by gate.crashing.org (8.14.1/8.14.1) with ESMTP id w78DIDkg003834; Wed, 8 Aug 2018 08:18:15 -0500 Message-ID: Subject: Re: [RFC 0/4] Virtio uses DMA API for all devices From: Benjamin Herrenschmidt To: Christoph Hellwig Cc: "Michael S. Tsirkin" , Will Deacon , Anshuman Khandual , virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, aik@ozlabs.ru, robh@kernel.org, joe@perches.com, elfring@users.sourceforge.net, david@gibson.dropbear.id.au, jasowang@redhat.com, mpe@ellerman.id.au, linuxram@us.ibm.com, haren@linux.vnet.ibm.com, paulus@samba.org, srikar@linux.vnet.ibm.com, robin.murphy@arm.com, jean-philippe.brucker@arm.com, marc.zyngier@arm.com Date: Wed, 08 Aug 2018 23:18:13 +1000 In-Reply-To: <20180808123036.GA2525@infradead.org> References: <20180805072930.GB23288@infradead.org> <20180806094243.GA16032@infradead.org> <6c707d6d33ac25a42265c2e9b521c2416d72c739.camel@kernel.crashing.org> <20180807062117.GD32709@infradead.org> <20180807135505.GA29034@infradead.org> <2103ecfe52d23cec03f185d08a87bfad9c9d82b5.camel@kernel.crashing.org> <20180808063158.GA2474@infradead.org> <4b596883892b5cb5560bef26fcd249e7107173ac.camel@kernel.crashing.org> <20180808123036.GA2525@infradead.org> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.4 (3.28.4-1.fc28) Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2018-08-08 at 05:30 -0700, Christoph Hellwig wrote: > On Wed, Aug 08, 2018 at 08:07:49PM +1000, Benjamin Herrenschmidt wrote: > > Qemu virtio bypasses that iommu when the VIRTIO_F_IOMMU_PLATFORM flag > > is not set (default) but there's nothing in the device-tree to tell the > > guest about this since it's a violation of our pseries architecture, so > > we just rely on Linux virtio "knowing" that it happens. It's a bit > > yucky but that's now history... > > That is ugly as hell, but it is how virtio works everywhere, so nothing > special so far. Yup. > > Essentially pseries "architecturally" does not have the concept of not > > having an iommu in the way and qemu violates that architecture today. > > > > (Remember it comes from pHyp, our priorietary HV, which we are somewhat > > mimmicing here). > > It shouldnt be too hard to have a dt property that communicates this, > should it? We could invent something I suppose. The additional problem then (yeah I know ... what a mess) is that qemu doesn't create the DT for PCI devices, the firmware (SLOF) inside the guest does using normal PCI probing. That said, that FW could know about all the virtio vendor/device IDs, check the VIRTIO_F_IOMMU_PLATFORM and set that property accordingly... messy but doable. It's not a bus property (see my other reply below as this could complicate things with your bus mask). But we are drifting from the problem at hand :-) You propose we do set VIRTIO_F_IOMMU_PLATFORM so we aren't in the above case, and the bypass stuff works, so no need to touch it. See my recap at the end of the email to make sure I understand fully what you suggest. > > So if we always set VIRTIO_F_IOMMU_PLATFORM, it *will* force all virtio > > through that iommu and performance will suffer (esp vhost I suspect), > > especially since adding/removing translations in the iommu is a > > hypercall. > Well, we'd nee to make sure that for this particular bus we skip the > actualy iommu. It's not a bus property. Qemu will happily mix up everything on the same bus, that includes emulated devices that go through the emulated iommu, real VFIO devices that go through an actual HW iommu and virtio that bypasses everything. This makes things tricky in general (not just in my powerpc secure VM case) since, at least on powerpc but I suppose elsewhere too, iommu related properties tend to be per "bus" while here, qemu will mix and match. But again, I think we are drifting away from the topic, see below > > > It would not be the same effect. The problem with that is that you must > > > now assumes that your qemu knows that for example you might be passing > > > a dma offset if the bus otherwise requires it. > > > > I would assume that arch_virtio_wants_dma_ops() only returns true when > > no such offsets are involved, at least in our case that would be what > > happens. > > That would work, but we're really piling hacĸs ontop of hacks here. Sort-of :-) At least none of what we are discussing now involves touching the dma_ops themselves so we are not in the way of your big cleanup operation here. But yeah, let's continue discussing your other solution below. > > > Or in other words: > > > you potentially break the contract between qemu and the guest of always > > > passing down physical addresses. If we explicitly change that contract > > > through using a flag that says you pass bus address everything is fine. > > > > For us a "bus address" is behind the iommu so that's what > > VIRTIO_F_IOMMU_PLATFORM does already. We don't have the concept of a > > bus address that is different. I suppose it's an ARMism to have DMA > > offsets that are separate from iommus ? > > No, a lot of platforms support a bus address that has an offset from > the physical address. including a lot of power platforms: Ok, just talking past each other :-) For all the powerpc ones, these *do* go through the iommu, which is what I meant. It's just a window of the iommu that provides some kind of direct mapping of memory. For pseries, there is no such thing however. What we do to avoid constant map/unmap of iommu PTEs in pseries guests is that we use hypercalls to create a 64-bit window and populate all its PTEs with an identity mapping. But that's not as efficient as a real bypass. There are good historical reasons for that, since pseries is a guest platform, its memory is never really where the guest thinks it is, so you always need an iommu to remap. Even for virtual devices, since for most of them, in the "IBM" pHyp model, the "peer" is actually another partition, so the virtual iommu handles translating accross the two partitions. Same goes with cell in HW, no real bypass, just the iommu being confiured with very large pages and a fixed mapping. powernv has a separate physical window that can be configured as a real bypass though, so does the U4 DART. Not sure about the FSL one. But yeah, your point stands, this is just implementation details. > arch/powerpc/kernel/pci-common.c: set_dma_offset(&dev->dev, PCI_DRAM_OFFSET); > arch/powerpc/platforms/cell/iommu.c: set_dma_offset(dev, cell_dma_nommu_offset); > arch/powerpc/platforms/cell/iommu.c: set_dma_offset(dev, addr); > arch/powerpc/platforms/powernv/pci-ioda.c: set_dma_offset(&pdev->dev, pe->tce_bypass_base); > arch/powerpc/platforms/powernv/pci-ioda.c: set_dma_offset(&pdev->dev, (1ULL << 32)); > arch/powerpc/platforms/powernv/pci-ioda.c: set_dma_offset(&dev->dev, pe->tce_bypass_base); > arch/powerpc/platforms/pseries/iommu.c: set_dma_offset(dev, dma_offset); > arch/powerpc/sysdev/dart_iommu.c: set_dma_offset(&dev->dev, DART_U4_BYPASS_BASE); > arch/powerpc/sysdev/fsl_pci.c: set_dma_offset(dev, pci64_dma_offset); > > to make things worse some platforms (at least on arm/arm64/mips/x86) can > also require additional banking where it isn't even a single linear map > but multiples windows. Sure, but all of this is just the configuration of the iommu. But I think we agree here, and your point remains valid, indeed my proposed hack: > if ((flags & VIRTIO_F_IOMMU_PLATFORM) || arch_virtio_wants_dma_ops()) Will only work if the IOMMU and non-IOMMU path are completely equivalent. We can provide that guarantee for our secure VM case, but not generally so if we were to go down the route of a quirk in virtio, it might be better to make it painfully obvious that it's specific to that one case with a different kind of turd: - if (xen_domain()) + if (xen_domain() || pseries_secure_vm()) return true; So to summarize, and make sure I'm not missing something, the two approaches at hand are either: 1- The above, which is a one liner and contained in the guest, so that's nice, but also means another turd in virtio which isn't ... 2- We force pseries to always set VIRTIO_F_IOMMU_PLATFORM, but with the current architecture on our side that will force virtio to always go through an emulated iommu, as pseries doesn't have the concept of a real bypass window, and thus will impact performance for both secure and non-secure VMs. 3- Invent a property that can be put in selected PCI device tree nodes that indicates that for that device specifically, the iommu can be bypassed, along with a hypercall to turn that bypass on/off. Virtio would then use VIRTIO_F_IOMMU_PLATFORM but its DT nodes would also have that property and Linux would notice it and turn bypass on. The resulting properties of those options are: 1- Is what I want because it's the simplest, provides the best performance now, and works without code changes to qemu or non-secure Linux. However it does add a tiny turd to virtio which is annoying. 2- This works but it puts the iommu in the way always, thus reducing virtio performance accross the board for pseries unless we only do that for secure VMs but that is difficult (as discussed earlier). 3- This would recover the performance lost in -2-, however it requires qemu *and* guest changes. Specifically, existing guests (RHEL 7 etc...) would get the performance hit of -2- unless modified to call that 'enable bypass' call, which isn't great. So imho we have to chose one of 3 not-great solutions here... Unless I missed something in your ideas of course. Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: [RFC 0/4] Virtio uses DMA API for all devices Date: Wed, 08 Aug 2018 23:18:13 +1000 Message-ID: References: <20180805072930.GB23288@infradead.org> <20180806094243.GA16032@infradead.org> <6c707d6d33ac25a42265c2e9b521c2416d72c739.camel@kernel.crashing.org> <20180807062117.GD32709@infradead.org> <20180807135505.GA29034@infradead.org> <2103ecfe52d23cec03f185d08a87bfad9c9d82b5.camel@kernel.crashing.org> <20180808063158.GA2474@infradead.org> <4b596883892b5cb5560bef26fcd249e7107173ac.camel@kernel.crashing.org> <20180808123036.GA2525@infradead.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20180808123036.GA2525@infradead.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Christoph Hellwig Cc: robh@kernel.org, srikar@linux.vnet.ibm.com, "Michael S. Tsirkin" , mpe@ellerman.id.au, Will Deacon , linux-kernel@vger.kernel.org, linuxram@us.ibm.com, virtualization@lists.linux-foundation.org, jean-philippe.brucker@arm.com, paulus@samba.org, marc.zyngier@arm.com, joe@perches.com, robin.murphy@arm.com, david@gibson.dropbear.id.au, linuxppc-dev@lists.ozlabs.org, elfring@users.sourceforge.net, haren@linux.vnet.ibm.com, Anshuman Khandual List-Id: virtualization@lists.linuxfoundation.org T24gV2VkLCAyMDE4LTA4LTA4IGF0IDA1OjMwIC0wNzAwLCBDaHJpc3RvcGggSGVsbHdpZyB3cm90 ZToKPiBPbiBXZWQsIEF1ZyAwOCwgMjAxOCBhdCAwODowNzo0OVBNICsxMDAwLCBCZW5qYW1pbiBI ZXJyZW5zY2htaWR0IHdyb3RlOgo+ID4gUWVtdSB2aXJ0aW8gYnlwYXNzZXMgdGhhdCBpb21tdSB3 aGVuIHRoZSBWSVJUSU9fRl9JT01NVV9QTEFURk9STSBmbGFnCj4gPiBpcyBub3Qgc2V0IChkZWZh dWx0KSBidXQgdGhlcmUncyBub3RoaW5nIGluIHRoZSBkZXZpY2UtdHJlZSB0byB0ZWxsIHRoZQo+ ID4gZ3Vlc3QgYWJvdXQgdGhpcyBzaW5jZSBpdCdzIGEgdmlvbGF0aW9uIG9mIG91ciBwc2VyaWVz IGFyY2hpdGVjdHVyZSwgc28KPiA+IHdlIGp1c3QgcmVseSBvbiBMaW51eCB2aXJ0aW8gImtub3dp bmciIHRoYXQgaXQgaGFwcGVucy4gSXQncyBhIGJpdAo+ID4geXVja3kgYnV0IHRoYXQncyBub3cg aGlzdG9yeS4uLgo+IAo+IFRoYXQgaXMgdWdseSBhcyBoZWxsLCBidXQgaXQgaXMgaG93IHZpcnRp byB3b3JrcyBldmVyeXdoZXJlLCBzbyBub3RoaW5nCj4gc3BlY2lhbCBzbyBmYXIuCgpZdXAuCgo+ ID4gRXNzZW50aWFsbHkgcHNlcmllcyAiYXJjaGl0ZWN0dXJhbGx5IiBkb2VzIG5vdCBoYXZlIHRo ZSBjb25jZXB0IG9mIG5vdAo+ID4gaGF2aW5nIGFuIGlvbW11IGluIHRoZSB3YXkgYW5kIHFlbXUg dmlvbGF0ZXMgdGhhdCBhcmNoaXRlY3R1cmUgdG9kYXkuCj4gPiAKPiA+IChSZW1lbWJlciBpdCBj b21lcyBmcm9tIHBIeXAsIG91ciBwcmlvcmlldGFyeSBIViwgd2hpY2ggd2UgYXJlIHNvbWV3aGF0 Cj4gPiBtaW1taWNpbmcgaGVyZSkuCj4gCj4gSXQgc2hvdWxkbnQgYmUgdG9vIGhhcmQgdG8gaGF2 ZSBhIGR0IHByb3BlcnR5IHRoYXQgY29tbXVuaWNhdGVzIHRoaXMsCj4gc2hvdWxkIGl0PwoKV2Ug Y291bGQgaW52ZW50IHNvbWV0aGluZyBJIHN1cHBvc2UuIFRoZSBhZGRpdGlvbmFsIHByb2JsZW0g dGhlbiAoeWVhaApJIGtub3cgLi4uIHdoYXQgYSBtZXNzKSBpcyB0aGF0IHFlbXUgZG9lc24ndCBj cmVhdGUgdGhlIERUIGZvciBQQ0kKZGV2aWNlcywgdGhlIGZpcm13YXJlIChTTE9GKSBpbnNpZGUg dGhlIGd1ZXN0IGRvZXMgdXNpbmcgbm9ybWFsIFBDSQpwcm9iaW5nLgoKVGhhdCBzYWlkLCB0aGF0 IEZXIGNvdWxkIGtub3cgYWJvdXQgYWxsIHRoZSB2aXJ0aW8gdmVuZG9yL2RldmljZSBJRHMsCmNo ZWNrIHRoZSBWSVJUSU9fRl9JT01NVV9QTEFURk9STSBhbmQgc2V0IHRoYXQgcHJvcGVydHkgYWNj b3JkaW5nbHkuLi4KbWVzc3kgYnV0IGRvYWJsZS4gSXQncyBub3QgYSBidXMgcHJvcGVydHkgKHNl ZSBteSBvdGhlciByZXBseSBiZWxvdyBhcwp0aGlzIGNvdWxkIGNvbXBsaWNhdGUgdGhpbmdzIHdp dGggeW91ciBidXMgbWFzaykuCgpCdXQgd2UgYXJlIGRyaWZ0aW5nIGZyb20gdGhlIHByb2JsZW0g YXQgaGFuZCA6LSkgWW91IHByb3Bvc2Ugd2UgZG8gc2V0ClZJUlRJT19GX0lPTU1VX1BMQVRGT1JN IHNvIHdlIGFyZW4ndCBpbiB0aGUgYWJvdmUgY2FzZSwgYW5kIHRoZSBieXBhc3MKc3R1ZmYgd29y a3MsIHNvIG5vIG5lZWQgdG8gdG91Y2ggaXQuCgpTZWUgbXkgcmVjYXAgYXQgdGhlIGVuZCBvZiB0 aGUgZW1haWwgdG8gbWFrZSBzdXJlIEkgdW5kZXJzdGFuZCBmdWxseQp3aGF0IHlvdSBzdWdnZXN0 LgoKPiA+IFNvIGlmIHdlIGFsd2F5cyBzZXQgVklSVElPX0ZfSU9NTVVfUExBVEZPUk0sIGl0ICp3 aWxsKiBmb3JjZSBhbGwgdmlydGlvCj4gPiB0aHJvdWdoIHRoYXQgaW9tbXUgYW5kIHBlcmZvcm1h bmNlIHdpbGwgc3VmZmVyIChlc3Agdmhvc3QgSSBzdXNwZWN0KSwKPiA+IGVzcGVjaWFsbHkgc2lu Y2UgYWRkaW5nL3JlbW92aW5nIHRyYW5zbGF0aW9ucyBpbiB0aGUgaW9tbXUgaXMgYQo+ID4gaHlw ZXJjYWxsLgoKPiBXZWxsLCB3ZSdkIG5lZSB0byBtYWtlIHN1cmUgdGhhdCBmb3IgdGhpcyBwYXJ0 aWN1bGFyIGJ1cyB3ZSBza2lwIHRoZQo+IGFjdHVhbHkgaW9tbXUuCgpJdCdzIG5vdCBhIGJ1cyBw cm9wZXJ0eS4gUWVtdSB3aWxsIGhhcHBpbHkgbWl4IHVwIGV2ZXJ5dGhpbmcgb24gdGhlCnNhbWUg YnVzLCB0aGF0IGluY2x1ZGVzIGVtdWxhdGVkIGRldmljZXMgdGhhdCBnbyB0aHJvdWdoIHRoZSBl bXVsYXRlZAppb21tdSwgcmVhbCBWRklPIGRldmljZXMgdGhhdCBnbyB0aHJvdWdoIGFuIGFjdHVh bCBIVyBpb21tdSBhbmQgdmlydGlvCnRoYXQgYnlwYXNzZXMgZXZlcnl0aGluZy4KClRoaXMgbWFr ZXMgdGhpbmdzIHRyaWNreSBpbiBnZW5lcmFsIChub3QganVzdCBpbiBteSBwb3dlcnBjIHNlY3Vy ZSBWTQpjYXNlKSBzaW5jZSwgYXQgbGVhc3Qgb24gcG93ZXJwYyBidXQgSSBzdXBwb3NlIGVsc2V3 aGVyZSB0b28sIGlvbW11CnJlbGF0ZWQgcHJvcGVydGllcyB0ZW5kIHRvIGJlIHBlciAiYnVzIiB3 aGlsZSBoZXJlLCBxZW11IHdpbGwgbWl4IGFuZAptYXRjaC4KCkJ1dCBhZ2FpbiwgSSB0aGluayB3 ZSBhcmUgZHJpZnRpbmcgYXdheSBmcm9tIHRoZSB0b3BpYywgc2VlIGJlbG93Cgo+ID4gPiBJdCB3 b3VsZCBub3QgYmUgdGhlIHNhbWUgZWZmZWN0LiAgVGhlIHByb2JsZW0gd2l0aCB0aGF0IGlzIHRo YXQgeW91IG11c3QKPiA+ID4gbm93IGFzc3VtZXMgdGhhdCB5b3VyIHFlbXUga25vd3MgdGhhdCBm b3IgZXhhbXBsZSB5b3UgbWlnaHQgYmUgcGFzc2luZwo+ID4gPiBhIGRtYSBvZmZzZXQgaWYgdGhl IGJ1cyBvdGhlcndpc2UgcmVxdWlyZXMgaXQuIAo+ID4gCj4gPiBJIHdvdWxkIGFzc3VtZSB0aGF0 IGFyY2hfdmlydGlvX3dhbnRzX2RtYV9vcHMoKSBvbmx5IHJldHVybnMgdHJ1ZSB3aGVuCj4gPiBu byBzdWNoIG9mZnNldHMgYXJlIGludm9sdmVkLCBhdCBsZWFzdCBpbiBvdXIgY2FzZSB0aGF0IHdv dWxkIGJlIHdoYXQKPiA+IGhhcHBlbnMuCj4gCj4gVGhhdCB3b3VsZCB3b3JrLCBidXQgd2UncmUg cmVhbGx5IHBpbGluZyBoYWPEuHMgb250b3Agb2YgaGFja3MgaGVyZS4KClNvcnQtb2YgOi0pIEF0 IGxlYXN0IG5vbmUgb2Ygd2hhdCB3ZSBhcmUgZGlzY3Vzc2luZyBub3cgaW52b2x2ZXMKdG91Y2hp bmcgdGhlIGRtYV9vcHMgdGhlbXNlbHZlcyBzbyB3ZSBhcmUgbm90IGluIHRoZSB3YXkgb2YgeW91 ciBiaWcKY2xlYW51cCBvcGVyYXRpb24gaGVyZS4gQnV0IHllYWgsIGxldCdzIGNvbnRpbnVlIGRp c2N1c3NpbmcgeW91ciBvdGhlcgpzb2x1dGlvbiBiZWxvdy4KCj4gPiA+ICBPciBpbiBvdGhlciB3 b3JkczoKPiA+ID4geW91IHBvdGVudGlhbGx5IGJyZWFrIHRoZSBjb250cmFjdCBiZXR3ZWVuIHFl bXUgYW5kIHRoZSBndWVzdCBvZiBhbHdheXMKPiA+ID4gcGFzc2luZyBkb3duIHBoeXNpY2FsIGFk ZHJlc3Nlcy4gIElmIHdlIGV4cGxpY2l0bHkgY2hhbmdlIHRoYXQgY29udHJhY3QKPiA+ID4gdGhy b3VnaCB1c2luZyBhIGZsYWcgdGhhdCBzYXlzIHlvdSBwYXNzIGJ1cyBhZGRyZXNzIGV2ZXJ5dGhp bmcgaXMgZmluZS4KPiA+IAo+ID4gRm9yIHVzIGEgImJ1cyBhZGRyZXNzIiBpcyBiZWhpbmQgdGhl IGlvbW11IHNvIHRoYXQncyB3aGF0Cj4gPiBWSVJUSU9fRl9JT01NVV9QTEFURk9STSBkb2VzIGFs cmVhZHkuIFdlIGRvbid0IGhhdmUgdGhlIGNvbmNlcHQgb2YgYQo+ID4gYnVzIGFkZHJlc3MgdGhh dCBpcyBkaWZmZXJlbnQuIEkgc3VwcG9zZSBpdCdzIGFuIEFSTWlzbSB0byBoYXZlIERNQQo+ID4g b2Zmc2V0cyB0aGF0IGFyZSBzZXBhcmF0ZSBmcm9tIGlvbW11cyA/IAo+IAo+IE5vLCBhIGxvdCBv ZiBwbGF0Zm9ybXMgc3VwcG9ydCBhIGJ1cyBhZGRyZXNzIHRoYXQgaGFzIGFuIG9mZnNldCBmcm9t Cj4gdGhlIHBoeXNpY2FsIGFkZHJlc3MuIGluY2x1ZGluZyBhIGxvdCBvZiBwb3dlciBwbGF0Zm9y bXM6CgpPaywganVzdCB0YWxraW5nIHBhc3QgZWFjaCBvdGhlciA6LSkgRm9yIGFsbCB0aGUgcG93 ZXJwYyBvbmVzLCB0aGVzZQoqZG8qIGdvIHRocm91Z2ggdGhlIGlvbW11LCB3aGljaCBpcyB3aGF0 IEkgbWVhbnQuIEl0J3MganVzdCBhIHdpbmRvdyBvZgp0aGUgaW9tbXUgdGhhdCBwcm92aWRlcyBz b21lIGtpbmQgb2YgZGlyZWN0IG1hcHBpbmcgb2YgbWVtb3J5LgoKRm9yIHBzZXJpZXMsIHRoZXJl IGlzIG5vIHN1Y2ggdGhpbmcgaG93ZXZlci4gV2hhdCB3ZSBkbyB0byBhdm9pZApjb25zdGFudCBt YXAvdW5tYXAgb2YgaW9tbXUgUFRFcyBpbiBwc2VyaWVzIGd1ZXN0cyBpcyB0aGF0IHdlIHVzZQpo eXBlcmNhbGxzIHRvIGNyZWF0ZSBhIDY0LWJpdCB3aW5kb3cgYW5kIHBvcHVsYXRlIGFsbCBpdHMg UFRFcyB3aXRoIGFuCmlkZW50aXR5IG1hcHBpbmcuIEJ1dCB0aGF0J3Mgbm90IGFzIGVmZmljaWVu dCBhcyBhIHJlYWwgYnlwYXNzLgoKVGhlcmUgYXJlIGdvb2QgaGlzdG9yaWNhbCByZWFzb25zIGZv ciB0aGF0LCBzaW5jZSBwc2VyaWVzIGlzIGEgZ3Vlc3QKcGxhdGZvcm0sIGl0cyBtZW1vcnkgaXMg bmV2ZXIgcmVhbGx5IHdoZXJlIHRoZSBndWVzdCB0aGlua3MgaXQgaXMsIHNvCnlvdSBhbHdheXMg bmVlZCBhbiBpb21tdSB0byByZW1hcC4gRXZlbiBmb3IgdmlydHVhbCBkZXZpY2VzLCBzaW5jZSBm b3IKbW9zdCBvZiB0aGVtLCBpbiB0aGUgIklCTSIgcEh5cCBtb2RlbCwgdGhlICJwZWVyIiBpcyBh Y3R1YWxseSBhbm90aGVyCnBhcnRpdGlvbiwgc28gdGhlIHZpcnR1YWwgaW9tbXUgaGFuZGxlcyB0 cmFuc2xhdGluZyBhY2Nyb3NzIHRoZSB0d28KcGFydGl0aW9ucy4KClNhbWUgZ29lcyB3aXRoIGNl bGwgaW4gSFcsIG5vIHJlYWwgYnlwYXNzLCBqdXN0IHRoZSBpb21tdSBiZWluZwpjb25maXVyZWQg d2l0aCB2ZXJ5IGxhcmdlIHBhZ2VzIGFuZCBhIGZpeGVkIG1hcHBpbmcuCgpwb3dlcm52IGhhcyBh IHNlcGFyYXRlIHBoeXNpY2FsIHdpbmRvdyB0aGF0IGNhbiBiZSBjb25maWd1cmVkIGFzIGEgcmVh bApieXBhc3MgdGhvdWdoLCBzbyBkb2VzIHRoZSBVNCBEQVJULiBOb3Qgc3VyZSBhYm91dCB0aGUg RlNMIG9uZS4KCkJ1dCB5ZWFoLCB5b3VyIHBvaW50IHN0YW5kcywgdGhpcyBpcyBqdXN0IGltcGxl bWVudGF0aW9uIGRldGFpbHMuCgo+IGFyY2gvcG93ZXJwYy9rZXJuZWwvcGNpLWNvbW1vbi5jOiAg ICAgICBzZXRfZG1hX29mZnNldCgmZGV2LT5kZXYsIFBDSV9EUkFNX09GRlNFVCk7Cj4gYXJjaC9w b3dlcnBjL3BsYXRmb3Jtcy9jZWxsL2lvbW11LmM6ICAgICAgICAgICAgc2V0X2RtYV9vZmZzZXQo ZGV2LCBjZWxsX2RtYV9ub21tdV9vZmZzZXQpOwo+IGFyY2gvcG93ZXJwYy9wbGF0Zm9ybXMvY2Vs bC9pb21tdS5jOiAgICAgICAgICAgIHNldF9kbWFfb2Zmc2V0KGRldiwgYWRkcik7Cj4gYXJjaC9w b3dlcnBjL3BsYXRmb3Jtcy9wb3dlcm52L3BjaS1pb2RhLmM6ICAgICAgc2V0X2RtYV9vZmZzZXQo JnBkZXYtPmRldiwgcGUtPnRjZV9ieXBhc3NfYmFzZSk7Cj4gYXJjaC9wb3dlcnBjL3BsYXRmb3Jt cy9wb3dlcm52L3BjaS1pb2RhLmM6ICAgICAgICAgICAgICAgICAgICAgIHNldF9kbWFfb2Zmc2V0 KCZwZGV2LT5kZXYsICgxVUxMIDw8IDMyKSk7Cj4gYXJjaC9wb3dlcnBjL3BsYXRmb3Jtcy9wb3dl cm52L3BjaS1pb2RhLmM6ICAgICAgICAgICAgICBzZXRfZG1hX29mZnNldCgmZGV2LT5kZXYsIHBl LT50Y2VfYnlwYXNzX2Jhc2UpOwo+IGFyY2gvcG93ZXJwYy9wbGF0Zm9ybXMvcHNlcmllcy9pb21t dS5jOiAgICAgICAgICAgICAgICAgICAgICAgICBzZXRfZG1hX29mZnNldChkZXYsIGRtYV9vZmZz ZXQpOwo+IGFyY2gvcG93ZXJwYy9zeXNkZXYvZGFydF9pb21tdS5jOiAgICAgICAgICAgICAgIHNl dF9kbWFfb2Zmc2V0KCZkZXYtPmRldiwgREFSVF9VNF9CWVBBU1NfQkFTRSk7Cj4gYXJjaC9wb3dl cnBjL3N5c2Rldi9mc2xfcGNpLmM6ICAgICAgICAgIHNldF9kbWFfb2Zmc2V0KGRldiwgcGNpNjRf ZG1hX29mZnNldCk7Cj4gCj4gdG8gbWFrZSB0aGluZ3Mgd29yc2Ugc29tZSBwbGF0Zm9ybXMgKGF0 IGxlYXN0IG9uIGFybS9hcm02NC9taXBzL3g4NikgY2FuCj4gYWxzbyByZXF1aXJlIGFkZGl0aW9u YWwgYmFua2luZyB3aGVyZSBpdCBpc24ndCBldmVuIGEgc2luZ2xlIGxpbmVhciBtYXAKPiBidXQg bXVsdGlwbGVzIHdpbmRvd3MuCgpTdXJlLCBidXQgYWxsIG9mIHRoaXMgaXMganVzdCB0aGUgY29u ZmlndXJhdGlvbiBvZiB0aGUgaW9tbXUuIEJ1dCBJCnRoaW5rIHdlIGFncmVlIGhlcmUsIGFuZCB5 b3VyIHBvaW50IHJlbWFpbnMgdmFsaWQsIGluZGVlZCBteSBwcm9wb3NlZApoYWNrOgoKPiAgICAg ICBpZiAoKGZsYWdzICYgVklSVElPX0ZfSU9NTVVfUExBVEZPUk0pIHx8IGFyY2hfdmlydGlvX3dh bnRzX2RtYV9vcHMoKSkKCldpbGwgb25seSB3b3JrIGlmIHRoZSBJT01NVSBhbmQgbm9uLUlPTU1V IHBhdGggYXJlIGNvbXBsZXRlbHkgZXF1aXZhbGVudC4KCldlIGNhbiBwcm92aWRlIHRoYXQgZ3Vh cmFudGVlIGZvciBvdXIgc2VjdXJlIFZNIGNhc2UsIGJ1dCBub3QgZ2VuZXJhbGx5IHNvIGlmCndl IHdlcmUgdG8gZ28gZG93biB0aGUgcm91dGUgb2YgYSBxdWlyayBpbiB2aXJ0aW8sIGl0IG1pZ2h0 IGJlIGJldHRlciB0bwptYWtlIGl0IHBhaW5mdWxseSBvYnZpb3VzIHRoYXQgaXQncyBzcGVjaWZp YyB0byB0aGF0IG9uZSBjYXNlIHdpdGggYSBkaWZmZXJlbnQKa2luZCBvZiB0dXJkOgoKLQlpZiAo eGVuX2RvbWFpbigpKQorCWlmICh4ZW5fZG9tYWluKCkgfHwgcHNlcmllc19zZWN1cmVfdm0oKSkK CQlyZXR1cm4gdHJ1ZTsKClNvIHRvIHN1bW1hcml6ZSwgYW5kIG1ha2Ugc3VyZSBJJ20gbm90IG1p c3Npbmcgc29tZXRoaW5nLCB0aGUgdHdvIGFwcHJvYWNoZXMKYXQgaGFuZCBhcmUgZWl0aGVyOgoK IDEtIFRoZSBhYm92ZSwgd2hpY2ggaXMgYSBvbmUgbGluZXIgYW5kIGNvbnRhaW5lZCBpbiB0aGUg Z3Vlc3QsIHNvIHRoYXQncyBuaWNlLCBidXQKYWxzbyBtZWFucyBhbm90aGVyIHR1cmQgaW4gdmly dGlvIHdoaWNoIGlzbid0IC4uLgoKIDItIFdlIGZvcmNlIHBzZXJpZXMgdG8gYWx3YXlzIHNldCBW SVJUSU9fRl9JT01NVV9QTEFURk9STSwgYnV0IHdpdGggdGhlIGN1cnJlbnQKYXJjaGl0ZWN0dXJl IG9uIG91ciBzaWRlIHRoYXQgd2lsbCBmb3JjZSB2aXJ0aW8gdG8gYWx3YXlzIGdvIHRocm91Z2gg YW4gZW11bGF0ZWQKaW9tbXUsIGFzIHBzZXJpZXMgZG9lc24ndCBoYXZlIHRoZSBjb25jZXB0IG9m IGEgcmVhbCBieXBhc3Mgd2luZG93LCBhbmQgdGh1cyB3aWxsCmltcGFjdCBwZXJmb3JtYW5jZSBm b3IgYm90aCBzZWN1cmUgYW5kIG5vbi1zZWN1cmUgVk1zLgoKIDMtIEludmVudCBhIHByb3BlcnR5 IHRoYXQgY2FuIGJlIHB1dCBpbiBzZWxlY3RlZCBQQ0kgZGV2aWNlIHRyZWUgbm9kZXMgdGhhdApp bmRpY2F0ZXMgdGhhdCBmb3IgdGhhdCBkZXZpY2Ugc3BlY2lmaWNhbGx5LCB0aGUgaW9tbXUgY2Fu IGJlIGJ5cGFzc2VkLCBhbG9uZyB3aXRoCmEgaHlwZXJjYWxsIHRvIHR1cm4gdGhhdCBieXBhc3Mg b24vb2ZmLiBWaXJ0aW8gd291bGQgdGhlbiB1c2UgVklSVElPX0ZfSU9NTVVfUExBVEZPUk0KYnV0 IGl0cyBEVCBub2RlcyB3b3VsZCBhbHNvIGhhdmUgdGhhdCBwcm9wZXJ0eSBhbmQgTGludXggd291 bGQgbm90aWNlIGl0IGFuZCB0dXJuCmJ5cGFzcyBvbi4KClRoZSByZXN1bHRpbmcgcHJvcGVydGll cyBvZiB0aG9zZSBvcHRpb25zIGFyZToKCjEtIElzIHdoYXQgSSB3YW50IGJlY2F1c2UgaXQncyB0 aGUgc2ltcGxlc3QsIHByb3ZpZGVzIHRoZSBiZXN0IHBlcmZvcm1hbmNlIG5vdywKICAgYW5kIHdv cmtzIHdpdGhvdXQgY29kZSBjaGFuZ2VzIHRvIHFlbXUgb3Igbm9uLXNlY3VyZSBMaW51eC4gSG93 ZXZlciBpdCBkb2VzCiAgIGFkZCBhIHRpbnkgdHVyZCB0byB2aXJ0aW8gd2hpY2ggaXMgYW5ub3lp bmcuCgoyLSBUaGlzIHdvcmtzIGJ1dCBpdCBwdXRzIHRoZSBpb21tdSBpbiB0aGUgd2F5IGFsd2F5 cywgdGh1cyByZWR1Y2luZyB2aXJ0aW8gcGVyZm9ybWFuY2UKICAgYWNjcm9zcyB0aGUgYm9hcmQg Zm9yIHBzZXJpZXMgdW5sZXNzIHdlIG9ubHkgZG8gdGhhdCBmb3Igc2VjdXJlIFZNcyBidXQgdGhh dCBpcwogICBkaWZmaWN1bHQgKGFzIGRpc2N1c3NlZCBlYXJsaWVyKS4KCjMtIFRoaXMgd291bGQg cmVjb3ZlciB0aGUgcGVyZm9ybWFuY2UgbG9zdCBpbiAtMi0sIGhvd2V2ZXIgaXQgcmVxdWlyZXMg cWVtdSAqYW5kKgogICBndWVzdCBjaGFuZ2VzLiBTcGVjaWZpY2FsbHksIGV4aXN0aW5nIGd1ZXN0 cyAoUkhFTCA3IGV0Yy4uLikgd291bGQgZ2V0IHRoZQogICBwZXJmb3JtYW5jZSBoaXQgb2YgLTIt IHVubGVzcyBtb2RpZmllZCB0byBjYWxsIHRoYXQgJ2VuYWJsZSBieXBhc3MnIGNhbGwsIHdoaWNo CiAgIGlzbid0IGdyZWF0LgoKU28gaW1obyB3ZSBoYXZlIHRvIGNob3NlIG9uZSBvZiAzIG5vdC1n cmVhdCBzb2x1dGlvbnMgaGVyZS4uLiBVbmxlc3MgSSBtaXNzZWQKc29tZXRoaW5nIGluIHlvdXIg aWRlYXMgb2YgY291cnNlLgoKQ2hlZXJzLApCZW4uCgoKCl9fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlzdApWaXJ0 dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4 Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg==