From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756069AbcEaTFB (ORCPT ); Tue, 31 May 2016 15:05:01 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:35730 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755623AbcEaTE7 (ORCPT ); Tue, 31 May 2016 15:04:59 -0400 Date: Tue, 31 May 2016 21:04:53 +0200 From: Daniel Vetter To: "Luis R. Rodriguez" Cc: Daniel Vetter , "Luis R. Rodriguez" , vw@iommu.org, Oded Gabbay , Joerg Roedel , Christian =?iso-8859-1?Q?K=F6nig?= , "alexander.deucher@amd.com" , Dave Airlie , iommu@lists.linux-foundation.org, Linux Kernel Mailing List , dri-devel , gregkh@linuxfoundation.org, hpa@zytor.com Subject: Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon Message-ID: <20160531190453.GL7231@phenom.ffwll.local> Mail-Followup-To: "Luis R. Rodriguez" , "Luis R. Rodriguez" , vw@iommu.org, Oded Gabbay , Joerg Roedel , Christian =?iso-8859-1?Q?K=F6nig?= , "alexander.deucher@amd.com" , Dave Airlie , iommu@lists.linux-foundation.org, Linux Kernel Mailing List , dri-devel , gregkh@linuxfoundation.org, hpa@zytor.com References: <20160527004642.GQ11948@wotan.suse.de> <1464311916-10065-1-git-send-email-mcgrof@kernel.org> <20160531165834.GG11948@wotan.suse.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160531165834.GG11948@wotan.suse.de> X-Operating-System: Linux phenom 4.6.0-rc5+ User-Agent: Mutt/1.6.0 (2016-04-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 31, 2016 at 06:58:34PM +0200, Luis R. Rodriguez wrote: > On Sun, May 29, 2016 at 08:27:07PM +0200, Daniel Vetter wrote: > > On Fri, May 27, 2016 at 3:18 AM, Luis R. Rodriguez wrote: > > > To get KFD support in radeon we need the following > > > initialization to happen in this order, their > > > respective driver file that has its init routine > > > listed next to it: > > > > > > 0. AMD IOMMUv1: arch/x86/kernel/pci-dma.c > > > 1. AMD IOMMUv2: drivers/iommu/amd_iommu_v2.c > > > 2. AMD KFD: drivers/gpu/drm/amd/amdkfd/kfd_module.c > > > 3. AMD Radeon: drivers/gpu/drm/radeon/radeon_drv.c > > > > > > Order is rather implicit, but these drivers can currently > > > only do so much given the amount of leg room available. > > > Below are the respective init routines and how they are > > > initialized: > > > > > > arch/x86/kernel/pci-dma.c rootfs_initcall(pci_iommu_init); > > > drivers/iommu/amd_iommu_v2.c module_init(amd_iommu_v2_init); > > > drivers/gpu/drm/amd/amdkfd/kfd_module.c module_init(kfd_module_init); > > > drivers/gpu/drm/radeon/radeon_drv.c module_init(radeon_init); > > > > > > When a driver is built-in module_init() folds to use > > > device_initcall(), and we have the following possible > > > orders: > > > > > > #define pure_initcall(fn) __define_initcall(fn, 0) > > > #define core_initcall(fn) __define_initcall(fn, 1) > > > #define postcore_initcall(fn)__define_initcall(fn, 2) > > > #define arch_initcall(fn) __define_initcall(fn, 3) > > > #define subsys_initcall(fn) __define_initcall(fn, 4) > > > #define fs_initcall(fn) __define_initcall(fn, 5) > > > --------------------------------------------------------- > > > #define rootfs_initcall(fn) __define_initcall(fn, rootfs) > > > #define device_initcall(fn) __define_initcall(fn, 6) > > > #define late_initcall(fn) __define_initcall(fn, 7) > > > > > > Since we start off from rootfs_initcall(), it gives us 3 more > > > levels of leg room to play with for order semantics, this isn't > > > enough to address all required levels of dependencies, this > > > is specially true given that AMD-KFD needs to be loaded before > > > the radeon driver -- -but this it not enforced by symbols. > > > If the AMD-KFD driver is not loaded prior to the radeon driver > > > because otherwise the radeon driver will not initialize the > > > AMD-KFD driver and you get no KFD functionality in userspace. > > > > > > Commit 1bacc894c227fad8a7 ("drivers: Move iommu/ before gpu/ in > > > Makefile") works around some of the possibe races between > > > the AMD IOMMU v2 and GPU drivers by changing the link order. > > > This is fragile, however its the bets we can do, given that > > > making the GPU drivers use late_initcall() would also implicate > > > a similar race between them. That possible race is fortunatley > > > addressed given that the drm Makefile currently has amdkfd > > > linked prior to radeon: > > > > > > drivers/gpu/drm/Makefile > > > ... > > > obj-$(CONFIG_HSA_AMD) += amd/amdkfd/ > > > obj-$(CONFIG_DRM_RADEON)+= radeon/ > > > ... > > > > > > Changing amdkfd and radeon to late_initcall() however is > > > still the right call in orde to annotate explicitly a > > > delayed dependency requirement between the GPU drivers > > > and the IOMMUs. > > > > > > We can't address the fragile nature of the link order > > > right now, but in the future that might be possible. > > > > > > Signed-off-by: Luis R. Rodriguez > > > --- > > > > > > Please note, the changes to drivers/Makefile are just > > > for the sake of forcing the possible race to occur, > > > if this works well the actual [PATCH] submission will > > > skip those changes as its pointless to remove those > > > work arounds as it stands, due to the limited nature > > > of the levels available for addressing requirements. > > > > > > Also, if you are aware of further dependency hell > > > things like these -- please do let me know as I am > > > interested in looking at addressing them. > > > > This should be fixed with -EDEFER_PROBE instead. Frobbing initcall > > levels should then just be done as an optimization to avoid too much > > reprobing. > > radeon already uses -EDEFER_PROBE but it assumes that amdkfd *is* loaded first, > and only if it is already loaded can it count on getting -EDEFER_PROBE. The > radeon driver will deffer probe *iff* kgd2kfd_init() returns -EDEFER_PROBE, > which only happens when amdkfd_init_completed is no longer present. If radeon > gets linked first though the symbol fetch for kgd2kfd_init() will make it as > not-present though. So the current heuristic used does not address proper link > / load order. Part of the issue mentioned on the commit log is another race > underneath the hood with the AMD IOMMU v2 which is needed for amdkfd. The > underlying issue however really is the lack of available clear semantics for > dependencies over 3 levels here. This is solved one way or another by link > order in the Makefiles, but as I've noted so far this has been rather implicit > and fragile. The change here makes at least the order of the GPU drivers > explicitly later than the IOMMUs. The specific race between radeon and amdfkd > is solved currently through link order through the Makefiles. In the future we > maybe able to make things more explicit. Sounds like the EDEFER_PROBE handling is broken - if the module isn't set up yet but selected in Kconfig, and needed for that hw generation then it should not just silently fail. > -EDEFER_PROBE also introduces a latency on load which we should not need if we > can handle proper link / load order dependency annotations. This change is a > small part of that work, but as it the commit log also notes future further > work is possible to build stronger semantics. Some of the work I'm doing with > linker-tables may help with this in the future [0], but for now this should > help with what the semantics we have in place. > > [0] http://lkml.kernel.org/r/1455889559-9428-1-git-send-email-mcgrof@kernel.org That's what I meant with "avoiding too much reprobing". But in the end the current solution to cross-driver deps we have is EDEFER_PROBE. Fiddling with the link order is all well for optimizing stuff, but imo _way_ too fragile for correctness. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Vetter Subject: Re: [RFT v3] drm: use late_initcall() for amdkfd and radeon Date: Tue, 31 May 2016 21:04:53 +0200 Message-ID: <20160531190453.GL7231@phenom.ffwll.local> References: <20160527004642.GQ11948@wotan.suse.de> <1464311916-10065-1-git-send-email-mcgrof@kernel.org> <20160531165834.GG11948@wotan.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20160531165834.GG11948@wotan.suse.de> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: "Luis R. Rodriguez" Cc: vw@iommu.org, gregkh@linuxfoundation.org, iommu@lists.linux-foundation.org, dri-devel , Linux Kernel Mailing List , "Luis R. Rodriguez" , hpa@zytor.com, "alexander.deucher@amd.com" , Christian =?iso-8859-1?Q?K=F6nig?= List-Id: iommu@lists.linux-foundation.org T24gVHVlLCBNYXkgMzEsIDIwMTYgYXQgMDY6NTg6MzRQTSArMDIwMCwgTHVpcyBSLiBSb2RyaWd1 ZXogd3JvdGU6Cj4gT24gU3VuLCBNYXkgMjksIDIwMTYgYXQgMDg6Mjc6MDdQTSArMDIwMCwgRGFu aWVsIFZldHRlciB3cm90ZToKPiA+IE9uIEZyaSwgTWF5IDI3LCAyMDE2IGF0IDM6MTggQU0sIEx1 aXMgUi4gUm9kcmlndWV6IDxtY2dyb2ZAa2VybmVsLm9yZz4gd3JvdGU6Cj4gPiA+IFRvIGdldCBL RkQgc3VwcG9ydCBpbiByYWRlb24gd2UgbmVlZCB0aGUgZm9sbG93aW5nCj4gPiA+IGluaXRpYWxp emF0aW9uIHRvIGhhcHBlbiBpbiB0aGlzIG9yZGVyLCB0aGVpcgo+ID4gPiByZXNwZWN0aXZlIGRy aXZlciBmaWxlIHRoYXQgaGFzIGl0cyBpbml0IHJvdXRpbmUKPiA+ID4gbGlzdGVkIG5leHQgdG8g aXQ6Cj4gPiA+Cj4gPiA+IDAuIEFNRCBJT01NVXYxOiAgICBhcmNoL3g4Ni9rZXJuZWwvcGNpLWRt YS5jCj4gPiA+IDEuIEFNRCBJT01NVXYyOiAgICBkcml2ZXJzL2lvbW11L2FtZF9pb21tdV92Mi5j Cj4gPiA+IDIuIEFNRCBLRkQ6ICAgICAgICBkcml2ZXJzL2dwdS9kcm0vYW1kL2FtZGtmZC9rZmRf bW9kdWxlLmMKPiA+ID4gMy4gQU1EIFJhZGVvbjogICAgIGRyaXZlcnMvZ3B1L2RybS9yYWRlb24v cmFkZW9uX2Rydi5jCj4gPiA+Cj4gPiA+IE9yZGVyIGlzIHJhdGhlciBpbXBsaWNpdCwgYnV0IHRo ZXNlIGRyaXZlcnMgY2FuIGN1cnJlbnRseQo+ID4gPiBvbmx5IGRvIHNvIG11Y2ggZ2l2ZW4gdGhl IGFtb3VudCBvZiBsZWcgcm9vbSBhdmFpbGFibGUuCj4gPiA+IEJlbG93IGFyZSB0aGUgcmVzcGVj dGl2ZSBpbml0IHJvdXRpbmVzIGFuZCBob3cgdGhleSBhcmUKPiA+ID4gaW5pdGlhbGl6ZWQ6Cj4g PiA+Cj4gPiA+IGFyY2gveDg2L2tlcm5lbC9wY2ktZG1hLmMgICAgICAgICAgICAgICByb290ZnNf aW5pdGNhbGwocGNpX2lvbW11X2luaXQpOwo+ID4gPiBkcml2ZXJzL2lvbW11L2FtZF9pb21tdV92 Mi5jICAgICAgICAgICAgbW9kdWxlX2luaXQoYW1kX2lvbW11X3YyX2luaXQpOwo+ID4gPiBkcml2 ZXJzL2dwdS9kcm0vYW1kL2FtZGtmZC9rZmRfbW9kdWxlLmMgbW9kdWxlX2luaXQoa2ZkX21vZHVs ZV9pbml0KTsKPiA+ID4gZHJpdmVycy9ncHUvZHJtL3JhZGVvbi9yYWRlb25fZHJ2LmMgICAgIG1v ZHVsZV9pbml0KHJhZGVvbl9pbml0KTsKPiA+ID4KPiA+ID4gV2hlbiBhIGRyaXZlciBpcyBidWls dC1pbiBtb2R1bGVfaW5pdCgpIGZvbGRzIHRvIHVzZQo+ID4gPiBkZXZpY2VfaW5pdGNhbGwoKSwg YW5kIHdlIGhhdmUgdGhlIGZvbGxvd2luZyBwb3NzaWJsZQo+ID4gPiBvcmRlcnM6Cj4gPiA+Cj4g PiA+ICAgICAgICAgI2RlZmluZSBwdXJlX2luaXRjYWxsKGZuKSAgICBfX2RlZmluZV9pbml0Y2Fs bChmbiwgMCkKPiA+ID4gICAgICAgICAjZGVmaW5lIGNvcmVfaW5pdGNhbGwoZm4pICAgIF9fZGVm aW5lX2luaXRjYWxsKGZuLCAxKQo+ID4gPiAgICAgICAgICNkZWZpbmUgcG9zdGNvcmVfaW5pdGNh bGwoZm4pX19kZWZpbmVfaW5pdGNhbGwoZm4sIDIpCj4gPiA+ICAgICAgICAgI2RlZmluZSBhcmNo X2luaXRjYWxsKGZuKSAgICBfX2RlZmluZV9pbml0Y2FsbChmbiwgMykKPiA+ID4gICAgICAgICAj ZGVmaW5lIHN1YnN5c19pbml0Y2FsbChmbikgIF9fZGVmaW5lX2luaXRjYWxsKGZuLCA0KQo+ID4g PiAgICAgICAgICNkZWZpbmUgZnNfaW5pdGNhbGwoZm4pICAgICAgX19kZWZpbmVfaW5pdGNhbGwo Zm4sIDUpCj4gPiA+ICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tCj4gPiA+ICAgICAgICAgI2RlZmluZSByb290ZnNfaW5pdGNh bGwoZm4pICBfX2RlZmluZV9pbml0Y2FsbChmbiwgcm9vdGZzKQo+ID4gPiAgICAgICAgICNkZWZp bmUgZGV2aWNlX2luaXRjYWxsKGZuKSAgX19kZWZpbmVfaW5pdGNhbGwoZm4sIDYpCj4gPiA+ICAg ICAgICAgI2RlZmluZSBsYXRlX2luaXRjYWxsKGZuKSAgICBfX2RlZmluZV9pbml0Y2FsbChmbiwg NykKPiA+ID4KPiA+ID4gU2luY2Ugd2Ugc3RhcnQgb2ZmIGZyb20gcm9vdGZzX2luaXRjYWxsKCks IGl0IGdpdmVzIHVzIDMgbW9yZQo+ID4gPiBsZXZlbHMgb2YgbGVnIHJvb20gdG8gcGxheSB3aXRo IGZvciBvcmRlciBzZW1hbnRpY3MsIHRoaXMgaXNuJ3QKPiA+ID4gZW5vdWdoIHRvIGFkZHJlc3Mg YWxsIHJlcXVpcmVkIGxldmVscyBvZiBkZXBlbmRlbmNpZXMsIHRoaXMKPiA+ID4gaXMgc3BlY2lh bGx5IHRydWUgZ2l2ZW4gdGhhdCBBTUQtS0ZEIG5lZWRzIHRvIGJlIGxvYWRlZCBiZWZvcmUKPiA+ ID4gdGhlIHJhZGVvbiBkcml2ZXIgLS0gLWJ1dCB0aGlzIGl0IG5vdCBlbmZvcmNlZCBieSBzeW1i b2xzLgo+ID4gPiBJZiB0aGUgQU1ELUtGRCBkcml2ZXIgaXMgbm90IGxvYWRlZCBwcmlvciB0byB0 aGUgcmFkZW9uIGRyaXZlcgo+ID4gPiBiZWNhdXNlIG90aGVyd2lzZSB0aGUgcmFkZW9uIGRyaXZl ciB3aWxsIG5vdCBpbml0aWFsaXplIHRoZQo+ID4gPiBBTUQtS0ZEIGRyaXZlciBhbmQgeW91IGdl dCBubyBLRkQgZnVuY3Rpb25hbGl0eSBpbiB1c2Vyc3BhY2UuCj4gPiA+Cj4gPiA+IENvbW1pdCAx YmFjYzg5NGMyMjdmYWQ4YTcgKCJkcml2ZXJzOiBNb3ZlIGlvbW11LyBiZWZvcmUgZ3B1LyBpbgo+ ID4gPiBNYWtlZmlsZSIpIHdvcmtzIGFyb3VuZCBzb21lIG9mIHRoZSBwb3NzaWJlIHJhY2VzIGJl dHdlZW4KPiA+ID4gdGhlIEFNRCBJT01NVSB2MiBhbmQgR1BVIGRyaXZlcnMgYnkgY2hhbmdpbmcg dGhlIGxpbmsgb3JkZXIuCj4gPiA+IFRoaXMgaXMgZnJhZ2lsZSwgaG93ZXZlciBpdHMgdGhlIGJl dHMgd2UgY2FuIGRvLCBnaXZlbiB0aGF0Cj4gPiA+IG1ha2luZyB0aGUgR1BVIGRyaXZlcnMgdXNl IGxhdGVfaW5pdGNhbGwoKSB3b3VsZCBhbHNvIGltcGxpY2F0ZQo+ID4gPiBhIHNpbWlsYXIgcmFj ZSBiZXR3ZWVuIHRoZW0uIFRoYXQgcG9zc2libGUgcmFjZSBpcyBmb3J0dW5hdGxleQo+ID4gPiBh ZGRyZXNzZWQgZ2l2ZW4gdGhhdCB0aGUgZHJtIE1ha2VmaWxlIGN1cnJlbnRseSBoYXMgYW1ka2Zk Cj4gPiA+IGxpbmtlZCBwcmlvciB0byByYWRlb246Cj4gPiA+Cj4gPiA+IGRyaXZlcnMvZ3B1L2Ry bS9NYWtlZmlsZQo+ID4gPiAuLi4KPiA+ID4gb2JqLSQoQ09ORklHX0hTQV9BTUQpICs9IGFtZC9h bWRrZmQvCj4gPiA+IG9iai0kKENPTkZJR19EUk1fUkFERU9OKSs9IHJhZGVvbi8KPiA+ID4gLi4u Cj4gPiA+Cj4gPiA+IENoYW5naW5nIGFtZGtmZCBhbmQgcmFkZW9uIHRvIGxhdGVfaW5pdGNhbGwo KSBob3dldmVyIGlzCj4gPiA+IHN0aWxsIHRoZSByaWdodCBjYWxsIGluIG9yZGUgdG8gYW5ub3Rh dGUgZXhwbGljaXRseSBhCj4gPiA+IGRlbGF5ZWQgZGVwZW5kZW5jeSByZXF1aXJlbWVudCBiZXR3 ZWVuIHRoZSBHUFUgZHJpdmVycwo+ID4gPiBhbmQgdGhlIElPTU1Vcy4KPiA+ID4KPiA+ID4gV2Ug Y2FuJ3QgYWRkcmVzcyB0aGUgZnJhZ2lsZSBuYXR1cmUgb2YgdGhlIGxpbmsgb3JkZXIKPiA+ID4g cmlnaHQgbm93LCBidXQgaW4gdGhlIGZ1dHVyZSB0aGF0IG1pZ2h0IGJlIHBvc3NpYmxlLgo+ID4g Pgo+ID4gPiBTaWduZWQtb2ZmLWJ5OiBMdWlzIFIuIFJvZHJpZ3VleiA8bWNncm9mQGtlcm5lbC5v cmc+Cj4gPiA+IC0tLQo+ID4gPgo+ID4gPiBQbGVhc2Ugbm90ZSwgdGhlIGNoYW5nZXMgdG8gZHJp dmVycy9NYWtlZmlsZSBhcmUganVzdAo+ID4gPiBmb3IgdGhlIHNha2Ugb2YgZm9yY2luZyB0aGUg cG9zc2libGUgcmFjZSB0byBvY2N1ciwKPiA+ID4gaWYgdGhpcyB3b3JrcyB3ZWxsIHRoZSBhY3R1 YWwgW1BBVENIXSBzdWJtaXNzaW9uIHdpbGwKPiA+ID4gc2tpcCB0aG9zZSBjaGFuZ2VzIGFzIGl0 cyBwb2ludGxlc3MgdG8gcmVtb3ZlIHRob3NlCj4gPiA+IHdvcmsgYXJvdW5kcyBhcyBpdCBzdGFu ZHMsIGR1ZSB0byB0aGUgbGltaXRlZCBuYXR1cmUKPiA+ID4gb2YgdGhlIGxldmVscyBhdmFpbGFi bGUgZm9yIGFkZHJlc3NpbmcgcmVxdWlyZW1lbnRzLgo+ID4gPgo+ID4gPiBBbHNvLCBpZiB5b3Ug YXJlIGF3YXJlIG9mIGZ1cnRoZXIgZGVwZW5kZW5jeSBoZWxsCj4gPiA+IHRoaW5ncyBsaWtlIHRo ZXNlIC0tIHBsZWFzZSBkbyBsZXQgbWUga25vdyBhcyBJIGFtCj4gPiA+IGludGVyZXN0ZWQgaW4g bG9va2luZyBhdCBhZGRyZXNzaW5nIHRoZW0uCj4gPiAKPiA+IFRoaXMgc2hvdWxkIGJlIGZpeGVk IHdpdGggLUVERUZFUl9QUk9CRSBpbnN0ZWFkLiBGcm9iYmluZyBpbml0Y2FsbAo+ID4gbGV2ZWxz IHNob3VsZCB0aGVuIGp1c3QgYmUgZG9uZSBhcyBhbiBvcHRpbWl6YXRpb24gdG8gYXZvaWQgdG9v IG11Y2gKPiA+IHJlcHJvYmluZy4KPiAKPiByYWRlb24gYWxyZWFkeSB1c2VzIC1FREVGRVJfUFJP QkUgYnV0IGl0IGFzc3VtZXMgdGhhdCBhbWRrZmQgKmlzKiBsb2FkZWQgZmlyc3QsCj4gYW5kIG9u bHkgaWYgaXQgaXMgYWxyZWFkeSBsb2FkZWQgY2FuIGl0IGNvdW50IG9uIGdldHRpbmcgLUVERUZF Ul9QUk9CRS4gVGhlCj4gcmFkZW9uIGRyaXZlciB3aWxsIGRlZmZlciBwcm9iZSAqaWZmKiBrZ2Qy a2ZkX2luaXQoKSByZXR1cm5zIC1FREVGRVJfUFJPQkUsCj4gd2hpY2ggb25seSBoYXBwZW5zIHdo ZW4gYW1ka2ZkX2luaXRfY29tcGxldGVkIGlzIG5vIGxvbmdlciBwcmVzZW50LiBJZiByYWRlb24K PiBnZXRzIGxpbmtlZCBmaXJzdCB0aG91Z2ggdGhlIHN5bWJvbCBmZXRjaCBmb3Iga2dkMmtmZF9p bml0KCkgd2lsbCBtYWtlIGl0IGFzCj4gbm90LXByZXNlbnQgdGhvdWdoLiBTbyB0aGUgY3VycmVu dCBoZXVyaXN0aWMgdXNlZCBkb2VzIG5vdCBhZGRyZXNzIHByb3BlciBsaW5rCj4gLyBsb2FkIG9y ZGVyLiBQYXJ0IG9mIHRoZSBpc3N1ZSBtZW50aW9uZWQgb24gdGhlIGNvbW1pdCBsb2cgaXMgYW5v dGhlciByYWNlCj4gdW5kZXJuZWF0aCB0aGUgaG9vZCB3aXRoIHRoZSBBTUQgSU9NTVUgdjIgd2hp Y2ggaXMgbmVlZGVkIGZvciBhbWRrZmQuIFRoZQo+IHVuZGVybHlpbmcgaXNzdWUgaG93ZXZlciBy ZWFsbHkgaXMgdGhlIGxhY2sgb2YgYXZhaWxhYmxlIGNsZWFyIHNlbWFudGljcyBmb3IKPiBkZXBl bmRlbmNpZXMgb3ZlciAzIGxldmVscyBoZXJlLiAgVGhpcyBpcyBzb2x2ZWQgb25lIHdheSBvciBh bm90aGVyIGJ5IGxpbmsKPiBvcmRlciBpbiB0aGUgTWFrZWZpbGVzLCBidXQgYXMgSSd2ZSBub3Rl ZCBzbyBmYXIgdGhpcyBoYXMgYmVlbiByYXRoZXIgaW1wbGljaXQKPiBhbmQgZnJhZ2lsZS4gVGhl IGNoYW5nZSBoZXJlIG1ha2VzIGF0IGxlYXN0IHRoZSBvcmRlciBvZiB0aGUgR1BVIGRyaXZlcnMK PiBleHBsaWNpdGx5IGxhdGVyIHRoYW4gdGhlIElPTU1Vcy4gVGhlIHNwZWNpZmljIHJhY2UgYmV0 d2VlbiByYWRlb24gYW5kIGFtZGZrZAo+IGlzIHNvbHZlZCBjdXJyZW50bHkgdGhyb3VnaCBsaW5r IG9yZGVyIHRocm91Z2ggdGhlIE1ha2VmaWxlcy4gSW4gdGhlIGZ1dHVyZSB3ZQo+IG1heWJlIGFi bGUgdG8gbWFrZSB0aGluZ3MgbW9yZSBleHBsaWNpdC4KClNvdW5kcyBsaWtlIHRoZSBFREVGRVJf UFJPQkUgaGFuZGxpbmcgaXMgYnJva2VuIC0gaWYgdGhlIG1vZHVsZSBpc24ndCBzZXQKdXAgeWV0 IGJ1dCBzZWxlY3RlZCBpbiBLY29uZmlnLCBhbmQgbmVlZGVkIGZvciB0aGF0IGh3IGdlbmVyYXRp b24gdGhlbiBpdApzaG91bGQgbm90IGp1c3Qgc2lsZW50bHkgZmFpbC4KCj4gLUVERUZFUl9QUk9C RSBhbHNvIGludHJvZHVjZXMgYSBsYXRlbmN5IG9uIGxvYWQgd2hpY2ggd2Ugc2hvdWxkIG5vdCBu ZWVkIGlmIHdlCj4gY2FuIGhhbmRsZSBwcm9wZXIgbGluayAvIGxvYWQgb3JkZXIgZGVwZW5kZW5j eSBhbm5vdGF0aW9ucy4gVGhpcyBjaGFuZ2UgaXMgYQo+IHNtYWxsIHBhcnQgb2YgdGhhdCB3b3Jr LCBidXQgYXMgaXQgdGhlIGNvbW1pdCBsb2cgYWxzbyBub3RlcyBmdXR1cmUgZnVydGhlcgo+IHdv cmsgaXMgcG9zc2libGUgdG8gYnVpbGQgc3Ryb25nZXIgc2VtYW50aWNzLiAgU29tZSBvZiB0aGUg d29yayBJJ20gZG9pbmcgd2l0aAo+IGxpbmtlci10YWJsZXMgbWF5IGhlbHAgd2l0aCB0aGlzIGlu IHRoZSBmdXR1cmUgWzBdLCBidXQgZm9yIG5vdyB0aGlzIHNob3VsZAo+IGhlbHAgd2l0aCB3aGF0 IHRoZSBzZW1hbnRpY3Mgd2UgaGF2ZSBpbiBwbGFjZS4KPiAKPiBbMF0gaHR0cDovL2xrbWwua2Vy bmVsLm9yZy9yLzE0NTU4ODk1NTktOTQyOC0xLWdpdC1zZW5kLWVtYWlsLW1jZ3JvZkBrZXJuZWwu b3JnCgpUaGF0J3Mgd2hhdCBJIG1lYW50IHdpdGggImF2b2lkaW5nIHRvbyBtdWNoIHJlcHJvYmlu ZyIuIEJ1dCBpbiB0aGUgZW5kIHRoZQpjdXJyZW50IHNvbHV0aW9uIHRvIGNyb3NzLWRyaXZlciBk ZXBzIHdlIGhhdmUgaXMgRURFRkVSX1BST0JFLiBGaWRkbGluZwp3aXRoIHRoZSBsaW5rIG9yZGVy IGlzIGFsbCB3ZWxsIGZvciBvcHRpbWl6aW5nIHN0dWZmLCBidXQgaW1vIF93YXlfIHRvbwpmcmFn aWxlIGZvciBjb3JyZWN0bmVzcy4KLURhbmllbAotLSAKRGFuaWVsIFZldHRlcgpTb2Z0d2FyZSBF bmdpbmVlciwgSW50ZWwgQ29ycG9yYXRpb24KaHR0cDovL2Jsb2cuZmZ3bGwuY2gKX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KZHJpLWRldmVsIG1haWxpbmcg bGlzdApkcmktZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRl c2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vZHJpLWRldmVsCg==