From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756216AbdDGPaZ (ORCPT ); Fri, 7 Apr 2017 11:30:25 -0400 Received: from mail.fireflyinternet.com ([109.228.58.192]:61544 "EHLO fireflyinternet.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1755811AbdDGPaS (ORCPT ); Fri, 7 Apr 2017 11:30:18 -0400 X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138; Date: Fri, 7 Apr 2017 16:30:11 +0100 From: Chris Wilson To: Andrea Arcangeli Cc: Martin Kepplinger , Thorsten Leemhuis , daniel.vetter@intel.com, Dave Airlie , intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org Subject: Re: [PATCH 2/5] i915: flush gem obj freeing workqueues to add accuracy to the i915 shrinker Message-ID: <20170407153011.GR10496@nuc-i3427.alporthouse.com> Mail-Followup-To: Chris Wilson , Andrea Arcangeli , Martin Kepplinger , Thorsten Leemhuis , daniel.vetter@intel.com, Dave Airlie , intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org References: <87pogtplxr.fsf@intel.com> <20170406232347.988-1-aarcange@redhat.com> <20170406232347.988-3-aarcange@redhat.com> <20170407100211.GG10496@nuc-i3427.alporthouse.com> <20170407130600.GA5035@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170407130600.GA5035@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Apr 07, 2017 at 03:06:00PM +0200, Andrea Arcangeli wrote: > On Fri, Apr 07, 2017 at 11:02:11AM +0100, Chris Wilson wrote: > > On Fri, Apr 07, 2017 at 01:23:44AM +0200, Andrea Arcangeli wrote: > > > Waiting a RCU grace period only guarantees the work gets queued, but > > > until after the queued workqueue returns, there's no guarantee the > > > memory was actually freed. So flush the work to provide better > > > guarantees to the reclaim code in addition of waiting a RCU grace > > > period to pass. > > > > We are not allowed to call flush_work() from the shrinker, the workqueue > > doesn't have and can't have the right reclaim flags. > > I figured the flush_work had to be conditional to "unlock" being true > too in the i915 shrinker (not only synchronize_rcu_expedited()), and I > already fixed that bit, but I didn't think it would be a problem to > wait for the workqueue as long as reclaim didn't recurse on the > struct_mutex (it is a problem if unlock is false of course as we would > be back to square one). I didn't get further hangs and I assume I've > been running a couple of synchronize_rcu_expedited() and flush_work (I > should add dynamic tracing to be sure). Not getting hangs is a good sign, but lockdep doesn't like it: [ 460.684901] WARNING: CPU: 1 PID: 172 at kernel/workqueue.c:2418 check_flush_dependency+0x92/0x130 [ 460.684924] workqueue: PF_MEMALLOC task 172(kworker/1:1H) is flushing !WQ_MEM_RECLAIM events:__i915_gem_free_work [i915] If I allocated the workqueue with WQ_MEM_RELCAIM, it complains bitterly as well. > Also note, I didn't get any lockdep warning when I reproduced the > workqueue hang in 4.11-rc5 so at least as far as lockdep is concerned > there's no problem to call synchronize_rcu_expedited and it couldn't > notice we were holding the struct_mutex while waiting for the new > workqueue to run. Yes, that is concerning. I think it's due to the coupling via the struct completion that is not being picked up lockdep, and I hope the "crossrelease" patches would fix the lack of warnings. > Also note recursing on the lock (unlock false case) is something > nothing else does, I'm not sure if it's worth the risk and if you > shouldn't just call mutex_trylock in the shrinker instead of > mutex_trylock_recursive. One thing was to recurse on the lock > internally in the same context, but recursing through the whole > reclaim is more dubious as safe. We know. We don't use trylock in order to reduce the frequency of users' oom. Peter added mutex_trylock_recursive() because we already were doing recursive locking in the shrinker and although we know we shouldn't, getting rid of the recursion is something we are doing, but slowly. > You could start dropping objects and wiping vmas and stuff in the > middle of some kmalloc/alloc_pages that doesn't expect it and then > crash for other reasons. So this reclaim recursion model of the > shinker is quite unique and quite challenging to proof as safe if you > keep using mutex_trylock_recursive in i915_gem_shrinker_scan. I know. Trying to stay on top of all the kmallocs under the struct_mutex and being aware that the shrinker can and will undo your objects as you work is a continual battle. And catches everyone working on i915.ko by surprise. Our policy to avoid surprises is based around pin before alloc. > Lock recursion in all other places could be dropped without runtime > downsides, the only place mutex_trylock_recursive makes a design > difference and makes sense to be used is in i915_gem_shrinker_scan, > the rest are implementation issues not fundamental shrinker design and > it'd be nice if those other mutex_trylock_recursive would all be > removed and the only one that is left is in i915_gem_shrinker_scan and > nowhere else (or to drop it also from i915_gem_shrinker_scan). We do need it for shrinker_count as well. If we just report 0 objects, the shrinker_scan callback will be skipped, iirc. All we do need it for direct calls to i915_gem_shrink() which themselves may or may not be underneath the struct_mutex at the time. > mutex_trylock_recursive() should also be patched to use > READ_ONCE(__mutex_owner(lock)) because currently it breaks C. I don't follow, static inline struct task_struct *__mutex_owner(struct mutex *lock) { return (struct task_struct *)(atomic_long_read(&lock->owner) & ~0x07); } The atomic read is equivalent to READ_ONCE(). What's broken here? (I guess strict aliasing and pointer cast?) > In the whole kernel i915 and msm drm are the only two users of such > function in fact. Yes, Peter will continue to remind us to fix our code and complain until it is. > Another thing is what value return from i915_gem_shrinker_scan when > unlock is false, and we can't possibly wait for the memory to be freed > let alone for a rcu grace period. For various reasons I think it's > safer to return the current "free" even if we could also return "0" in > such case. There are different tradeoffs, returning "free" is less > likely to trigger an early OOM as the VM thinks it's still making > progress and in fact it will get more free memory shortly, while > returning SHRINK_STOP would also be an option and it would insist more > on the other slabs so it would be more reliable at freeing memory > timely, but it would be more at risk of early OOM. I think returning > "free" is the better tradeoff of the two, but I suggest to add a > comment as it's not exactly obvious what is better. Ah. The RCU freeing is only for the small fry, the slabs from which requests and objects are allocated. It's the gigabytes of pages we have pinned that can be released by i915_gem_shrink() are what we count as freed, even though we only return them to the system and hope the lru anon page scanner will make available for reuse (if they are dirty, they will need to be swapped out, but some will be returned to the system directly by truncating the shmemfs filp backing an object.) -Chris -- Chris Wilson, Intel Open Source Technology Centre From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Wilson Subject: Re: [PATCH 2/5] i915: flush gem obj freeing workqueues to add accuracy to the i915 shrinker Date: Fri, 7 Apr 2017 16:30:11 +0100 Message-ID: <20170407153011.GR10496@nuc-i3427.alporthouse.com> References: <87pogtplxr.fsf@intel.com> <20170406232347.988-1-aarcange@redhat.com> <20170406232347.988-3-aarcange@redhat.com> <20170407100211.GG10496@nuc-i3427.alporthouse.com> <20170407130600.GA5035@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20170407130600.GA5035@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Andrea Arcangeli Cc: intel-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Thorsten Leemhuis , Martin Kepplinger , daniel.vetter@intel.com List-Id: dri-devel@lists.freedesktop.org T24gRnJpLCBBcHIgMDcsIDIwMTcgYXQgMDM6MDY6MDBQTSArMDIwMCwgQW5kcmVhIEFyY2FuZ2Vs aSB3cm90ZToKPiBPbiBGcmksIEFwciAwNywgMjAxNyBhdCAxMTowMjoxMUFNICswMTAwLCBDaHJp cyBXaWxzb24gd3JvdGU6Cj4gPiBPbiBGcmksIEFwciAwNywgMjAxNyBhdCAwMToyMzo0NEFNICsw MjAwLCBBbmRyZWEgQXJjYW5nZWxpIHdyb3RlOgo+ID4gPiBXYWl0aW5nIGEgUkNVIGdyYWNlIHBl cmlvZCBvbmx5IGd1YXJhbnRlZXMgdGhlIHdvcmsgZ2V0cyBxdWV1ZWQsIGJ1dAo+ID4gPiB1bnRp bCBhZnRlciB0aGUgcXVldWVkIHdvcmtxdWV1ZSByZXR1cm5zLCB0aGVyZSdzIG5vIGd1YXJhbnRl ZSB0aGUKPiA+ID4gbWVtb3J5IHdhcyBhY3R1YWxseSBmcmVlZC4gU28gZmx1c2ggdGhlIHdvcmsg dG8gcHJvdmlkZSBiZXR0ZXIKPiA+ID4gZ3VhcmFudGVlcyB0byB0aGUgcmVjbGFpbSBjb2RlIGlu IGFkZGl0aW9uIG9mIHdhaXRpbmcgYSBSQ1UgZ3JhY2UKPiA+ID4gcGVyaW9kIHRvIHBhc3MuCj4g PiAKPiA+IFdlIGFyZSBub3QgYWxsb3dlZCB0byBjYWxsIGZsdXNoX3dvcmsoKSBmcm9tIHRoZSBz aHJpbmtlciwgdGhlIHdvcmtxdWV1ZQo+ID4gZG9lc24ndCBoYXZlIGFuZCBjYW4ndCBoYXZlIHRo ZSByaWdodCByZWNsYWltIGZsYWdzLgo+IAo+IEkgZmlndXJlZCB0aGUgZmx1c2hfd29yayBoYWQg dG8gYmUgY29uZGl0aW9uYWwgdG8gInVubG9jayIgYmVpbmcgdHJ1ZQo+IHRvbyBpbiB0aGUgaTkx NSBzaHJpbmtlciAobm90IG9ubHkgc3luY2hyb25pemVfcmN1X2V4cGVkaXRlZCgpKSwgYW5kIEkK PiBhbHJlYWR5IGZpeGVkIHRoYXQgYml0LCBidXQgSSBkaWRuJ3QgdGhpbmsgaXQgd291bGQgYmUg YSBwcm9ibGVtIHRvCj4gd2FpdCBmb3IgdGhlIHdvcmtxdWV1ZSBhcyBsb25nIGFzIHJlY2xhaW0g ZGlkbid0IHJlY3Vyc2Ugb24gdGhlCj4gc3RydWN0X211dGV4IChpdCBpcyBhIHByb2JsZW0gaWYg dW5sb2NrIGlzIGZhbHNlIG9mIGNvdXJzZSBhcyB3ZSB3b3VsZAo+IGJlIGJhY2sgdG8gc3F1YXJl IG9uZSkuIEkgZGlkbid0IGdldCBmdXJ0aGVyIGhhbmdzIGFuZCBJIGFzc3VtZSBJJ3ZlCj4gYmVl biBydW5uaW5nIGEgY291cGxlIG9mIHN5bmNocm9uaXplX3JjdV9leHBlZGl0ZWQoKSBhbmQgZmx1 c2hfd29yayAoSQo+IHNob3VsZCBhZGQgZHluYW1pYyB0cmFjaW5nIHRvIGJlIHN1cmUpLgoKTm90 IGdldHRpbmcgaGFuZ3MgaXMgYSBnb29kIHNpZ24sIGJ1dCBsb2NrZGVwIGRvZXNuJ3QgbGlrZSBp dDoKClsgIDQ2MC42ODQ5MDFdIFdBUk5JTkc6IENQVTogMSBQSUQ6IDE3MiBhdCBrZXJuZWwvd29y a3F1ZXVlLmM6MjQxOCBjaGVja19mbHVzaF9kZXBlbmRlbmN5KzB4OTIvMHgxMzAKWyAgNDYwLjY4 NDkyNF0gd29ya3F1ZXVlOiBQRl9NRU1BTExPQyB0YXNrIDE3Mihrd29ya2VyLzE6MUgpIGlzIGZs dXNoaW5nICFXUV9NRU1fUkVDTEFJTSBldmVudHM6X19pOTE1X2dlbV9mcmVlX3dvcmsgW2k5MTVd CgpJZiBJIGFsbG9jYXRlZCB0aGUgd29ya3F1ZXVlIHdpdGggV1FfTUVNX1JFTENBSU0sIGl0IGNv bXBsYWlucyBiaXR0ZXJseQphcyB3ZWxsLgoKPiBBbHNvIG5vdGUsIEkgZGlkbid0IGdldCBhbnkg bG9ja2RlcCB3YXJuaW5nIHdoZW4gSSByZXByb2R1Y2VkIHRoZQo+IHdvcmtxdWV1ZSBoYW5nIGlu IDQuMTEtcmM1IHNvIGF0IGxlYXN0IGFzIGZhciBhcyBsb2NrZGVwIGlzIGNvbmNlcm5lZAo+IHRo ZXJlJ3Mgbm8gcHJvYmxlbSB0byBjYWxsIHN5bmNocm9uaXplX3JjdV9leHBlZGl0ZWQgYW5kIGl0 IGNvdWxkbid0Cj4gbm90aWNlIHdlIHdlcmUgaG9sZGluZyB0aGUgc3RydWN0X211dGV4IHdoaWxl IHdhaXRpbmcgZm9yIHRoZSBuZXcKPiB3b3JrcXVldWUgdG8gcnVuLgoKWWVzLCB0aGF0IGlzIGNv bmNlcm5pbmcuIEkgdGhpbmsgaXQncyBkdWUgdG8gdGhlIGNvdXBsaW5nIHZpYQp0aGUgc3RydWN0 IGNvbXBsZXRpb24gdGhhdCBpcyBub3QgYmVpbmcgcGlja2VkIHVwIGxvY2tkZXAsIGFuZCBJIGhv cGUKdGhlICJjcm9zc3JlbGVhc2UiIHBhdGNoZXMgd291bGQgZml4IHRoZSBsYWNrIG9mIHdhcm5p bmdzLgoKPiBBbHNvIG5vdGUgcmVjdXJzaW5nIG9uIHRoZSBsb2NrICh1bmxvY2sgZmFsc2UgY2Fz ZSkgaXMgc29tZXRoaW5nCj4gbm90aGluZyBlbHNlIGRvZXMsIEknbSBub3Qgc3VyZSBpZiBpdCdz IHdvcnRoIHRoZSByaXNrIGFuZCBpZiB5b3UKPiBzaG91bGRuJ3QganVzdCBjYWxsIG11dGV4X3Ry eWxvY2sgaW4gdGhlIHNocmlua2VyIGluc3RlYWQgb2YKPiBtdXRleF90cnlsb2NrX3JlY3Vyc2l2 ZS4gT25lIHRoaW5nIHdhcyB0byByZWN1cnNlIG9uIHRoZSBsb2NrCj4gaW50ZXJuYWxseSBpbiB0 aGUgc2FtZSBjb250ZXh0LCBidXQgcmVjdXJzaW5nIHRocm91Z2ggdGhlIHdob2xlCj4gcmVjbGFp bSBpcyBtb3JlIGR1YmlvdXMgYXMgc2FmZS4KCldlIGtub3cuIFdlIGRvbid0IHVzZSB0cnlsb2Nr IGluIG9yZGVyIHRvIHJlZHVjZSB0aGUgZnJlcXVlbmN5IG9mIHVzZXJzJwpvb20uIFBldGVyIGFk ZGVkIG11dGV4X3RyeWxvY2tfcmVjdXJzaXZlKCkgYmVjYXVzZSB3ZSBhbHJlYWR5IHdlcmUgZG9p bmcKcmVjdXJzaXZlIGxvY2tpbmcgaW4gdGhlIHNocmlua2VyIGFuZCBhbHRob3VnaCB3ZSBrbm93 IHdlIHNob3VsZG4ndCwKZ2V0dGluZyByaWQgb2YgdGhlIHJlY3Vyc2lvbiBpcyBzb21ldGhpbmcg d2UgYXJlIGRvaW5nLCBidXQgc2xvd2x5LgoKPiBZb3UgY291bGQgc3RhcnQgZHJvcHBpbmcgb2Jq ZWN0cyBhbmQgd2lwaW5nIHZtYXMgYW5kIHN0dWZmIGluIHRoZQo+IG1pZGRsZSBvZiBzb21lIGtt YWxsb2MvYWxsb2NfcGFnZXMgdGhhdCBkb2Vzbid0IGV4cGVjdCBpdCBhbmQgdGhlbgo+IGNyYXNo IGZvciBvdGhlciByZWFzb25zLiBTbyB0aGlzIHJlY2xhaW0gcmVjdXJzaW9uIG1vZGVsIG9mIHRo ZQo+IHNoaW5rZXIgaXMgcXVpdGUgdW5pcXVlIGFuZCBxdWl0ZSBjaGFsbGVuZ2luZyB0byBwcm9v ZiBhcyBzYWZlIGlmIHlvdQo+IGtlZXAgdXNpbmcgbXV0ZXhfdHJ5bG9ja19yZWN1cnNpdmUgaW4g aTkxNV9nZW1fc2hyaW5rZXJfc2Nhbi4KCkkga25vdy4gVHJ5aW5nIHRvIHN0YXkgb24gdG9wIG9m IGFsbCB0aGUga21hbGxvY3MgdW5kZXIgdGhlIHN0cnVjdF9tdXRleAphbmQgYmVpbmcgYXdhcmUg dGhhdCB0aGUgc2hyaW5rZXIgY2FuIGFuZCB3aWxsIHVuZG8geW91ciBvYmplY3RzIGFzIHlvdQp3 b3JrIGlzIGEgY29udGludWFsIGJhdHRsZS4gQW5kIGNhdGNoZXMgZXZlcnlvbmUgd29ya2luZyBv biBpOTE1LmtvIGJ5CnN1cnByaXNlLiBPdXIgcG9saWN5IHRvIGF2b2lkIHN1cnByaXNlcyBpcyBi YXNlZCBhcm91bmQgcGluIGJlZm9yZSBhbGxvYy4KIAo+IExvY2sgcmVjdXJzaW9uIGluIGFsbCBv dGhlciBwbGFjZXMgY291bGQgYmUgZHJvcHBlZCB3aXRob3V0IHJ1bnRpbWUKPiBkb3duc2lkZXMs IHRoZSBvbmx5IHBsYWNlIG11dGV4X3RyeWxvY2tfcmVjdXJzaXZlIG1ha2VzIGEgZGVzaWduCj4g ZGlmZmVyZW5jZSBhbmQgbWFrZXMgc2Vuc2UgdG8gYmUgdXNlZCBpcyBpbiBpOTE1X2dlbV9zaHJp bmtlcl9zY2FuLAo+IHRoZSByZXN0IGFyZSBpbXBsZW1lbnRhdGlvbiBpc3N1ZXMgbm90IGZ1bmRh bWVudGFsIHNocmlua2VyIGRlc2lnbiBhbmQKPiBpdCdkIGJlIG5pY2UgaWYgdGhvc2Ugb3RoZXIg bXV0ZXhfdHJ5bG9ja19yZWN1cnNpdmUgd291bGQgYWxsIGJlCj4gcmVtb3ZlZCBhbmQgdGhlIG9u bHkgb25lIHRoYXQgaXMgbGVmdCBpcyBpbiBpOTE1X2dlbV9zaHJpbmtlcl9zY2FuIGFuZAo+IG5v d2hlcmUgZWxzZSAob3IgdG8gZHJvcCBpdCBhbHNvIGZyb20gaTkxNV9nZW1fc2hyaW5rZXJfc2Nh bikuCgpXZSBkbyBuZWVkIGl0IGZvciBzaHJpbmtlcl9jb3VudCBhcyB3ZWxsLiBJZiB3ZSBqdXN0 IHJlcG9ydCAwIG9iamVjdHMsCnRoZSBzaHJpbmtlcl9zY2FuIGNhbGxiYWNrIHdpbGwgYmUgc2tp cHBlZCwgaWlyYy4gQWxsIHdlIGRvIG5lZWQgaXQgZm9yCmRpcmVjdCBjYWxscyB0byBpOTE1X2dl bV9zaHJpbmsoKSB3aGljaCB0aGVtc2VsdmVzIG1heSBvciBtYXkgbm90IGJlCnVuZGVybmVhdGgg dGhlIHN0cnVjdF9tdXRleCBhdCB0aGUgdGltZS4KCj4gbXV0ZXhfdHJ5bG9ja19yZWN1cnNpdmUo KSBzaG91bGQgYWxzbyBiZSBwYXRjaGVkIHRvIHVzZQo+IFJFQURfT05DRShfX211dGV4X293bmVy KGxvY2spKSBiZWNhdXNlIGN1cnJlbnRseSBpdCBicmVha3MgQy4KCkkgZG9uJ3QgZm9sbG93LAoK c3RhdGljIGlubGluZSBzdHJ1Y3QgdGFza19zdHJ1Y3QgKl9fbXV0ZXhfb3duZXIoc3RydWN0IG11 dGV4ICpsb2NrKQp7CiAgICAgICAgcmV0dXJuIChzdHJ1Y3QgdGFza19zdHJ1Y3QgKikoYXRvbWlj X2xvbmdfcmVhZCgmbG9jay0+b3duZXIpICYgfjB4MDcpOwp9CgpUaGUgYXRvbWljIHJlYWQgaXMg ZXF1aXZhbGVudCB0byBSRUFEX09OQ0UoKS4gV2hhdCdzIGJyb2tlbiBoZXJlPyAoSQpndWVzcyBz dHJpY3QgYWxpYXNpbmcgYW5kIHBvaW50ZXIgY2FzdD8pCgo+IEluIHRoZSB3aG9sZSBrZXJuZWwg aTkxNSBhbmQgbXNtIGRybSBhcmUgdGhlIG9ubHkgdHdvIHVzZXJzIG9mIHN1Y2gKPiBmdW5jdGlv biBpbiBmYWN0LgoKWWVzLCBQZXRlciB3aWxsIGNvbnRpbnVlIHRvIHJlbWluZCB1cyB0byBmaXgg b3VyIGNvZGUgYW5kIGNvbXBsYWluIHVudGlsCml0IGlzLgoKPiBBbm90aGVyIHRoaW5nIGlzIHdo YXQgdmFsdWUgcmV0dXJuIGZyb20gaTkxNV9nZW1fc2hyaW5rZXJfc2NhbiB3aGVuCj4gdW5sb2Nr IGlzIGZhbHNlLCBhbmQgd2UgY2FuJ3QgcG9zc2libHkgd2FpdCBmb3IgdGhlIG1lbW9yeSB0byBi ZSBmcmVlZAo+IGxldCBhbG9uZSBmb3IgYSByY3UgZ3JhY2UgcGVyaW9kLiBGb3IgdmFyaW91cyBy ZWFzb25zIEkgdGhpbmsgaXQncwo+IHNhZmVyIHRvIHJldHVybiB0aGUgY3VycmVudCAiZnJlZSIg ZXZlbiBpZiB3ZSBjb3VsZCBhbHNvIHJldHVybiAiMCIgaW4KPiBzdWNoIGNhc2UuIFRoZXJlIGFy ZSBkaWZmZXJlbnQgdHJhZGVvZmZzLCByZXR1cm5pbmcgImZyZWUiIGlzIGxlc3MKPiBsaWtlbHkg dG8gdHJpZ2dlciBhbiBlYXJseSBPT00gYXMgdGhlIFZNIHRoaW5rcyBpdCdzIHN0aWxsIG1ha2lu Zwo+IHByb2dyZXNzIGFuZCBpbiBmYWN0IGl0IHdpbGwgZ2V0IG1vcmUgZnJlZSBtZW1vcnkgc2hv cnRseSwgd2hpbGUKPiByZXR1cm5pbmcgU0hSSU5LX1NUT1Agd291bGQgYWxzbyBiZSBhbiBvcHRp b24gYW5kIGl0IHdvdWxkIGluc2lzdCBtb3JlCj4gb24gdGhlIG90aGVyIHNsYWJzIHNvIGl0IHdv dWxkIGJlIG1vcmUgcmVsaWFibGUgYXQgZnJlZWluZyBtZW1vcnkKPiB0aW1lbHksIGJ1dCBpdCB3 b3VsZCBiZSBtb3JlIGF0IHJpc2sgb2YgZWFybHkgT09NLiBJIHRoaW5rIHJldHVybmluZwo+ICJm cmVlIiBpcyB0aGUgYmV0dGVyIHRyYWRlb2ZmIG9mIHRoZSB0d28sIGJ1dCBJIHN1Z2dlc3QgdG8g YWRkIGEKPiBjb21tZW50IGFzIGl0J3Mgbm90IGV4YWN0bHkgb2J2aW91cyB3aGF0IGlzIGJldHRl ci4KCkFoLiBUaGUgUkNVIGZyZWVpbmcgaXMgb25seSBmb3IgdGhlIHNtYWxsIGZyeSwgdGhlIHNs YWJzIGZyb20gd2hpY2gKcmVxdWVzdHMgYW5kIG9iamVjdHMgYXJlIGFsbG9jYXRlZC4gSXQncyB0 aGUgZ2lnYWJ5dGVzIG9mIHBhZ2VzIHdlIGhhdmUKcGlubmVkIHRoYXQgY2FuIGJlIHJlbGVhc2Vk IGJ5IGk5MTVfZ2VtX3NocmluaygpIGFyZSB3aGF0IHdlIGNvdW50IGFzCmZyZWVkLCBldmVuIHRo b3VnaCB3ZSBvbmx5IHJldHVybiB0aGVtIHRvIHRoZSBzeXN0ZW0gYW5kIGhvcGUgdGhlIGxydQph bm9uIHBhZ2Ugc2Nhbm5lciB3aWxsIG1ha2UgYXZhaWxhYmxlIGZvciByZXVzZSAoaWYgdGhleSBh cmUgZGlydHksIHRoZXkKd2lsbCBuZWVkIHRvIGJlIHN3YXBwZWQgb3V0LCBidXQgc29tZSB3aWxs IGJlIHJldHVybmVkIHRvIHRoZSBzeXN0ZW0KZGlyZWN0bHkgYnkgdHJ1bmNhdGluZyB0aGUgc2ht ZW1mcyBmaWxwIGJhY2tpbmcgYW4gb2JqZWN0LikKLUNocmlzCgotLSAKQ2hyaXMgV2lsc29uLCBJ bnRlbCBPcGVuIFNvdXJjZSBUZWNobm9sb2d5IENlbnRyZQpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpJbnRlbC1nZnggbWFpbGluZyBsaXN0CkludGVsLWdm eEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFp bG1hbi9saXN0aW5mby9pbnRlbC1nZngK