From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1752091AbdJFLeJ (ORCPT <rfc822;w@1wt.eu>);
        Fri, 6 Oct 2017 07:34:09 -0400
Received: from mga07.intel.com ([134.134.136.100]:39614 "EHLO mga07.intel.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1751557AbdJFLeG (ORCPT <rfc822;linux-kernel@vger.kernel.org>);
        Fri, 6 Oct 2017 07:34:06 -0400
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.42,483,1500966000"; 
   d="scan'208";a="159595411"
Subject: Re: [Intel-gfx] [PATCH 1/2] drm/i915: Preallocate our mmu notifier
 workequeu to unbreak cpu hotplug deadlock
To: Daniel Vetter <daniel.vetter@ffwll.ch>,
        Intel Graphics Development <intel-gfx@lists.freedesktop.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
        LKML <linux-kernel@vger.kernel.org>, Tejun Heo <tj@kernel.org>,
        Daniel Vetter <daniel.vetter@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Sasha Levin <alexander.levin@verizon.com>
References: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
From: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Message-ID: <9f61d57c-12c4-eb01-a008-e674727810d6@linux.intel.com>
Date: Fri, 6 Oct 2017 12:34:02 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-GB
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 06/10/2017 10:06, Daniel Vetter wrote:
> 4.14-rc1 gained the fancy new cross-release support in lockdep, which
> seems to have uncovered a few more rules about what is allowed and
> isn't.
> 
> This one here seems to indicate that allocating a work-queue while
> holding mmap_sem is a no-go, so let's try to preallocate it.
> 
> Of course another way to break this chain would be somewhere in the
> cpu hotplug code, since this isn't the only trace we're finding now
> which goes through msr_create_device.
> 
> Full lockdep splat:
> 
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.14.0-rc1-CI-CI_DRM_3118+ #1 Tainted: G     U
> ------------------------------------------------------
> prime_mmap/1551 is trying to acquire lock:
>   (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8109dbb7>] apply_workqueue_attrs+0x17/0x50
> 
> but task is already holding lock:
>   (&dev_priv->mm_lock){+.+.}, at: [<ffffffffa01a7b2a>] i915_gem_userptr_init__mmu_notifier+0x14a/0x270 [i915]
> 
> which lock already depends on the new lock.
> 
> the existing dependency chain (in reverse order) is:
> 
> -> #6 (&dev_priv->mm_lock){+.+.}:
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         __mutex_lock+0x86/0x9b0
>         mutex_lock_nested+0x1b/0x20
>         i915_gem_userptr_init__mmu_notifier+0x14a/0x270 [i915]
>         i915_gem_userptr_ioctl+0x222/0x2c0 [i915]
>         drm_ioctl_kernel+0x69/0xb0
>         drm_ioctl+0x2f9/0x3d0
>         do_vfs_ioctl+0x94/0x670
>         SyS_ioctl+0x41/0x70
>         entry_SYSCALL_64_fastpath+0x1c/0xb1
> 
> -> #5 (&mm->mmap_sem){++++}:
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         __might_fault+0x68/0x90
>         _copy_to_user+0x23/0x70
>         filldir+0xa5/0x120
>         dcache_readdir+0xf9/0x170
>         iterate_dir+0x69/0x1a0
>         SyS_getdents+0xa5/0x140
>         entry_SYSCALL_64_fastpath+0x1c/0xb1
> 
> -> #4 (&sb->s_type->i_mutex_key#5){++++}:
>         down_write+0x3b/0x70
>         handle_create+0xcb/0x1e0
>         devtmpfsd+0x139/0x180
>         kthread+0x152/0x190
>         ret_from_fork+0x27/0x40
> 
> -> #3 ((complete)&req.done){+.+.}:
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         wait_for_common+0x58/0x210
>         wait_for_completion+0x1d/0x20
>         devtmpfs_create_node+0x13d/0x160
>         device_add+0x5eb/0x620
>         device_create_groups_vargs+0xe0/0xf0
>         device_create+0x3a/0x40
>         msr_device_create+0x2b/0x40
>         cpuhp_invoke_callback+0xa3/0x840
>         cpuhp_thread_fun+0x7a/0x150
>         smpboot_thread_fn+0x18a/0x280
>         kthread+0x152/0x190
>         ret_from_fork+0x27/0x40
> 
> -> #2 (cpuhp_state){+.+.}:
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         cpuhp_issue_call+0x10b/0x170
>         __cpuhp_setup_state_cpuslocked+0x134/0x2a0
>         __cpuhp_setup_state+0x46/0x60
>         page_writeback_init+0x43/0x67
>         pagecache_init+0x3d/0x42
>         start_kernel+0x3a8/0x3fc
>         x86_64_start_reservations+0x2a/0x2c
>         x86_64_start_kernel+0x6d/0x70
>         verify_cpu+0x0/0xfb
> 
> -> #1 (cpuhp_state_mutex){+.+.}:
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         __mutex_lock+0x86/0x9b0
>         mutex_lock_nested+0x1b/0x20
>         __cpuhp_setup_state_cpuslocked+0x52/0x2a0
>         __cpuhp_setup_state+0x46/0x60
>         page_alloc_init+0x28/0x30
>         start_kernel+0x145/0x3fc
>         x86_64_start_reservations+0x2a/0x2c
>         x86_64_start_kernel+0x6d/0x70
>         verify_cpu+0x0/0xfb
> 
> -> #0 (cpu_hotplug_lock.rw_sem){++++}:
>         check_prev_add+0x430/0x840
>         __lock_acquire+0x1420/0x15e0
>         lock_acquire+0xb0/0x200
>         cpus_read_lock+0x3d/0xb0
>         apply_workqueue_attrs+0x17/0x50
>         __alloc_workqueue_key+0x1d8/0x4d9
>         i915_gem_userptr_init__mmu_notifier+0x1fb/0x270 [i915]
>         i915_gem_userptr_ioctl+0x222/0x2c0 [i915]
>         drm_ioctl_kernel+0x69/0xb0
>         drm_ioctl+0x2f9/0x3d0
>         do_vfs_ioctl+0x94/0x670
>         SyS_ioctl+0x41/0x70
>         entry_SYSCALL_64_fastpath+0x1c/0xb1
> 
> other info that might help us debug this:
> 
> Chain exists of:
>    cpu_hotplug_lock.rw_sem --> &mm->mmap_sem --> &dev_priv->mm_lock
> 
>   Possible unsafe locking scenario:
> 
>         CPU0                    CPU1
>         ----                    ----
>    lock(&dev_priv->mm_lock);
>                                 lock(&mm->mmap_sem);
>                                 lock(&dev_priv->mm_lock);
>    lock(cpu_hotplug_lock.rw_sem);
> 
>   *** DEADLOCK ***
> 
> 2 locks held by prime_mmap/1551:
>   #0:  (&mm->mmap_sem){++++}, at: [<ffffffffa01a7b18>] i915_gem_userptr_init__mmu_notifier+0x138/0x270 [i915]
>   #1:  (&dev_priv->mm_lock){+.+.}, at: [<ffffffffa01a7b2a>] i915_gem_userptr_init__mmu_notifier+0x14a/0x270 [i915]
> 
> stack backtrace:
> CPU: 4 PID: 1551 Comm: prime_mmap Tainted: G     U          4.14.0-rc1-CI-CI_DRM_3118+ #1
> Hardware name: Dell Inc. XPS 8300  /0Y2MRG, BIOS A06 10/17/2011
> Call Trace:
>   dump_stack+0x68/0x9f
>   print_circular_bug+0x235/0x3c0
>   ? lockdep_init_map_crosslock+0x20/0x20
>   check_prev_add+0x430/0x840
>   __lock_acquire+0x1420/0x15e0
>   ? __lock_acquire+0x1420/0x15e0
>   ? lockdep_init_map_crosslock+0x20/0x20
>   lock_acquire+0xb0/0x200
>   ? apply_workqueue_attrs+0x17/0x50
>   cpus_read_lock+0x3d/0xb0
>   ? apply_workqueue_attrs+0x17/0x50
>   apply_workqueue_attrs+0x17/0x50
>   __alloc_workqueue_key+0x1d8/0x4d9
>   ? __lockdep_init_map+0x57/0x1c0
>   i915_gem_userptr_init__mmu_notifier+0x1fb/0x270 [i915]
>   i915_gem_userptr_ioctl+0x222/0x2c0 [i915]
>   ? i915_gem_userptr_release+0x140/0x140 [i915]
>   drm_ioctl_kernel+0x69/0xb0
>   drm_ioctl+0x2f9/0x3d0
>   ? i915_gem_userptr_release+0x140/0x140 [i915]
>   ? __do_page_fault+0x2a4/0x570
>   do_vfs_ioctl+0x94/0x670
>   ? entry_SYSCALL_64_fastpath+0x5/0xb1
>   ? __this_cpu_preempt_check+0x13/0x20
>   ? trace_hardirqs_on_caller+0xe3/0x1b0
>   SyS_ioctl+0x41/0x70
>   entry_SYSCALL_64_fastpath+0x1c/0xb1
> RIP: 0033:0x7fbb83c39587
> RSP: 002b:00007fff188dc228 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: ffffffff81492963 RCX: 00007fbb83c39587
> RDX: 00007fff188dc260 RSI: 00000000c0186473 RDI: 0000000000000003
> RBP: ffffc90001487f88 R08: 0000000000000000 R09: 00007fff188dc2ac
> R10: 00007fbb83efcb58 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000003 R14: 00000000c0186473 R15: 00007fff188dc2ac
>   ? __this_cpu_preempt_check+0x13/0x20
> 
> v2: Set ret correctly when we raced with another thread.
> 
> v3: Use Chris' diff. Attach the right lockdep splat.
> 
> Cc: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Sasha Levin <alexander.levin@verizon.com>
> Cc: Marta Lofstedt <marta.lofstedt@intel.com>
> Cc: Tejun Heo <tj@kernel.org>
> References: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_3180/shard-hsw3/igt@prime_mmap@test_userptr.html
> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102939
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> ---
>   drivers/gpu/drm/i915/i915_gem_userptr.c | 35 +++++++++++++++++++--------------
>   1 file changed, 20 insertions(+), 15 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_gem_userptr.c b/drivers/gpu/drm/i915/i915_gem_userptr.c
> index 2d4996de7331..f9b3406401af 100644
> --- a/drivers/gpu/drm/i915/i915_gem_userptr.c
> +++ b/drivers/gpu/drm/i915/i915_gem_userptr.c
> @@ -164,7 +164,6 @@ static struct i915_mmu_notifier *
>   i915_mmu_notifier_create(struct mm_struct *mm)
>   {
>   	struct i915_mmu_notifier *mn;
> -	int ret;
>   
>   	mn = kmalloc(sizeof(*mn), GFP_KERNEL);
>   	if (mn == NULL)
> @@ -179,14 +178,6 @@ i915_mmu_notifier_create(struct mm_struct *mm)
>   		return ERR_PTR(-ENOMEM);
>   	}
>   
> -	 /* Protected by mmap_sem (write-lock) */
> -	ret = __mmu_notifier_register(&mn->mn, mm);
> -	if (ret) {
> -		destroy_workqueue(mn->wq);
> -		kfree(mn);
> -		return ERR_PTR(ret);
> -	}
> -
>   	return mn;
>   }
>   
> @@ -210,23 +201,37 @@ i915_gem_userptr_release__mmu_notifier(struct drm_i915_gem_object *obj)
>   static struct i915_mmu_notifier *
>   i915_mmu_notifier_find(struct i915_mm_struct *mm)
>   {
> -	struct i915_mmu_notifier *mn = mm->mn;
> +	struct i915_mmu_notifier *mn;
> +	int err;
>   
>   	mn = mm->mn;
>   	if (mn)
>   		return mn;
>   
> +	mn = i915_mmu_notifier_create(mm->mm);
> +	if (IS_ERR(mn))
> +		return mn;

Strictly speaking we don't want to fail just yet, only it we actually 
needed a new notifier and we failed to create it.

> +
> +	err = 0;
>   	down_write(&mm->mm->mmap_sem);
>   	mutex_lock(&mm->i915->mm_lock);
> -	if ((mn = mm->mn) == NULL) {
> -		mn = i915_mmu_notifier_create(mm->mm);
> -		if (!IS_ERR(mn))
> -			mm->mn = mn;
> +	if (mm->mn == NULL) {
> +		/* Protected by mmap_sem (write-lock) */
> +		err = __mmu_notifier_register(&mn->mn, mm->mm);
> +		if (!err) {
> +			/* Protected by mm_lock */
> +			mm->mn = fetch_and_zero(&mn);
> +		}
>   	}
>   	mutex_unlock(&mm->i915->mm_lock);
>   	up_write(&mm->mm->mmap_sem);
>   
> -	return mn;
> +	if (mn) {
> +		destroy_workqueue(mn->wq);
> +		kfree(mn);
> +	}
> +
> +	return err ? ERR_PTR(err) : mm->mn;
>   }
>   
>   static int
> 

Otherwise looks good to me.

I would also put a note in the commit on how working around the locking 
issue is also beneficial to performance with moving the allocation step 
outside the mmap_sem.

Regards,

Tvrtko

From mboxrd@z Thu Jan  1 00:00:00 1970
From: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Subject: Re: [PATCH 1/2] drm/i915: Preallocate our mmu notifier
 workequeu to unbreak cpu hotplug deadlock
Date: Fri, 6 Oct 2017 12:34:02 +0100
Message-ID: <9f61d57c-12c4-eb01-a008-e674727810d6@linux.intel.com>
References: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
Mime-Version: 1.0
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Content-Transfer-Encoding: base64
Return-path: <intel-gfx-bounces@lists.freedesktop.org>
Received: from mga05.intel.com (mga05.intel.com [192.55.52.43])
 by gabe.freedesktop.org (Postfix) with ESMTPS id 74B9C6E932
 for <intel-gfx@lists.freedesktop.org>; Fri,  6 Oct 2017 11:34:05 +0000 (UTC)
In-Reply-To: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
Content-Language: en-GB
List-Unsubscribe: <https://lists.freedesktop.org/mailman/options/intel-gfx>,
 <mailto:intel-gfx-request@lists.freedesktop.org?subject=unsubscribe>
List-Archive: <https://lists.freedesktop.org/archives/intel-gfx>
List-Post: <mailto:intel-gfx@lists.freedesktop.org>
List-Help: <mailto:intel-gfx-request@lists.freedesktop.org?subject=help>
List-Subscribe: <https://lists.freedesktop.org/mailman/listinfo/intel-gfx>,
 <mailto:intel-gfx-request@lists.freedesktop.org?subject=subscribe>
Errors-To: intel-gfx-bounces@lists.freedesktop.org
Sender: "Intel-gfx" <intel-gfx-bounces@lists.freedesktop.org>
To: Daniel Vetter <daniel.vetter@ffwll.ch>, Intel Graphics Development <intel-gfx@lists.freedesktop.org>
Cc: Peter Zijlstra <peterz@infradead.org>, LKML <linux-kernel@vger.kernel.org>, Sasha Levin <alexander.levin@verizon.com>, Tejun Heo <tj@kernel.org>, Daniel Vetter <daniel.vetter@intel.com>, Thomas Gleixner <tglx@linutronix.de>
List-Id: intel-gfx@lists.freedesktop.org

Ck9uIDA2LzEwLzIwMTcgMTA6MDYsIERhbmllbCBWZXR0ZXIgd3JvdGU6Cj4gNC4xNC1yYzEgZ2Fp
bmVkIHRoZSBmYW5jeSBuZXcgY3Jvc3MtcmVsZWFzZSBzdXBwb3J0IGluIGxvY2tkZXAsIHdoaWNo
Cj4gc2VlbXMgdG8gaGF2ZSB1bmNvdmVyZWQgYSBmZXcgbW9yZSBydWxlcyBhYm91dCB3aGF0IGlz
IGFsbG93ZWQgYW5kCj4gaXNuJ3QuCj4gCj4gVGhpcyBvbmUgaGVyZSBzZWVtcyB0byBpbmRpY2F0
ZSB0aGF0IGFsbG9jYXRpbmcgYSB3b3JrLXF1ZXVlIHdoaWxlCj4gaG9sZGluZyBtbWFwX3NlbSBp
cyBhIG5vLWdvLCBzbyBsZXQncyB0cnkgdG8gcHJlYWxsb2NhdGUgaXQuCj4gCj4gT2YgY291cnNl
IGFub3RoZXIgd2F5IHRvIGJyZWFrIHRoaXMgY2hhaW4gd291bGQgYmUgc29tZXdoZXJlIGluIHRo
ZQo+IGNwdSBob3RwbHVnIGNvZGUsIHNpbmNlIHRoaXMgaXNuJ3QgdGhlIG9ubHkgdHJhY2Ugd2Un
cmUgZmluZGluZyBub3cKPiB3aGljaCBnb2VzIHRocm91Z2ggbXNyX2NyZWF0ZV9kZXZpY2UuCj4g
Cj4gRnVsbCBsb2NrZGVwIHNwbGF0Ogo+IAo+ID09PT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PQo+IFdBUk5JTkc6IHBvc3NpYmxlIGNpcmN1bGFyIGxv
Y2tpbmcgZGVwZW5kZW5jeSBkZXRlY3RlZAo+IDQuMTQuMC1yYzEtQ0ktQ0lfRFJNXzMxMTgrICMx
IFRhaW50ZWQ6IEcgICAgIFUKPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0KPiBwcmltZV9tbWFwLzE1NTEgaXMgdHJ5aW5nIHRvIGFjcXVpcmUg
bG9jazoKPiAgIChjcHVfaG90cGx1Z19sb2NrLnJ3X3NlbSl7KysrK30sIGF0OiBbPGZmZmZmZmZm
ODEwOWRiYjc+XSBhcHBseV93b3JrcXVldWVfYXR0cnMrMHgxNy8weDUwCj4gCj4gYnV0IHRhc2sg
aXMgYWxyZWFkeSBob2xkaW5nIGxvY2s6Cj4gICAoJmRldl9wcml2LT5tbV9sb2NrKXsrLisufSwg
YXQ6IFs8ZmZmZmZmZmZhMDFhN2IyYT5dIGk5MTVfZ2VtX3VzZXJwdHJfaW5pdF9fbW11X25vdGlm
aWVyKzB4MTRhLzB4MjcwIFtpOTE1XQo+IAo+IHdoaWNoIGxvY2sgYWxyZWFkeSBkZXBlbmRzIG9u
IHRoZSBuZXcgbG9jay4KPiAKPiB0aGUgZXhpc3RpbmcgZGVwZW5kZW5jeSBjaGFpbiAoaW4gcmV2
ZXJzZSBvcmRlcikgaXM6Cj4gCj4gLT4gIzYgKCZkZXZfcHJpdi0+bW1fbG9jayl7Ky4rLn06Cj4g
ICAgICAgICBfX2xvY2tfYWNxdWlyZSsweDE0MjAvMHgxNWUwCj4gICAgICAgICBsb2NrX2FjcXVp
cmUrMHhiMC8weDIwMAo+ICAgICAgICAgX19tdXRleF9sb2NrKzB4ODYvMHg5YjAKPiAgICAgICAg
IG11dGV4X2xvY2tfbmVzdGVkKzB4MWIvMHgyMAo+ICAgICAgICAgaTkxNV9nZW1fdXNlcnB0cl9p
bml0X19tbXVfbm90aWZpZXIrMHgxNGEvMHgyNzAgW2k5MTVdCj4gICAgICAgICBpOTE1X2dlbV91
c2VycHRyX2lvY3RsKzB4MjIyLzB4MmMwIFtpOTE1XQo+ICAgICAgICAgZHJtX2lvY3RsX2tlcm5l
bCsweDY5LzB4YjAKPiAgICAgICAgIGRybV9pb2N0bCsweDJmOS8weDNkMAo+ICAgICAgICAgZG9f
dmZzX2lvY3RsKzB4OTQvMHg2NzAKPiAgICAgICAgIFN5U19pb2N0bCsweDQxLzB4NzAKPiAgICAg
ICAgIGVudHJ5X1NZU0NBTExfNjRfZmFzdHBhdGgrMHgxYy8weGIxCj4gCj4gLT4gIzUgKCZtbS0+
bW1hcF9zZW0peysrKyt9Ogo+ICAgICAgICAgX19sb2NrX2FjcXVpcmUrMHgxNDIwLzB4MTVlMAo+
ICAgICAgICAgbG9ja19hY3F1aXJlKzB4YjAvMHgyMDAKPiAgICAgICAgIF9fbWlnaHRfZmF1bHQr
MHg2OC8weDkwCj4gICAgICAgICBfY29weV90b191c2VyKzB4MjMvMHg3MAo+ICAgICAgICAgZmls
bGRpcisweGE1LzB4MTIwCj4gICAgICAgICBkY2FjaGVfcmVhZGRpcisweGY5LzB4MTcwCj4gICAg
ICAgICBpdGVyYXRlX2RpcisweDY5LzB4MWEwCj4gICAgICAgICBTeVNfZ2V0ZGVudHMrMHhhNS8w
eDE0MAo+ICAgICAgICAgZW50cnlfU1lTQ0FMTF82NF9mYXN0cGF0aCsweDFjLzB4YjEKPiAKPiAt
PiAjNCAoJnNiLT5zX3R5cGUtPmlfbXV0ZXhfa2V5IzUpeysrKyt9Ogo+ICAgICAgICAgZG93bl93
cml0ZSsweDNiLzB4NzAKPiAgICAgICAgIGhhbmRsZV9jcmVhdGUrMHhjYi8weDFlMAo+ICAgICAg
ICAgZGV2dG1wZnNkKzB4MTM5LzB4MTgwCj4gICAgICAgICBrdGhyZWFkKzB4MTUyLzB4MTkwCj4g
ICAgICAgICByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MAo+IAo+IC0+ICMzICgoY29tcGxldGUpJnJl
cS5kb25lKXsrLisufToKPiAgICAgICAgIF9fbG9ja19hY3F1aXJlKzB4MTQyMC8weDE1ZTAKPiAg
ICAgICAgIGxvY2tfYWNxdWlyZSsweGIwLzB4MjAwCj4gICAgICAgICB3YWl0X2Zvcl9jb21tb24r
MHg1OC8weDIxMAo+ICAgICAgICAgd2FpdF9mb3JfY29tcGxldGlvbisweDFkLzB4MjAKPiAgICAg
ICAgIGRldnRtcGZzX2NyZWF0ZV9ub2RlKzB4MTNkLzB4MTYwCj4gICAgICAgICBkZXZpY2VfYWRk
KzB4NWViLzB4NjIwCj4gICAgICAgICBkZXZpY2VfY3JlYXRlX2dyb3Vwc192YXJncysweGUwLzB4
ZjAKPiAgICAgICAgIGRldmljZV9jcmVhdGUrMHgzYS8weDQwCj4gICAgICAgICBtc3JfZGV2aWNl
X2NyZWF0ZSsweDJiLzB4NDAKPiAgICAgICAgIGNwdWhwX2ludm9rZV9jYWxsYmFjaysweGEzLzB4
ODQwCj4gICAgICAgICBjcHVocF90aHJlYWRfZnVuKzB4N2EvMHgxNTAKPiAgICAgICAgIHNtcGJv
b3RfdGhyZWFkX2ZuKzB4MThhLzB4MjgwCj4gICAgICAgICBrdGhyZWFkKzB4MTUyLzB4MTkwCj4g
ICAgICAgICByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MAo+IAo+IC0+ICMyIChjcHVocF9zdGF0ZSl7
Ky4rLn06Cj4gICAgICAgICBfX2xvY2tfYWNxdWlyZSsweDE0MjAvMHgxNWUwCj4gICAgICAgICBs
b2NrX2FjcXVpcmUrMHhiMC8weDIwMAo+ICAgICAgICAgY3B1aHBfaXNzdWVfY2FsbCsweDEwYi8w
eDE3MAo+ICAgICAgICAgX19jcHVocF9zZXR1cF9zdGF0ZV9jcHVzbG9ja2VkKzB4MTM0LzB4MmEw
Cj4gICAgICAgICBfX2NwdWhwX3NldHVwX3N0YXRlKzB4NDYvMHg2MAo+ICAgICAgICAgcGFnZV93
cml0ZWJhY2tfaW5pdCsweDQzLzB4NjcKPiAgICAgICAgIHBhZ2VjYWNoZV9pbml0KzB4M2QvMHg0
Mgo+ICAgICAgICAgc3RhcnRfa2VybmVsKzB4M2E4LzB4M2ZjCj4gICAgICAgICB4ODZfNjRfc3Rh
cnRfcmVzZXJ2YXRpb25zKzB4MmEvMHgyYwo+ICAgICAgICAgeDg2XzY0X3N0YXJ0X2tlcm5lbCsw
eDZkLzB4NzAKPiAgICAgICAgIHZlcmlmeV9jcHUrMHgwLzB4ZmIKPiAKPiAtPiAjMSAoY3B1aHBf
c3RhdGVfbXV0ZXgpeysuKy59Ogo+ICAgICAgICAgX19sb2NrX2FjcXVpcmUrMHgxNDIwLzB4MTVl
MAo+ICAgICAgICAgbG9ja19hY3F1aXJlKzB4YjAvMHgyMDAKPiAgICAgICAgIF9fbXV0ZXhfbG9j
aysweDg2LzB4OWIwCj4gICAgICAgICBtdXRleF9sb2NrX25lc3RlZCsweDFiLzB4MjAKPiAgICAg
ICAgIF9fY3B1aHBfc2V0dXBfc3RhdGVfY3B1c2xvY2tlZCsweDUyLzB4MmEwCj4gICAgICAgICBf
X2NwdWhwX3NldHVwX3N0YXRlKzB4NDYvMHg2MAo+ICAgICAgICAgcGFnZV9hbGxvY19pbml0KzB4
MjgvMHgzMAo+ICAgICAgICAgc3RhcnRfa2VybmVsKzB4MTQ1LzB4M2ZjCj4gICAgICAgICB4ODZf
NjRfc3RhcnRfcmVzZXJ2YXRpb25zKzB4MmEvMHgyYwo+ICAgICAgICAgeDg2XzY0X3N0YXJ0X2tl
cm5lbCsweDZkLzB4NzAKPiAgICAgICAgIHZlcmlmeV9jcHUrMHgwLzB4ZmIKPiAKPiAtPiAjMCAo
Y3B1X2hvdHBsdWdfbG9jay5yd19zZW0peysrKyt9Ogo+ICAgICAgICAgY2hlY2tfcHJldl9hZGQr
MHg0MzAvMHg4NDAKPiAgICAgICAgIF9fbG9ja19hY3F1aXJlKzB4MTQyMC8weDE1ZTAKPiAgICAg
ICAgIGxvY2tfYWNxdWlyZSsweGIwLzB4MjAwCj4gICAgICAgICBjcHVzX3JlYWRfbG9jaysweDNk
LzB4YjAKPiAgICAgICAgIGFwcGx5X3dvcmtxdWV1ZV9hdHRycysweDE3LzB4NTAKPiAgICAgICAg
IF9fYWxsb2Nfd29ya3F1ZXVlX2tleSsweDFkOC8weDRkOQo+ICAgICAgICAgaTkxNV9nZW1fdXNl
cnB0cl9pbml0X19tbXVfbm90aWZpZXIrMHgxZmIvMHgyNzAgW2k5MTVdCj4gICAgICAgICBpOTE1
X2dlbV91c2VycHRyX2lvY3RsKzB4MjIyLzB4MmMwIFtpOTE1XQo+ICAgICAgICAgZHJtX2lvY3Rs
X2tlcm5lbCsweDY5LzB4YjAKPiAgICAgICAgIGRybV9pb2N0bCsweDJmOS8weDNkMAo+ICAgICAg
ICAgZG9fdmZzX2lvY3RsKzB4OTQvMHg2NzAKPiAgICAgICAgIFN5U19pb2N0bCsweDQxLzB4NzAK
PiAgICAgICAgIGVudHJ5X1NZU0NBTExfNjRfZmFzdHBhdGgrMHgxYy8weGIxCj4gCj4gb3RoZXIg
aW5mbyB0aGF0IG1pZ2h0IGhlbHAgdXMgZGVidWcgdGhpczoKPiAKPiBDaGFpbiBleGlzdHMgb2Y6
Cj4gICAgY3B1X2hvdHBsdWdfbG9jay5yd19zZW0gLS0+ICZtbS0+bW1hcF9zZW0gLS0+ICZkZXZf
cHJpdi0+bW1fbG9jawo+IAo+ICAgUG9zc2libGUgdW5zYWZlIGxvY2tpbmcgc2NlbmFyaW86Cj4g
Cj4gICAgICAgICBDUFUwICAgICAgICAgICAgICAgICAgICBDUFUxCj4gICAgICAgICAtLS0tICAg
ICAgICAgICAgICAgICAgICAtLS0tCj4gICAgbG9jaygmZGV2X3ByaXYtPm1tX2xvY2spOwo+ICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG9jaygmbW0tPm1tYXBfc2VtKTsKPiAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxvY2soJmRldl9wcml2LT5tbV9sb2NrKTsKPiAg
ICBsb2NrKGNwdV9ob3RwbHVnX2xvY2sucndfc2VtKTsKPiAKPiAgICoqKiBERUFETE9DSyAqKioK
PiAKPiAyIGxvY2tzIGhlbGQgYnkgcHJpbWVfbW1hcC8xNTUxOgo+ICAgIzA6ICAoJm1tLT5tbWFw
X3NlbSl7KysrK30sIGF0OiBbPGZmZmZmZmZmYTAxYTdiMTg+XSBpOTE1X2dlbV91c2VycHRyX2lu
aXRfX21tdV9ub3RpZmllcisweDEzOC8weDI3MCBbaTkxNV0KPiAgICMxOiAgKCZkZXZfcHJpdi0+
bW1fbG9jayl7Ky4rLn0sIGF0OiBbPGZmZmZmZmZmYTAxYTdiMmE+XSBpOTE1X2dlbV91c2VycHRy
X2luaXRfX21tdV9ub3RpZmllcisweDE0YS8weDI3MCBbaTkxNV0KPiAKPiBzdGFjayBiYWNrdHJh
Y2U6Cj4gQ1BVOiA0IFBJRDogMTU1MSBDb21tOiBwcmltZV9tbWFwIFRhaW50ZWQ6IEcgICAgIFUg
ICAgICAgICAgNC4xNC4wLXJjMS1DSS1DSV9EUk1fMzExOCsgIzEKPiBIYXJkd2FyZSBuYW1lOiBE
ZWxsIEluYy4gWFBTIDgzMDAgIC8wWTJNUkcsIEJJT1MgQTA2IDEwLzE3LzIwMTEKPiBDYWxsIFRy
YWNlOgo+ICAgZHVtcF9zdGFjaysweDY4LzB4OWYKPiAgIHByaW50X2NpcmN1bGFyX2J1ZysweDIz
NS8weDNjMAo+ICAgPyBsb2NrZGVwX2luaXRfbWFwX2Nyb3NzbG9jaysweDIwLzB4MjAKPiAgIGNo
ZWNrX3ByZXZfYWRkKzB4NDMwLzB4ODQwCj4gICBfX2xvY2tfYWNxdWlyZSsweDE0MjAvMHgxNWUw
Cj4gICA/IF9fbG9ja19hY3F1aXJlKzB4MTQyMC8weDE1ZTAKPiAgID8gbG9ja2RlcF9pbml0X21h
cF9jcm9zc2xvY2srMHgyMC8weDIwCj4gICBsb2NrX2FjcXVpcmUrMHhiMC8weDIwMAo+ICAgPyBh
cHBseV93b3JrcXVldWVfYXR0cnMrMHgxNy8weDUwCj4gICBjcHVzX3JlYWRfbG9jaysweDNkLzB4
YjAKPiAgID8gYXBwbHlfd29ya3F1ZXVlX2F0dHJzKzB4MTcvMHg1MAo+ICAgYXBwbHlfd29ya3F1
ZXVlX2F0dHJzKzB4MTcvMHg1MAo+ICAgX19hbGxvY193b3JrcXVldWVfa2V5KzB4MWQ4LzB4NGQ5
Cj4gICA/IF9fbG9ja2RlcF9pbml0X21hcCsweDU3LzB4MWMwCj4gICBpOTE1X2dlbV91c2VycHRy
X2luaXRfX21tdV9ub3RpZmllcisweDFmYi8weDI3MCBbaTkxNV0KPiAgIGk5MTVfZ2VtX3VzZXJw
dHJfaW9jdGwrMHgyMjIvMHgyYzAgW2k5MTVdCj4gICA/IGk5MTVfZ2VtX3VzZXJwdHJfcmVsZWFz
ZSsweDE0MC8weDE0MCBbaTkxNV0KPiAgIGRybV9pb2N0bF9rZXJuZWwrMHg2OS8weGIwCj4gICBk
cm1faW9jdGwrMHgyZjkvMHgzZDAKPiAgID8gaTkxNV9nZW1fdXNlcnB0cl9yZWxlYXNlKzB4MTQw
LzB4MTQwIFtpOTE1XQo+ICAgPyBfX2RvX3BhZ2VfZmF1bHQrMHgyYTQvMHg1NzAKPiAgIGRvX3Zm
c19pb2N0bCsweDk0LzB4NjcwCj4gICA/IGVudHJ5X1NZU0NBTExfNjRfZmFzdHBhdGgrMHg1LzB4
YjEKPiAgID8gX190aGlzX2NwdV9wcmVlbXB0X2NoZWNrKzB4MTMvMHgyMAo+ICAgPyB0cmFjZV9o
YXJkaXJxc19vbl9jYWxsZXIrMHhlMy8weDFiMAo+ICAgU3lTX2lvY3RsKzB4NDEvMHg3MAo+ICAg
ZW50cnlfU1lTQ0FMTF82NF9mYXN0cGF0aCsweDFjLzB4YjEKPiBSSVA6IDAwMzM6MHg3ZmJiODNj
Mzk1ODcKPiBSU1A6IDAwMmI6MDAwMDdmZmYxODhkYzIyOCBFRkxBR1M6IDAwMDAwMjQ2IE9SSUdf
UkFYOiAwMDAwMDAwMDAwMDAwMDEwCj4gUkFYOiBmZmZmZmZmZmZmZmZmZmRhIFJCWDogZmZmZmZm
ZmY4MTQ5Mjk2MyBSQ1g6IDAwMDA3ZmJiODNjMzk1ODcKPiBSRFg6IDAwMDA3ZmZmMTg4ZGMyNjAg
UlNJOiAwMDAwMDAwMGMwMTg2NDczIFJESTogMDAwMDAwMDAwMDAwMDAwMwo+IFJCUDogZmZmZmM5
MDAwMTQ4N2Y4OCBSMDg6IDAwMDAwMDAwMDAwMDAwMDAgUjA5OiAwMDAwN2ZmZjE4OGRjMmFjCj4g
UjEwOiAwMDAwN2ZiYjgzZWZjYjU4IFIxMTogMDAwMDAwMDAwMDAwMDI0NiBSMTI6IDAwMDAwMDAw
MDAwMDAwMDAKPiBSMTM6IDAwMDAwMDAwMDAwMDAwMDMgUjE0OiAwMDAwMDAwMGMwMTg2NDczIFIx
NTogMDAwMDdmZmYxODhkYzJhYwo+ICAgPyBfX3RoaXNfY3B1X3ByZWVtcHRfY2hlY2srMHgxMy8w
eDIwCj4gCj4gdjI6IFNldCByZXQgY29ycmVjdGx5IHdoZW4gd2UgcmFjZWQgd2l0aCBhbm90aGVy
IHRocmVhZC4KPiAKPiB2MzogVXNlIENocmlzJyBkaWZmLiBBdHRhY2ggdGhlIHJpZ2h0IGxvY2tk
ZXAgc3BsYXQuCj4gCj4gQ2M6IENocmlzIFdpbHNvbiA8Y2hyaXNAY2hyaXMtd2lsc29uLmNvLnVr
Pgo+IENjOiBUdnJ0a28gVXJzdWxpbiA8dHZydGtvLnVyc3VsaW5AaW50ZWwuY29tPgo+IENjOiBK
b29uYXMgTGFodGluZW4gPGpvb25hcy5sYWh0aW5lbkBsaW51eC5pbnRlbC5jb20+Cj4gQ2M6IFBl
dGVyIFppamxzdHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz4KPiBDYzogVGhvbWFzIEdsZWl4bmVy
IDx0Z2x4QGxpbnV0cm9uaXguZGU+Cj4gQ2M6IFNhc2hhIExldmluIDxhbGV4YW5kZXIubGV2aW5A
dmVyaXpvbi5jb20+Cj4gQ2M6IE1hcnRhIExvZnN0ZWR0IDxtYXJ0YS5sb2ZzdGVkdEBpbnRlbC5j
b20+Cj4gQ2M6IFRlanVuIEhlbyA8dGpAa2VybmVsLm9yZz4KPiBSZWZlcmVuY2VzOiBodHRwczov
L2ludGVsLWdmeC1jaS4wMS5vcmcvdHJlZS9kcm0tdGlwL0NJX0RSTV8zMTgwL3NoYXJkLWhzdzMv
aWd0QHByaW1lX21tYXBAdGVzdF91c2VycHRyLmh0bWwKPiBCdWd6aWxsYTogaHR0cHM6Ly9idWdz
LmZyZWVkZXNrdG9wLm9yZy9zaG93X2J1Zy5jZ2k/aWQ9MTAyOTM5Cj4gU2lnbmVkLW9mZi1ieTog
RGFuaWVsIFZldHRlciA8ZGFuaWVsLnZldHRlckBpbnRlbC5jb20+Cj4gLS0tCj4gICBkcml2ZXJz
L2dwdS9kcm0vaTkxNS9pOTE1X2dlbV91c2VycHRyLmMgfCAzNSArKysrKysrKysrKysrKysrKysr
LS0tLS0tLS0tLS0tLS0KPiAgIDEgZmlsZSBjaGFuZ2VkLCAyMCBpbnNlcnRpb25zKCspLCAxNSBk
ZWxldGlvbnMoLSkKPiAKPiBkaWZmIC0tZ2l0IGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvaTkxNV9n
ZW1fdXNlcnB0ci5jIGIvZHJpdmVycy9ncHUvZHJtL2k5MTUvaTkxNV9nZW1fdXNlcnB0ci5jCj4g
aW5kZXggMmQ0OTk2ZGU3MzMxLi5mOWIzNDA2NDAxYWYgMTAwNjQ0Cj4gLS0tIGEvZHJpdmVycy9n
cHUvZHJtL2k5MTUvaTkxNV9nZW1fdXNlcnB0ci5jCj4gKysrIGIvZHJpdmVycy9ncHUvZHJtL2k5
MTUvaTkxNV9nZW1fdXNlcnB0ci5jCj4gQEAgLTE2NCw3ICsxNjQsNiBAQCBzdGF0aWMgc3RydWN0
IGk5MTVfbW11X25vdGlmaWVyICoKPiAgIGk5MTVfbW11X25vdGlmaWVyX2NyZWF0ZShzdHJ1Y3Qg
bW1fc3RydWN0ICptbSkKPiAgIHsKPiAgIAlzdHJ1Y3QgaTkxNV9tbXVfbm90aWZpZXIgKm1uOwo+
IC0JaW50IHJldDsKPiAgIAo+ICAgCW1uID0ga21hbGxvYyhzaXplb2YoKm1uKSwgR0ZQX0tFUk5F
TCk7Cj4gICAJaWYgKG1uID09IE5VTEwpCj4gQEAgLTE3OSwxNCArMTc4LDYgQEAgaTkxNV9tbXVf
bm90aWZpZXJfY3JlYXRlKHN0cnVjdCBtbV9zdHJ1Y3QgKm1tKQo+ICAgCQlyZXR1cm4gRVJSX1BU
UigtRU5PTUVNKTsKPiAgIAl9Cj4gICAKPiAtCSAvKiBQcm90ZWN0ZWQgYnkgbW1hcF9zZW0gKHdy
aXRlLWxvY2spICovCj4gLQlyZXQgPSBfX21tdV9ub3RpZmllcl9yZWdpc3RlcigmbW4tPm1uLCBt
bSk7Cj4gLQlpZiAocmV0KSB7Cj4gLQkJZGVzdHJveV93b3JrcXVldWUobW4tPndxKTsKPiAtCQlr
ZnJlZShtbik7Cj4gLQkJcmV0dXJuIEVSUl9QVFIocmV0KTsKPiAtCX0KPiAtCj4gICAJcmV0dXJu
IG1uOwo+ICAgfQo+ICAgCj4gQEAgLTIxMCwyMyArMjAxLDM3IEBAIGk5MTVfZ2VtX3VzZXJwdHJf
cmVsZWFzZV9fbW11X25vdGlmaWVyKHN0cnVjdCBkcm1faTkxNV9nZW1fb2JqZWN0ICpvYmopCj4g
ICBzdGF0aWMgc3RydWN0IGk5MTVfbW11X25vdGlmaWVyICoKPiAgIGk5MTVfbW11X25vdGlmaWVy
X2ZpbmQoc3RydWN0IGk5MTVfbW1fc3RydWN0ICptbSkKPiAgIHsKPiAtCXN0cnVjdCBpOTE1X21t
dV9ub3RpZmllciAqbW4gPSBtbS0+bW47Cj4gKwlzdHJ1Y3QgaTkxNV9tbXVfbm90aWZpZXIgKm1u
Owo+ICsJaW50IGVycjsKPiAgIAo+ICAgCW1uID0gbW0tPm1uOwo+ICAgCWlmIChtbikKPiAgIAkJ
cmV0dXJuIG1uOwo+ICAgCj4gKwltbiA9IGk5MTVfbW11X25vdGlmaWVyX2NyZWF0ZShtbS0+bW0p
Owo+ICsJaWYgKElTX0VSUihtbikpCj4gKwkJcmV0dXJuIG1uOwoKU3RyaWN0bHkgc3BlYWtpbmcg
d2UgZG9uJ3Qgd2FudCB0byBmYWlsIGp1c3QgeWV0LCBvbmx5IGl0IHdlIGFjdHVhbGx5IApuZWVk
ZWQgYSBuZXcgbm90aWZpZXIgYW5kIHdlIGZhaWxlZCB0byBjcmVhdGUgaXQuCgo+ICsKPiArCWVy
ciA9IDA7Cj4gICAJZG93bl93cml0ZSgmbW0tPm1tLT5tbWFwX3NlbSk7Cj4gICAJbXV0ZXhfbG9j
aygmbW0tPmk5MTUtPm1tX2xvY2spOwo+IC0JaWYgKChtbiA9IG1tLT5tbikgPT0gTlVMTCkgewo+
IC0JCW1uID0gaTkxNV9tbXVfbm90aWZpZXJfY3JlYXRlKG1tLT5tbSk7Cj4gLQkJaWYgKCFJU19F
UlIobW4pKQo+IC0JCQltbS0+bW4gPSBtbjsKPiArCWlmIChtbS0+bW4gPT0gTlVMTCkgewo+ICsJ
CS8qIFByb3RlY3RlZCBieSBtbWFwX3NlbSAod3JpdGUtbG9jaykgKi8KPiArCQllcnIgPSBfX21t
dV9ub3RpZmllcl9yZWdpc3RlcigmbW4tPm1uLCBtbS0+bW0pOwo+ICsJCWlmICghZXJyKSB7Cj4g
KwkJCS8qIFByb3RlY3RlZCBieSBtbV9sb2NrICovCj4gKwkJCW1tLT5tbiA9IGZldGNoX2FuZF96
ZXJvKCZtbik7Cj4gKwkJfQo+ICAgCX0KPiAgIAltdXRleF91bmxvY2soJm1tLT5pOTE1LT5tbV9s
b2NrKTsKPiAgIAl1cF93cml0ZSgmbW0tPm1tLT5tbWFwX3NlbSk7Cj4gICAKPiAtCXJldHVybiBt
bjsKPiArCWlmIChtbikgewo+ICsJCWRlc3Ryb3lfd29ya3F1ZXVlKG1uLT53cSk7Cj4gKwkJa2Zy
ZWUobW4pOwo+ICsJfQo+ICsKPiArCXJldHVybiBlcnIgPyBFUlJfUFRSKGVycikgOiBtbS0+bW47
Cj4gICB9Cj4gICAKPiAgIHN0YXRpYyBpbnQKPiAKCk90aGVyd2lzZSBsb29rcyBnb29kIHRvIG1l
LgoKSSB3b3VsZCBhbHNvIHB1dCBhIG5vdGUgaW4gdGhlIGNvbW1pdCBvbiBob3cgd29ya2luZyBh
cm91bmQgdGhlIGxvY2tpbmcgCmlzc3VlIGlzIGFsc28gYmVuZWZpY2lhbCB0byBwZXJmb3JtYW5j
ZSB3aXRoIG1vdmluZyB0aGUgYWxsb2NhdGlvbiBzdGVwIApvdXRzaWRlIHRoZSBtbWFwX3NlbS4K
ClJlZ2FyZHMsCgpUdnJ0a28KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0
b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50
ZWwtZ2Z4Cg==