From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1751971AbdJFLEL convert rfc822-to-8bit (ORCPT <rfc822;w@1wt.eu>);
        Fri, 6 Oct 2017 07:04:11 -0400
Received: from mail.fireflyinternet.com ([109.228.58.192]:63830 "EHLO
        fireflyinternet.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org
        with ESMTP id S1750849AbdJFLEK (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Fri, 6 Oct 2017 07:04:10 -0400
X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=78.156.65.138;
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 8BIT
To: Daniel Vetter <daniel.vetter@ffwll.ch>,
        "Intel Graphics Development" <intel-gfx@lists.freedesktop.org>
From: Chris Wilson <chris@chris-wilson.co.uk>
In-Reply-To: <20171006090637.25545-2-daniel.vetter@ffwll.ch>
Cc: "LKML" <linux-kernel@vger.kernel.org>,
        "Daniel Vetter" <daniel.vetter@ffwll.ch>,
        "Mika Kuoppala" <mika.kuoppala@intel.com>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        "Marta Lofstedt" <marta.lofstedt@intel.com>,
        "Daniel Vetter" <daniel.vetter@intel.com>
References: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
 <20171006090637.25545-2-daniel.vetter@ffwll.ch>
Message-ID: <150728782908.14660.8438180235890645166@mail.alporthouse.com>
User-Agent: alot/0.3.6
Subject: Re: [PATCH 2/2] drm/i915: Use rcu instead of stop_machine in set_wedged
Date: Fri, 06 Oct 2017 12:03:49 +0100
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Quoting Daniel Vetter (2017-10-06 10:06:37)
> stop_machine is not really a locking primitive we should use, except
> when the hw folks tell us the hw is broken and that's the only way to
> work around it.
> 
> This patch tries to address the locking abuse of stop_machine() from
> 
> commit 20e4933c478a1ca694b38fa4ac44d99e659941f5
> Author: Chris Wilson <chris@chris-wilson.co.uk>
> Date:   Tue Nov 22 14:41:21 2016 +0000
> 
>     drm/i915: Stop the machine as we install the wedged submit_request handler
> 
> Chris said parts of the reasons for going with stop_machine() was that
> it's no overhead for the fast-path. But these callbacks use irqsave
> spinlocks and do a bunch of MMIO, and rcu_read_lock is _real_ fast.
> 
> To stay as close as possible to the stop_machine semantics we first
> update all the submit function pointers to the nop handler, then call
> synchronize_rcu() to make sure no new requests can be submitted. This
> should give us exactly the huge barrier we want.
> 
> I pondered whether we should annotate engine->submit_request as __rcu
> and use rcu_assign_pointer and rcu_dereference on it. But the reason
> behind those is to make sure the compiler/cpu barriers are there for
> when you have an actual data structure you point at, to make sure all
> the writes are seen correctly on the read side. But we just have a
> function pointer, and .text isn't changed, so no need for these
> barriers and hence no need for annotations.
> 
> This should fix the followwing lockdep splat:
> 
> ======================================================
> WARNING: possible circular locking dependency detected
> 4.14.0-rc3-CI-CI_DRM_3179+ #1 Tainted: G     U
> ------------------------------------------------------
> kworker/3:4/562 is trying to acquire lock:
>  (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8113d4bc>] stop_machine+0x1c/0x40
> 
> but task is already holding lock:
>  (&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
> 
> which lock already depends on the new lock.
> 
> the existing dependency chain (in reverse order) is:
> 
> -> #6 (&dev->struct_mutex){+.+.}:
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        __mutex_lock+0x86/0x9b0
>        mutex_lock_interruptible_nested+0x1b/0x20
>        i915_mutex_lock_interruptible+0x51/0x130 [i915]
>        i915_gem_fault+0x209/0x650 [i915]
>        __do_fault+0x1e/0x80
>        __handle_mm_fault+0xa08/0xed0
>        handle_mm_fault+0x156/0x300
>        __do_page_fault+0x2c5/0x570
>        do_page_fault+0x28/0x250
>        page_fault+0x22/0x30
> 
> -> #5 (&mm->mmap_sem){++++}:
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        __might_fault+0x68/0x90
>        _copy_to_user+0x23/0x70
>        filldir+0xa5/0x120
>        dcache_readdir+0xf9/0x170
>        iterate_dir+0x69/0x1a0
>        SyS_getdents+0xa5/0x140
>        entry_SYSCALL_64_fastpath+0x1c/0xb1
> 
> -> #4 (&sb->s_type->i_mutex_key#5){++++}:
>        down_write+0x3b/0x70
>        handle_create+0xcb/0x1e0
>        devtmpfsd+0x139/0x180
>        kthread+0x152/0x190
>        ret_from_fork+0x27/0x40
> 
> -> #3 ((complete)&req.done){+.+.}:
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        wait_for_common+0x58/0x210
>        wait_for_completion+0x1d/0x20
>        devtmpfs_create_node+0x13d/0x160
>        device_add+0x5eb/0x620
>        device_create_groups_vargs+0xe0/0xf0
>        device_create+0x3a/0x40
>        msr_device_create+0x2b/0x40
>        cpuhp_invoke_callback+0xc9/0xbf0
>        cpuhp_thread_fun+0x17b/0x240
>        smpboot_thread_fn+0x18a/0x280
>        kthread+0x152/0x190
>        ret_from_fork+0x27/0x40
> 
> -> #2 (cpuhp_state-up){+.+.}:
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        cpuhp_issue_call+0x133/0x1c0
>        __cpuhp_setup_state_cpuslocked+0x139/0x2a0
>        __cpuhp_setup_state+0x46/0x60
>        page_writeback_init+0x43/0x67
>        pagecache_init+0x3d/0x42
>        start_kernel+0x3a8/0x3fc
>        x86_64_start_reservations+0x2a/0x2c
>        x86_64_start_kernel+0x6d/0x70
>        verify_cpu+0x0/0xfb
> 
> -> #1 (cpuhp_state_mutex){+.+.}:
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        __mutex_lock+0x86/0x9b0
>        mutex_lock_nested+0x1b/0x20
>        __cpuhp_setup_state_cpuslocked+0x53/0x2a0
>        __cpuhp_setup_state+0x46/0x60
>        page_alloc_init+0x28/0x30
>        start_kernel+0x145/0x3fc
>        x86_64_start_reservations+0x2a/0x2c
>        x86_64_start_kernel+0x6d/0x70
>        verify_cpu+0x0/0xfb
> 
> -> #0 (cpu_hotplug_lock.rw_sem){++++}:
>        check_prev_add+0x430/0x840
>        __lock_acquire+0x1420/0x15e0
>        lock_acquire+0xb0/0x200
>        cpus_read_lock+0x3d/0xb0
>        stop_machine+0x1c/0x40
>        i915_gem_set_wedged+0x1a/0x20 [i915]
>        i915_reset+0xb9/0x230 [i915]
>        i915_reset_device+0x1f6/0x260 [i915]
>        i915_handle_error+0x2d8/0x430 [i915]
>        hangcheck_declare_hang+0xd3/0xf0 [i915]
>        i915_hangcheck_elapsed+0x262/0x2d0 [i915]
>        process_one_work+0x233/0x660
>        worker_thread+0x4e/0x3b0
>        kthread+0x152/0x190
>        ret_from_fork+0x27/0x40
> 
> other info that might help us debug this:
> 
> Chain exists of:
>   cpu_hotplug_lock.rw_sem --> &mm->mmap_sem --> &dev->struct_mutex
> 
>  Possible unsafe locking scenario:
> 
>        CPU0                    CPU1
>        ----                    ----
>   lock(&dev->struct_mutex);
>                                lock(&mm->mmap_sem);
>                                lock(&dev->struct_mutex);
>   lock(cpu_hotplug_lock.rw_sem);
> 
>  *** DEADLOCK ***
> 
> 3 locks held by kworker/3:4/562:
>  #0:  ("events_long"){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
>  #1:  ((&(&i915->gpu_error.hangcheck_work)->work)){+.+.}, at: [<ffffffff8109c64a>] process_one_work+0x1aa/0x660
>  #2:  (&dev->struct_mutex){+.+.}, at: [<ffffffffa0136588>] i915_reset_device+0x1e8/0x260 [i915]
> 
> stack backtrace:
> CPU: 3 PID: 562 Comm: kworker/3:4 Tainted: G     U          4.14.0-rc3-CI-CI_DRM_3179+ #1
> Hardware name:                  /NUC7i5BNB, BIOS BNKBL357.86A.0048.2017.0704.1415 07/04/2017
> Workqueue: events_long i915_hangcheck_elapsed [i915]
> Call Trace:
>  dump_stack+0x68/0x9f
>  print_circular_bug+0x235/0x3c0
>  ? lockdep_init_map_crosslock+0x20/0x20
>  check_prev_add+0x430/0x840
>  ? irq_work_queue+0x86/0xe0
>  ? wake_up_klogd+0x53/0x70
>  __lock_acquire+0x1420/0x15e0
>  ? __lock_acquire+0x1420/0x15e0
>  ? lockdep_init_map_crosslock+0x20/0x20
>  lock_acquire+0xb0/0x200
>  ? stop_machine+0x1c/0x40
>  ? i915_gem_object_truncate+0x50/0x50 [i915]
>  cpus_read_lock+0x3d/0xb0
>  ? stop_machine+0x1c/0x40
>  stop_machine+0x1c/0x40
>  i915_gem_set_wedged+0x1a/0x20 [i915]
>  i915_reset+0xb9/0x230 [i915]
>  i915_reset_device+0x1f6/0x260 [i915]
>  ? gen8_gt_irq_ack+0x170/0x170 [i915]
>  ? work_on_cpu_safe+0x60/0x60
>  i915_handle_error+0x2d8/0x430 [i915]
>  ? vsnprintf+0xd1/0x4b0
>  ? scnprintf+0x3a/0x70
>  hangcheck_declare_hang+0xd3/0xf0 [i915]
>  ? intel_runtime_pm_put+0x56/0xa0 [i915]
>  i915_hangcheck_elapsed+0x262/0x2d0 [i915]
>  process_one_work+0x233/0x660
>  worker_thread+0x4e/0x3b0
>  kthread+0x152/0x190
>  ? process_one_work+0x660/0x660
>  ? kthread_create_on_node+0x40/0x40
>  ret_from_fork+0x27/0x40
> Setting dangerous option reset - tainting kernel
> i915 0000:00:02.0: Resetting chip after gpu hang
> Setting dangerous option reset - tainting kernel
> i915 0000:00:02.0: Resetting chip after gpu hang
> 
> v2: Have 1 global synchronize_rcu() barrier across all engines, and
> improve commit message.
> 
> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=102886
> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103096
> Cc: Chris Wilson <chris@chris-wilson.co.uk>
> Cc: Mika Kuoppala <mika.kuoppala@intel.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Marta Lofstedt <marta.lofstedt@intel.com>
> Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
> ---
>  drivers/gpu/drm/i915/i915_gem.c                   | 31 +++++++++--------------
>  drivers/gpu/drm/i915/i915_gem_request.c           |  2 ++
>  drivers/gpu/drm/i915/selftests/i915_gem_request.c |  2 ++
>  3 files changed, 16 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
> index ab8c6946fea4..e79a6ca60265 100644
> --- a/drivers/gpu/drm/i915/i915_gem.c
> +++ b/drivers/gpu/drm/i915/i915_gem.c
> @@ -3020,16 +3020,8 @@ static void nop_submit_request(struct drm_i915_gem_request *request)
>         intel_engine_init_global_seqno(request->engine, request->global_seqno);
>  }
>  
> -static void engine_set_wedged(struct intel_engine_cs *engine)
> +static void engine_complete_requests(struct intel_engine_cs *engine)
>  {
> -       /* We need to be sure that no thread is running the old callback as
> -        * we install the nop handler (otherwise we would submit a request
> -        * to hardware that will never complete). In order to prevent this
> -        * race, we wait until the machine is idle before making the swap
> -        * (using stop_machine()).
> -        */
> -       engine->submit_request = nop_submit_request;
> -
>         /* Mark all executing requests as skipped */
>         engine->cancel_requests(engine);

How are we planning to serialise the intel_engine_init_global_seqno()
here with the in-flight nop_submit? With sufficient thrust we will get a
stale breadcrumb and an incomplete request.
-Chris

From mboxrd@z Thu Jan  1 00:00:00 1970
From: Chris Wilson <chris@chris-wilson.co.uk>
Subject: Re: [PATCH 2/2] drm/i915: Use rcu instead of
	stop_machine in set_wedged
Date: Fri, 06 Oct 2017 12:03:49 +0100
Message-ID: <150728782908.14660.8438180235890645166@mail.alporthouse.com>
References: <20171006090637.25545-1-daniel.vetter@ffwll.ch>
 <20171006090637.25545-2-daniel.vetter@ffwll.ch>
Mime-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Return-path: <intel-gfx-bounces@lists.freedesktop.org>
Received: from fireflyinternet.com (mail.fireflyinternet.com [109.228.58.192])
 by gabe.freedesktop.org (Postfix) with ESMTPS id A1BE56E927
 for <intel-gfx@lists.freedesktop.org>; Fri,  6 Oct 2017 11:04:13 +0000 (UTC)
In-Reply-To: <20171006090637.25545-2-daniel.vetter@ffwll.ch>
List-Unsubscribe: <https://lists.freedesktop.org/mailman/options/intel-gfx>,
 <mailto:intel-gfx-request@lists.freedesktop.org?subject=unsubscribe>
List-Archive: <https://lists.freedesktop.org/archives/intel-gfx>
List-Post: <mailto:intel-gfx@lists.freedesktop.org>
List-Help: <mailto:intel-gfx-request@lists.freedesktop.org?subject=help>
List-Subscribe: <https://lists.freedesktop.org/mailman/listinfo/intel-gfx>,
 <mailto:intel-gfx-request@lists.freedesktop.org?subject=subscribe>
Errors-To: intel-gfx-bounces@lists.freedesktop.org
Sender: "Intel-gfx" <intel-gfx-bounces@lists.freedesktop.org>
To: Intel Graphics Development <intel-gfx@lists.freedesktop.org>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>, LKML <linux-kernel@vger.kernel.org>, Daniel Vetter <daniel.vetter@intel.com>, Thomas Gleixner <tglx@linutronix.de>, Mika Kuoppala <mika.kuoppala@intel.com>
List-Id: intel-gfx@lists.freedesktop.org

UXVvdGluZyBEYW5pZWwgVmV0dGVyICgyMDE3LTEwLTA2IDEwOjA2OjM3KQo+IHN0b3BfbWFjaGlu
ZSBpcyBub3QgcmVhbGx5IGEgbG9ja2luZyBwcmltaXRpdmUgd2Ugc2hvdWxkIHVzZSwgZXhjZXB0
Cj4gd2hlbiB0aGUgaHcgZm9sa3MgdGVsbCB1cyB0aGUgaHcgaXMgYnJva2VuIGFuZCB0aGF0J3Mg
dGhlIG9ubHkgd2F5IHRvCj4gd29yayBhcm91bmQgaXQuCj4gCj4gVGhpcyBwYXRjaCB0cmllcyB0
byBhZGRyZXNzIHRoZSBsb2NraW5nIGFidXNlIG9mIHN0b3BfbWFjaGluZSgpIGZyb20KPiAKPiBj
b21taXQgMjBlNDkzM2M0NzhhMWNhNjk0YjM4ZmE0YWM0NGQ5OWU2NTk5NDFmNQo+IEF1dGhvcjog
Q2hyaXMgV2lsc29uIDxjaHJpc0BjaHJpcy13aWxzb24uY28udWs+Cj4gRGF0ZTogICBUdWUgTm92
IDIyIDE0OjQxOjIxIDIwMTYgKzAwMDAKPiAKPiAgICAgZHJtL2k5MTU6IFN0b3AgdGhlIG1hY2hp
bmUgYXMgd2UgaW5zdGFsbCB0aGUgd2VkZ2VkIHN1Ym1pdF9yZXF1ZXN0IGhhbmRsZXIKPiAKPiBD
aHJpcyBzYWlkIHBhcnRzIG9mIHRoZSByZWFzb25zIGZvciBnb2luZyB3aXRoIHN0b3BfbWFjaGlu
ZSgpIHdhcyB0aGF0Cj4gaXQncyBubyBvdmVyaGVhZCBmb3IgdGhlIGZhc3QtcGF0aC4gQnV0IHRo
ZXNlIGNhbGxiYWNrcyB1c2UgaXJxc2F2ZQo+IHNwaW5sb2NrcyBhbmQgZG8gYSBidW5jaCBvZiBN
TUlPLCBhbmQgcmN1X3JlYWRfbG9jayBpcyBfcmVhbF8gZmFzdC4KPiAKPiBUbyBzdGF5IGFzIGNs
b3NlIGFzIHBvc3NpYmxlIHRvIHRoZSBzdG9wX21hY2hpbmUgc2VtYW50aWNzIHdlIGZpcnN0Cj4g
dXBkYXRlIGFsbCB0aGUgc3VibWl0IGZ1bmN0aW9uIHBvaW50ZXJzIHRvIHRoZSBub3AgaGFuZGxl
ciwgdGhlbiBjYWxsCj4gc3luY2hyb25pemVfcmN1KCkgdG8gbWFrZSBzdXJlIG5vIG5ldyByZXF1
ZXN0cyBjYW4gYmUgc3VibWl0dGVkLiBUaGlzCj4gc2hvdWxkIGdpdmUgdXMgZXhhY3RseSB0aGUg
aHVnZSBiYXJyaWVyIHdlIHdhbnQuCj4gCj4gSSBwb25kZXJlZCB3aGV0aGVyIHdlIHNob3VsZCBh
bm5vdGF0ZSBlbmdpbmUtPnN1Ym1pdF9yZXF1ZXN0IGFzIF9fcmN1Cj4gYW5kIHVzZSByY3VfYXNz
aWduX3BvaW50ZXIgYW5kIHJjdV9kZXJlZmVyZW5jZSBvbiBpdC4gQnV0IHRoZSByZWFzb24KPiBi
ZWhpbmQgdGhvc2UgaXMgdG8gbWFrZSBzdXJlIHRoZSBjb21waWxlci9jcHUgYmFycmllcnMgYXJl
IHRoZXJlIGZvcgo+IHdoZW4geW91IGhhdmUgYW4gYWN0dWFsIGRhdGEgc3RydWN0dXJlIHlvdSBw
b2ludCBhdCwgdG8gbWFrZSBzdXJlIGFsbAo+IHRoZSB3cml0ZXMgYXJlIHNlZW4gY29ycmVjdGx5
IG9uIHRoZSByZWFkIHNpZGUuIEJ1dCB3ZSBqdXN0IGhhdmUgYQo+IGZ1bmN0aW9uIHBvaW50ZXIs
IGFuZCAudGV4dCBpc24ndCBjaGFuZ2VkLCBzbyBubyBuZWVkIGZvciB0aGVzZQo+IGJhcnJpZXJz
IGFuZCBoZW5jZSBubyBuZWVkIGZvciBhbm5vdGF0aW9ucy4KPiAKPiBUaGlzIHNob3VsZCBmaXgg
dGhlIGZvbGxvd3dpbmcgbG9ja2RlcCBzcGxhdDoKPiAKPiA9PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KPiBXQVJOSU5HOiBwb3NzaWJsZSBjaXJj
dWxhciBsb2NraW5nIGRlcGVuZGVuY3kgZGV0ZWN0ZWQKPiA0LjE0LjAtcmMzLUNJLUNJX0RSTV8z
MTc5KyAjMSBUYWludGVkOiBHICAgICBVCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCj4ga3dvcmtlci8zOjQvNTYyIGlzIHRyeWluZyB0byBh
Y3F1aXJlIGxvY2s6Cj4gIChjcHVfaG90cGx1Z19sb2NrLnJ3X3NlbSl7KysrK30sIGF0OiBbPGZm
ZmZmZmZmODExM2Q0YmM+XSBzdG9wX21hY2hpbmUrMHgxYy8weDQwCj4gCj4gYnV0IHRhc2sgaXMg
YWxyZWFkeSBob2xkaW5nIGxvY2s6Cj4gICgmZGV2LT5zdHJ1Y3RfbXV0ZXgpeysuKy59LCBhdDog
WzxmZmZmZmZmZmEwMTM2NTg4Pl0gaTkxNV9yZXNldF9kZXZpY2UrMHgxZTgvMHgyNjAgW2k5MTVd
Cj4gCj4gd2hpY2ggbG9jayBhbHJlYWR5IGRlcGVuZHMgb24gdGhlIG5ldyBsb2NrLgo+IAo+IHRo
ZSBleGlzdGluZyBkZXBlbmRlbmN5IGNoYWluIChpbiByZXZlcnNlIG9yZGVyKSBpczoKPiAKPiAt
PiAjNiAoJmRldi0+c3RydWN0X211dGV4KXsrLisufToKPiAgICAgICAgX19sb2NrX2FjcXVpcmUr
MHgxNDIwLzB4MTVlMAo+ICAgICAgICBsb2NrX2FjcXVpcmUrMHhiMC8weDIwMAo+ICAgICAgICBf
X211dGV4X2xvY2srMHg4Ni8weDliMAo+ICAgICAgICBtdXRleF9sb2NrX2ludGVycnVwdGlibGVf
bmVzdGVkKzB4MWIvMHgyMAo+ICAgICAgICBpOTE1X211dGV4X2xvY2tfaW50ZXJydXB0aWJsZSsw
eDUxLzB4MTMwIFtpOTE1XQo+ICAgICAgICBpOTE1X2dlbV9mYXVsdCsweDIwOS8weDY1MCBbaTkx
NV0KPiAgICAgICAgX19kb19mYXVsdCsweDFlLzB4ODAKPiAgICAgICAgX19oYW5kbGVfbW1fZmF1
bHQrMHhhMDgvMHhlZDAKPiAgICAgICAgaGFuZGxlX21tX2ZhdWx0KzB4MTU2LzB4MzAwCj4gICAg
ICAgIF9fZG9fcGFnZV9mYXVsdCsweDJjNS8weDU3MAo+ICAgICAgICBkb19wYWdlX2ZhdWx0KzB4
MjgvMHgyNTAKPiAgICAgICAgcGFnZV9mYXVsdCsweDIyLzB4MzAKPiAKPiAtPiAjNSAoJm1tLT5t
bWFwX3NlbSl7KysrK306Cj4gICAgICAgIF9fbG9ja19hY3F1aXJlKzB4MTQyMC8weDE1ZTAKPiAg
ICAgICAgbG9ja19hY3F1aXJlKzB4YjAvMHgyMDAKPiAgICAgICAgX19taWdodF9mYXVsdCsweDY4
LzB4OTAKPiAgICAgICAgX2NvcHlfdG9fdXNlcisweDIzLzB4NzAKPiAgICAgICAgZmlsbGRpcisw
eGE1LzB4MTIwCj4gICAgICAgIGRjYWNoZV9yZWFkZGlyKzB4ZjkvMHgxNzAKPiAgICAgICAgaXRl
cmF0ZV9kaXIrMHg2OS8weDFhMAo+ICAgICAgICBTeVNfZ2V0ZGVudHMrMHhhNS8weDE0MAo+ICAg
ICAgICBlbnRyeV9TWVNDQUxMXzY0X2Zhc3RwYXRoKzB4MWMvMHhiMQo+IAo+IC0+ICM0ICgmc2It
PnNfdHlwZS0+aV9tdXRleF9rZXkjNSl7KysrK306Cj4gICAgICAgIGRvd25fd3JpdGUrMHgzYi8w
eDcwCj4gICAgICAgIGhhbmRsZV9jcmVhdGUrMHhjYi8weDFlMAo+ICAgICAgICBkZXZ0bXBmc2Qr
MHgxMzkvMHgxODAKPiAgICAgICAga3RocmVhZCsweDE1Mi8weDE5MAo+ICAgICAgICByZXRfZnJv
bV9mb3JrKzB4MjcvMHg0MAo+IAo+IC0+ICMzICgoY29tcGxldGUpJnJlcS5kb25lKXsrLisufToK
PiAgICAgICAgX19sb2NrX2FjcXVpcmUrMHgxNDIwLzB4MTVlMAo+ICAgICAgICBsb2NrX2FjcXVp
cmUrMHhiMC8weDIwMAo+ICAgICAgICB3YWl0X2Zvcl9jb21tb24rMHg1OC8weDIxMAo+ICAgICAg
ICB3YWl0X2Zvcl9jb21wbGV0aW9uKzB4MWQvMHgyMAo+ICAgICAgICBkZXZ0bXBmc19jcmVhdGVf
bm9kZSsweDEzZC8weDE2MAo+ICAgICAgICBkZXZpY2VfYWRkKzB4NWViLzB4NjIwCj4gICAgICAg
IGRldmljZV9jcmVhdGVfZ3JvdXBzX3ZhcmdzKzB4ZTAvMHhmMAo+ICAgICAgICBkZXZpY2VfY3Jl
YXRlKzB4M2EvMHg0MAo+ICAgICAgICBtc3JfZGV2aWNlX2NyZWF0ZSsweDJiLzB4NDAKPiAgICAg
ICAgY3B1aHBfaW52b2tlX2NhbGxiYWNrKzB4YzkvMHhiZjAKPiAgICAgICAgY3B1aHBfdGhyZWFk
X2Z1bisweDE3Yi8weDI0MAo+ICAgICAgICBzbXBib290X3RocmVhZF9mbisweDE4YS8weDI4MAo+
ICAgICAgICBrdGhyZWFkKzB4MTUyLzB4MTkwCj4gICAgICAgIHJldF9mcm9tX2ZvcmsrMHgyNy8w
eDQwCj4gCj4gLT4gIzIgKGNwdWhwX3N0YXRlLXVwKXsrLisufToKPiAgICAgICAgX19sb2NrX2Fj
cXVpcmUrMHgxNDIwLzB4MTVlMAo+ICAgICAgICBsb2NrX2FjcXVpcmUrMHhiMC8weDIwMAo+ICAg
ICAgICBjcHVocF9pc3N1ZV9jYWxsKzB4MTMzLzB4MWMwCj4gICAgICAgIF9fY3B1aHBfc2V0dXBf
c3RhdGVfY3B1c2xvY2tlZCsweDEzOS8weDJhMAo+ICAgICAgICBfX2NwdWhwX3NldHVwX3N0YXRl
KzB4NDYvMHg2MAo+ICAgICAgICBwYWdlX3dyaXRlYmFja19pbml0KzB4NDMvMHg2Nwo+ICAgICAg
ICBwYWdlY2FjaGVfaW5pdCsweDNkLzB4NDIKPiAgICAgICAgc3RhcnRfa2VybmVsKzB4M2E4LzB4
M2ZjCj4gICAgICAgIHg4Nl82NF9zdGFydF9yZXNlcnZhdGlvbnMrMHgyYS8weDJjCj4gICAgICAg
IHg4Nl82NF9zdGFydF9rZXJuZWwrMHg2ZC8weDcwCj4gICAgICAgIHZlcmlmeV9jcHUrMHgwLzB4
ZmIKPiAKPiAtPiAjMSAoY3B1aHBfc3RhdGVfbXV0ZXgpeysuKy59Ogo+ICAgICAgICBfX2xvY2tf
YWNxdWlyZSsweDE0MjAvMHgxNWUwCj4gICAgICAgIGxvY2tfYWNxdWlyZSsweGIwLzB4MjAwCj4g
ICAgICAgIF9fbXV0ZXhfbG9jaysweDg2LzB4OWIwCj4gICAgICAgIG11dGV4X2xvY2tfbmVzdGVk
KzB4MWIvMHgyMAo+ICAgICAgICBfX2NwdWhwX3NldHVwX3N0YXRlX2NwdXNsb2NrZWQrMHg1My8w
eDJhMAo+ICAgICAgICBfX2NwdWhwX3NldHVwX3N0YXRlKzB4NDYvMHg2MAo+ICAgICAgICBwYWdl
X2FsbG9jX2luaXQrMHgyOC8weDMwCj4gICAgICAgIHN0YXJ0X2tlcm5lbCsweDE0NS8weDNmYwo+
ICAgICAgICB4ODZfNjRfc3RhcnRfcmVzZXJ2YXRpb25zKzB4MmEvMHgyYwo+ICAgICAgICB4ODZf
NjRfc3RhcnRfa2VybmVsKzB4NmQvMHg3MAo+ICAgICAgICB2ZXJpZnlfY3B1KzB4MC8weGZiCj4g
Cj4gLT4gIzAgKGNwdV9ob3RwbHVnX2xvY2sucndfc2VtKXsrKysrfToKPiAgICAgICAgY2hlY2tf
cHJldl9hZGQrMHg0MzAvMHg4NDAKPiAgICAgICAgX19sb2NrX2FjcXVpcmUrMHgxNDIwLzB4MTVl
MAo+ICAgICAgICBsb2NrX2FjcXVpcmUrMHhiMC8weDIwMAo+ICAgICAgICBjcHVzX3JlYWRfbG9j
aysweDNkLzB4YjAKPiAgICAgICAgc3RvcF9tYWNoaW5lKzB4MWMvMHg0MAo+ICAgICAgICBpOTE1
X2dlbV9zZXRfd2VkZ2VkKzB4MWEvMHgyMCBbaTkxNV0KPiAgICAgICAgaTkxNV9yZXNldCsweGI5
LzB4MjMwIFtpOTE1XQo+ICAgICAgICBpOTE1X3Jlc2V0X2RldmljZSsweDFmNi8weDI2MCBbaTkx
NV0KPiAgICAgICAgaTkxNV9oYW5kbGVfZXJyb3IrMHgyZDgvMHg0MzAgW2k5MTVdCj4gICAgICAg
IGhhbmdjaGVja19kZWNsYXJlX2hhbmcrMHhkMy8weGYwIFtpOTE1XQo+ICAgICAgICBpOTE1X2hh
bmdjaGVja19lbGFwc2VkKzB4MjYyLzB4MmQwIFtpOTE1XQo+ICAgICAgICBwcm9jZXNzX29uZV93
b3JrKzB4MjMzLzB4NjYwCj4gICAgICAgIHdvcmtlcl90aHJlYWQrMHg0ZS8weDNiMAo+ICAgICAg
ICBrdGhyZWFkKzB4MTUyLzB4MTkwCj4gICAgICAgIHJldF9mcm9tX2ZvcmsrMHgyNy8weDQwCj4g
Cj4gb3RoZXIgaW5mbyB0aGF0IG1pZ2h0IGhlbHAgdXMgZGVidWcgdGhpczoKPiAKPiBDaGFpbiBl
eGlzdHMgb2Y6Cj4gICBjcHVfaG90cGx1Z19sb2NrLnJ3X3NlbSAtLT4gJm1tLT5tbWFwX3NlbSAt
LT4gJmRldi0+c3RydWN0X211dGV4Cj4gCj4gIFBvc3NpYmxlIHVuc2FmZSBsb2NraW5nIHNjZW5h
cmlvOgo+IAo+ICAgICAgICBDUFUwICAgICAgICAgICAgICAgICAgICBDUFUxCj4gICAgICAgIC0t
LS0gICAgICAgICAgICAgICAgICAgIC0tLS0KPiAgIGxvY2soJmRldi0+c3RydWN0X211dGV4KTsK
PiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG9jaygmbW0tPm1tYXBfc2VtKTsKPiAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbG9jaygmZGV2LT5zdHJ1Y3RfbXV0ZXgpOwo+
ICAgbG9jayhjcHVfaG90cGx1Z19sb2NrLnJ3X3NlbSk7Cj4gCj4gICoqKiBERUFETE9DSyAqKioK
PiAKPiAzIGxvY2tzIGhlbGQgYnkga3dvcmtlci8zOjQvNTYyOgo+ICAjMDogICgiZXZlbnRzX2xv
bmciKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA5YzY0YT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgx
YWEvMHg2NjAKPiAgIzE6ICAoKCYoJmk5MTUtPmdwdV9lcnJvci5oYW5nY2hlY2tfd29yayktPndv
cmspKXsrLisufSwgYXQ6IFs8ZmZmZmZmZmY4MTA5YzY0YT5dIHByb2Nlc3Nfb25lX3dvcmsrMHgx
YWEvMHg2NjAKPiAgIzI6ICAoJmRldi0+c3RydWN0X211dGV4KXsrLisufSwgYXQ6IFs8ZmZmZmZm
ZmZhMDEzNjU4OD5dIGk5MTVfcmVzZXRfZGV2aWNlKzB4MWU4LzB4MjYwIFtpOTE1XQo+IAo+IHN0
YWNrIGJhY2t0cmFjZToKPiBDUFU6IDMgUElEOiA1NjIgQ29tbToga3dvcmtlci8zOjQgVGFpbnRl
ZDogRyAgICAgVSAgICAgICAgICA0LjE0LjAtcmMzLUNJLUNJX0RSTV8zMTc5KyAjMQo+IEhhcmR3
YXJlIG5hbWU6ICAgICAgICAgICAgICAgICAgL05VQzdpNUJOQiwgQklPUyBCTktCTDM1Ny44NkEu
MDA0OC4yMDE3LjA3MDQuMTQxNSAwNy8wNC8yMDE3Cj4gV29ya3F1ZXVlOiBldmVudHNfbG9uZyBp
OTE1X2hhbmdjaGVja19lbGFwc2VkIFtpOTE1XQo+IENhbGwgVHJhY2U6Cj4gIGR1bXBfc3RhY2sr
MHg2OC8weDlmCj4gIHByaW50X2NpcmN1bGFyX2J1ZysweDIzNS8weDNjMAo+ICA/IGxvY2tkZXBf
aW5pdF9tYXBfY3Jvc3Nsb2NrKzB4MjAvMHgyMAo+ICBjaGVja19wcmV2X2FkZCsweDQzMC8weDg0
MAo+ICA/IGlycV93b3JrX3F1ZXVlKzB4ODYvMHhlMAo+ICA/IHdha2VfdXBfa2xvZ2QrMHg1My8w
eDcwCj4gIF9fbG9ja19hY3F1aXJlKzB4MTQyMC8weDE1ZTAKPiAgPyBfX2xvY2tfYWNxdWlyZSsw
eDE0MjAvMHgxNWUwCj4gID8gbG9ja2RlcF9pbml0X21hcF9jcm9zc2xvY2srMHgyMC8weDIwCj4g
IGxvY2tfYWNxdWlyZSsweGIwLzB4MjAwCj4gID8gc3RvcF9tYWNoaW5lKzB4MWMvMHg0MAo+ICA/
IGk5MTVfZ2VtX29iamVjdF90cnVuY2F0ZSsweDUwLzB4NTAgW2k5MTVdCj4gIGNwdXNfcmVhZF9s
b2NrKzB4M2QvMHhiMAo+ICA/IHN0b3BfbWFjaGluZSsweDFjLzB4NDAKPiAgc3RvcF9tYWNoaW5l
KzB4MWMvMHg0MAo+ICBpOTE1X2dlbV9zZXRfd2VkZ2VkKzB4MWEvMHgyMCBbaTkxNV0KPiAgaTkx
NV9yZXNldCsweGI5LzB4MjMwIFtpOTE1XQo+ICBpOTE1X3Jlc2V0X2RldmljZSsweDFmNi8weDI2
MCBbaTkxNV0KPiAgPyBnZW44X2d0X2lycV9hY2srMHgxNzAvMHgxNzAgW2k5MTVdCj4gID8gd29y
a19vbl9jcHVfc2FmZSsweDYwLzB4NjAKPiAgaTkxNV9oYW5kbGVfZXJyb3IrMHgyZDgvMHg0MzAg
W2k5MTVdCj4gID8gdnNucHJpbnRmKzB4ZDEvMHg0YjAKPiAgPyBzY25wcmludGYrMHgzYS8weDcw
Cj4gIGhhbmdjaGVja19kZWNsYXJlX2hhbmcrMHhkMy8weGYwIFtpOTE1XQo+ICA/IGludGVsX3J1
bnRpbWVfcG1fcHV0KzB4NTYvMHhhMCBbaTkxNV0KPiAgaTkxNV9oYW5nY2hlY2tfZWxhcHNlZCsw
eDI2Mi8weDJkMCBbaTkxNV0KPiAgcHJvY2Vzc19vbmVfd29yaysweDIzMy8weDY2MAo+ICB3b3Jr
ZXJfdGhyZWFkKzB4NGUvMHgzYjAKPiAga3RocmVhZCsweDE1Mi8weDE5MAo+ICA/IHByb2Nlc3Nf
b25lX3dvcmsrMHg2NjAvMHg2NjAKPiAgPyBrdGhyZWFkX2NyZWF0ZV9vbl9ub2RlKzB4NDAvMHg0
MAo+ICByZXRfZnJvbV9mb3JrKzB4MjcvMHg0MAo+IFNldHRpbmcgZGFuZ2Vyb3VzIG9wdGlvbiBy
ZXNldCAtIHRhaW50aW5nIGtlcm5lbAo+IGk5MTUgMDAwMDowMDowMi4wOiBSZXNldHRpbmcgY2hp
cCBhZnRlciBncHUgaGFuZwo+IFNldHRpbmcgZGFuZ2Vyb3VzIG9wdGlvbiByZXNldCAtIHRhaW50
aW5nIGtlcm5lbAo+IGk5MTUgMDAwMDowMDowMi4wOiBSZXNldHRpbmcgY2hpcCBhZnRlciBncHUg
aGFuZwo+IAo+IHYyOiBIYXZlIDEgZ2xvYmFsIHN5bmNocm9uaXplX3JjdSgpIGJhcnJpZXIgYWNy
b3NzIGFsbCBlbmdpbmVzLCBhbmQKPiBpbXByb3ZlIGNvbW1pdCBtZXNzYWdlLgo+IAo+IEJ1Z3pp
bGxhOiBodHRwczovL2J1Z3MuZnJlZWRlc2t0b3Aub3JnL3Nob3dfYnVnLmNnaT9pZD0xMDI4ODYK
PiBCdWd6aWxsYTogaHR0cHM6Ly9idWdzLmZyZWVkZXNrdG9wLm9yZy9zaG93X2J1Zy5jZ2k/aWQ9
MTAzMDk2Cj4gQ2M6IENocmlzIFdpbHNvbiA8Y2hyaXNAY2hyaXMtd2lsc29uLmNvLnVrPgo+IENj
OiBNaWthIEt1b3BwYWxhIDxtaWthLmt1b3BwYWxhQGludGVsLmNvbT4KPiBDYzogVGhvbWFzIEds
ZWl4bmVyIDx0Z2x4QGxpbnV0cm9uaXguZGU+Cj4gQ2M6IE1hcnRhIExvZnN0ZWR0IDxtYXJ0YS5s
b2ZzdGVkdEBpbnRlbC5jb20+Cj4gU2lnbmVkLW9mZi1ieTogRGFuaWVsIFZldHRlciA8ZGFuaWVs
LnZldHRlckBpbnRlbC5jb20+Cj4gLS0tCj4gIGRyaXZlcnMvZ3B1L2RybS9pOTE1L2k5MTVfZ2Vt
LmMgICAgICAgICAgICAgICAgICAgfCAzMSArKysrKysrKystLS0tLS0tLS0tLS0tLQo+ICBkcml2
ZXJzL2dwdS9kcm0vaTkxNS9pOTE1X2dlbV9yZXF1ZXN0LmMgICAgICAgICAgIHwgIDIgKysKPiAg
ZHJpdmVycy9ncHUvZHJtL2k5MTUvc2VsZnRlc3RzL2k5MTVfZ2VtX3JlcXVlc3QuYyB8ICAyICsr
Cj4gIDMgZmlsZXMgY2hhbmdlZCwgMTYgaW5zZXJ0aW9ucygrKSwgMTkgZGVsZXRpb25zKC0pCj4g
Cj4gZGlmZiAtLWdpdCBhL2RyaXZlcnMvZ3B1L2RybS9pOTE1L2k5MTVfZ2VtLmMgYi9kcml2ZXJz
L2dwdS9kcm0vaTkxNS9pOTE1X2dlbS5jCj4gaW5kZXggYWI4YzY5NDZmZWE0Li5lNzlhNmNhNjAy
NjUgMTAwNjQ0Cj4gLS0tIGEvZHJpdmVycy9ncHUvZHJtL2k5MTUvaTkxNV9nZW0uYwo+ICsrKyBi
L2RyaXZlcnMvZ3B1L2RybS9pOTE1L2k5MTVfZ2VtLmMKPiBAQCAtMzAyMCwxNiArMzAyMCw4IEBA
IHN0YXRpYyB2b2lkIG5vcF9zdWJtaXRfcmVxdWVzdChzdHJ1Y3QgZHJtX2k5MTVfZ2VtX3JlcXVl
c3QgKnJlcXVlc3QpCj4gICAgICAgICBpbnRlbF9lbmdpbmVfaW5pdF9nbG9iYWxfc2Vxbm8ocmVx
dWVzdC0+ZW5naW5lLCByZXF1ZXN0LT5nbG9iYWxfc2Vxbm8pOwo+ICB9Cj4gIAo+IC1zdGF0aWMg
dm9pZCBlbmdpbmVfc2V0X3dlZGdlZChzdHJ1Y3QgaW50ZWxfZW5naW5lX2NzICplbmdpbmUpCj4g
K3N0YXRpYyB2b2lkIGVuZ2luZV9jb21wbGV0ZV9yZXF1ZXN0cyhzdHJ1Y3QgaW50ZWxfZW5naW5l
X2NzICplbmdpbmUpCj4gIHsKPiAtICAgICAgIC8qIFdlIG5lZWQgdG8gYmUgc3VyZSB0aGF0IG5v
IHRocmVhZCBpcyBydW5uaW5nIHRoZSBvbGQgY2FsbGJhY2sgYXMKPiAtICAgICAgICAqIHdlIGlu
c3RhbGwgdGhlIG5vcCBoYW5kbGVyIChvdGhlcndpc2Ugd2Ugd291bGQgc3VibWl0IGEgcmVxdWVz
dAo+IC0gICAgICAgICogdG8gaGFyZHdhcmUgdGhhdCB3aWxsIG5ldmVyIGNvbXBsZXRlKS4gSW4g
b3JkZXIgdG8gcHJldmVudCB0aGlzCj4gLSAgICAgICAgKiByYWNlLCB3ZSB3YWl0IHVudGlsIHRo
ZSBtYWNoaW5lIGlzIGlkbGUgYmVmb3JlIG1ha2luZyB0aGUgc3dhcAo+IC0gICAgICAgICogKHVz
aW5nIHN0b3BfbWFjaGluZSgpKS4KPiAtICAgICAgICAqLwo+IC0gICAgICAgZW5naW5lLT5zdWJt
aXRfcmVxdWVzdCA9IG5vcF9zdWJtaXRfcmVxdWVzdDsKPiAtCj4gICAgICAgICAvKiBNYXJrIGFs
bCBleGVjdXRpbmcgcmVxdWVzdHMgYXMgc2tpcHBlZCAqLwo+ICAgICAgICAgZW5naW5lLT5jYW5j
ZWxfcmVxdWVzdHMoZW5naW5lKTsKCkhvdyBhcmUgd2UgcGxhbm5pbmcgdG8gc2VyaWFsaXNlIHRo
ZSBpbnRlbF9lbmdpbmVfaW5pdF9nbG9iYWxfc2Vxbm8oKQpoZXJlIHdpdGggdGhlIGluLWZsaWdo
dCBub3Bfc3VibWl0PyBXaXRoIHN1ZmZpY2llbnQgdGhydXN0IHdlIHdpbGwgZ2V0IGEKc3RhbGUg
YnJlYWRjcnVtYiBhbmQgYW4gaW5jb21wbGV0ZSByZXF1ZXN0LgotQ2hyaXMKX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlz
dApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHBzOi8vbGlzdHMuZnJlZWRlc2t0
b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg==