From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95511C433E6 for ; Thu, 4 Mar 2021 15:32:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6870C64F6C for ; Thu, 4 Mar 2021 15:32:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234907AbhCDPbf (ORCPT ); Thu, 4 Mar 2021 10:31:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234006AbhCDPb0 (ORCPT ); Thu, 4 Mar 2021 10:31:26 -0500 Received: from mail-oo1-xc2d.google.com (mail-oo1-xc2d.google.com [IPv6:2607:f8b0:4864:20::c2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F568C061756 for ; Thu, 4 Mar 2021 07:30:46 -0800 (PST) Received: by mail-oo1-xc2d.google.com with SMTP id f26so6675416oog.5 for ; Thu, 04 Mar 2021 07:30:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=ZbSTgcxFwVsXX8cacvNImXqBk6nL/JKSCQkK8scwbdaJ7P4GYL5DdRitYj772F3UVb nHY0yD6KlXNqJE+L9Pi7OVrD3aQQIplqSeILnDuA84nAGcWj/AxOngi3I6P1vRiwv+hW idOi6ga0uJkAuYZMCZZgt1W49Wd8gkdaZum+jYs97edYqkMZqdpehf5Hx9yVUsmISKa1 NLIoME0vYh8Js3r4lhGvxJkgX2lazM75fPrd0jIvoYaeAqTv2SaEUimY7MOjszdisZ6o zftRIiWcuRRMDCB8Qim3QODe0HfKi/AelrXYsQEIVw7szPOcXJDLEyCsJiR+U14eGnBb IMqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=QEg2lol2IPvfRMH4qB2mwpCjJkX2TYN8dtoydvH9JD7jvY8K3fgqJCmpdr1aqMs2bn BZA2wConw1+mUuaWcniVS6fwLDBwFb1AMYNWPvp34bgNPZsAv8XNDnm2ZdaCVx31gxdk e7to8sdRPS/EVYeZaEj1DCL7quRz0dIatdA2zVFGGeXZ9AsyEDZrPPi1yzYxfNx9A8SJ PJwIRt4y3+2BkqcpvKciRd06yY+PKKkSyKQn/7djW6/2KzM1tR3meoho/G+OGqDwr4q5 KmhywXoCzszwAjbRN5PpBVDaiLFztDnxP4llMisCSLzch4w3zvOqiC93nSexmw93Pxbs LNtg== X-Gm-Message-State: AOAM530p/RhWfAJlQFZ3+EHB7/cFaDdaeTPbU1FqOzm382/KrD2PmurS u/+o45PnA+iWXYJ9DYGVwf9FjG4+koe1xZwedM+J4gwDbjY= X-Google-Smtp-Source: ABdhPJxwWfhCLZ/Ry11Sw8qKBy3AkxAR0n0dJiFMrBMUt5ATFagcLEL89wFGXjzG8+FRsArb8luNHCOKKV+KiQWPxL8= X-Received: by 2002:a4a:a105:: with SMTP id i5mr3765483ool.54.1614871845744; Thu, 04 Mar 2021 07:30:45 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> In-Reply-To: <20210304145730.GC54534@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 16:30:34 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Cc: Christophe Leroy , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , LKML , linuxppc-dev@lists.ozlabs.org, kasan-dev , Catalin Marinas , Will Deacon , Linux ARM , broonie@kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 4 Mar 2021 at 15:57, Mark Rutland wrote: > [adding Mark Brown] > > On Wed, Mar 03, 2021 at 04:20:43PM +0100, Marco Elver wrote: > > On Wed, Mar 03, 2021 at 03:52PM +0100, Christophe Leroy wrote: > > > Le 03/03/2021 =C3=AF=C2=BF=C2=BD 15:38, Marco Elver a =C3=AF=C2=BF=C2= =BDcrit=C3=AF=C2=BF=C2=BD: > > > > On Wed, 3 Mar 2021 at 15:09, Christophe Leroy > > > > wrote: > > > > > > > > > > It seems like all other sane architectures, namely x86 and arm64 > > > > > at least, include the running function as top entry when saving > > > > > stack trace. > > > > > > > > > > Functionnalities like KFENCE expect it. > > > > > > > > > > Do the same on powerpc, it allows KFENCE to properly identify the= faulting > > > > > function as depicted below. Before the patch KFENCE was identifyi= ng > > > > > finish_task_switch.isra as the faulting function. > > > > > > > > > > [ 14.937370] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > [ 14.948692] BUG: KFENCE: invalid read in test_invalid_access+0= x54/0x108 > > > > > [ 14.948692] > > > > > [ 14.956814] Invalid read at 0xdf98800a: > > > > > [ 14.960664] test_invalid_access+0x54/0x108 > > > > > [ 14.964876] finish_task_switch.isra.0+0x54/0x23c > > > > > [ 14.969606] kunit_try_run_case+0x5c/0xd0 > > > > > [ 14.973658] kunit_generic_run_threadfn_adapter+0x24/0x30 > > > > > [ 14.979079] kthread+0x15c/0x174 > > > > > [ 14.982342] ret_from_kernel_thread+0x14/0x1c > > > > > [ 14.986731] > > > > > [ 14.988236] CPU: 0 PID: 111 Comm: kunit_try_catch Tainted: G = B 5.12.0-rc1-01537-g95f6e2088d7e-dirty #4682 > > > > > [ 14.999795] NIP: c016ec2c LR: c02f517c CTR: c016ebd8 > > > > > [ 15.004851] REGS: e2449d90 TRAP: 0301 Tainted: G B = (5.12.0-rc1-01537-g95f6e2088d7e-dirty) > > > > > [ 15.015274] MSR: 00009032 CR: 22000004 XER= : 00000000 > > > > > [ 15.022043] DAR: df98800a DSISR: 20000000 > > > > > [ 15.022043] GPR00: c02f517c e2449e50 c1142080 e100dd24 c084b13= c 00000008 c084b32b c016ebd8 > > > > > [ 15.022043] GPR08: c0850000 df988000 c0d10000 e2449eb0 2200028= 8 > > > > > [ 15.040581] NIP [c016ec2c] test_invalid_access+0x54/0x108 > > > > > [ 15.046010] LR [c02f517c] kunit_try_run_case+0x5c/0xd0 > > > > > [ 15.051181] Call Trace: > > > > > [ 15.053637] [e2449e50] [c005a68c] finish_task_switch.isra.0+0x= 54/0x23c (unreliable) > > > > > [ 15.061338] [e2449eb0] [c02f517c] kunit_try_run_case+0x5c/0xd0 > > > > > [ 15.067215] [e2449ed0] [c02f648c] kunit_generic_run_threadfn_a= dapter+0x24/0x30 > > > > > [ 15.074472] [e2449ef0] [c004e7b0] kthread+0x15c/0x174 > > > > > [ 15.079571] [e2449f30] [c001317c] ret_from_kernel_thread+0x14/= 0x1c > > > > > [ 15.085798] Instruction dump: > > > > > [ 15.088784] 8129d608 38e7ebd8 81020280 911f004c 39000000 995f0= 024 907f0028 90ff001c > > > > > [ 15.096613] 3949000a 915f0020 3d40c0d1 3d00c085 <8929000a> 390= 8adb0 812a4b98 3d40c02f > > > > > [ 15.104612] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > > > Signed-off-by: Christophe Leroy > > > > > > > > Acked-by: Marco Elver > > > > > > > > Thank you, I think this looks like the right solution. Just a quest= ion below: > > > > > > > ... > > > > > > > > @@ -59,23 +70,26 @@ void save_stack_trace(struct stack_trace *tra= ce) > > > > > > > > > > sp =3D current_stack_frame(); > > > > > > > > > > - save_context_stack(trace, sp, current, 1); > > > > > + save_context_stack(trace, sp, (unsigned long)save_stack_t= race, current, 1); > > > > > > > > This causes ip =3D=3D save_stack_trace and also below for > > > > save_stack_trace_tsk. Does this mean save_stack_trace() is included= in > > > > the trace? Looking at kernel/stacktrace.c, I think the library want= s > > > > to exclude itself from the trace, as it does '.skip =3D skipnr + 1'= (and > > > > '.skip =3D skipnr + (current =3D=3D tsk)' for the _tsk variant). > > > > > > > > If the arch-helper here is included, should this use _RET_IP_ inste= ad? > > > > > > > > > > Don't really know, I was inspired by arm64 which has: > > > > > > void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cook= ie, > > > struct task_struct *task, struct pt_regs *regs) > > > { > > > struct stackframe frame; > > > > > > if (regs) > > > start_backtrace(&frame, regs->regs[29], regs->pc); > > > else if (task =3D=3D current) > > > start_backtrace(&frame, > > > (unsigned long)__builtin_frame_address(0)= , > > > (unsigned long)arch_stack_walk); > > > else > > > start_backtrace(&frame, thread_saved_fp(task), > > > thread_saved_pc(task)); > > > > > > walk_stackframe(task, &frame, consume_entry, cookie); > > > } > > > > > > But looking at x86 you may be right, so what should be done really ? > > > > x86: > > > > [ 2.843292] calling stack_trace_save: > > [ 2.843705] test_func+0x6c/0x118 > > [ 2.844184] do_one_initcall+0x58/0x270 > > [ 2.844618] kernel_init_freeable+0x1da/0x23a > > [ 2.845110] kernel_init+0xc/0x166 > > [ 2.845494] ret_from_fork+0x22/0x30 > > > > [ 2.867525] calling stack_trace_save_tsk: > > [ 2.868017] test_func+0xa9/0x118 > > [ 2.868530] do_one_initcall+0x58/0x270 > > [ 2.869003] kernel_init_freeable+0x1da/0x23a > > [ 2.869535] kernel_init+0xc/0x166 > > [ 2.869957] ret_from_fork+0x22/0x30 > > > > arm64: > > > > [ 3.786911] calling stack_trace_save: > > [ 3.787147] stack_trace_save+0x50/0x78 > > [ 3.787443] test_func+0x84/0x13c > > [ 3.787738] do_one_initcall+0x5c/0x310 > > [ 3.788099] kernel_init_freeable+0x214/0x294 > > [ 3.788363] kernel_init+0x18/0x164 > > [ 3.788585] ret_from_fork+0x10/0x30 > > > > [ 3.803615] calling stack_trace_save_tsk: > > [ 3.804266] stack_trace_save_tsk+0x9c/0x100 > > [ 3.804541] test_func+0xc4/0x13c > > [ 3.804803] do_one_initcall+0x5c/0x310 > > [ 3.805031] kernel_init_freeable+0x214/0x294 > > [ 3.805284] kernel_init+0x18/0x164 > > [ 3.805505] ret_from_fork+0x10/0x30 > > > > +Cc arm64 folks. > > > > So I think the arm64 version also has a bug, because I think a user of > > really doesn't care about the library function > > itself. And from reading kernel/stacktrace.c I think it wants to exclud= e > > itself entirely. > > > > It's a shame that isn't better documented, but I'm > > pretty sure that including the library functions in the trace is not > > useful. > > I agree this behaviour isn't desireable, and that the lack of > documentation is unfortunate. > > It looks like GCC is happy to give us the function-entry-time FP if we us= e > __builtin_frame_address(1), and assuming clang is similarly happy we can = do: > > | diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktr= ace.c > | index ad20981dfda4..5dfbf915eb7f 100644 > | --- a/arch/arm64/kernel/stacktrace.c > | +++ b/arch/arm64/kernel/stacktrace.c > | @@ -203,8 +203,8 @@ void arch_stack_walk(stack_trace_consume_fn consume= _entry, void *cookie, > | start_backtrace(&frame, regs->regs[29], regs->pc); > | else if (task =3D=3D current) > | start_backtrace(&frame, > | - (unsigned long)__builtin_frame_address(= 0), > | - (unsigned long)arch_stack_walk); > | + (unsigned long)__builtin_frame_address(= 1), > | + (unsigned long)__builtin_return_address= (0)); > | else > | start_backtrace(&frame, thread_saved_fp(task), > | thread_saved_pc(task)); > > ... such that arch_stack_walk() will try to avoid including itself in a > trace, and so the existing skipping should (w/ caveats below) skip > stack_trace_save() or stack_trace_save_tsk(). Thank you! Yes, that works. > If that works for you, I can spin that as a patch, though we'll need to > check that doesn't introduce a new fencepost error elsewhere. > > The bigger problem here is that skipping is dodgy to begin with, and > this is still liable to break in some cases. One big concern is that > (especially with LTO) we cannot guarantee the compiler will not inline > or outline functions, causing the skipp value to be too large or too > small. That's liable to happen to callers, and in theory (though > unlikely in practice), portions of arch_stack_walk() or > stack_trace_save() could get outlined too. > > Unless we can get some strong guarantees from compiler folk such that we > can guarantee a specific function acts boundary for unwinding (and > doesn't itself get split, etc), the only reliable way I can think to > solve this requires an assembly trampoline. Whatever we do is liable to > need some invasive rework. Will LTO and friends respect 'noinline'? One thing I also noticed is that tail calls would also cause the stack trace to appear somewhat incomplete (for some of my tests I've disabled tail call optimizations). Is there a way to also mark a function non-tail-callable? But I'm also not sure if with all that we'd be guaranteed the code we want, even though in practice it might. Thanks, -- Marco From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5DEBC433DB for ; Thu, 4 Mar 2021 15:31:19 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4007464F1E for ; Thu, 4 Mar 2021 15:31:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4007464F1E Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4DrvwP4k5Vz3dCZ for ; Fri, 5 Mar 2021 02:31:17 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ZbSTgcxF; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::c32; helo=mail-oo1-xc32.google.com; envelope-from=elver@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20161025 header.b=ZbSTgcxF; dkim-atps=neutral Received: from mail-oo1-xc32.google.com (mail-oo1-xc32.google.com [IPv6:2607:f8b0:4864:20::c32]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Drvvw081cz3ccF for ; Fri, 5 Mar 2021 02:30:49 +1100 (AEDT) Received: by mail-oo1-xc32.google.com with SMTP id l5so6663003ooj.7 for ; Thu, 04 Mar 2021 07:30:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=ZbSTgcxFwVsXX8cacvNImXqBk6nL/JKSCQkK8scwbdaJ7P4GYL5DdRitYj772F3UVb nHY0yD6KlXNqJE+L9Pi7OVrD3aQQIplqSeILnDuA84nAGcWj/AxOngi3I6P1vRiwv+hW idOi6ga0uJkAuYZMCZZgt1W49Wd8gkdaZum+jYs97edYqkMZqdpehf5Hx9yVUsmISKa1 NLIoME0vYh8Js3r4lhGvxJkgX2lazM75fPrd0jIvoYaeAqTv2SaEUimY7MOjszdisZ6o zftRIiWcuRRMDCB8Qim3QODe0HfKi/AelrXYsQEIVw7szPOcXJDLEyCsJiR+U14eGnBb IMqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=S0ji4SXLGlAhabYuAOC6fftnFII5uJ69LeVl1j6EiSlBJqJYdxKA6E8/IqZLrWYKLn bHD6RnletZpLRf2F92tVb86nPWC7sY47jYsYeXK7ETtEYJBuDyhdoTZepvsTtMrBy4hb /Ly3TwDe1TTQMBURt3x5libjP9H+hBWAiRDmDVr4ZuHY5a6fzKfL8sHBrBD4SMEUZ724 1IN6EPXkH5hjroXgpdXjGM4eXaCY7iSGI5wFsGwSEAN6Rnk1INmYrpf3nO0CSh6J+Rak xyybvfS3ISZO6a8czDwWkGOr33+ZtvD9Y6MFDcaVy3AiUN/vhs0BASA3POh88nmYdYZo F/OQ== X-Gm-Message-State: AOAM530sMGuZg2LZDXg7pvlxHr7hSsQuQm1GOgmGaMhztwmOeB6q2sO/ 6TIZEDTQsD4dF6ICHaoWU3hRVpnbJai8zatl59JK3A== X-Google-Smtp-Source: ABdhPJxwWfhCLZ/Ry11Sw8qKBy3AkxAR0n0dJiFMrBMUt5ATFagcLEL89wFGXjzG8+FRsArb8luNHCOKKV+KiQWPxL8= X-Received: by 2002:a4a:a105:: with SMTP id i5mr3765483ool.54.1614871845744; Thu, 04 Mar 2021 07:30:45 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> In-Reply-To: <20210304145730.GC54534@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 16:30:34 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Catalin Marinas , Will Deacon , LKML , broonie@kernel.org, Paul Mackerras , kasan-dev , linuxppc-dev@lists.ozlabs.org, Linux ARM Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, 4 Mar 2021 at 15:57, Mark Rutland wrote: > [adding Mark Brown] > > On Wed, Mar 03, 2021 at 04:20:43PM +0100, Marco Elver wrote: > > On Wed, Mar 03, 2021 at 03:52PM +0100, Christophe Leroy wrote: > > > Le 03/03/2021 =C3=AF=C2=BF=C2=BD 15:38, Marco Elver a =C3=AF=C2=BF=C2= =BDcrit=C3=AF=C2=BF=C2=BD: > > > > On Wed, 3 Mar 2021 at 15:09, Christophe Leroy > > > > wrote: > > > > > > > > > > It seems like all other sane architectures, namely x86 and arm64 > > > > > at least, include the running function as top entry when saving > > > > > stack trace. > > > > > > > > > > Functionnalities like KFENCE expect it. > > > > > > > > > > Do the same on powerpc, it allows KFENCE to properly identify the= faulting > > > > > function as depicted below. Before the patch KFENCE was identifyi= ng > > > > > finish_task_switch.isra as the faulting function. > > > > > > > > > > [ 14.937370] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > [ 14.948692] BUG: KFENCE: invalid read in test_invalid_access+0= x54/0x108 > > > > > [ 14.948692] > > > > > [ 14.956814] Invalid read at 0xdf98800a: > > > > > [ 14.960664] test_invalid_access+0x54/0x108 > > > > > [ 14.964876] finish_task_switch.isra.0+0x54/0x23c > > > > > [ 14.969606] kunit_try_run_case+0x5c/0xd0 > > > > > [ 14.973658] kunit_generic_run_threadfn_adapter+0x24/0x30 > > > > > [ 14.979079] kthread+0x15c/0x174 > > > > > [ 14.982342] ret_from_kernel_thread+0x14/0x1c > > > > > [ 14.986731] > > > > > [ 14.988236] CPU: 0 PID: 111 Comm: kunit_try_catch Tainted: G = B 5.12.0-rc1-01537-g95f6e2088d7e-dirty #4682 > > > > > [ 14.999795] NIP: c016ec2c LR: c02f517c CTR: c016ebd8 > > > > > [ 15.004851] REGS: e2449d90 TRAP: 0301 Tainted: G B = (5.12.0-rc1-01537-g95f6e2088d7e-dirty) > > > > > [ 15.015274] MSR: 00009032 CR: 22000004 XER= : 00000000 > > > > > [ 15.022043] DAR: df98800a DSISR: 20000000 > > > > > [ 15.022043] GPR00: c02f517c e2449e50 c1142080 e100dd24 c084b13= c 00000008 c084b32b c016ebd8 > > > > > [ 15.022043] GPR08: c0850000 df988000 c0d10000 e2449eb0 2200028= 8 > > > > > [ 15.040581] NIP [c016ec2c] test_invalid_access+0x54/0x108 > > > > > [ 15.046010] LR [c02f517c] kunit_try_run_case+0x5c/0xd0 > > > > > [ 15.051181] Call Trace: > > > > > [ 15.053637] [e2449e50] [c005a68c] finish_task_switch.isra.0+0x= 54/0x23c (unreliable) > > > > > [ 15.061338] [e2449eb0] [c02f517c] kunit_try_run_case+0x5c/0xd0 > > > > > [ 15.067215] [e2449ed0] [c02f648c] kunit_generic_run_threadfn_a= dapter+0x24/0x30 > > > > > [ 15.074472] [e2449ef0] [c004e7b0] kthread+0x15c/0x174 > > > > > [ 15.079571] [e2449f30] [c001317c] ret_from_kernel_thread+0x14/= 0x1c > > > > > [ 15.085798] Instruction dump: > > > > > [ 15.088784] 8129d608 38e7ebd8 81020280 911f004c 39000000 995f0= 024 907f0028 90ff001c > > > > > [ 15.096613] 3949000a 915f0020 3d40c0d1 3d00c085 <8929000a> 390= 8adb0 812a4b98 3d40c02f > > > > > [ 15.104612] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > > > > > Signed-off-by: Christophe Leroy > > > > > > > > Acked-by: Marco Elver > > > > > > > > Thank you, I think this looks like the right solution. Just a quest= ion below: > > > > > > > ... > > > > > > > > @@ -59,23 +70,26 @@ void save_stack_trace(struct stack_trace *tra= ce) > > > > > > > > > > sp =3D current_stack_frame(); > > > > > > > > > > - save_context_stack(trace, sp, current, 1); > > > > > + save_context_stack(trace, sp, (unsigned long)save_stack_t= race, current, 1); > > > > > > > > This causes ip =3D=3D save_stack_trace and also below for > > > > save_stack_trace_tsk. Does this mean save_stack_trace() is included= in > > > > the trace? Looking at kernel/stacktrace.c, I think the library want= s > > > > to exclude itself from the trace, as it does '.skip =3D skipnr + 1'= (and > > > > '.skip =3D skipnr + (current =3D=3D tsk)' for the _tsk variant). > > > > > > > > If the arch-helper here is included, should this use _RET_IP_ inste= ad? > > > > > > > > > > Don't really know, I was inspired by arm64 which has: > > > > > > void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cook= ie, > > > struct task_struct *task, struct pt_regs *regs) > > > { > > > struct stackframe frame; > > > > > > if (regs) > > > start_backtrace(&frame, regs->regs[29], regs->pc); > > > else if (task =3D=3D current) > > > start_backtrace(&frame, > > > (unsigned long)__builtin_frame_address(0)= , > > > (unsigned long)arch_stack_walk); > > > else > > > start_backtrace(&frame, thread_saved_fp(task), > > > thread_saved_pc(task)); > > > > > > walk_stackframe(task, &frame, consume_entry, cookie); > > > } > > > > > > But looking at x86 you may be right, so what should be done really ? > > > > x86: > > > > [ 2.843292] calling stack_trace_save: > > [ 2.843705] test_func+0x6c/0x118 > > [ 2.844184] do_one_initcall+0x58/0x270 > > [ 2.844618] kernel_init_freeable+0x1da/0x23a > > [ 2.845110] kernel_init+0xc/0x166 > > [ 2.845494] ret_from_fork+0x22/0x30 > > > > [ 2.867525] calling stack_trace_save_tsk: > > [ 2.868017] test_func+0xa9/0x118 > > [ 2.868530] do_one_initcall+0x58/0x270 > > [ 2.869003] kernel_init_freeable+0x1da/0x23a > > [ 2.869535] kernel_init+0xc/0x166 > > [ 2.869957] ret_from_fork+0x22/0x30 > > > > arm64: > > > > [ 3.786911] calling stack_trace_save: > > [ 3.787147] stack_trace_save+0x50/0x78 > > [ 3.787443] test_func+0x84/0x13c > > [ 3.787738] do_one_initcall+0x5c/0x310 > > [ 3.788099] kernel_init_freeable+0x214/0x294 > > [ 3.788363] kernel_init+0x18/0x164 > > [ 3.788585] ret_from_fork+0x10/0x30 > > > > [ 3.803615] calling stack_trace_save_tsk: > > [ 3.804266] stack_trace_save_tsk+0x9c/0x100 > > [ 3.804541] test_func+0xc4/0x13c > > [ 3.804803] do_one_initcall+0x5c/0x310 > > [ 3.805031] kernel_init_freeable+0x214/0x294 > > [ 3.805284] kernel_init+0x18/0x164 > > [ 3.805505] ret_from_fork+0x10/0x30 > > > > +Cc arm64 folks. > > > > So I think the arm64 version also has a bug, because I think a user of > > really doesn't care about the library function > > itself. And from reading kernel/stacktrace.c I think it wants to exclud= e > > itself entirely. > > > > It's a shame that isn't better documented, but I'm > > pretty sure that including the library functions in the trace is not > > useful. > > I agree this behaviour isn't desireable, and that the lack of > documentation is unfortunate. > > It looks like GCC is happy to give us the function-entry-time FP if we us= e > __builtin_frame_address(1), and assuming clang is similarly happy we can = do: > > | diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktr= ace.c > | index ad20981dfda4..5dfbf915eb7f 100644 > | --- a/arch/arm64/kernel/stacktrace.c > | +++ b/arch/arm64/kernel/stacktrace.c > | @@ -203,8 +203,8 @@ void arch_stack_walk(stack_trace_consume_fn consume= _entry, void *cookie, > | start_backtrace(&frame, regs->regs[29], regs->pc); > | else if (task =3D=3D current) > | start_backtrace(&frame, > | - (unsigned long)__builtin_frame_address(= 0), > | - (unsigned long)arch_stack_walk); > | + (unsigned long)__builtin_frame_address(= 1), > | + (unsigned long)__builtin_return_address= (0)); > | else > | start_backtrace(&frame, thread_saved_fp(task), > | thread_saved_pc(task)); > > ... such that arch_stack_walk() will try to avoid including itself in a > trace, and so the existing skipping should (w/ caveats below) skip > stack_trace_save() or stack_trace_save_tsk(). Thank you! Yes, that works. > If that works for you, I can spin that as a patch, though we'll need to > check that doesn't introduce a new fencepost error elsewhere. > > The bigger problem here is that skipping is dodgy to begin with, and > this is still liable to break in some cases. One big concern is that > (especially with LTO) we cannot guarantee the compiler will not inline > or outline functions, causing the skipp value to be too large or too > small. That's liable to happen to callers, and in theory (though > unlikely in practice), portions of arch_stack_walk() or > stack_trace_save() could get outlined too. > > Unless we can get some strong guarantees from compiler folk such that we > can guarantee a specific function acts boundary for unwinding (and > doesn't itself get split, etc), the only reliable way I can think to > solve this requires an assembly trampoline. Whatever we do is liable to > need some invasive rework. Will LTO and friends respect 'noinline'? One thing I also noticed is that tail calls would also cause the stack trace to appear somewhat incomplete (for some of my tests I've disabled tail call optimizations). Is there a way to also mark a function non-tail-callable? But I'm also not sure if with all that we'd be guaranteed the code we want, even though in practice it might. Thanks, -- Marco From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB515C433E0 for ; Thu, 4 Mar 2021 15:32:59 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1B85664EEE for ; Thu, 4 Mar 2021 15:32:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B85664EEE Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From:In-Reply-To: References:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=b2dXbBFqhOB+kXSjAFRQvS/eMI2aA8iv9yZFAugW9D8=; b=QEfDI859iRNJA4Al9IlWQOVOP M57lWXcCEl8K0K+Vh/zPo6usrjw5+xea/hZesJ/C4uDoz4j5V9u2z2MP505fVf3Bhyw+yy6PWy/ic vrFYUySC8wW7R531y4HJ0AM7a6G913QHVCuINb6LGNvT94PfkrMM/B76H+fA2bSAqZo6Gm1zIr2yl pmIPu47FQKRc6SkXYYTq31ha6h8Cua+e5UvkeUvvUheh23eImJDNJSw5EWR1FWT6HlVjwjFNXeoHa waCgQ8hCnHLFeXEUlwilTFwwoaXKB65Lo9lyz2ZUEfwvl5N25LjDqrL0oluOA4eQWbwleVBHnczit kZl5W+EbA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lHpwO-0093WZ-Gb; Thu, 04 Mar 2021 15:31:04 +0000 Received: from mail-oo1-xc30.google.com ([2607:f8b0:4864:20::c30]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lHpwA-0093Th-Tt for linux-arm-kernel@lists.infradead.org; Thu, 04 Mar 2021 15:30:53 +0000 Received: by mail-oo1-xc30.google.com with SMTP id s23so6656748oot.12 for ; Thu, 04 Mar 2021 07:30:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=ZbSTgcxFwVsXX8cacvNImXqBk6nL/JKSCQkK8scwbdaJ7P4GYL5DdRitYj772F3UVb nHY0yD6KlXNqJE+L9Pi7OVrD3aQQIplqSeILnDuA84nAGcWj/AxOngi3I6P1vRiwv+hW idOi6ga0uJkAuYZMCZZgt1W49Wd8gkdaZum+jYs97edYqkMZqdpehf5Hx9yVUsmISKa1 NLIoME0vYh8Js3r4lhGvxJkgX2lazM75fPrd0jIvoYaeAqTv2SaEUimY7MOjszdisZ6o zftRIiWcuRRMDCB8Qim3QODe0HfKi/AelrXYsQEIVw7szPOcXJDLEyCsJiR+U14eGnBb IMqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YsbrBjQ4ejiQFiBXd0K2I2t5lmeYk0KLBpHXPT2nL3g=; b=HnBT5aZCyWBmyOpnldBhYstBv8oTFnEJar65oXJfSssXFse+iLA+bd5A35TIQXwlpC NKnl4DIaFwMoleUe7Uz1NstqFicpdVjJ49JHBnYY2Gm7xUiFAWB8sGD/K9AM//M5ZtDH rXWmuoXwWw+CQaTTPFY5I8jJR2RDowV1nS2sHJ5xB39p0D1hWrUt081q+aTMsyOrmY0r PMpUI6mbhJH2EJsttJQ/XDY0KM39uVbaKBVMXmj24+lXul+mcIaqIS1Hrf08hDPMFKot NZ8ccqeTBtUYtdXn6j1wMMHJssMRVIFUL1/g7eoat37P/khfpIaB+QtF1LhWBTYzm2pw lnCw== X-Gm-Message-State: AOAM532FPAGUUNu9h+jXzvxswSAdSdxRrHhL9so50uhWmNHvAJUXUrML 6eDyAunnZ0ifvnT5RYGIHQuUcpm39DSM9DvznaTEmA== X-Google-Smtp-Source: ABdhPJxwWfhCLZ/Ry11Sw8qKBy3AkxAR0n0dJiFMrBMUt5ATFagcLEL89wFGXjzG8+FRsArb8luNHCOKKV+KiQWPxL8= X-Received: by 2002:a4a:a105:: with SMTP id i5mr3765483ool.54.1614871845744; Thu, 04 Mar 2021 07:30:45 -0800 (PST) MIME-Version: 1.0 References: <1802be3e-dc1a-52e0-1754-a40f0ea39658@csgroup.eu> <20210304145730.GC54534@C02TD0UTHF1T.local> In-Reply-To: <20210304145730.GC54534@C02TD0UTHF1T.local> From: Marco Elver Date: Thu, 4 Mar 2021 16:30:34 +0100 Message-ID: Subject: Re: [PATCH v1] powerpc: Include running function as first entry in save_stack_trace() and friends To: Mark Rutland Cc: Christophe Leroy , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , LKML , linuxppc-dev@lists.ozlabs.org, kasan-dev , Catalin Marinas , Will Deacon , Linux ARM , broonie@kernel.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1LCA0IE1hciAyMDIxIGF0IDE1OjU3LCBNYXJrIFJ1dGxhbmQgPG1hcmsucnV0bGFuZEBh cm0uY29tPiB3cm90ZToKPiBbYWRkaW5nIE1hcmsgQnJvd25dCj4KPiBPbiBXZWQsIE1hciAwMywg MjAyMSBhdCAwNDoyMDo0M1BNICswMTAwLCBNYXJjbyBFbHZlciB3cm90ZToKPiA+IE9uIFdlZCwg TWFyIDAzLCAyMDIxIGF0IDAzOjUyUE0gKzAxMDAsIENocmlzdG9waGUgTGVyb3kgd3JvdGU6Cj4g PiA+IExlIDAzLzAzLzIwMjEgw6/Cv8K9IDE1OjM4LCBNYXJjbyBFbHZlciBhIMOvwr/CvWNyaXTD r8K/wr06Cj4gPiA+ID4gT24gV2VkLCAzIE1hciAyMDIxIGF0IDE1OjA5LCBDaHJpc3RvcGhlIExl cm95Cj4gPiA+ID4gPGNocmlzdG9waGUubGVyb3lAY3Nncm91cC5ldT4gd3JvdGU6Cj4gPiA+ID4g Pgo+ID4gPiA+ID4gSXQgc2VlbXMgbGlrZSBhbGwgb3RoZXIgc2FuZSBhcmNoaXRlY3R1cmVzLCBu YW1lbHkgeDg2IGFuZCBhcm02NAo+ID4gPiA+ID4gYXQgbGVhc3QsIGluY2x1ZGUgdGhlIHJ1bm5p bmcgZnVuY3Rpb24gYXMgdG9wIGVudHJ5IHdoZW4gc2F2aW5nCj4gPiA+ID4gPiBzdGFjayB0cmFj ZS4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBGdW5jdGlvbm5hbGl0aWVzIGxpa2UgS0ZFTkNFIGV4cGVj dCBpdC4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBEbyB0aGUgc2FtZSBvbiBwb3dlcnBjLCBpdCBhbGxv d3MgS0ZFTkNFIHRvIHByb3Blcmx5IGlkZW50aWZ5IHRoZSBmYXVsdGluZwo+ID4gPiA+ID4gZnVu Y3Rpb24gYXMgZGVwaWN0ZWQgYmVsb3cuIEJlZm9yZSB0aGUgcGF0Y2ggS0ZFTkNFIHdhcyBpZGVu dGlmeWluZwo+ID4gPiA+ID4gZmluaXNoX3Rhc2tfc3dpdGNoLmlzcmEgYXMgdGhlIGZhdWx0aW5n IGZ1bmN0aW9uLgo+ID4gPiA+ID4KPiA+ID4gPiA+IFsgICAxNC45MzczNzBdID09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQo+ ID4gPiA+ID4gWyAgIDE0Ljk0ODY5Ml0gQlVHOiBLRkVOQ0U6IGludmFsaWQgcmVhZCBpbiB0ZXN0 X2ludmFsaWRfYWNjZXNzKzB4NTQvMHgxMDgKPiA+ID4gPiA+IFsgICAxNC45NDg2OTJdCj4gPiA+ ID4gPiBbICAgMTQuOTU2ODE0XSBJbnZhbGlkIHJlYWQgYXQgMHhkZjk4ODAwYToKPiA+ID4gPiA+ IFsgICAxNC45NjA2NjRdICB0ZXN0X2ludmFsaWRfYWNjZXNzKzB4NTQvMHgxMDgKPiA+ID4gPiA+ IFsgICAxNC45NjQ4NzZdICBmaW5pc2hfdGFza19zd2l0Y2guaXNyYS4wKzB4NTQvMHgyM2MKPiA+ ID4gPiA+IFsgICAxNC45Njk2MDZdICBrdW5pdF90cnlfcnVuX2Nhc2UrMHg1Yy8weGQwCj4gPiA+ ID4gPiBbICAgMTQuOTczNjU4XSAga3VuaXRfZ2VuZXJpY19ydW5fdGhyZWFkZm5fYWRhcHRlcisw eDI0LzB4MzAKPiA+ID4gPiA+IFsgICAxNC45NzkwNzldICBrdGhyZWFkKzB4MTVjLzB4MTc0Cj4g PiA+ID4gPiBbICAgMTQuOTgyMzQyXSAgcmV0X2Zyb21fa2VybmVsX3RocmVhZCsweDE0LzB4MWMK PiA+ID4gPiA+IFsgICAxNC45ODY3MzFdCj4gPiA+ID4gPiBbICAgMTQuOTg4MjM2XSBDUFU6IDAg UElEOiAxMTEgQ29tbToga3VuaXRfdHJ5X2NhdGNoIFRhaW50ZWQ6IEcgICAgQiAgICAgICAgICAg ICA1LjEyLjAtcmMxLTAxNTM3LWc5NWY2ZTIwODhkN2UtZGlydHkgIzQ2ODIKPiA+ID4gPiA+IFsg ICAxNC45OTk3OTVdIE5JUDogIGMwMTZlYzJjIExSOiBjMDJmNTE3YyBDVFI6IGMwMTZlYmQ4Cj4g PiA+ID4gPiBbICAgMTUuMDA0ODUxXSBSRUdTOiBlMjQ0OWQ5MCBUUkFQOiAwMzAxICAgVGFpbnRl ZDogRyAgICBCICAgICAgICAgICAgICAoNS4xMi4wLXJjMS0wMTUzNy1nOTVmNmUyMDg4ZDdlLWRp cnR5KQo+ID4gPiA+ID4gWyAgIDE1LjAxNTI3NF0gTVNSOiAgMDAwMDkwMzIgPEVFLE1FLElSLERS LFJJPiAgQ1I6IDIyMDAwMDA0ICBYRVI6IDAwMDAwMDAwCj4gPiA+ID4gPiBbICAgMTUuMDIyMDQz XSBEQVI6IGRmOTg4MDBhIERTSVNSOiAyMDAwMDAwMAo+ID4gPiA+ID4gWyAgIDE1LjAyMjA0M10g R1BSMDA6IGMwMmY1MTdjIGUyNDQ5ZTUwIGMxMTQyMDgwIGUxMDBkZDI0IGMwODRiMTNjIDAwMDAw MDA4IGMwODRiMzJiIGMwMTZlYmQ4Cj4gPiA+ID4gPiBbICAgMTUuMDIyMDQzXSBHUFIwODogYzA4 NTAwMDAgZGY5ODgwMDAgYzBkMTAwMDAgZTI0NDllYjAgMjIwMDAyODgKPiA+ID4gPiA+IFsgICAx NS4wNDA1ODFdIE5JUCBbYzAxNmVjMmNdIHRlc3RfaW52YWxpZF9hY2Nlc3MrMHg1NC8weDEwOAo+ ID4gPiA+ID4gWyAgIDE1LjA0NjAxMF0gTFIgW2MwMmY1MTdjXSBrdW5pdF90cnlfcnVuX2Nhc2Ur MHg1Yy8weGQwCj4gPiA+ID4gPiBbICAgMTUuMDUxMTgxXSBDYWxsIFRyYWNlOgo+ID4gPiA+ID4g WyAgIDE1LjA1MzYzN10gW2UyNDQ5ZTUwXSBbYzAwNWE2OGNdIGZpbmlzaF90YXNrX3N3aXRjaC5p c3JhLjArMHg1NC8weDIzYyAodW5yZWxpYWJsZSkKPiA+ID4gPiA+IFsgICAxNS4wNjEzMzhdIFtl MjQ0OWViMF0gW2MwMmY1MTdjXSBrdW5pdF90cnlfcnVuX2Nhc2UrMHg1Yy8weGQwCj4gPiA+ID4g PiBbICAgMTUuMDY3MjE1XSBbZTI0NDllZDBdIFtjMDJmNjQ4Y10ga3VuaXRfZ2VuZXJpY19ydW5f dGhyZWFkZm5fYWRhcHRlcisweDI0LzB4MzAKPiA+ID4gPiA+IFsgICAxNS4wNzQ0NzJdIFtlMjQ0 OWVmMF0gW2MwMDRlN2IwXSBrdGhyZWFkKzB4MTVjLzB4MTc0Cj4gPiA+ID4gPiBbICAgMTUuMDc5 NTcxXSBbZTI0NDlmMzBdIFtjMDAxMzE3Y10gcmV0X2Zyb21fa2VybmVsX3RocmVhZCsweDE0LzB4 MWMKPiA+ID4gPiA+IFsgICAxNS4wODU3OThdIEluc3RydWN0aW9uIGR1bXA6Cj4gPiA+ID4gPiBb ICAgMTUuMDg4Nzg0XSA4MTI5ZDYwOCAzOGU3ZWJkOCA4MTAyMDI4MCA5MTFmMDA0YyAzOTAwMDAw MCA5OTVmMDAyNCA5MDdmMDAyOCA5MGZmMDAxYwo+ID4gPiA+ID4gWyAgIDE1LjA5NjYxM10gMzk0 OTAwMGEgOTE1ZjAwMjAgM2Q0MGMwZDEgM2QwMGMwODUgPDg5MjkwMDBhPiAzOTA4YWRiMCA4MTJh NGI5OCAzZDQwYzAyZgo+ID4gPiA+ID4gWyAgIDE1LjEwNDYxMl0gPT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Cj4gPiA+ID4g Pgo+ID4gPiA+ID4gU2lnbmVkLW9mZi1ieTogQ2hyaXN0b3BoZSBMZXJveSA8Y2hyaXN0b3BoZS5s ZXJveUBjc2dyb3VwLmV1Pgo+ID4gPiA+Cj4gPiA+ID4gQWNrZWQtYnk6IE1hcmNvIEVsdmVyIDxl bHZlckBnb29nbGUuY29tPgo+ID4gPiA+Cj4gPiA+ID4gVGhhbmsgeW91LCBJIHRoaW5rIHRoaXMg bG9va3MgbGlrZSB0aGUgcmlnaHQgc29sdXRpb24uIEp1c3QgYSBxdWVzdGlvbiBiZWxvdzoKPiA+ ID4gPgo+ID4gPiAuLi4KPiA+ID4KPiA+ID4gPiA+IEBAIC01OSwyMyArNzAsMjYgQEAgdm9pZCBz YXZlX3N0YWNrX3RyYWNlKHN0cnVjdCBzdGFja190cmFjZSAqdHJhY2UpCj4gPiA+ID4gPgo+ID4g PiA+ID4gICAgICAgICAgc3AgPSBjdXJyZW50X3N0YWNrX2ZyYW1lKCk7Cj4gPiA+ID4gPgo+ID4g PiA+ID4gLSAgICAgICBzYXZlX2NvbnRleHRfc3RhY2sodHJhY2UsIHNwLCBjdXJyZW50LCAxKTsK PiA+ID4gPiA+ICsgICAgICAgc2F2ZV9jb250ZXh0X3N0YWNrKHRyYWNlLCBzcCwgKHVuc2lnbmVk IGxvbmcpc2F2ZV9zdGFja190cmFjZSwgY3VycmVudCwgMSk7Cj4gPiA+ID4KPiA+ID4gPiBUaGlz IGNhdXNlcyBpcCA9PSBzYXZlX3N0YWNrX3RyYWNlIGFuZCBhbHNvIGJlbG93IGZvcgo+ID4gPiA+ IHNhdmVfc3RhY2tfdHJhY2VfdHNrLiBEb2VzIHRoaXMgbWVhbiBzYXZlX3N0YWNrX3RyYWNlKCkg aXMgaW5jbHVkZWQgaW4KPiA+ID4gPiB0aGUgdHJhY2U/IExvb2tpbmcgYXQga2VybmVsL3N0YWNr dHJhY2UuYywgSSB0aGluayB0aGUgbGlicmFyeSB3YW50cwo+ID4gPiA+IHRvIGV4Y2x1ZGUgaXRz ZWxmIGZyb20gdGhlIHRyYWNlLCBhcyBpdCBkb2VzICcuc2tpcCA9IHNraXBuciArIDEnIChhbmQK PiA+ID4gPiAnLnNraXAgICA9IHNraXBuciArIChjdXJyZW50ID09IHRzayknIGZvciB0aGUgX3Rz ayB2YXJpYW50KS4KPiA+ID4gPgo+ID4gPiA+IElmIHRoZSBhcmNoLWhlbHBlciBoZXJlIGlzIGlu Y2x1ZGVkLCBzaG91bGQgdGhpcyB1c2UgX1JFVF9JUF8gaW5zdGVhZD8KPiA+ID4gPgo+ID4gPgo+ ID4gPiBEb24ndCByZWFsbHkga25vdywgSSB3YXMgaW5zcGlyZWQgYnkgYXJtNjQgd2hpY2ggaGFz Ogo+ID4gPgo+ID4gPiB2b2lkIGFyY2hfc3RhY2tfd2FsayhzdGFja190cmFjZV9jb25zdW1lX2Zu IGNvbnN1bWVfZW50cnksIHZvaWQgKmNvb2tpZSwKPiA+ID4gICAgICAgICAgICAgICAgICBzdHJ1 Y3QgdGFza19zdHJ1Y3QgKnRhc2ssIHN0cnVjdCBwdF9yZWdzICpyZWdzKQo+ID4gPiB7Cj4gPiA+ ICAgICBzdHJ1Y3Qgc3RhY2tmcmFtZSBmcmFtZTsKPiA+ID4KPiA+ID4gICAgIGlmIChyZWdzKQo+ ID4gPiAgICAgICAgICAgICBzdGFydF9iYWNrdHJhY2UoJmZyYW1lLCByZWdzLT5yZWdzWzI5XSwg cmVncy0+cGMpOwo+ID4gPiAgICAgZWxzZSBpZiAodGFzayA9PSBjdXJyZW50KQo+ID4gPiAgICAg ICAgICAgICBzdGFydF9iYWNrdHJhY2UoJmZyYW1lLAo+ID4gPiAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgKHVuc2lnbmVkIGxvbmcpX19idWlsdGluX2ZyYW1lX2FkZHJlc3MoMCksCj4gPiA+ ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAodW5zaWduZWQgbG9uZylhcmNoX3N0YWNrX3dh bGspOwo+ID4gPiAgICAgZWxzZQo+ID4gPiAgICAgICAgICAgICBzdGFydF9iYWNrdHJhY2UoJmZy YW1lLCB0aHJlYWRfc2F2ZWRfZnAodGFzayksCj4gPiA+ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB0aHJlYWRfc2F2ZWRfcGModGFzaykpOwo+ID4gPgo+ID4gPiAgICAgd2Fsa19zdGFja2Zy YW1lKHRhc2ssICZmcmFtZSwgY29uc3VtZV9lbnRyeSwgY29va2llKTsKPiA+ID4gfQo+ID4gPgo+ ID4gPiBCdXQgbG9va2luZyBhdCB4ODYgeW91IG1heSBiZSByaWdodCwgc28gd2hhdCBzaG91bGQg YmUgZG9uZSByZWFsbHkgPwo+ID4KPiA+IHg4NjoKPiA+Cj4gPiBbICAgIDIuODQzMjkyXSBjYWxs aW5nIHN0YWNrX3RyYWNlX3NhdmU6Cj4gPiBbICAgIDIuODQzNzA1XSAgdGVzdF9mdW5jKzB4NmMv MHgxMTgKPiA+IFsgICAgMi44NDQxODRdICBkb19vbmVfaW5pdGNhbGwrMHg1OC8weDI3MAo+ID4g WyAgICAyLjg0NDYxOF0gIGtlcm5lbF9pbml0X2ZyZWVhYmxlKzB4MWRhLzB4MjNhCj4gPiBbICAg IDIuODQ1MTEwXSAga2VybmVsX2luaXQrMHhjLzB4MTY2Cj4gPiBbICAgIDIuODQ1NDk0XSAgcmV0 X2Zyb21fZm9yaysweDIyLzB4MzAKPiA+Cj4gPiBbICAgIDIuODY3NTI1XSBjYWxsaW5nIHN0YWNr X3RyYWNlX3NhdmVfdHNrOgo+ID4gWyAgICAyLjg2ODAxN10gIHRlc3RfZnVuYysweGE5LzB4MTE4 Cj4gPiBbICAgIDIuODY4NTMwXSAgZG9fb25lX2luaXRjYWxsKzB4NTgvMHgyNzAKPiA+IFsgICAg Mi44NjkwMDNdICBrZXJuZWxfaW5pdF9mcmVlYWJsZSsweDFkYS8weDIzYQo+ID4gWyAgICAyLjg2 OTUzNV0gIGtlcm5lbF9pbml0KzB4Yy8weDE2Ngo+ID4gWyAgICAyLjg2OTk1N10gIHJldF9mcm9t X2ZvcmsrMHgyMi8weDMwCj4gPgo+ID4gYXJtNjQ6Cj4gPgo+ID4gWyAgICAzLjc4NjkxMV0gY2Fs bGluZyBzdGFja190cmFjZV9zYXZlOgo+ID4gWyAgICAzLjc4NzE0N10gIHN0YWNrX3RyYWNlX3Nh dmUrMHg1MC8weDc4Cj4gPiBbICAgIDMuNzg3NDQzXSAgdGVzdF9mdW5jKzB4ODQvMHgxM2MKPiA+ IFsgICAgMy43ODc3MzhdICBkb19vbmVfaW5pdGNhbGwrMHg1Yy8weDMxMAo+ID4gWyAgICAzLjc4 ODA5OV0gIGtlcm5lbF9pbml0X2ZyZWVhYmxlKzB4MjE0LzB4Mjk0Cj4gPiBbICAgIDMuNzg4MzYz XSAga2VybmVsX2luaXQrMHgxOC8weDE2NAo+ID4gWyAgICAzLjc4ODU4NV0gIHJldF9mcm9tX2Zv cmsrMHgxMC8weDMwCj4gPgo+ID4gWyAgICAzLjgwMzYxNV0gY2FsbGluZyBzdGFja190cmFjZV9z YXZlX3RzazoKPiA+IFsgICAgMy44MDQyNjZdICBzdGFja190cmFjZV9zYXZlX3RzaysweDljLzB4 MTAwCj4gPiBbICAgIDMuODA0NTQxXSAgdGVzdF9mdW5jKzB4YzQvMHgxM2MKPiA+IFsgICAgMy44 MDQ4MDNdICBkb19vbmVfaW5pdGNhbGwrMHg1Yy8weDMxMAo+ID4gWyAgICAzLjgwNTAzMV0gIGtl cm5lbF9pbml0X2ZyZWVhYmxlKzB4MjE0LzB4Mjk0Cj4gPiBbICAgIDMuODA1Mjg0XSAga2VybmVs X2luaXQrMHgxOC8weDE2NAo+ID4gWyAgICAzLjgwNTUwNV0gIHJldF9mcm9tX2ZvcmsrMHgxMC8w eDMwCj4gPgo+ID4gK0NjIGFybTY0IGZvbGtzLgo+ID4KPiA+IFNvIEkgdGhpbmsgdGhlIGFybTY0 IHZlcnNpb24gYWxzbyBoYXMgYSBidWcsIGJlY2F1c2UgSSB0aGluayBhIHVzZXIgb2YKPiA+IDxs aW51eC9zdGFja3RyYWNlLmg+IHJlYWxseSBkb2Vzbid0IGNhcmUgYWJvdXQgdGhlIGxpYnJhcnkg ZnVuY3Rpb24KPiA+IGl0c2VsZi4gQW5kIGZyb20gcmVhZGluZyBrZXJuZWwvc3RhY2t0cmFjZS5j IEkgdGhpbmsgaXQgd2FudHMgdG8gZXhjbHVkZQo+ID4gaXRzZWxmIGVudGlyZWx5Lgo+ID4KPiA+ IEl0J3MgYSBzaGFtZSB0aGF0IDxsaW51eC9zdGFja3RyYWNlLmg+IGlzbid0IGJldHRlciBkb2N1 bWVudGVkLCBidXQgSSdtCj4gPiBwcmV0dHkgc3VyZSB0aGF0IGluY2x1ZGluZyB0aGUgbGlicmFy eSBmdW5jdGlvbnMgaW4gdGhlIHRyYWNlIGlzIG5vdAo+ID4gdXNlZnVsLgo+Cj4gSSBhZ3JlZSB0 aGlzIGJlaGF2aW91ciBpc24ndCBkZXNpcmVhYmxlLCBhbmQgdGhhdCB0aGUgbGFjayBvZgo+IGRv Y3VtZW50YXRpb24gaXMgdW5mb3J0dW5hdGUuCj4KPiBJdCBsb29rcyBsaWtlIEdDQyBpcyBoYXBw eSB0byBnaXZlIHVzIHRoZSBmdW5jdGlvbi1lbnRyeS10aW1lIEZQIGlmIHdlIHVzZQo+IF9fYnVp bHRpbl9mcmFtZV9hZGRyZXNzKDEpLCBhbmQgYXNzdW1pbmcgY2xhbmcgaXMgc2ltaWxhcmx5IGhh cHB5IHdlIGNhbiBkbzoKPgo+IHwgZGlmZiAtLWdpdCBhL2FyY2gvYXJtNjQva2VybmVsL3N0YWNr dHJhY2UuYyBiL2FyY2gvYXJtNjQva2VybmVsL3N0YWNrdHJhY2UuYwo+IHwgaW5kZXggYWQyMDk4 MWRmZGE0Li41ZGZiZjkxNWViN2YgMTAwNjQ0Cj4gfCAtLS0gYS9hcmNoL2FybTY0L2tlcm5lbC9z dGFja3RyYWNlLmMKPiB8ICsrKyBiL2FyY2gvYXJtNjQva2VybmVsL3N0YWNrdHJhY2UuYwo+IHwg QEAgLTIwMyw4ICsyMDMsOCBAQCB2b2lkIGFyY2hfc3RhY2tfd2FsayhzdGFja190cmFjZV9jb25z dW1lX2ZuIGNvbnN1bWVfZW50cnksIHZvaWQgKmNvb2tpZSwKPiB8ICAgICAgICAgICAgICAgICBz dGFydF9iYWNrdHJhY2UoJmZyYW1lLCByZWdzLT5yZWdzWzI5XSwgcmVncy0+cGMpOwo+IHwgICAg ICAgICBlbHNlIGlmICh0YXNrID09IGN1cnJlbnQpCj4gfCAgICAgICAgICAgICAgICAgc3RhcnRf YmFja3RyYWNlKCZmcmFtZSwKPiB8IC0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKHVu c2lnbmVkIGxvbmcpX19idWlsdGluX2ZyYW1lX2FkZHJlc3MoMCksCj4gfCAtICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICh1bnNpZ25lZCBsb25nKWFyY2hfc3RhY2tfd2Fsayk7Cj4gfCAr ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICh1bnNpZ25lZCBsb25nKV9fYnVpbHRpbl9m cmFtZV9hZGRyZXNzKDEpLAo+IHwgKyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAodW5z aWduZWQgbG9uZylfX2J1aWx0aW5fcmV0dXJuX2FkZHJlc3MoMCkpOwo+IHwgICAgICAgICBlbHNl Cj4gfCAgICAgICAgICAgICAgICAgc3RhcnRfYmFja3RyYWNlKCZmcmFtZSwgdGhyZWFkX3NhdmVk X2ZwKHRhc2spLAo+IHwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0aHJlYWRfc2F2 ZWRfcGModGFzaykpOwo+Cj4gLi4uIHN1Y2ggdGhhdCBhcmNoX3N0YWNrX3dhbGsoKSB3aWxsIHRy eSB0byBhdm9pZCBpbmNsdWRpbmcgaXRzZWxmIGluIGEKPiB0cmFjZSwgYW5kIHNvIHRoZSBleGlz dGluZyBza2lwcGluZyBzaG91bGQgKHcvIGNhdmVhdHMgYmVsb3cpIHNraXAKPiBzdGFja190cmFj ZV9zYXZlKCkgb3Igc3RhY2tfdHJhY2Vfc2F2ZV90c2soKS4KClRoYW5rIHlvdSEgWWVzLCB0aGF0 IHdvcmtzLgoKPiBJZiB0aGF0IHdvcmtzIGZvciB5b3UsIEkgY2FuIHNwaW4gdGhhdCBhcyBhIHBh dGNoLCB0aG91Z2ggd2UnbGwgbmVlZCB0bwo+IGNoZWNrIHRoYXQgZG9lc24ndCBpbnRyb2R1Y2Ug YSBuZXcgZmVuY2Vwb3N0IGVycm9yIGVsc2V3aGVyZS4KPgo+IFRoZSBiaWdnZXIgcHJvYmxlbSBo ZXJlIGlzIHRoYXQgc2tpcHBpbmcgaXMgZG9kZ3kgdG8gYmVnaW4gd2l0aCwgYW5kCj4gdGhpcyBp cyBzdGlsbCBsaWFibGUgdG8gYnJlYWsgaW4gc29tZSBjYXNlcy4gT25lIGJpZyBjb25jZXJuIGlz IHRoYXQKPiAoZXNwZWNpYWxseSB3aXRoIExUTykgd2UgY2Fubm90IGd1YXJhbnRlZSB0aGUgY29t cGlsZXIgd2lsbCBub3QgaW5saW5lCj4gb3Igb3V0bGluZSBmdW5jdGlvbnMsIGNhdXNpbmcgdGhl IHNraXBwIHZhbHVlIHRvIGJlIHRvbyBsYXJnZSBvciB0b28KPiBzbWFsbC4gVGhhdCdzIGxpYWJs ZSB0byBoYXBwZW4gdG8gY2FsbGVycywgYW5kIGluIHRoZW9yeSAodGhvdWdoCj4gdW5saWtlbHkg aW4gcHJhY3RpY2UpLCBwb3J0aW9ucyBvZiBhcmNoX3N0YWNrX3dhbGsoKSBvcgo+IHN0YWNrX3Ry YWNlX3NhdmUoKSBjb3VsZCBnZXQgb3V0bGluZWQgdG9vLgo+Cj4gVW5sZXNzIHdlIGNhbiBnZXQg c29tZSBzdHJvbmcgZ3VhcmFudGVlcyBmcm9tIGNvbXBpbGVyIGZvbGsgc3VjaCB0aGF0IHdlCj4g Y2FuIGd1YXJhbnRlZSBhIHNwZWNpZmljIGZ1bmN0aW9uIGFjdHMgYm91bmRhcnkgZm9yIHVud2lu ZGluZyAoYW5kCj4gZG9lc24ndCBpdHNlbGYgZ2V0IHNwbGl0LCBldGMpLCB0aGUgb25seSByZWxp YWJsZSB3YXkgSSBjYW4gdGhpbmsgdG8KPiBzb2x2ZSB0aGlzIHJlcXVpcmVzIGFuIGFzc2VtYmx5 IHRyYW1wb2xpbmUuIFdoYXRldmVyIHdlIGRvIGlzIGxpYWJsZSB0bwo+IG5lZWQgc29tZSBpbnZh c2l2ZSByZXdvcmsuCgpXaWxsIExUTyBhbmQgZnJpZW5kcyByZXNwZWN0ICdub2lubGluZSc/IE9u ZSB0aGluZyBJIGFsc28gbm90aWNlZCBpcwp0aGF0IHRhaWwgY2FsbHMgd291bGQgYWxzbyBjYXVz ZSB0aGUgc3RhY2sgdHJhY2UgdG8gYXBwZWFyIHNvbWV3aGF0CmluY29tcGxldGUgKGZvciBzb21l IG9mIG15IHRlc3RzIEkndmUgZGlzYWJsZWQgdGFpbCBjYWxsCm9wdGltaXphdGlvbnMpLiBJcyB0 aGVyZSBhIHdheSB0byBhbHNvIG1hcmsgYSBmdW5jdGlvbgpub24tdGFpbC1jYWxsYWJsZT8gQnV0 IEknbSBhbHNvIG5vdCBzdXJlIGlmIHdpdGggYWxsIHRoYXQgd2UnZCBiZQpndWFyYW50ZWVkIHRo ZSBjb2RlIHdlIHdhbnQsIGV2ZW4gdGhvdWdoIGluIHByYWN0aWNlIGl0IG1pZ2h0LgoKVGhhbmtz LAotLSBNYXJjbwoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5p bmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8v bGludXgtYXJtLWtlcm5lbAo=