From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DABBC433E0 for ; Mon, 1 Feb 2021 23:02:24 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4740064EAA for ; Mon, 1 Feb 2021 23:02:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4740064EAA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=QPiHuAQfqLkPPMTfM1DIDJv/d0sLXmuY8TZPLz22NCg=; b=VoieyCajZDUOCDgTJbZQouJSB j9BhrUJ9x3KCriHDPpVSCg8a3UnvawikExpCkFZEgqP1Z9YWxpHNfz14wIrLv515kWnSnaWG0G77R 9Of7lo8ygdE0omYEEw91hY4td8j2Gp27BHylmvlJBamd4SDj3+PvGwIOKW9QGEu0jypMbqyzk6IGp 9gGuAJ2FnFlD9jdklEnZDaMcbWiAzUZmkajy5ZIF+W78g1ENSQS64xZ9gMobHgJj6jz4fiwVCnaoX izVwLb94DN6+f0A1xFdTTBZ/UnwBAPxCoxovf2+pgq5gHD9mcxzxo/qHDH4/xW/uWSGYmwucXKjN5 fBA6qdwSw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6iBa-0000QC-Hw; Mon, 01 Feb 2021 23:00:46 +0000 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l6iBY-0000Pl-67 for linux-arm-kernel@lists.infradead.org; Mon, 01 Feb 2021 23:00:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612220443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=zX9a/Cs2Fey5jY4Rdi7ss2uJdfIyLKTNcUYtchNiu0w=; b=Y7fvEanpd4ezrbcocNHOnSoI7L5VYPA7d2B+TOSQ5UooHS5uliMcspTgLkowKxYPuBm7te khvJoBuUTa2m7X5Kbc7ep+rvvYk+u5tX775Fpyk8QCJgWqTGZjMhygc/uWgDqJPBYIkMSI BvHrKn8F9Vznf+aKxIQGikHv5FpQ09U= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-550-om_lOsePOlSXRPkjbkTmxA-1; Mon, 01 Feb 2021 18:00:37 -0500 X-MC-Unique: om_lOsePOlSXRPkjbkTmxA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id EE9D215720; Mon, 1 Feb 2021 23:00:35 +0000 (UTC) Received: from treble (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 90B956F924; Mon, 1 Feb 2021 23:00:34 +0000 (UTC) Date: Mon, 1 Feb 2021 17:00:32 -0600 From: Josh Poimboeuf To: "Madhavan T. Venkataraman" Subject: Re: [RFC PATCH 0/3] arm64: Implement reliable stack trace Message-ID: <20210201230032.syuv2nrbbureszbu@treble> References: <20201012172605.10715-1-broonie@kernel.org> <13095563-ff6d-b806-1bf3-efde4383456e@linux.microsoft.com> <20210128142250.GC4537@sirena.org.uk> <20210128152649.6zin3hzim3etbv2p@treble> <20210201160225.GD66060@C02TD0UTHF1T.local> MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jpoimboe@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210201_180044_515493_5E3CCB42 X-CRM114-Status: GOOD ( 36.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Julien Thierry , Catalin Marinas , Mark Brown , Miroslav Benes , Will Deacon , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Feb 01, 2021 at 03:38:53PM -0600, Madhavan T. Venkataraman wrote: > So, I have a few questions from a livepatch perspective. > > For livepatch, the kernel makes sure that task is not running when its stack is checked, > correct? Correct. > The only possibility I can think of is that the task could have received an > interrupt and could have been preempted at the end of the interrupt. The interrupt > could have happened during the frame pointer prolog or epilog. Is this the problem case > for livepatch? > > If the unwinder could check a flag in the task that indicates that the task was interrupted, > the unwinder could declare the stack trace unreliable. E.g., a (hacky) solution could > be to set and clear the flag in preempt_schedule_irq() which takes a task off a CPU > when it is preempted at the end of an interrupt. The flag would remain set while the task is not > on a CPU. > > Similarly, for exceptions, can we set a flag in a task indicating that it is processing > an exception? Is there a top level exception handler where we can do this? Is there common > code that exception handlers use where we can set this? Or, can we deduce this from ptregs->pstate > that is saved for the task? > > Mind you, the flag is advisory. If the unwinder has some way to unwind through an exception, > more power to it. For x86 (frame pointers), entry code uses ENCODE_FRAME_POINTER, which creates a special pt_regs frame. When the reliable unwinder sees the encoded regs on the stack, it knows it encountered some asynchronous event, like preemption, and it marks the stack unreliable. > > Given that, I think that assuming we must use a shadow stack for > > reliable unwinding would be jumping the gun. > > > > So, this is the problem I was considering. Let us say that a function properly sets up the > frame pointer at the beginning and properly restores it to the previous value when it > returns. But because of compiler bugs or some inline assembly code or other errant code, > the frame pointer gets modified in the middle of the function. Then, the function calls > another function. Then, the unwinder tries to unwind the stack. The unwinder has no > way of knowing that the frame pointer was modified. To tackle this problem, Objtool > has to laboriously walk all the code paths and track every modification to the stack and > the frame pointer. And, if there are frame modifications, it has to fail the kernel build. > Did I understand it correctly? Yes, though it generally warns instead of failing the build. But we keep the warnings to zero as best we can. BTW, the most common inline asm frame pointer bug we saw on x86 was a call instruction which got inserted by GCC before the prologue -- or sometimes there was no prologue because it was otherwise considered a leaf function. > In these cases, the shadow stack can be used to unwind the stack. The shadow stack has > return addresses pushed on it. For livepatch purposes, this good enough. We try to fix every warning. For the few warnings we whitelist instead of fixing, we make sure it's not a risk for live patching. > >> Objtool will check for the no-ops. If they are present, it will replace the no-ops with > >> the shadow stack prolog and epilog. It can also check the frame pointer prolog and > >> epilog. > > > > I suspect this will interact poorly with patchable-function-entry, which > > prefixes each instrumentable function with some NOPs. > > > > Objtool knows if the kernel was configured with tracing. The compiler inserts a fixed, > known number of no-ops for tracing purposes. So, why is it difficult for objtool to > find the prolog/epilog no-ops? Objtool tries to stay out of the code generation business. Because then who's going to validate objtool's code :-) And the compiler already does a decent job at generating it. > > I think at this point, we haven't gained anything from using a shadow > > stack, and I'd much rather we used objtool to gather any metadata needed > > to make unwinding reliable without mandating a shadow stack. > > > > I think we have gained something. Pushing the return addresses on the shadow stack > and using them to unwind means that objtool does not have to decode every single > instruction and track the changes to the stack and frame state. It also means > that the kernel build does not have to be failed when some frame modification is > detected by objtool. How do we know the kernel has full and accurate CFI coverage? The original version of objtool was an awk script which basically just crudely looked for the prologue/epilogue instructions. It mostly worked. But it wasn't 100%, and these days the prologue isn't always at the beginning, and the epilogue is usually buried in the middle. And sometimes there are more stack pushes/pops hidden outside of the formal prologue/epilogue. Not to mention asm code which does all kinds of crazy things. And other edge cases, like leaf functions which don't require frame pointers, and alternatives patching/paravirt/etc which can muck with the stack layout at runtime. Eventually we realized a "full coverage" objtool is the wisest approach. Also, a simpler version of objtool isn't really an option on the x86 side, since we now have a lot of other features relying on its full coverage. Other than the decoder, most of the objtool logic is arch-independent. -- Josh _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel