From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 860C7C433E9 for ; Tue, 26 Jan 2021 18:20:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5E16F207BC for ; Tue, 26 Jan 2021 18:20:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2404049AbhAZSU0 (ORCPT ); Tue, 26 Jan 2021 13:20:26 -0500 Received: from linux.microsoft.com ([13.77.154.182]:45182 "EHLO linux.microsoft.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390578AbhAZSEb (ORCPT ); Tue, 26 Jan 2021 13:04:31 -0500 Received: from [192.168.254.32] (unknown [47.187.219.45]) by linux.microsoft.com (Postfix) with ESMTPSA id 40D9120B7192; Tue, 26 Jan 2021 10:03:32 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 40D9120B7192 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1611684212; bh=VkWouM927sS4tAA7jtXNPFYjeYSo0tacxiplhxttVOs=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=q60zSDdvN+Rup5Vcwic4qbLREsKILOQ/wVvLBox2wYRIjVRrbvyno2Oid33hAh0NW o1gw3Z0fLDgLG0YC/0FgxJNA63izCpq1/7ynmb+E+VWquERtyrP7a/RBQRXg/u8zK6 D7gaINkh0KFPqnQN5CkX8Ys/UMniFZpNYdi04Gq8= Subject: Re: Live patching on ARM64 To: Mark Rutland Cc: linux-arm-kernel@lists.infradead.org, Mark Brown , Julien Thierry , jpoimboe@redhat.com, live-patching@vger.kernel.org, linux-kernel@vger.kernel.org References: <20210115123347.GB39776@C02TD0UTHF1T.local> From: "Madhavan T. Venkataraman" Message-ID: Date: Tue, 26 Jan 2021 12:03:31 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20210115123347.GB39776@C02TD0UTHF1T.local> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, Mark Rutland had sent me some ideas on what work is pending for ARM64 live patching. I sent some questions to Mark Rutland. I forgot to include everyone in the email. Sorry about that. I have reproduced my questions and his responses below. Please chime in with any comments: Thanks! On Mon, Jan 25, 2021 at 11:58:47AM -0600, Madhavan T. Venkataraman wrote: > Some questions below: I've answered thos below. If possible, I'd prefer to handle future queries on a public list (so that others can chime in, and so that it gets archived), so if you could direct further questions to a thread on LAKML, that would be much appreciated. > On 1/15/21 6:33 AM, Mark Rutland wrote: > [...] >> >> One general thing that I believe we'll need to do is to rework code to >> be patch-safe (which implies being noinstr-safe too). For example, we'll >> need to rework the instruction patching code such that this cannot end >> up patching itself (or anything that has instrumented it) in an unsafe >> way. >> > > OK. I understand that. Are there are other scenarios that make patching > unsafe? I suspect so; these are simply the cases I'm immediately aware of. I suspect there are other cases that we will need to consider that don't immediately spring to mind. > I expect the kernel already handles scenarios such as two CPUs patching > the same location at the same time or a thread executing at a location that is > currently being patched. IIRC that is supposed to be catered for by ftrace (and so I assume for livepatching too); I'm not certain about kprobes. In addition to synchronization in the core ftrace code, arm64's ftrace_modify_code() has a sanity-check with a non-atomic RMW sequence. We might be able to make that more robust wiuth a faultable cmpxchg, and some changes around ftrace_update_ftrace_func() and ftrace_make_nop() to get rid of the unvalidated cases. > Any other scenarios to be considered? I'm not immediately aware of others, but suspect more cases will become apparent as work progresses on the bits we already know about. >> Once we have objtool it should be possible to identify those cases >> automatically. Currently I'm aware that we'll need to do something in at >> least the following places: >> > > OK. AFAIK, objtool checks for the following: > > - returning from noinstr function with instrumentation enabled > > - calling instrumentable functions from noinstr code without: > > instrumentation_begin(); > instrumentation_end(); > > Is that what you mean? That's what I was thinking of, yes -- this should highlight some places that will need attention. > Does objtool check other things as well that is relevant to (un)safe > patching? I'm not entirely familiar with objtool, so I'm not exactly sure what it can do; I expect Josh and Julien can give more detail here. >> * The insn framework (which is used by some patching code), since the >> bulk of it lives in arch/arm64/kernel/insn.c and isn't marked noinstr. >> >> We can probably shift the bulk of the aarch64_insn_gen_*() and >> aarch64_get_*() helpers into a header as __always_inline functions, >> which would allow them to be used in noinstr code. As those are >> typically invoked with a number of constant arguments that the >> compiler can fold, this /might/ work out as an optimization if the >> compiler can elide the error paths. > > OK. I will take a look at the insn code. IIRC Julien's objtool series had some patches had some patches moving the insn code about, so it'd be worth checking whether that's a help or a hindrance. If it's possible to split out a set of preparatory patches that make that ready both for objtool and the kernel, that would make it easier to review that and queue it early. >> * The alternatives code, since we call instrumentable and patchable >> functions between updating instructions and performing all the >> necessary maintenance. There are a number of cases within >> __apply_alternatives(), e.g. >> >> - test_bit() >> - cpus_have_cap() >> - pr_info_once() >> - lm_alias() >> - alt_cb, if the callback is not marked as noinstr, or if it calls >> instrumentable code (e.g. from the insn framework). >> - clean_dcache_range_nopatch(), as read_sanitised_ftr_reg() and >> related code can be instrumented. >> >> This might need some underlying rework elsewhere (e.g. in the >> cpufeature code, or atomics framework). >> >> So on the kernel side, maybe a first step would be to try to headerize >> the insn generation code as __always_inline, and see whether that looks >> ok? With that out of the way it'd be a bit easier to rework patching >> code depending on the insn framework. > > OK. I will study this. Great, thanks! Mark. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70409C433DB for ; Tue, 26 Jan 2021 18:05:43 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 270E622228 for ; Tue, 26 Jan 2021 18:05:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 270E622228 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.microsoft.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Date:Message-ID:From: References:To:Subject:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=N6Wj3SdcJDW5AcSicxps+TccCG12H95XOAv6v5sCKYk=; b=1L1LUTGB6X00uZxLLEeKiqMCu F0bHntQUHsO2yzDaUPIvPIz0wVym+pmY/EO0mOcMiU7mVHb1hYTudMZNZHnUvFK/y2FxUvm0KKHzi AwZO4CBX5xjC3wFm5wfDYlV2purDhn/83oCWG+x9WJkSaHWblCYm3tBB1or4q6YD7oT2/V1kZNskm XL1ktd6rQfK22Oe4a2zwbwAcgYyDIVqC68mR/sjkIyEVMyGeNwdUFDQMHth6goVG6MMC7famKGm28 KPXrdyMT6cJ8NU0dwmFGthietLuIie7tsWpmJ+xi9d3hRhAmc406x0rCkolluWd+xTNRND9NWg+DN 8+NeBHskw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4Sgk-0004e9-Cl; Tue, 26 Jan 2021 18:03:38 +0000 Received: from linux.microsoft.com ([13.77.154.182]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4Sgg-0004c3-SO for linux-arm-kernel@lists.infradead.org; Tue, 26 Jan 2021 18:03:36 +0000 Received: from [192.168.254.32] (unknown [47.187.219.45]) by linux.microsoft.com (Postfix) with ESMTPSA id 40D9120B7192; Tue, 26 Jan 2021 10:03:32 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 40D9120B7192 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1611684212; bh=VkWouM927sS4tAA7jtXNPFYjeYSo0tacxiplhxttVOs=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=q60zSDdvN+Rup5Vcwic4qbLREsKILOQ/wVvLBox2wYRIjVRrbvyno2Oid33hAh0NW o1gw3Z0fLDgLG0YC/0FgxJNA63izCpq1/7ynmb+E+VWquERtyrP7a/RBQRXg/u8zK6 D7gaINkh0KFPqnQN5CkX8Ys/UMniFZpNYdi04Gq8= Subject: Re: Live patching on ARM64 To: Mark Rutland References: <20210115123347.GB39776@C02TD0UTHF1T.local> From: "Madhavan T. Venkataraman" Message-ID: Date: Tue, 26 Jan 2021 12:03:31 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <20210115123347.GB39776@C02TD0UTHF1T.local> Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210126_130335_246806_AAB1F797 X-CRM114-Status: GOOD ( 33.76 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Julien Thierry , linux-kernel@vger.kernel.org, Mark Brown , jpoimboe@redhat.com, live-patching@vger.kernel.org, linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi all, Mark Rutland had sent me some ideas on what work is pending for ARM64 live patching. I sent some questions to Mark Rutland. I forgot to include everyone in the email. Sorry about that. I have reproduced my questions and his responses below. Please chime in with any comments: Thanks! On Mon, Jan 25, 2021 at 11:58:47AM -0600, Madhavan T. Venkataraman wrote: > Some questions below: I've answered thos below. If possible, I'd prefer to handle future queries on a public list (so that others can chime in, and so that it gets archived), so if you could direct further questions to a thread on LAKML, that would be much appreciated. > On 1/15/21 6:33 AM, Mark Rutland wrote: > [...] >> >> One general thing that I believe we'll need to do is to rework code to >> be patch-safe (which implies being noinstr-safe too). For example, we'll >> need to rework the instruction patching code such that this cannot end >> up patching itself (or anything that has instrumented it) in an unsafe >> way. >> > > OK. I understand that. Are there are other scenarios that make patching > unsafe? I suspect so; these are simply the cases I'm immediately aware of. I suspect there are other cases that we will need to consider that don't immediately spring to mind. > I expect the kernel already handles scenarios such as two CPUs patching > the same location at the same time or a thread executing at a location that is > currently being patched. IIRC that is supposed to be catered for by ftrace (and so I assume for livepatching too); I'm not certain about kprobes. In addition to synchronization in the core ftrace code, arm64's ftrace_modify_code() has a sanity-check with a non-atomic RMW sequence. We might be able to make that more robust wiuth a faultable cmpxchg, and some changes around ftrace_update_ftrace_func() and ftrace_make_nop() to get rid of the unvalidated cases. > Any other scenarios to be considered? I'm not immediately aware of others, but suspect more cases will become apparent as work progresses on the bits we already know about. >> Once we have objtool it should be possible to identify those cases >> automatically. Currently I'm aware that we'll need to do something in at >> least the following places: >> > > OK. AFAIK, objtool checks for the following: > > - returning from noinstr function with instrumentation enabled > > - calling instrumentable functions from noinstr code without: > > instrumentation_begin(); > instrumentation_end(); > > Is that what you mean? That's what I was thinking of, yes -- this should highlight some places that will need attention. > Does objtool check other things as well that is relevant to (un)safe > patching? I'm not entirely familiar with objtool, so I'm not exactly sure what it can do; I expect Josh and Julien can give more detail here. >> * The insn framework (which is used by some patching code), since the >> bulk of it lives in arch/arm64/kernel/insn.c and isn't marked noinstr. >> >> We can probably shift the bulk of the aarch64_insn_gen_*() and >> aarch64_get_*() helpers into a header as __always_inline functions, >> which would allow them to be used in noinstr code. As those are >> typically invoked with a number of constant arguments that the >> compiler can fold, this /might/ work out as an optimization if the >> compiler can elide the error paths. > > OK. I will take a look at the insn code. IIRC Julien's objtool series had some patches had some patches moving the insn code about, so it'd be worth checking whether that's a help or a hindrance. If it's possible to split out a set of preparatory patches that make that ready both for objtool and the kernel, that would make it easier to review that and queue it early. >> * The alternatives code, since we call instrumentable and patchable >> functions between updating instructions and performing all the >> necessary maintenance. There are a number of cases within >> __apply_alternatives(), e.g. >> >> - test_bit() >> - cpus_have_cap() >> - pr_info_once() >> - lm_alias() >> - alt_cb, if the callback is not marked as noinstr, or if it calls >> instrumentable code (e.g. from the insn framework). >> - clean_dcache_range_nopatch(), as read_sanitised_ftr_reg() and >> related code can be instrumented. >> >> This might need some underlying rework elsewhere (e.g. in the >> cpufeature code, or atomics framework). >> >> So on the kernel side, maybe a first step would be to try to headerize >> the insn generation code as __always_inline, and see whether that looks >> ok? With that out of the way it'd be a bit easier to rework patching >> code depending on the insn framework. > > OK. I will study this. Great, thanks! Mark. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel