From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7AACC433B4 for ; Tue, 6 Apr 2021 11:01:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A95AD61055 for ; Tue, 6 Apr 2021 11:01:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245318AbhDFLBS (ORCPT ); Tue, 6 Apr 2021 07:01:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232292AbhDFLBO (ORCPT ); Tue, 6 Apr 2021 07:01:14 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B662C06174A for ; Tue, 6 Apr 2021 04:01:06 -0700 (PDT) From: John Ogness DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1617706863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=S2O1k9yZlYySt7+8gnNrDK/Yp/8kwoWX62PxlbcqwbA=; b=nl9GFkH778lC7VCrJOuaoV80FdfH0mtXyzGk6zcuRJPpzUbwbdCttQk6fZN0TWXbTNFRHS 9OP4F05zxprZGg/xrYgT3Dc78sX2wQoXJufiMTarERPRiF1HwtW87KNXnnLCLPBwAEUI8l Lyi4cptRgwjQyWy4f5OzjrnYIUFDU3znx7RetH9ffo5pac9HdBj/wgI4zvpLnhJw7e2oAi h34q2YaMqx653oO2+7Tqu8iGaVsxQwJhHRPYLrqPOAfOpobaf1NbF3s91jgtawE7IIWdKE UV94lIL4ee0lGL/6gJg9KEoMtmeOBzCgkHyqOsp+irX0SlOmZB2rI9/coqhytg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1617706863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=S2O1k9yZlYySt7+8gnNrDK/Yp/8kwoWX62PxlbcqwbA=; b=yzGkWAKS3iH7xchc7y3QLe7xFMXuhjofzIxtdxB9f2DojJ4aTiCvWNPgQVcmr4T/abFsf7 grBHNd4e9TDvqEDQ== To: Petr Mladek Cc: Sergey Senozhatsky , Sergey Senozhatsky , Steven Rostedt , Thomas Gleixner , linux-kernel@vger.kernel.org, Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , Eric Biederman , Christophe Leroy , Nicholas Piggin , Alistair Popple , Jordan Niethe , Peter Zijlstra , "Aneesh Kumar K.V" , Andrew Morton , Kees Cook , Tiezhu Yang , Alexey Kardashevskiy , Yue Hu , Rafael Aquini , "Guilherme G. Piccoli" , "Paul E. McKenney" , linuxppc-dev@lists.ozlabs.org, kexec@lists.infradead.org Subject: Re: [PATCH printk v2 2/5] printk: remove safe buffers In-Reply-To: References: <20210330153512.1182-1-john.ogness@linutronix.de> <20210330153512.1182-3-john.ogness@linutronix.de> <87a6qiqgzr.fsf@jogness.linutronix.de> Date: Tue, 06 Apr 2021 13:01:02 +0200 Message-ID: <87im4zoexd.fsf@jogness.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2021-04-01, Petr Mladek wrote: >> Caller-id solves this problem and is easy to sort for anyone with >> `grep'. Yes, it is a shame that `dmesg' does not show it, but >> directly using any of the printk interfaces does show it (kmsg_dump, >> /dev/kmsg, syslog, console). > > True but frankly, the current situation is _far_ from convenient: > > + consoles do not show it by default > + none userspace tool (dmesg, journalctl, crash) is able to show it > + grep is a nightmare, especially if you have more than handful of CPUs > > Yes, everything is solvable but not easily. > >> > I get this with "echo l >/proc/sysrq-trigger" and this patchset: >> >> Of course. Without caller-id, it is a mess. But this has nothing to do >> with NMI. The same problem exists for WARN_ON() on multiple CPUs >> simultaneously. If the user is not using caller-id, they are >> lost. Caller-id is the current solution to the interlaced logs. > > Sure. But in reality, the risk of mixed WARN_ONs is small. While > this patch makes backtraces from all CPUs always unusable without > caller_id and non-trivial effort. I would prefer we solve the situation for non-NMI as well, not just for the sysrq "l" case. >> For the long term, we should introduce a printk-context API that allows >> callers to perfectly pack their multi-line output into a single >> entry. We discussed [0][1] this back in August 2020. > > We need a "short" term solution. There are currently 3 solutions: > > 1. Keep nmi_safe() and all the hacks around. > > 2. Serialize nmi_cpu_backtrace() by a spin lock and later by > the special lock used also by atomic consoles. > > 3. Tell complaining people how to sort the messed logs. Or we look into the long term solution now. If caller-id's cannot not be used as the solution (because nobody turns it on, nobody knows about it, and/or distros do not enable it), then we should look at how to make at least the backtraces contiguous. I have a few ideas here. John Ogness From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 533F8C433ED for ; Tue, 6 Apr 2021 11:01:39 +0000 (UTC) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9E6A561154 for ; Tue, 6 Apr 2021 11:01:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E6A561154 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4FF4N115SRz30FD for ; Tue, 6 Apr 2021 21:01:37 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; secure) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=nl9GFkH7; dkim=fail reason="signature verification failed" header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=yzGkWAKS; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linutronix.de (client-ip=193.142.43.55; helo=galois.linutronix.de; envelope-from=john.ogness@linutronix.de; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=linutronix.de header.i=@linutronix.de header.a=rsa-sha256 header.s=2020 header.b=nl9GFkH7; dkim=pass header.d=linutronix.de header.i=@linutronix.de header.a=ed25519-sha256 header.s=2020e header.b=yzGkWAKS; dkim-atps=neutral Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4FF4MW0CMGz2xZL for ; Tue, 6 Apr 2021 21:01:10 +1000 (AEST) From: John Ogness DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1617706863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=S2O1k9yZlYySt7+8gnNrDK/Yp/8kwoWX62PxlbcqwbA=; b=nl9GFkH778lC7VCrJOuaoV80FdfH0mtXyzGk6zcuRJPpzUbwbdCttQk6fZN0TWXbTNFRHS 9OP4F05zxprZGg/xrYgT3Dc78sX2wQoXJufiMTarERPRiF1HwtW87KNXnnLCLPBwAEUI8l Lyi4cptRgwjQyWy4f5OzjrnYIUFDU3znx7RetH9ffo5pac9HdBj/wgI4zvpLnhJw7e2oAi h34q2YaMqx653oO2+7Tqu8iGaVsxQwJhHRPYLrqPOAfOpobaf1NbF3s91jgtawE7IIWdKE UV94lIL4ee0lGL/6gJg9KEoMtmeOBzCgkHyqOsp+irX0SlOmZB2rI9/coqhytg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1617706863; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=S2O1k9yZlYySt7+8gnNrDK/Yp/8kwoWX62PxlbcqwbA=; b=yzGkWAKS3iH7xchc7y3QLe7xFMXuhjofzIxtdxB9f2DojJ4aTiCvWNPgQVcmr4T/abFsf7 grBHNd4e9TDvqEDQ== To: Petr Mladek Subject: Re: [PATCH printk v2 2/5] printk: remove safe buffers In-Reply-To: References: <20210330153512.1182-1-john.ogness@linutronix.de> <20210330153512.1182-3-john.ogness@linutronix.de> <87a6qiqgzr.fsf@jogness.linutronix.de> Date: Tue, 06 Apr 2021 13:01:02 +0200 Message-ID: <87im4zoexd.fsf@jogness.linutronix.de> MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sergey Senozhatsky , Alexey Kardashevskiy , Paul Mackerras , Tiezhu Yang , Rafael Aquini , "Aneesh Kumar K.V" , Peter Zijlstra , Yue Hu , Jordan Niethe , Kees Cook , "Paul E. McKenney" , Alistair Popple , "Guilherme G. Piccoli" , Nicholas Piggin , Steven Rostedt , Thomas Gleixner , kexec@lists.infradead.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Eric Biederman , Andrew Morton , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On 2021-04-01, Petr Mladek wrote: >> Caller-id solves this problem and is easy to sort for anyone with >> `grep'. Yes, it is a shame that `dmesg' does not show it, but >> directly using any of the printk interfaces does show it (kmsg_dump, >> /dev/kmsg, syslog, console). > > True but frankly, the current situation is _far_ from convenient: > > + consoles do not show it by default > + none userspace tool (dmesg, journalctl, crash) is able to show it > + grep is a nightmare, especially if you have more than handful of CPUs > > Yes, everything is solvable but not easily. > >> > I get this with "echo l >/proc/sysrq-trigger" and this patchset: >> >> Of course. Without caller-id, it is a mess. But this has nothing to do >> with NMI. The same problem exists for WARN_ON() on multiple CPUs >> simultaneously. If the user is not using caller-id, they are >> lost. Caller-id is the current solution to the interlaced logs. > > Sure. But in reality, the risk of mixed WARN_ONs is small. While > this patch makes backtraces from all CPUs always unusable without > caller_id and non-trivial effort. I would prefer we solve the situation for non-NMI as well, not just for the sysrq "l" case. >> For the long term, we should introduce a printk-context API that allows >> callers to perfectly pack their multi-line output into a single >> entry. We discussed [0][1] this back in August 2020. > > We need a "short" term solution. There are currently 3 solutions: > > 1. Keep nmi_safe() and all the hacks around. > > 2. Serialize nmi_cpu_backtrace() by a spin lock and later by > the special lock used also by atomic consoles. > > 3. Tell complaining people how to sort the messed logs. Or we look into the long term solution now. If caller-id's cannot not be used as the solution (because nobody turns it on, nobody knows about it, and/or distros do not enable it), then we should look at how to make at least the backtraces contiguous. I have a few ideas here. John Ogness