From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4BF1C43460 for ; Wed, 21 Apr 2021 15:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 78A856144B for ; Wed, 21 Apr 2021 15:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240411AbhDUPMp (ORCPT ); Wed, 21 Apr 2021 11:12:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240325AbhDUPMo (ORCPT ); Wed, 21 Apr 2021 11:12:44 -0400 Received: from mail-oo1-xc29.google.com (mail-oo1-xc29.google.com [IPv6:2607:f8b0:4864:20::c29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A5A09C06138B for ; Wed, 21 Apr 2021 08:12:09 -0700 (PDT) Received: by mail-oo1-xc29.google.com with SMTP id s1-20020a4ac1010000b02901cfd9170ce2so9270080oop.12 for ; Wed, 21 Apr 2021 08:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FEXEAgUao8/U1XoW751IUwiCrSmDVaqLIc6z62zqqMg=; b=Vo5glLC6k7D+/dE1nb/qouPscezMFfLgYz7H9PDhiS+225DnfgsTkqbIQ3YKQxcbKH N9WNrIISNRSCIXZ0Xa2geQUHPwPnz1H0/r2shySWTDVl9W9/1wWxlwISt92QkyHI1/D5 is4kUk+NU/HEU91rjPdqiJG5y3zN7p9qXt+Hmpoxbd5ZzCUXzUI+sZiJlvFysiAqPOzk 2JWPjDS8PDVfeR7oLmiyTUp1U5Sj+Recg00TFzL2m2g9M6d82OTR14ZDv16kvT/l4M2D BpDuIcq7ihOTMmVUiwcrQGjhGXbhQPyPPMY7gQ1x2zK9oJETG0hcADBQKWBqljmgmIpW xnPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FEXEAgUao8/U1XoW751IUwiCrSmDVaqLIc6z62zqqMg=; b=TSuvyYUC/RVgn2EXhFHm5d1W1svuc7ccCRQ5PGVh1o6AhEWXSbZhzCDovaoOGavLv1 ZPxRmuYh5/C8dJJPysRXKXPM8AIb7BdYbhOxzLxFX1v7NxnGolL14314Gz9/qYv62Xaa ZEGV2Rn2un5UV/9RQdjat8Rot4DlLPCtmtjmuK2eOFBzjm/Z+xQr5tRQhiefE2Lzd3Mh bBOEiHOcnWTpQIvHiiSPpfAOcKTF+73ULZmOFFoiz0Rh3e6djJQCg7mGZSgpxI/14ugM llJux5w4oDuR5Zc3RybCIMr0tYfj+2S7iaDNroXagjaIkOi/GV0i4nrNnfFl9qkoIGZa QMzw== X-Gm-Message-State: AOAM533MwWBq5H/XrGEF0pBE0WTsNtYqC2lRcBrg9hfk0XGRwm1ITuJM S+6Gzu/SLBhfq1iyem2TNL2sbKrwTKxOVc7im9R9PA== X-Google-Smtp-Source: ABdhPJyng+pvseRT0sVGDwFEJZ8TI5HsU+LGWY1XrOff8CcFEH9YHvKH2/GavzRvVwXXrQ+JymLm5bTa/K1m4eP+ih4= X-Received: by 2002:a4a:d80e:: with SMTP id f14mr18296328oov.54.1619017928732; Wed, 21 Apr 2021 08:12:08 -0700 (PDT) MIME-Version: 1.0 References: <20210408103605.1676875-1-elver@google.com> <20210408103605.1676875-6-elver@google.com> <1fbf3429-42e5-0959-9a5c-91de80f02b6a@samsung.com> <43f8a3bf-34c5-0fc9-c335-7f92eaf23022@samsung.com> <740077ce-efe1-b171-f807-bc5fd95a32ba@samsung.com> In-Reply-To: From: Marco Elver Date: Wed, 21 Apr 2021 17:11:57 +0200 Message-ID: Subject: Re: [PATCH v4 05/10] signal: Introduce TRAP_PERF si_code and si_perf to siginfo To: Marek Szyprowski Cc: Peter Zijlstra , Alexander Shishkin , Arnaldo Carvalho de Melo , Ingo Molnar , Jiri Olsa , Mark Rutland , Namhyung Kim , Thomas Gleixner , Alexander Potapenko , Al Viro , Arnd Bergmann , Christian Brauner , Dmitry Vyukov , Jann Horn , Jens Axboe , Matt Morehouse , Peter Collingbourne , Ian Rogers , Oleg Nesterov , kasan-dev , linux-arch , linux-fsdevel , LKML , "the arch/x86 maintainers" , "open list:KERNEL SELFTEST FRAMEWORK" , Geert Uytterhoeven , Bartlomiej Zolnierkiewicz , Linux ARM , linux-tegra@vger.kernel.org, jonathanh@nvidia.com Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org +Cc linux-arm-kernel On Wed, 21 Apr 2021 at 15:19, Marek Szyprowski wrote: > > Hi Marco, > > On 21.04.2021 13:03, Marco Elver wrote: > > On Wed, 21 Apr 2021 at 12:57, Marek Szyprowski wrote: > >> On 21.04.2021 11:35, Marek Szyprowski wrote: > >>> On 21.04.2021 10:11, Marco Elver wrote: > >>>> On Wed, 21 Apr 2021 at 09:35, Marek Szyprowski > >>>> wrote: > >>>>> On 21.04.2021 08:21, Marek Szyprowski wrote: > >>>>>> On 21.04.2021 00:42, Marco Elver wrote: > >>>>>>> On Tue, 20 Apr 2021 at 23:26, Marek Szyprowski > >>>>>>> wrote: > >>>>>>>> On 08.04.2021 12:36, Marco Elver wrote: > >>>>>>>>> Introduces the TRAP_PERF si_code, and associated siginfo_t field > >>>>>>>>> si_perf. These will be used by the perf event subsystem to send > >>>>>>>>> signals > >>>>>>>>> (if requested) to the task where an event occurred. > >>>>>>>>> > >>>>>>>>> Acked-by: Geert Uytterhoeven # m68k > >>>>>>>>> Acked-by: Arnd Bergmann # asm-generic > >>>>>>>>> Signed-off-by: Marco Elver > >>>>>>>> This patch landed in linux-next as commit fb6cc127e0b6 ("signal: > >>>>>>>> Introduce TRAP_PERF si_code and si_perf to siginfo"). It causes > >>>>>>>> regression on my test systems (arm 32bit and 64bit). Most systems > >>>>>>>> fails > >>>>>>>> to boot in the given time frame. I've observed that there is a > >>>>>>>> timeout > >>>>>>>> waiting for udev to populate /dev and then also during the network > >>>>>>>> interfaces configuration. Reverting this commit, together with > >>>>>>>> 97ba62b27867 ("perf: Add support for SIGTRAP on perf events") to > >>>>>>>> let it > >>>>>>>> compile, on top of next-20210420 fixes the issue. > >>>>>>> Thanks, this is weird for sure and nothing in particular stands out. > >>>>>>> > >>>>>>> I have questions: > >>>>>>> -- Can you please share your config? > >>>>>> This happens with standard multi_v7_defconfig (arm) or just defconfig > >>>>>> for arm64. > >>>>>> > >>>>>>> -- Also, can you share how you run this? Can it be reproduced in > >>>>>>> qemu? > >>>>>> Nothing special. I just boot my test systems and see that they are > >>>>>> waiting lots of time during the udev populating /dev and network > >>>>>> interfaces configuration. I didn't try with qemu yet. > >>>>>>> -- How did you derive this patch to be at fault? Why not just > >>>>>>> 97ba62b27867, given you also need to revert it? > >>>>>> Well, I've just run my boot tests with automated 'git bisect' and that > >>>>>> was its result. It was a bit late in the evening, so I didn't analyze > >>>>>> it further, I've just posted a report about the issue I've found. It > >>>>>> looks that bisecting pointed to a wrong commit somehow. > >>>>>>> If you are unsure which patch exactly it is, can you try just > >>>>>>> reverting 97ba62b27867 and see what happens? > >>>>>> Indeed, this is a real faulty commit. Initially I've decided to revert > >>>>>> it to let kernel compile (it uses some symbols introduced by this > >>>>>> commit). Reverting only it on top of linux-next 20210420 also fixes > >>>>>> the issue. I'm sorry for the noise in this thread. I hope we will find > >>>>>> what really causes the issue. > >>>>> This was a premature conclusion. It looks that during the test I've did > >>>>> while writing that reply, the modules were not deployed properly and a > >>>>> test board (RPi4) booted without modules. In that case the board booted > >>>>> fine and there was no udev timeout. After deploying kernel modules, the > >>>>> udev timeout is back. > >>>> I'm confused now. Can you confirm that the problem is due to your > >>>> kernel modules, or do you think it's still due to 97ba62b27867? Or > >>>> fb6cc127e0b6 (this patch)? > >>> I don't use any custom kernel modules. I just deploy all modules that > >>> are being built from the given kernel defconfig (arm > >>> multi_v7_defconfig or arm64 default) and they are automatically loaded > >>> during the boot by udev. I've checked again and bisect was right. The > >>> kernel built from fb6cc127e0b6 suffers from the described issue, while > >>> the one build from the previous commit (2e498d0a74e5) works fine. > >> I've managed to reproduce this issue with qemu. I've compiled the kernel > >> for arm 32bit with multi_v7_defconfig and used some older Debian rootfs > >> image. The log and qemu parameters are here: > >> https://protect2.fireeye.com/v1/url?k=7cfc23a2-23671aa9-7cfda8ed-002590f5b904-dab7e2ec39dae1f9&q=1&e=36a5ed13-6ad5-430c-8f44-e95c4f0af5c3&u=https%3A%2F%2Fpaste.debian.net%2F1194526%2F > >> > >> Check the timestamp for the 'EXT4-fs (vda): re-mounted' message and > >> 'done (timeout)' status for the 'Waiting for /dev to be fully populated' > >> message. This happens only when kernel modules build from the > >> multi_v7_defconfig are deployed on the rootfs. > > Still hard to say what is going on and what is at fault. But being > > able to repro this in qemu helps debug quicker -- would you also be > > able to share the precise rootfs.img, i.e. upload it somewhere I can > > fetch it? And just to be sure, please also share your .config, as it > > might have compiler-version dependent configuration that might help > > repro (unlikely, but you never know). > > I've managed to reproduce this issue with a public Raspberry Pi OS Lite > rootfs image, even without deploying kernel modules: > > https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip > > # qemu-system-arm -M virt -smp 2 -m 512 -kernel zImage -append "earlycon > console=ttyAMA0 root=/dev/vda2 rw rootwait" -serial stdio -display none > -monitor null -device virtio-blk-device,drive=virtio-blk -drive > file=/tmp/2021-03-04-raspios-buster-armhf-lite.img,id=virtio-blk,if=none,format=raw > -netdev user,id=user -device virtio-net-device,netdev=user > > The above one doesn't boot if zImage z compiled from commit fb6cc127e0b6 > and boots if compiled from 2e498d0a74e5. In both cases I've used default > arm/multi_v7_defconfig and > gcc-linaro-6.4.1-2017.11-x86_64_arm-linux-gnueabi toolchain. Yup, I've narrowed it down to the addition of "__u64 _perf" to siginfo_t. My guess is the __u64 causes a different alignment for a bunch of adjacent fields. It seems that x86 and m68k are the only ones that have compile-time tests for the offsets. Arm should probably add those -- I have added a bucket of static_assert() in arch/arm/kernel/signal.c and see that something's off. I'll hopefully have a fix in a day or so. Thanks, -- Marco