From: Marek Szyprowski <m.szyprowski@samsung.com>
To: Marco Elver <elver@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Ingo Molnar <mingo@redhat.com>, Jiri Olsa <jolsa@redhat.com>,
Mark Rutland <mark.rutland@arm.com>,
Namhyung Kim <namhyung@kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Alexander Potapenko <glider@google.com>,
Al Viro <viro@zeniv.linux.org.uk>, Arnd Bergmann <arnd@arndb.de>,
Christian Brauner <christian@brauner.io>,
Dmitry Vyukov <dvyukov@google.com>, Jann Horn <jannh@google.com>,
Jens Axboe <axboe@kernel.dk>, Matt Morehouse <mascasa@google.com>,
Peter Collingbourne <pcc@google.com>,
Ian Rogers <irogers@google.com>, Oleg Nesterov <oleg@redhat.com>,
kasan-dev <kasan-dev@googlegroups.com>,
linux-arch <linux-arch@vger.kernel.org>,
linux-fsdevel <linux-fsdevel@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
the arch/x86 maintainers <x86@kernel.org>,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>,
Geert Uytterhoeven <geert@linux-m68k.org>,
Bartlomiej Zolnierkiewicz <b.zolnierkie@samsung.com>
Subject: Re: [PATCH v4 05/10] signal: Introduce TRAP_PERF si_code and si_perf to siginfo
Date: Wed, 21 Apr 2021 15:19:25 +0200 [thread overview]
Message-ID: <cf6ed5cd-3202-65ce-86bc-6f1eba1b7d17@samsung.com> (raw)
In-Reply-To: <CANpmjNM6bQpc49teN-9qQhCXoJXaek5stFGR2kPwDroSFBc0fw@mail.gmail.com>
Hi Marco,
On 21.04.2021 13:03, Marco Elver wrote:
> On Wed, 21 Apr 2021 at 12:57, Marek Szyprowski <m.szyprowski@samsung.com> wrote:
>> On 21.04.2021 11:35, Marek Szyprowski wrote:
>>> On 21.04.2021 10:11, Marco Elver wrote:
>>>> On Wed, 21 Apr 2021 at 09:35, Marek Szyprowski
>>>> <m.szyprowski@samsung.com> wrote:
>>>>> On 21.04.2021 08:21, Marek Szyprowski wrote:
>>>>>> On 21.04.2021 00:42, Marco Elver wrote:
>>>>>>> On Tue, 20 Apr 2021 at 23:26, Marek Szyprowski
>>>>>>> <m.szyprowski@samsung.com> wrote:
>>>>>>>> On 08.04.2021 12:36, Marco Elver wrote:
>>>>>>>>> Introduces the TRAP_PERF si_code, and associated siginfo_t field
>>>>>>>>> si_perf. These will be used by the perf event subsystem to send
>>>>>>>>> signals
>>>>>>>>> (if requested) to the task where an event occurred.
>>>>>>>>>
>>>>>>>>> Acked-by: Geert Uytterhoeven <geert@linux-m68k.org> # m68k
>>>>>>>>> Acked-by: Arnd Bergmann <arnd@arndb.de> # asm-generic
>>>>>>>>> Signed-off-by: Marco Elver <elver@google.com>
>>>>>>>> This patch landed in linux-next as commit fb6cc127e0b6 ("signal:
>>>>>>>> Introduce TRAP_PERF si_code and si_perf to siginfo"). It causes
>>>>>>>> regression on my test systems (arm 32bit and 64bit). Most systems
>>>>>>>> fails
>>>>>>>> to boot in the given time frame. I've observed that there is a
>>>>>>>> timeout
>>>>>>>> waiting for udev to populate /dev and then also during the network
>>>>>>>> interfaces configuration. Reverting this commit, together with
>>>>>>>> 97ba62b27867 ("perf: Add support for SIGTRAP on perf events") to
>>>>>>>> let it
>>>>>>>> compile, on top of next-20210420 fixes the issue.
>>>>>>> Thanks, this is weird for sure and nothing in particular stands out.
>>>>>>>
>>>>>>> I have questions:
>>>>>>> -- Can you please share your config?
>>>>>> This happens with standard multi_v7_defconfig (arm) or just defconfig
>>>>>> for arm64.
>>>>>>
>>>>>>> -- Also, can you share how you run this? Can it be reproduced in
>>>>>>> qemu?
>>>>>> Nothing special. I just boot my test systems and see that they are
>>>>>> waiting lots of time during the udev populating /dev and network
>>>>>> interfaces configuration. I didn't try with qemu yet.
>>>>>>> -- How did you derive this patch to be at fault? Why not just
>>>>>>> 97ba62b27867, given you also need to revert it?
>>>>>> Well, I've just run my boot tests with automated 'git bisect' and that
>>>>>> was its result. It was a bit late in the evening, so I didn't analyze
>>>>>> it further, I've just posted a report about the issue I've found. It
>>>>>> looks that bisecting pointed to a wrong commit somehow.
>>>>>>> If you are unsure which patch exactly it is, can you try just
>>>>>>> reverting 97ba62b27867 and see what happens?
>>>>>> Indeed, this is a real faulty commit. Initially I've decided to revert
>>>>>> it to let kernel compile (it uses some symbols introduced by this
>>>>>> commit). Reverting only it on top of linux-next 20210420 also fixes
>>>>>> the issue. I'm sorry for the noise in this thread. I hope we will find
>>>>>> what really causes the issue.
>>>>> This was a premature conclusion. It looks that during the test I've did
>>>>> while writing that reply, the modules were not deployed properly and a
>>>>> test board (RPi4) booted without modules. In that case the board booted
>>>>> fine and there was no udev timeout. After deploying kernel modules, the
>>>>> udev timeout is back.
>>>> I'm confused now. Can you confirm that the problem is due to your
>>>> kernel modules, or do you think it's still due to 97ba62b27867? Or
>>>> fb6cc127e0b6 (this patch)?
>>> I don't use any custom kernel modules. I just deploy all modules that
>>> are being built from the given kernel defconfig (arm
>>> multi_v7_defconfig or arm64 default) and they are automatically loaded
>>> during the boot by udev. I've checked again and bisect was right. The
>>> kernel built from fb6cc127e0b6 suffers from the described issue, while
>>> the one build from the previous commit (2e498d0a74e5) works fine.
>> I've managed to reproduce this issue with qemu. I've compiled the kernel
>> for arm 32bit with multi_v7_defconfig and used some older Debian rootfs
>> image. The log and qemu parameters are here:
>> https://protect2.fireeye.com/v1/url?k=7cfc23a2-23671aa9-7cfda8ed-002590f5b904-dab7e2ec39dae1f9&q=1&e=36a5ed13-6ad5-430c-8f44-e95c4f0af5c3&u=https%3A%2F%2Fpaste.debian.net%2F1194526%2F
>>
>> Check the timestamp for the 'EXT4-fs (vda): re-mounted' message and
>> 'done (timeout)' status for the 'Waiting for /dev to be fully populated'
>> message. This happens only when kernel modules build from the
>> multi_v7_defconfig are deployed on the rootfs.
> Still hard to say what is going on and what is at fault. But being
> able to repro this in qemu helps debug quicker -- would you also be
> able to share the precise rootfs.img, i.e. upload it somewhere I can
> fetch it? And just to be sure, please also share your .config, as it
> might have compiler-version dependent configuration that might help
> repro (unlikely, but you never know).
I've managed to reproduce this issue with a public Raspberry Pi OS Lite
rootfs image, even without deploying kernel modules:
https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2021-03-25/2021-03-04-raspios-buster-armhf-lite.zip
# qemu-system-arm -M virt -smp 2 -m 512 -kernel zImage -append "earlycon
console=ttyAMA0 root=/dev/vda2 rw rootwait" -serial stdio -display none
-monitor null -device virtio-blk-device,drive=virtio-blk -drive
file=/tmp/2021-03-04-raspios-buster-armhf-lite.img,id=virtio-blk,if=none,format=raw
-netdev user,id=user -device virtio-net-device,netdev=user
The above one doesn't boot if zImage z compiled from commit fb6cc127e0b6
and boots if compiled from 2e498d0a74e5. In both cases I've used default
arm/multi_v7_defconfig and
gcc-linaro-6.4.1-2017.11-x86_64_arm-linux-gnueabi toolchain.
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
next prev parent reply other threads:[~2021-04-21 13:22 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-08 10:35 [PATCH v4 00/10] Add support for synchronous signals on perf events Marco Elver
2021-04-08 10:35 ` [PATCH v4 01/10] perf: Rework perf_event_exit_event() Marco Elver
2021-04-08 10:35 ` [PATCH v4 02/10] perf: Apply PERF_EVENT_IOC_MODIFY_ATTRIBUTES to children Marco Elver
2021-04-08 10:35 ` [PATCH v4 03/10] perf: Support only inheriting events if cloned with CLONE_THREAD Marco Elver
2021-04-08 10:35 ` [PATCH v4 04/10] perf: Add support for event removal on exec Marco Elver
2021-04-08 10:36 ` [PATCH v4 05/10] signal: Introduce TRAP_PERF si_code and si_perf to siginfo Marco Elver
[not found] ` <CGME20210420212618eucas1p102b427d1af9c682217dfe093f3eac3e8@eucas1p1.samsung.com>
2021-04-20 21:26 ` Marek Szyprowski
2021-04-20 22:42 ` Marco Elver
2021-04-21 6:21 ` Marek Szyprowski
2021-04-21 7:35 ` Marek Szyprowski
2021-04-21 8:11 ` Marco Elver
2021-04-21 9:35 ` Marek Szyprowski
2021-04-21 10:57 ` Marek Szyprowski
2021-04-21 11:03 ` Marco Elver
2021-04-21 13:19 ` Marek Szyprowski [this message]
2021-04-21 15:11 ` Marco Elver
2021-04-21 16:27 ` Marco Elver
2021-04-21 18:23 ` Marco Elver
2021-04-22 6:12 ` Marek Szyprowski
2021-04-22 6:47 ` Marco Elver
2021-04-22 8:16 ` Jon Hunter
2021-04-26 7:35 ` Alexander Egorenkov
2021-04-21 15:07 ` Jon Hunter
2021-04-08 10:36 ` [PATCH v4 06/10] perf: Add support for SIGTRAP on perf events Marco Elver
2021-04-08 10:36 ` [PATCH v4 07/10] selftests/perf_events: Add kselftest for process-wide sigtrap handling Marco Elver
2021-04-08 10:36 ` [PATCH v4 08/10] selftests/perf_events: Add kselftest for remove_on_exec Marco Elver
2021-04-08 10:36 ` [PATCH v4 09/10] tools headers uapi: Sync tools/include/uapi/linux/perf_event.h Marco Elver
2021-04-08 10:36 ` [PATCH v4 10/10] perf test: Add basic stress test for sigtrap handling Marco Elver
2021-04-14 8:37 ` [PATCH v4 00/10] Add support for synchronous signals on perf events Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cf6ed5cd-3202-65ce-86bc-6f1eba1b7d17@samsung.com \
--to=m.szyprowski@samsung.com \
--cc=acme@kernel.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=arnd@arndb.de \
--cc=axboe@kernel.dk \
--cc=b.zolnierkie@samsung.com \
--cc=christian@brauner.io \
--cc=dvyukov@google.com \
--cc=elver@google.com \
--cc=geert@linux-m68k.org \
--cc=glider@google.com \
--cc=irogers@google.com \
--cc=jannh@google.com \
--cc=jolsa@redhat.com \
--cc=kasan-dev@googlegroups.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mascasa@google.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=oleg@redhat.com \
--cc=pcc@google.com \
--cc=peterz@infradead.org \
--cc=tglx@linutronix.de \
--cc=viro@zeniv.linux.org.uk \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).