All of lore.kernel.org
 help / color / mirror / Atom feed
From: Alexander Egorenkov <egorenar@linux.ibm.com>
To: Luis Chamberlain <mcgrof@kernel.org>,
	Bruno Goncalves <bgoncalv@redhat.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>,
	akpm@linux-foundation.org, bp@alien8.de, corbet@lwn.net,
	gregkh@linuxfoundation.org, jeyu@kernel.org,
	linux-kernel@vger.kernel.org,
	Nick Desaulniers <ndesaulniers@google.com>,
	torvalds@linux-foundation.org, Dave Young <dyoung@redhat.com>
Subject: Re: [PATCH v3 1/2] init/initramfs.c: do unpacking asynchronously
Date: Wed, 28 Jul 2021 12:44:14 +0200	[thread overview]
Message-ID: <87mtq6g1kx.fsf@oc8242746057.ibm.com> (raw)
In-Reply-To: <YQAbZhCwLS7vb/ag@bombadil.infradead.org>
In-Reply-To: 

Luis Chamberlain <mcgrof@kernel.org> writes:

> On Tue, Jul 27, 2021 at 04:27:08PM +0200, Bruno Goncalves wrote:
>> On Tue, Jul 27, 2021 at 4:21 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>> >
>> > On Tue, Jul 27, 2021 at 04:12:54PM +0200, Bruno Goncalves wrote:
>> > > On Tue, Jul 27, 2021 at 3:55 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>> > > >
>> > > > On Tue, Jul 27, 2021 at 09:31:54AM +0200, Bruno Goncalves wrote:
>> > > > > On Mon, Jul 26, 2021 at 1:46 PM Rasmus Villemoes
>> > > > > <linux@rasmusvillemoes.dk> wrote:
>> > > > > >
>> > > > > > On 24/07/2021 09.46, Alexander Egorenkov wrote:
>> > > > > > > Hello,
>> > > > > > >
>> > > > > > > since e7cb072eb988 ("init/initramfs.c: do unpacking asynchronously"), we
>> > > > > > > started seeing the following problem on s390 arch regularly:
>> > > > > > >
>> > > > > > > [    5.039734] wait_for_initramfs() called before rootfs_initcalls
>> > > >
>> > > > So some context here, which might help.
>> > > >
>> > > > The initramfs_cookie is initialized until a a rootfs_initcall() is
>> > > > called, in this case populate_rootfs(). The code is small, so might
>> > > > as well include it:
>> > > >
>> > > > static int __init populate_rootfs(void)
>> > > > {
>> > > >         initramfs_cookie = async_schedule_domain(do_populate_rootfs, NULL,
>> > > >                                                  &initramfs_domain);
>> > > >         if (!initramfs_async)
>> > > >                 wait_for_initramfs();
>> > > >         return 0;
>> > > > }
>> > > > rootfs_initcall(populate_rootfs);
>> > > >
>> > > > The warning you see comes from a situation where a wait_for_initramfs()
>> > > > gets called but we haven't yet initialized initramfs_cookie.  There are
>> > > > only a few calls for wait_for_initramfs() in the kernel, and the only
>> > > > thing I can think of is that somehow s390 may rely on a usermode helper
>> > > > early on, but not every time.
>> > > >
>> > > > What umh calls does s390 issue?
>> > > >
>> > > > > Unfortunately, we haven't been able to find the root cause, but since
>> > > > > June 23rd we haven't hit this panic...
>> > > > >
>> > > > > Btw, this panic we were hitting only when testing kernels from "scsi"
>> > > > > and "block" trees.
>> > > >
>> > > > Do you use drdb maybe?
>> > >
>> > > No, the machines we were able to reproduce the problem don't have drdb.
>> >
>> > Are there *any* umh calls early on boot on the s390 systems? If so
>> > chances are that is the droid you are looking for.
>> 
>> Sorry Luis,
>> 
>> I was just replying the question mentioning an old thread
>> (https://lore.kernel.org/lkml/CA+QYu4qxf2CYe2gC6EYnOHXPKS-+cEXL=MnUvqRFaN7W1i6ahQ@mail.gmail.com/T/#u)
>> on ppc64le.
>> 
>> regarding the "umh" it doesn't show anything on ppc64le boot.
>
> There is not a single pr_*() call on kernel/umh.c, and so unless the
> respective ppc64le / s390 umh callers have a print, we won't know if you
> really did use a print.

I instrumented the UMH code and it seems that all wait_for_initramfs()
are triggered by request_module() from drbg.

>
> Can you reproduce the failure? How often?
>
>   Luis

The failure can be reproduced almost daily but on only one special test
machine and not immediately but after running many tests. I instrumented
our devel kernel in order to find out when/how the initramfs is being corrupted.

Still not reproducible on my own test machine. Very weird.

I'll report back as soon as we have something tangible.

Regards
Alex

  parent reply	other threads:[~2021-07-28 10:44 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-13 21:25 [PATCH v3 0/2] background initramfs unpacking, and CONFIG_MODPROBE_PATH Rasmus Villemoes
2021-03-13 21:25 ` [PATCH v3 1/2] init/initramfs.c: do unpacking asynchronously Rasmus Villemoes
2021-03-15 20:21   ` Luis Chamberlain
2021-03-15 21:33   ` Andrew Morton
2021-03-15 21:59     ` Rasmus Villemoes
2021-07-24  7:46   ` Alexander Egorenkov
2021-07-26 11:46     ` Rasmus Villemoes
2021-07-27  7:31       ` Bruno Goncalves
2021-07-27 13:54         ` Luis Chamberlain
2021-07-27 14:12           ` Bruno Goncalves
2021-07-27 14:21             ` Luis Chamberlain
2021-07-27 14:27               ` Bruno Goncalves
2021-07-27 14:42                 ` Luis Chamberlain
2021-07-27 14:48                   ` Bruno Goncalves
2021-07-28 10:44                   ` Alexander Egorenkov [this message]
2021-07-28 10:38           ` Alexander Egorenkov
2021-07-28 10:36       ` Alexander Egorenkov
2021-07-28 11:49         ` Rasmus Villemoes
2021-03-13 21:25 ` [PATCH v3 2/2] modules: add CONFIG_MODPROBE_PATH Rasmus Villemoes
2021-03-15 20:06   ` Luis Chamberlain
2021-03-15 20:23 ` [PATCH v3 0/2] background initramfs unpacking, and CONFIG_MODPROBE_PATH Luis Chamberlain

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87mtq6g1kx.fsf@oc8242746057.ibm.com \
    --to=egorenar@linux.ibm.com \
    --cc=akpm@linux-foundation.org \
    --cc=bgoncalv@redhat.com \
    --cc=bp@alien8.de \
    --cc=corbet@lwn.net \
    --cc=dyoung@redhat.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jeyu@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@rasmusvillemoes.dk \
    --cc=mcgrof@kernel.org \
    --cc=ndesaulniers@google.com \
    --cc=torvalds@linux-foundation.org \
    --subject='Re: [PATCH v3 1/2] init/initramfs.c: do unpacking asynchronously' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.