All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ard Biesheuvel <ardb@kernel.org>
To: Arnd Bergmann <arnd@arndb.de>
Cc: Yegor Yefremov <yegorslists@googlemail.com>,
	Tony Lindgren <tony@atomide.com>,
	 Linux-OMAP <linux-omap@vger.kernel.org>,
	linux-clk <linux-clk@vger.kernel.org>,
	 Stephen Boyd <sboyd@kernel.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: am335x: 5.18.x: system stalling
Date: Mon, 30 May 2022 17:14:58 +0200	[thread overview]
Message-ID: <CAMj1kXEvxP8ULqy7ajT_cSxMzYLJuCjEZGfYBb=F9qOwz-AFaQ@mail.gmail.com> (raw)
In-Reply-To: <CAK8P3a02uFq4edc_VzPaNQXp_cuLXUMbF4c=k6KATApS9hNHkw@mail.gmail.com>

On Mon, 30 May 2022 at 15:54, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Sat, May 28, 2022 at 9:28 PM Yegor Yefremov
> <yegorslists@googlemail.com> wrote:
> >
> > On Sat, May 28, 2022 at 3:14 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > >
> > > On Sat, May 28, 2022 at 3:01 PM Yegor Yefremov
> > > <yegorslists@googlemail.com> wrote:
> > > > On Sat, May 28, 2022 at 11:07 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > In file included from ./include/linux/irqflags.h:17,
> > > >                  from ./arch/arm/include/asm/bitops.h:28,
> > > >                  from ./include/linux/bitops.h:33,
> > > >                  from ./include/linux/log2.h:12,
> > > >                  from kernel/bounds.c:13:
> > > > ./arch/arm/include/asm/percpu.h: In function ‘__my_cpu_offset’:
> > > > ./arch/arm/include/asm/percpu.h:32:9: error: ‘__per_cpu_offset’
> > > > undeclared (first use in this function); did you mean
> > > > ‘__my_cpu_offset’?
> > > >    32 |  return __per_cpu_offset[0];
> > > >       |         ^~~~~~~~~~~~~~~~
> > > >       |         __my_cpu_offset
> > > > ./arch/arm/include/asm/percpu.h:32:9: note: each undeclared identifier
> > > > is reported only once for each function it appears in
> > >
> > > I think you just missed the line in my patch that adds the
> > > "extern unsigned long __per_cpu_offset[];" variable declaration.
> >
> > So, I tried both variants and both led to stalls.
>
> I'm running out of ideas here.  Going to back to the original bisection,
> I rebased Ard's patches in a way that you should be able to build the
> config for each patch, and I split up the "ARM: implement
> THREAD_INFO_IN_TASK for uniprocessor systems" commit in yet
> another way, hoping to get something left over that points to the
> bug. Can you try bisecting through the top commits of
>
> https://kernel.org/pub/scm/linux/kernel/git/soc/soc.git am335x-stall-test
>
> starting maybe with "52d240871760 irqchip: nvic: Use
> GENERIC_IRQ_MULTI_HANDLER" as the patch that is almost certainly
> going to be ok?
>
> At some point I fear we may have to give up and just mark the v6+SMP
> configuration as broken, which is something we have considered in the
> past but ended up always keeping around for the purpose of testing
> omap2plus_defconfig and imx_v6_v7_defconfig. Note that on production
> systems you probably don't want to use that config anway, and should
> either stick to a uniprocessor build, or disable the ARMv6 support.
>

Yeah, I am also running out of ideas. One question, though: does the
RCU detected stall always occur in the same place? I.e., how similar
are the backtraces of the stalls between different occurrences?
Perhaps we could narrow down where in the code we are stalling, and
gain some more understanding of the root cause.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

WARNING: multiple messages have this Message-ID (diff)
From: Ard Biesheuvel <ardb@kernel.org>
To: Arnd Bergmann <arnd@arndb.de>
Cc: Yegor Yefremov <yegorslists@googlemail.com>,
	Tony Lindgren <tony@atomide.com>,
	Linux-OMAP <linux-omap@vger.kernel.org>,
	linux-clk <linux-clk@vger.kernel.org>,
	Stephen Boyd <sboyd@kernel.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: am335x: 5.18.x: system stalling
Date: Mon, 30 May 2022 17:14:58 +0200	[thread overview]
Message-ID: <CAMj1kXEvxP8ULqy7ajT_cSxMzYLJuCjEZGfYBb=F9qOwz-AFaQ@mail.gmail.com> (raw)
In-Reply-To: <CAK8P3a02uFq4edc_VzPaNQXp_cuLXUMbF4c=k6KATApS9hNHkw@mail.gmail.com>

On Mon, 30 May 2022 at 15:54, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Sat, May 28, 2022 at 9:28 PM Yegor Yefremov
> <yegorslists@googlemail.com> wrote:
> >
> > On Sat, May 28, 2022 at 3:14 PM Arnd Bergmann <arnd@arndb.de> wrote:
> > >
> > > On Sat, May 28, 2022 at 3:01 PM Yegor Yefremov
> > > <yegorslists@googlemail.com> wrote:
> > > > On Sat, May 28, 2022 at 11:07 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > In file included from ./include/linux/irqflags.h:17,
> > > >                  from ./arch/arm/include/asm/bitops.h:28,
> > > >                  from ./include/linux/bitops.h:33,
> > > >                  from ./include/linux/log2.h:12,
> > > >                  from kernel/bounds.c:13:
> > > > ./arch/arm/include/asm/percpu.h: In function ‘__my_cpu_offset’:
> > > > ./arch/arm/include/asm/percpu.h:32:9: error: ‘__per_cpu_offset’
> > > > undeclared (first use in this function); did you mean
> > > > ‘__my_cpu_offset’?
> > > >    32 |  return __per_cpu_offset[0];
> > > >       |         ^~~~~~~~~~~~~~~~
> > > >       |         __my_cpu_offset
> > > > ./arch/arm/include/asm/percpu.h:32:9: note: each undeclared identifier
> > > > is reported only once for each function it appears in
> > >
> > > I think you just missed the line in my patch that adds the
> > > "extern unsigned long __per_cpu_offset[];" variable declaration.
> >
> > So, I tried both variants and both led to stalls.
>
> I'm running out of ideas here.  Going to back to the original bisection,
> I rebased Ard's patches in a way that you should be able to build the
> config for each patch, and I split up the "ARM: implement
> THREAD_INFO_IN_TASK for uniprocessor systems" commit in yet
> another way, hoping to get something left over that points to the
> bug. Can you try bisecting through the top commits of
>
> https://kernel.org/pub/scm/linux/kernel/git/soc/soc.git am335x-stall-test
>
> starting maybe with "52d240871760 irqchip: nvic: Use
> GENERIC_IRQ_MULTI_HANDLER" as the patch that is almost certainly
> going to be ok?
>
> At some point I fear we may have to give up and just mark the v6+SMP
> configuration as broken, which is something we have considered in the
> past but ended up always keeping around for the purpose of testing
> omap2plus_defconfig and imx_v6_v7_defconfig. Note that on production
> systems you probably don't want to use that config anway, and should
> either stick to a uniprocessor build, or disable the ARMv6 support.
>

Yeah, I am also running out of ideas. One question, though: does the
RCU detected stall always occur in the same place? I.e., how similar
are the backtraces of the stalls between different occurrences?
Perhaps we could narrow down where in the code we are stalling, and
gain some more understanding of the root cause.

  reply	other threads:[~2022-05-30 15:16 UTC|newest]

Thread overview: 115+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-05-04 10:35 am335x: 5.18.x: system stalling Yegor Yefremov
2022-05-05  5:08 ` Tony Lindgren
2022-05-11 14:16   ` Yegor Yefremov
2022-05-12  5:41     ` Tony Lindgren
2022-05-12  5:41       ` Tony Lindgren
2022-05-12  8:14       ` Arnd Bergmann
2022-05-12  8:14         ` Arnd Bergmann
2022-05-12  8:42       ` Arnd Bergmann
2022-05-12  8:42         ` Arnd Bergmann
2022-05-12 10:20         ` Yegor Yefremov
2022-05-12 10:20           ` Yegor Yefremov
2022-05-19 16:52           ` Yegor Yefremov
2022-05-19 16:52             ` Yegor Yefremov
2022-05-21 19:41             ` Arnd Bergmann
2022-05-21 19:41               ` Arnd Bergmann
2022-05-24 13:38               ` Yegor Yefremov
2022-05-24 13:38                 ` Yegor Yefremov
2022-05-24 14:19                 ` Tony Lindgren
2022-05-24 14:19                   ` Tony Lindgren
2022-05-26  5:49                   ` Yegor Yefremov
2022-05-26  5:49                     ` Yegor Yefremov
2022-05-26  6:20                     ` Tony Lindgren
2022-05-26  6:20                       ` Tony Lindgren
2022-05-26  8:19                       ` Ard Biesheuvel
2022-05-26  8:19                         ` Ard Biesheuvel
2022-05-26 12:37                         ` Yegor Yefremov
2022-05-26 12:37                           ` Yegor Yefremov
2022-05-26 14:15                           ` Arnd Bergmann
2022-05-26 14:15                             ` Arnd Bergmann
2022-05-27  4:44                             ` Yegor Yefremov
2022-05-27  4:44                               ` Yegor Yefremov
2022-05-27  6:38                               ` Arnd Bergmann
2022-05-27  6:38                                 ` Arnd Bergmann
2022-05-27  6:50                                 ` Tony Lindgren
2022-05-27  6:50                                   ` Tony Lindgren
2022-05-27  6:57                                   ` Arnd Bergmann
2022-05-27  6:57                                     ` Arnd Bergmann
2022-05-27  8:17                                     ` Yegor Yefremov
2022-05-27  8:17                                       ` Yegor Yefremov
2022-05-27  8:38                                       ` Arnd Bergmann
2022-05-27  8:38                                         ` Arnd Bergmann
2022-05-27  9:50                                         ` Yegor Yefremov
2022-05-27  9:50                                           ` Yegor Yefremov
2022-05-27 12:53                                           ` Arnd Bergmann
2022-05-27 12:53                                             ` Arnd Bergmann
2022-05-27 13:12                                             ` Ard Biesheuvel
2022-05-27 13:12                                               ` Ard Biesheuvel
2022-05-27 14:12                                               ` Arnd Bergmann
2022-05-27 14:12                                                 ` Arnd Bergmann
2022-05-28  5:48                                                 ` Yegor Yefremov
2022-05-28  5:48                                                   ` Yegor Yefremov
2022-05-28  7:53                                                   ` Arnd Bergmann
2022-05-28  7:53                                                     ` Arnd Bergmann
2022-05-28  8:29                                                     ` Yegor Yefremov
2022-05-28  8:29                                                       ` Yegor Yefremov
2022-05-28  9:07                                                       ` Ard Biesheuvel
2022-05-28  9:07                                                         ` Ard Biesheuvel
2022-05-28 13:01                                                         ` Yegor Yefremov
2022-05-28 13:01                                                           ` Yegor Yefremov
2022-05-28 13:13                                                           ` Arnd Bergmann
2022-05-28 13:13                                                             ` Arnd Bergmann
2022-05-28 19:28                                                             ` Yegor Yefremov
2022-05-28 19:28                                                               ` Yegor Yefremov
2022-05-30 10:16                                                               ` Ard Biesheuvel
2022-05-30 10:16                                                                 ` Ard Biesheuvel
2022-05-30 12:09                                                                 ` Yegor Yefremov
2022-05-30 12:09                                                                   ` Yegor Yefremov
2022-05-30 13:54                                                               ` Arnd Bergmann
2022-05-30 13:54                                                                 ` Arnd Bergmann
2022-05-30 15:14                                                                 ` Ard Biesheuvel [this message]
2022-05-30 15:14                                                                   ` Ard Biesheuvel
2022-05-31  8:36                                                                   ` Yegor Yefremov
2022-05-31  8:36                                                                     ` Yegor Yefremov
2022-05-31 14:16                                                                     ` Yegor Yefremov
2022-05-31 14:16                                                                       ` Yegor Yefremov
2022-05-31 15:22                                                                       ` Arnd Bergmann
2022-05-31 15:22                                                                         ` Arnd Bergmann
2022-06-01  7:36                                                                         ` Yegor Yefremov
2022-06-01  7:36                                                                           ` Yegor Yefremov
2022-06-01  7:59                                                                           ` Arnd Bergmann
2022-06-01  7:59                                                                             ` Arnd Bergmann
2022-06-01  8:08                                                                             ` Ard Biesheuvel
2022-06-01  8:08                                                                               ` Ard Biesheuvel
2022-06-01  9:27                                                                               ` Ard Biesheuvel
2022-06-01  9:27                                                                                 ` Ard Biesheuvel
2022-06-01 10:03                                                                                 ` Yegor Yefremov
2022-06-01 10:03                                                                                   ` Yegor Yefremov
2022-06-01 10:06                                                                                   ` Ard Biesheuvel
2022-06-01 10:06                                                                                     ` Ard Biesheuvel
2022-06-01 10:46                                                                                     ` Yegor Yefremov
2022-06-01 10:46                                                                                       ` Yegor Yefremov
2022-06-01 10:49                                                                                       ` Ard Biesheuvel
2022-06-01 10:49                                                                                         ` Ard Biesheuvel
2022-06-02 10:17                                                                                         ` Yegor Yefremov
2022-06-02 10:17                                                                                           ` Yegor Yefremov
2022-06-02 10:37                                                                                           ` Ard Biesheuvel
2022-06-02 10:37                                                                                             ` Ard Biesheuvel
2022-06-02 12:27                                                                                             ` Yegor Yefremov
2022-06-02 12:27                                                                                               ` Yegor Yefremov
2022-06-03  8:54                                                                                               ` Yegor Yefremov
2022-06-03  8:54                                                                                                 ` Yegor Yefremov
2022-06-03  9:32                                                                                                 ` Arnd Bergmann
2022-06-03  9:32                                                                                                   ` Arnd Bergmann
2022-06-03 19:11                                                                                                   ` Yegor Yefremov
2022-06-03 19:11                                                                                                     ` Yegor Yefremov
2022-06-03 20:46                                                                                                     ` Arnd Bergmann
2022-06-03 20:46                                                                                                       ` Arnd Bergmann
2022-06-05 14:59                                                                                                       ` Ard Biesheuvel
2022-06-05 14:59                                                                                                         ` Ard Biesheuvel
2022-06-07  8:55                                                                                                         ` Yegor Yefremov
2022-06-07  8:55                                                                                                           ` Yegor Yefremov
2022-08-12  7:35                                                                                                           ` Arnd Bergmann
2022-08-12  7:35                                                                                                             ` Arnd Bergmann
2022-05-24 14:36                 ` Arnd Bergmann
2022-05-24 14:36                   ` Arnd Bergmann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAMj1kXEvxP8ULqy7ajT_cSxMzYLJuCjEZGfYBb=F9qOwz-AFaQ@mail.gmail.com' \
    --to=ardb@kernel.org \
    --cc=arnd@arndb.de \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-clk@vger.kernel.org \
    --cc=linux-omap@vger.kernel.org \
    --cc=sboyd@kernel.org \
    --cc=tony@atomide.com \
    --cc=yegorslists@googlemail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.