From: Joel Fernandes <joel@joelfernandes.org>
To: Guenter Roeck <linux@roeck-us.net>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
stable@vger.kernel.org, patches@lists.linux.dev,
linux-kernel@vger.kernel.org, torvalds@linux-foundation.org,
akpm@linux-foundation.org, shuah@kernel.org,
patches@kernelci.org, lkft-triage@lists.linaro.org,
pavel@denx.de, jonathanh@nvidia.com, f.fainelli@gmail.com,
sudipm.mukherjee@gmail.com, srw@sladewatkins.net, rwarsow@gmx.de,
conor@kernel.org, paulmck@kernel.org
Subject: Re: [PATCH 5.15 00/92] 5.15.126-rc1 review
Date: Wed, 9 Aug 2023 14:39:37 -0400 [thread overview]
Message-ID: <CAEXW_YRi8cW+jHad8Dm_XTL+-OX7-S0X9TbZOyeOL=Tu9sV_4Q@mail.gmail.com> (raw)
In-Reply-To: <CAEXW_YQ4GqPwvUF8=8CWmdj=cD56v_eEVK-EirsObQXyBDFVpg@mail.gmail.com>
On Wed, Aug 9, 2023 at 2:35 PM Joel Fernandes <joel@joelfernandes.org> wrote:
>
> On Wed, Aug 9, 2023 at 12:18 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >
> > On 8/9/23 06:53, Joel Fernandes wrote:
> > > On Wed, Aug 09, 2023 at 12:40:36PM +0200, Greg Kroah-Hartman wrote:
> > >> This is the start of the stable review cycle for the 5.15.126 release.
> > >> There are 92 patches in this series, all will be posted as a response
> > >> to this one. If anyone has any issues with these being applied, please
> > >> let me know.
> > >>
> > >> Responses should be made by Fri, 11 Aug 2023 10:36:10 +0000.
> > >> Anything received after that time might be too late.
> > >>
> > >> The whole patch series can be found in one patch at:
> > >> https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.15.126-rc1.gz
> > >> or in the git tree and branch at:
> > >> git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.15.y
> > >> and the diffstat can be found below.
> > >
> > > Not necesscarily new with 5.15 stable but 3 of the 19 rcutorture scenarios
> > > hang with this -rc: TREE04, TREE07, TASKS03.
> > >
> > > 5.15 has a known stop machine issue where it hangs after 1.5 hours with cpu
> > > hotplug rcutorture testing. Me and tglx are continuing to debug this. The
> > > issue does not show up on anything but 5.15 stable kernels and neither on
> > > mainline.
> > >
> >
> > Do you by any have a crash pattern that we could possibly use to find the crash
> > in ChromeOS crash logs ? No idea if that would help, but it could provide some
> > additional data points.
>
> The pattern shows as a hard hang, the system is unresponsive and all CPUs
> are stuck in stop_machine. Sometimes it recovers on its own from the
> hang and then RCU immediately gives stall warnings. It takes 1.5 hour
> to reproduce and sometimes never happens for several hours.
>
> It appears related to CPU hotplug since gdb showed me most of the CPUs
> are spinning in multi_cpu_stop() / stop machine after the hang.
>
Adding to this, it appears one of the CPUs is constantly firing and
reprogramming hrtimer events for some reason every few 100
microseconds (I see this in gdb). My debug angle right now is to
figure out why it does that but collecting a trace is hard as it
appears even trace collection may not be happening once hung and the
only traces I am getting are the ones after the hang recovers, not
during the hang. I am also trying to see if multi_cpu_stop() can
panic the kernel if it sits there too long.
- Joel
next prev parent reply other threads:[~2023-08-09 18:39 UTC|newest]
Thread overview: 121+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-09 10:40 [PATCH 5.15 00/92] 5.15.126-rc1 review Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 01/92] io_uring: gate iowait schedule on having pending requests Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 02/92] perf: Fix function pointer case Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 03/92] net/mlx5: Free irqs only on shutdown callback Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 04/92] arm64: errata: Add workaround for TSB flush failures Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 05/92] arm64: errata: Add detection for TRBE write to out-of-range Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 06/92] iommu/arm-smmu-v3: Work around MMU-600 erratum 1076982 Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 07/92] iommu/arm-smmu-v3: Document MMU-700 erratum 2812531 Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 08/92] iommu/arm-smmu-v3: Add explicit feature for nesting Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 09/92] iommu/arm-smmu-v3: Document nesting-related errata Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 10/92] arm64: dts: imx8mn-var-som: add missing pull-up for onboard PHY reset pinmux Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 11/92] firmware: arm_scmi: Fix chan_free cleanup on SMC Greg Kroah-Hartman
2023-08-09 19:33 ` Florian Fainelli
2023-08-10 8:45 ` Sudeep Holla
2023-08-09 10:40 ` [PATCH 5.15 12/92] word-at-a-time: use the same return type for has_zero regardless of endianness Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 13/92] KVM: s390: fix sthyi error handling Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 14/92] wifi: cfg80211: Fix return value in scan logic Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 15/92] net/mlx5: DR, fix memory leak in mlx5dr_cmd_create_reformat_ctx Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 16/92] net/mlx5e: fix return value check in mlx5e_ipsec_remove_trailer() Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 17/92] bpf: Add length check for SK_DIAG_BPF_STORAGE_REQ_MAP_FD parsing Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 18/92] rtnetlink: let rtnl_bridge_setlink checks IFLA_BRIDGE_MODE length Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 19/92] net: dsa: fix value check in bcm_sf2_sw_probe() Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 20/92] perf test uprobe_from_different_cu: Skip if there is no gcc Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 21/92] net: sched: cls_u32: Fix match key mis-addressing Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 22/92] mISDN: hfcpci: Fix potential deadlock on &hc->lock Greg Kroah-Hartman
2023-08-09 10:40 ` [PATCH 5.15 23/92] qed: Fix kernel-doc warnings Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 24/92] qed: Fix scheduling in a tasklet while getting stats Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 25/92] net: annotate data-races around sk->sk_max_pacing_rate Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 26/92] net: add missing READ_ONCE(sk->sk_rcvlowat) annotation Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 27/92] net: add missing READ_ONCE(sk->sk_sndbuf) annotation Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 28/92] net: add missing READ_ONCE(sk->sk_rcvbuf) annotation Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 29/92] net: add missing data-race annotations around sk->sk_peek_off Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 30/92] net: add missing data-race annotation for sk_ll_usec Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 31/92] net/sched: taprio: Limit TCA_TAPRIO_ATTR_SCHED_CYCLE_TIME to INT_MAX Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 32/92] bpf, cpumap: Handle skb as well when clean up ptr_ring Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 33/92] net/sched: cls_u32: No longer copy tcf_result on update to avoid use-after-free Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 34/92] net/sched: cls_fw: " Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 35/92] net/sched: cls_route: " Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 36/92] bpf: sockmap: Remove preempt_disable in sock_map_sk_acquire Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 37/92] net: ll_temac: Switch to use dev_err_probe() helper Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 38/92] net: ll_temac: fix error checking of irq_of_parse_and_map() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 39/92] net: korina: handle clk prepare error in korina_probe() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 40/92] net: netsec: Ignore phy-mode on SynQuacer in DT mode Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 41/92] net: dcb: choose correct policy to parse DCB_ATTR_BCN Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 42/92] s390/qeth: Dont call dev_close/dev_open (DOWN/UP) Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 43/92] ip6mr: Fix skb_under_panic in ip6mr_cache_report() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 44/92] vxlan: Fix nexthop hash size Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 45/92] net/mlx5: fs_core: Make find_closest_ft more generic Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 46/92] net/mlx5: fs_core: Skip the FTs in the same FS_TYPE_PRIO_CHAINS fs_prio Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 47/92] prestera: fix fallback to previous version on same major version Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 48/92] tcp_metrics: fix addr_same() helper Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 49/92] tcp_metrics: annotate data-races around tm->tcpm_stamp Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 50/92] tcp_metrics: annotate data-races around tm->tcpm_lock Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 51/92] tcp_metrics: annotate data-races around tm->tcpm_vals[] Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 52/92] tcp_metrics: annotate data-races around tm->tcpm_net Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 53/92] tcp_metrics: fix data-race in tcpm_suck_dst() vs fastopen Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 54/92] scsi: zfcp: Defer fc_rport blocking until after ADISC response Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 55/92] scsi: storvsc: Limit max_sectors for virtual Fibre Channel devices Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 56/92] libceph: fix potential hang in ceph_osdc_notify() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 57/92] USB: zaurus: Add ID for A-300/B-500/C-700 Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 58/92] ceph: defer stopping mdsc delayed_work Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 59/92] firmware: arm_scmi: Drop OF node reference in the transport channel setup Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 60/92] exfat: use kvmalloc_array/kvfree instead of kmalloc_array/kfree Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 61/92] exfat: release s_lock before calling dir_emit() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 62/92] mtd: spinand: toshiba: Fix ecc_get_status Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 63/92] mtd: rawnand: meson: fix OOB available bytes for ECC Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 64/92] arm64: dts: stratix10: fix incorrect I2C property for SCL signal Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 65/92] net: tun_chr_open(): set sk_uid from current_fsuid() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 66/92] net: tap_open(): " Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 67/92] wifi: mt76: mt7615: do not advertise 5 GHz on first phy of MT7615D (DBDC) Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 68/92] rbd: prevent busy loop when requesting exclusive lock Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 69/92] bpf: Disable preemption in bpf_event_output Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 70/92] open: make RESOLVE_CACHED correctly test for O_TMPFILE Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 71/92] drm/ttm: check null pointer before accessing when swapping Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 72/92] bpf, cpumap: Make sure kthread is running before map update returns Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 73/92] file: reinstate f_pos locking optimization for regular files Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 74/92] fs/ntfs3: Use __GFP_NOWARN allocation at ntfs_load_attr_list() Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 75/92] fs/sysv: Null check to prevent null-ptr-deref bug Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 76/92] Bluetooth: L2CAP: Fix use-after-free in l2cap_sock_ready_cb Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 77/92] net: usbnet: Fix WARNING in usbnet_start_xmit/usb_submit_urb Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 78/92] fs: Protect reconfiguration of sb read-write from racing writes Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 79/92] ext2: Drop fragment support Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 80/92] mtd: rawnand: omap_elm: Fix incorrect type in assignment Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 81/92] mtd: rawnand: rockchip: fix oobfree offset and description Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 82/92] mtd: rawnand: rockchip: Align hwecc vs. raw page helper layouts Greg Kroah-Hartman
2023-08-09 10:41 ` [PATCH 5.15 83/92] mtd: rawnand: fsl_upm: Fix an off-by one test in fun_exec_op() Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 84/92] powerpc/mm/altmap: Fix altmap boundary check Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 85/92] drm/fsl-dcu: Use drm_plane_helper_destroy() Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 86/92] drm/imx/ipuv3: Fix front porch adjustment upon hactive aligning Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 87/92] selftests/rseq: check if libc rseq support is registered Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 88/92] selftests/rseq: Play nice with binaries statically linked against glibc 2.35+ Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 89/92] soundwire: bus: pm_runtime_request_resume on peripheral attachment Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 90/92] soundwire: fix enumeration completion Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 91/92] PM / wakeirq: support enabling wake-up irq after runtime_suspend called Greg Kroah-Hartman
2023-08-09 10:42 ` [PATCH 5.15 92/92] PM: sleep: wakeirq: fix wake irq arming Greg Kroah-Hartman
2023-08-09 13:53 ` [PATCH 5.15 00/92] 5.15.126-rc1 review Joel Fernandes
2023-08-09 16:18 ` Guenter Roeck
2023-08-09 18:35 ` Joel Fernandes
2023-08-09 18:39 ` Joel Fernandes [this message]
2023-08-09 19:25 ` Guenter Roeck
2023-08-09 20:14 ` Joel Fernandes
2023-08-09 20:38 ` Guenter Roeck
2023-08-09 20:39 ` Joel Fernandes
2023-08-09 21:45 ` Guenter Roeck
2023-08-10 17:55 ` Paul E. McKenney
2023-08-10 21:54 ` Joel Fernandes
2023-08-10 22:14 ` Joel Fernandes
2023-08-10 22:34 ` Paul E. McKenney
2023-08-10 22:55 ` Guenter Roeck
2023-08-10 23:13 ` Joel Fernandes
2023-08-09 17:05 ` SeongJae Park
2023-08-09 19:38 ` Florian Fainelli
2023-08-10 10:16 ` Harshit Mogalapalli
2023-08-10 10:25 ` Guenter Roeck
2023-08-10 10:24 ` Guenter Roeck
2023-08-10 16:25 ` Florian Fainelli
2023-08-11 10:02 ` Greg Kroah-Hartman
2023-08-10 16:06 ` Guenter Roeck
2023-08-11 10:05 ` Greg Kroah-Hartman
2023-08-10 21:11 ` Ron Economos
2023-08-10 21:54 ` Daniel Díaz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAEXW_YRi8cW+jHad8Dm_XTL+-OX7-S0X9TbZOyeOL=Tu9sV_4Q@mail.gmail.com' \
--to=joel@joelfernandes.org \
--cc=akpm@linux-foundation.org \
--cc=conor@kernel.org \
--cc=f.fainelli@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=jonathanh@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@roeck-us.net \
--cc=lkft-triage@lists.linaro.org \
--cc=patches@kernelci.org \
--cc=patches@lists.linux.dev \
--cc=paulmck@kernel.org \
--cc=pavel@denx.de \
--cc=rwarsow@gmx.de \
--cc=shuah@kernel.org \
--cc=srw@sladewatkins.net \
--cc=stable@vger.kernel.org \
--cc=sudipm.mukherjee@gmail.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).