* Linux regressions report for mainline [2021-11-24]
@ 2021-11-24 9:25 Regzbot (on behalf of Thorsten Leemhuis)
2021-11-24 10:01 ` Thorsten Leemhuis
2021-11-24 18:13 ` Linus Torvalds
0 siblings, 2 replies; 6+ messages in thread
From: Regzbot (on behalf of Thorsten Leemhuis) @ 2021-11-24 9:25 UTC (permalink / raw)
To: LKML, Linus Torvalds, Linux regressions mailing list; +Cc: Regzbot
From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info>
Hi, this is regzbot, the Linux kernel regression tracking bot.
Currently I'm aware of 15 regressions in linux-mainline. Find the
current status below and the latest on the web:
https://linux-regtracking.leemhuis.info/regzbot/mainline/
Bye bye, hope to see you soon for the next report.
Regzbot (on behalf of Thorsten Leemhuis)
========================================================
current cycle (v5.15.. aka v5.16-rc), culprit identified
========================================================
Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years
-----------------------------------------------------------------------------------------------
https://lore.kernel.org/regressions/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com
By Niklas Schnelle, 4 days ago; 3 activities, latest 3 days ago.
Introduced in 32def4120e48 (v5.16-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com
=========================================================================================
previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months
=========================================================================================
CONFIG_SYSFB_SIMPLEFB breaks console scrolling
----------------------------------------------
https://lore.kernel.org/lkml/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de
By Harald Dunkel, 8 days ago; 1 activities, latest 8 days ago.
Introduced in 8633ef82f101 (v5.15-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de
==================================================================================
older cycles (..v5.14), culprit identified, with activity in the past three months
==================================================================================
wireless AP (Raspberry Pi with rt2x00usb) crashes every hour or so
------------------------------------------------------------------
https://lore.kernel.org/lkml/20211118132556.GD334428@darkstar.musicnaut.iki.fi
By Aaro Koskinen, 5 days ago; 5 activities, latest 1 days ago.
Introduced in 03c3911d2d67 (v5.14-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/20211118132556.GD334428@darkstar.musicnaut.iki.fi
Latest activity with a patch:
* [PATCH 5.16] mac80211: fix rate control for retransmitted frames
https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name
1 days ago, by Felix Fietkau; thread monitored.
Bluetooth: Query LE tx power on startup broke Bluetooth on MacBookPro16,1
-------------------------------------------------------------------------
https://lore.kernel.org/regressions/4970a940-211b-25d6-edab-21a815313954@protonmail.com
By Orlando Chamberlain, 56 days ago; 30 activities, latest 4 days ago.
Introduced in 7c395ea521e6 (v5.11-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/4970a940-211b-25d6-edab-21a815313954@protonmail.com
Latest activity with a patch:
* Re: [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power
https://lore.kernel.org/lkml/CABBYNZLjSfcG_KqTEbL6NOSvHhA5-b1t_S=3FQP4=GwW21kuzg@mail.gmail.com
18 days ago, by Luiz Augusto von Dentz
Noteworthy links:
* Re: [PATCH] Bluetooth: add quirk disabling query LE tx power
https://lore.kernel.org/lkml/43fb97ad-69eb-95ad-d50a-b8f1113dbee6@leemhuis.info
54 days ago, by Thorsten Leemhuis; thread monitored.
* [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power
https://lore.kernel.org/lkml/20211001083412.3078-1-redecorating@protonmail.com
54 days ago, by Orlando Chamberlain; thread monitored.
Re: rt2x00 regression
---------------------
https://lore.kernel.org/regressions/87czop5j33.fsf@tynnyri.adurom.net
By Kalle Valo, 54 days ago; 16 activities, latest 4 days ago.
Introduced in e383c70474db (v5.2-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/87czop5j33.fsf@tynnyri.adurom.net
Latest activity with a patch:
* [PATCH] rt2x00: do not mark device gone on EPROTO errors during start
https://lore.kernel.org/regressions/20211111141003.GA134627@wp.pl
12 days ago, by Stanislaw Gruszka
Noteworthy links:
* rt2x00 regression
https://lore.kernel.org/linux-wireless/bff7d309-a816-6a75-51b6-5928ef4f7a8c@exuvo.se
789 days ago, by Anton Olsson; thread monitored.
PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization
-----------------------------------------------------------------------------------
https://lore.kernel.org/linux-pci/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com
By Stéphane Graber, 5 days ago; 4 activities, latest 5 days ago.
Introduced in 6dce5aa59e0b (v5.5-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com
Latest activity with a patch:
* Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization
https://lore.kernel.org/linux-pci/CAL_JsqKrfpDtQZMMuhA_tURit6fO82FzPbKA40o6_8jWRewm8g@mail.gmail.com
5 days ago, by Rob Herring
mhi: ath11k resume fails on some devices
----------------------------------------
https://lore.kernel.org/regressions/871r5p0x2u.fsf@codeaurora.org
By Kalle Valo, 69 days ago; 23 activities, latest 5 days ago.
Introduced in 020d3b26c07a (v5.13-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/871r5p0x2u.fsf@codeaurora.org
bug: usb: gadget: FSL_UDC_CORE Corrupted request list leads to unrecoverable loop.
----------------------------------------------------------------------------------
https://lore.kernel.org/linuxppc-dev/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com
By Eugene Bordenkircher, 25 days ago; 7 activities, latest 7 days ago.
Introduced in f79a60b8785 (v3.4-rc4)
https://linux-regtracking.leemhuis.info/regzbot/regression/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com
Noteworthy links:
* https://lore.kernel.org/all/CADRPPNSrhiwr8jmBb2h4cFYqHtuDKK8rL0i6Bkg7+xEyXJPATA@mail.gmail.com/
22 days ago, by Thorsten Leemhuis
* https://lore.kernel.org/all/2c275adc278477e1e512ea6ecc0c1f4dcc46969d.camel@infinera.com/
22 days ago, by Thorsten Leemhuis
Bug in Memory Layout of rx_desc for QCA6174
-------------------------------------------
https://lore.kernel.org/ath10k/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com
By Francesco Magliocca, 159 days ago; 5 activities, latest 24 days ago.
Introduced in e3def6f7ddf8 (v4.16-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com
Noteworthy links:
* Bug in Memory Layout of rx_desc for QCA6174
https://lore.kernel.org/ath10k/CAH4F6uvX=xtTnBDaj1BVHSx_FDSUbpc4TRC2DGTHBmGJSD2oEA@mail.gmail.com
25 days ago, by Francesco Magliocca; thread monitored.
====================================================
current cycle (v5.15.. aka v5.16-rc), unkown culprit
====================================================
mm: reclaim_throttle leads to stall in near-OOM conditions
----------------------------------------------------------
https://lore.kernel.org/lkml/20211124011954.7cab9bb4@mail.inbox.lv
By Alexey Avramov, 0 days ago; 1 activities, latest 0 days ago.
Introduced in v5.15..v5.16-rc1
https://linux-regtracking.leemhuis.info/regzbot/regression/20211124011954.7cab9bb4@mail.inbox.lv
mm: LTP/memcg testcase regression induced by 8cd7c588decf..66ce520bb7c2 series
------------------------------------------------------------------------------
https://lore.kernel.org/lkml/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de
By Mike Galbraith, 2 days ago; 8 activities, latest 0 days ago.
Introduced in 8cd7c588decf..66ce520bb7c2 (v5.15..v5.16-rc1)
https://linux-regtracking.leemhuis.info/regzbot/regression/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de
====================================================================================
previous cycle (v5.14..v5.15), unkown culprit, with activity in the past three weeks
====================================================================================
Kernel 5.15 reboots / freezes upon ifup/ifdown
----------------------------------------------
https://lore.kernel.org/stable/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de
By Stefan Dietrich, 0 days ago; 5 activities, latest 0 days ago.
Introduced in v5.14..v5.15
https://linux-regtracking.leemhuis.info/regzbot/regression/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de
kernel 5.15.1: AMD RX 6700 XT - Fails to resume after screen blank
------------------------------------------------------------------
https://lore.kernel.org/stable/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk
By Mark Boddington, 13 days ago; 7 activities, latest 11 days ago.
Introduced in v5.14..v5.15
https://linux-regtracking.leemhuis.info/regzbot/regression/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk
=============================================================================
older cycles (..v5.14), unkown culprit, with activity in the past three weeks
=============================================================================
Ralink RT2800 kernel deference issue since kernel 5.14
------------------------------------------------------
https://lore.kernel.org/linux-wireless/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net
By Robert W, 11 days ago; 5 activities, latest 1 days ago.
Introduced in v5.13..v5.14
https://linux-regtracking.leemhuis.info/regzbot/regression/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net
Latest activity with a patch:
* [PATCH 5.16] mac80211: fix rate control for retransmitted frames
https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name
1 days ago, by Felix Fietkau; thread monitored.
====================================================================
all others with unkown culprit and activity in the past three months
====================================================================
idle power increased from ~20 to ~28 watts
------------------------------------------
https://lore.kernel.org/lkml/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org
By Idzibear, 22 days ago; 3 activities, latest 22 days ago.
Introduced in v5.14..v5.15
https://linux-regtracking.leemhuis.info/regzbot/regression/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org
=============
End of report
=============
--
P.S.: Wanna know more about regzbot or how to use it to track regressions
for your subsystem? Then check out the getting started guide or the
reference documentation:
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md
The short version: if you see a regression report you want to see
tracked, just send a reply to the report where you Cc
regressions@lists.linux.dev with a line like this:
#regzbot introduced: v5.13..v5.14-rc1
If you want to fix a tracked regression, just do what is expected
anyway: add a 'Link:' tag with the url to the report, e.g.:
Link: https://lore.kernel.org/all/30th.anniversary.repost@klaava.Helsinki.FI/
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux regressions report for mainline [2021-11-24]
2021-11-24 9:25 Linux regressions report for mainline [2021-11-24] Regzbot (on behalf of Thorsten Leemhuis)
@ 2021-11-24 10:01 ` Thorsten Leemhuis
2021-11-24 11:03 ` Greg KH
2021-11-24 18:13 ` Linus Torvalds
1 sibling, 1 reply; 6+ messages in thread
From: Thorsten Leemhuis @ 2021-11-24 10:01 UTC (permalink / raw)
To: LKML, Linus Torvalds, Linux regressions mailing list
Lo! This was the first report sent by regzbot, my Linux kernel
regression tracking bot, so I guess it might be a good idea to highlight
a few of it's aspects:
On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote:
> From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info>
>
> Hi, this is regzbot, the Linux kernel regression tracking bot.
>
> Currently I'm aware of 15 regressions in linux-mainline. Find the
> current status below and the latest on the web:
>
> https://linux-regtracking.leemhuis.info/regzbot/mainline/
>
> Bye bye, hope to see you soon for the next report.
> Regzbot (on behalf of Thorsten Leemhuis)
From now on I plan to make regzbot send such reports on Monday mornings,
e.g. usually a few hours after Linus released a new RC.
Let me know what you think about the format.
A few random thoughts and explanations about the current format:
- next weeks report will highlight regressions that get added to regzbot
over the next few days
- I chose to categorize the regressions by identification status and by
version line. Those where the culprit is identified come first, as they
are the ones which most of the time can be solved by reverting the culprit
- the entries in each section are ordered by time of last activity,
which makes it easy to spot those where nothing happened recently, as
they are near the end of a section.
- the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ )
links to the latest five activities regzbot noticed (an activity most of
the time will be a mail send in reply to the report or a related thread
that regzbot monitors). I for now chose to not do that in the report, as
it would make it much longer and might be something that spam filters
consider suspicious.
That's it from my side. Let me know what you think.
Ciao, Thorsten
> ========================================================
> current cycle (v5.15.. aka v5.16-rc), culprit identified
> ========================================================
>
>
> Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years
> -----------------------------------------------------------------------------------------------
> https://lore.kernel.org/regressions/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com
> By Niklas Schnelle, 4 days ago; 3 activities, latest 3 days ago.
> Introduced in 32def4120e48 (v5.16-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com
>
>
> =========================================================================================
> previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months
> =========================================================================================
>
>
> CONFIG_SYSFB_SIMPLEFB breaks console scrolling
> ----------------------------------------------
> https://lore.kernel.org/lkml/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de
> By Harald Dunkel, 8 days ago; 1 activities, latest 8 days ago.
> Introduced in 8633ef82f101 (v5.15-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de
>
>
> ==================================================================================
> older cycles (..v5.14), culprit identified, with activity in the past three months
> ==================================================================================
>
>
> wireless AP (Raspberry Pi with rt2x00usb) crashes every hour or so
> ------------------------------------------------------------------
> https://lore.kernel.org/lkml/20211118132556.GD334428@darkstar.musicnaut.iki.fi
> By Aaro Koskinen, 5 days ago; 5 activities, latest 1 days ago.
> Introduced in 03c3911d2d67 (v5.14-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/20211118132556.GD334428@darkstar.musicnaut.iki.fi
>
> Latest activity with a patch:
> * [PATCH 5.16] mac80211: fix rate control for retransmitted frames
> https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name
> 1 days ago, by Felix Fietkau; thread monitored.
>
>
> Bluetooth: Query LE tx power on startup broke Bluetooth on MacBookPro16,1
> -------------------------------------------------------------------------
> https://lore.kernel.org/regressions/4970a940-211b-25d6-edab-21a815313954@protonmail.com
> By Orlando Chamberlain, 56 days ago; 30 activities, latest 4 days ago.
> Introduced in 7c395ea521e6 (v5.11-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/4970a940-211b-25d6-edab-21a815313954@protonmail.com
>
> Latest activity with a patch:
> * Re: [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power
> https://lore.kernel.org/lkml/CABBYNZLjSfcG_KqTEbL6NOSvHhA5-b1t_S=3FQP4=GwW21kuzg@mail.gmail.com
> 18 days ago, by Luiz Augusto von Dentz
>
> Noteworthy links:
> * Re: [PATCH] Bluetooth: add quirk disabling query LE tx power
> https://lore.kernel.org/lkml/43fb97ad-69eb-95ad-d50a-b8f1113dbee6@leemhuis.info
> 54 days ago, by Thorsten Leemhuis; thread monitored.
> * [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power
> https://lore.kernel.org/lkml/20211001083412.3078-1-redecorating@protonmail.com
> 54 days ago, by Orlando Chamberlain; thread monitored.
>
>
> Re: rt2x00 regression
> ---------------------
> https://lore.kernel.org/regressions/87czop5j33.fsf@tynnyri.adurom.net
> By Kalle Valo, 54 days ago; 16 activities, latest 4 days ago.
> Introduced in e383c70474db (v5.2-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/87czop5j33.fsf@tynnyri.adurom.net
>
> Latest activity with a patch:
> * [PATCH] rt2x00: do not mark device gone on EPROTO errors during start
> https://lore.kernel.org/regressions/20211111141003.GA134627@wp.pl
> 12 days ago, by Stanislaw Gruszka
>
> Noteworthy links:
> * rt2x00 regression
> https://lore.kernel.org/linux-wireless/bff7d309-a816-6a75-51b6-5928ef4f7a8c@exuvo.se
> 789 days ago, by Anton Olsson; thread monitored.
>
>
> PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization
> -----------------------------------------------------------------------------------
> https://lore.kernel.org/linux-pci/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com
> By Stéphane Graber, 5 days ago; 4 activities, latest 5 days ago.
> Introduced in 6dce5aa59e0b (v5.5-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com
>
> Latest activity with a patch:
> * Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization
> https://lore.kernel.org/linux-pci/CAL_JsqKrfpDtQZMMuhA_tURit6fO82FzPbKA40o6_8jWRewm8g@mail.gmail.com
> 5 days ago, by Rob Herring
>
>
> mhi: ath11k resume fails on some devices
> ----------------------------------------
> https://lore.kernel.org/regressions/871r5p0x2u.fsf@codeaurora.org
> By Kalle Valo, 69 days ago; 23 activities, latest 5 days ago.
> Introduced in 020d3b26c07a (v5.13-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/871r5p0x2u.fsf@codeaurora.org
>
>
> bug: usb: gadget: FSL_UDC_CORE Corrupted request list leads to unrecoverable loop.
> ----------------------------------------------------------------------------------
> https://lore.kernel.org/linuxppc-dev/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com
> By Eugene Bordenkircher, 25 days ago; 7 activities, latest 7 days ago.
> Introduced in f79a60b8785 (v3.4-rc4)
> https://linux-regtracking.leemhuis.info/regzbot/regression/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com
>
> Noteworthy links:
> * https://lore.kernel.org/all/CADRPPNSrhiwr8jmBb2h4cFYqHtuDKK8rL0i6Bkg7+xEyXJPATA@mail.gmail.com/
> 22 days ago, by Thorsten Leemhuis
> * https://lore.kernel.org/all/2c275adc278477e1e512ea6ecc0c1f4dcc46969d.camel@infinera.com/
> 22 days ago, by Thorsten Leemhuis
>
>
> Bug in Memory Layout of rx_desc for QCA6174
> -------------------------------------------
> https://lore.kernel.org/ath10k/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com
> By Francesco Magliocca, 159 days ago; 5 activities, latest 24 days ago.
> Introduced in e3def6f7ddf8 (v4.16-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com
>
> Noteworthy links:
> * Bug in Memory Layout of rx_desc for QCA6174
> https://lore.kernel.org/ath10k/CAH4F6uvX=xtTnBDaj1BVHSx_FDSUbpc4TRC2DGTHBmGJSD2oEA@mail.gmail.com
> 25 days ago, by Francesco Magliocca; thread monitored.
>
>
> ====================================================
> current cycle (v5.15.. aka v5.16-rc), unkown culprit
> ====================================================
>
>
> mm: reclaim_throttle leads to stall in near-OOM conditions
> ----------------------------------------------------------
> https://lore.kernel.org/lkml/20211124011954.7cab9bb4@mail.inbox.lv
> By Alexey Avramov, 0 days ago; 1 activities, latest 0 days ago.
> Introduced in v5.15..v5.16-rc1
> https://linux-regtracking.leemhuis.info/regzbot/regression/20211124011954.7cab9bb4@mail.inbox.lv
>
>
> mm: LTP/memcg testcase regression induced by 8cd7c588decf..66ce520bb7c2 series
> ------------------------------------------------------------------------------
> https://lore.kernel.org/lkml/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de
> By Mike Galbraith, 2 days ago; 8 activities, latest 0 days ago.
> Introduced in 8cd7c588decf..66ce520bb7c2 (v5.15..v5.16-rc1)
> https://linux-regtracking.leemhuis.info/regzbot/regression/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de
>
>
> ====================================================================================
> previous cycle (v5.14..v5.15), unkown culprit, with activity in the past three weeks
> ====================================================================================
>
>
> Kernel 5.15 reboots / freezes upon ifup/ifdown
> ----------------------------------------------
> https://lore.kernel.org/stable/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de
> By Stefan Dietrich, 0 days ago; 5 activities, latest 0 days ago.
> Introduced in v5.14..v5.15
> https://linux-regtracking.leemhuis.info/regzbot/regression/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de
>
>
> kernel 5.15.1: AMD RX 6700 XT - Fails to resume after screen blank
> ------------------------------------------------------------------
> https://lore.kernel.org/stable/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk
> By Mark Boddington, 13 days ago; 7 activities, latest 11 days ago.
> Introduced in v5.14..v5.15
> https://linux-regtracking.leemhuis.info/regzbot/regression/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk
>
>
> =============================================================================
> older cycles (..v5.14), unkown culprit, with activity in the past three weeks
> =============================================================================
>
>
> Ralink RT2800 kernel deference issue since kernel 5.14
> ------------------------------------------------------
> https://lore.kernel.org/linux-wireless/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net
> By Robert W, 11 days ago; 5 activities, latest 1 days ago.
> Introduced in v5.13..v5.14
> https://linux-regtracking.leemhuis.info/regzbot/regression/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net
>
> Latest activity with a patch:
> * [PATCH 5.16] mac80211: fix rate control for retransmitted frames
> https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name
> 1 days ago, by Felix Fietkau; thread monitored.
>
>
> ====================================================================
> all others with unkown culprit and activity in the past three months
> ====================================================================
>
>
> idle power increased from ~20 to ~28 watts
> ------------------------------------------
> https://lore.kernel.org/lkml/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org
> By Idzibear, 22 days ago; 3 activities, latest 22 days ago.
> Introduced in v5.14..v5.15
> https://linux-regtracking.leemhuis.info/regzbot/regression/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org
>
>
> =============
> End of report
> =============
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux regressions report for mainline [2021-11-24]
2021-11-24 10:01 ` Thorsten Leemhuis
@ 2021-11-24 11:03 ` Greg KH
2021-11-24 12:52 ` Thorsten Leemhuis
0 siblings, 1 reply; 6+ messages in thread
From: Greg KH @ 2021-11-24 11:03 UTC (permalink / raw)
To: Thorsten Leemhuis; +Cc: LKML, Linus Torvalds, Linux regressions mailing list
On Wed, Nov 24, 2021 at 11:01:52AM +0100, Thorsten Leemhuis wrote:
> Lo! This was the first report sent by regzbot, my Linux kernel
> regression tracking bot, so I guess it might be a good idea to highlight
> a few of it's aspects:
>
> On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote:
> > From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info>
> >
> > Hi, this is regzbot, the Linux kernel regression tracking bot.
> >
> > Currently I'm aware of 15 regressions in linux-mainline. Find the
> > current status below and the latest on the web:
> >
> > https://linux-regtracking.leemhuis.info/regzbot/mainline/
> >
> > Bye bye, hope to see you soon for the next report.
> > Regzbot (on behalf of Thorsten Leemhuis)
>
> >From now on I plan to make regzbot send such reports on Monday mornings,
> e.g. usually a few hours after Linus released a new RC.
>
> Let me know what you think about the format.
>
> A few random thoughts and explanations about the current format:
>
> - next weeks report will highlight regressions that get added to regzbot
> over the next few days
>
> - I chose to categorize the regressions by identification status and by
> version line. Those where the culprit is identified come first, as they
> are the ones which most of the time can be solved by reverting the culprit
>
> - the entries in each section are ordered by time of last activity,
> which makes it easy to spot those where nothing happened recently, as
> they are near the end of a section.
>
> - the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ )
> links to the latest five activities regzbot noticed (an activity most of
> the time will be a mail send in reply to the report or a related thread
> that regzbot monitors). I for now chose to not do that in the report, as
> it would make it much longer and might be something that spam filters
> consider suspicious.
>
> That's it from my side. Let me know what you think.
I like it, but as a maintainer, I find this hard to know what to do with
it.
As a maintainer, I want to be reminded of what regressions have been
reported for my subsystem, so I know what to do and who to go poke about
them.
I could dig through the list and try to see if these are relevant to me,
but it's not always obvious.
How about something like "one email per issue" as a response from the
first report, that would cc: the needed maintainers (or people from
MAINTAINERS, you should be able to get that automatically from
get_maintainer.pl) and the subsystem mailing list (again from
get_maintainers.pl), so that I am constantly reminded that "you need to
get this fixed!".
Pester me, it's my job as a maintainer to get regressions resolved. If
I see a long list with no names on it, It's easier for me to just ignore
:)
Anyway, I know that's more work for you, don't do it if you don't want
to, but as you have individual items in your database already, maybe it
is easy to do, I don't know. Thanks again for doing this work, it's
great to see happen.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux regressions report for mainline [2021-11-24]
2021-11-24 11:03 ` Greg KH
@ 2021-11-24 12:52 ` Thorsten Leemhuis
0 siblings, 0 replies; 6+ messages in thread
From: Thorsten Leemhuis @ 2021-11-24 12:52 UTC (permalink / raw)
To: Greg KH; +Cc: LKML, Linus Torvalds, Linux regressions mailing list
On 24.11.21 12:03, Greg KH wrote:
> On Wed, Nov 24, 2021 at 11:01:52AM +0100, Thorsten Leemhuis wrote:
>> Lo! This was the first report sent by regzbot, my Linux kernel
>> regression tracking bot, so I guess it might be a good idea to highlight
>> a few of it's aspects:
>>
>> On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote:
>>> From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info>
>>>
>>> Hi, this is regzbot, the Linux kernel regression tracking bot.
>>>
>>> Currently I'm aware of 15 regressions in linux-mainline. Find the
>>> current status below and the latest on the web:
>>>
>>> https://linux-regtracking.leemhuis.info/regzbot/mainline/
>>>
>>> Bye bye, hope to see you soon for the next report.
>>> Regzbot (on behalf of Thorsten Leemhuis)
>>
>> >From now on I plan to make regzbot send such reports on Monday mornings,
>> e.g. usually a few hours after Linus released a new RC.
>>
>> Let me know what you think about the format.
>>
>> A few random thoughts and explanations about the current format:
>>
>> - next weeks report will highlight regressions that get added to regzbot
>> over the next few days
>>
>> - I chose to categorize the regressions by identification status and by
>> version line. Those where the culprit is identified come first, as they
>> are the ones which most of the time can be solved by reverting the culprit
>>
>> - the entries in each section are ordered by time of last activity,
>> which makes it easy to spot those where nothing happened recently, as
>> they are near the end of a section.
>>
>> - the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ )
>> links to the latest five activities regzbot noticed (an activity most of
>> the time will be a mail send in reply to the report or a related thread
>> that regzbot monitors). I for now chose to not do that in the report, as
>> it would make it much longer and might be something that spam filters
>> consider suspicious.
>>
>> That's it from my side. Let me know what you think.
>
> I like it,
Glad to hear. Many thx for the feedback!
> but as a maintainer, I find this hard to know what to do with
> it.
Well, I designed this a bit more for the tree maintainer, so Linus can:
(1) see when things are not moving forward as quickly as he'd like to
and get involved in the discussions or revert a commit
(2) decide if he wants to release another rc or the final
> As a maintainer, I want to be reminded of what regressions have been
> reported for my subsystem, so I know what to do and who to go poke about
> them.
>
> I could dig through the list and try to see if these are relevant to me,
> but it's not always obvious.
Yeah, that true. :-/
> How about something like "one email per issue" as a response from the
> first report, that would cc: the needed maintainers (or people from
> MAINTAINERS, you should be able to get that automatically from
> get_maintainer.pl) and the subsystem mailing list (again from
> get_maintainers.pl), so that I am constantly reminded that "you need to
> get this fixed!".
I'll keep this idea in mind. But that would result in a lot of mails.
That's something I like to avoid, as we all get enough mails already: if
regzbot starts to spam people, then some people might set up a mental or
procmail filter to ignore all of them.
That's why I settled on a different strategy:
> Pester me, it's my job as a maintainer to get regressions resolved.
I'm pestering developers and maintainers already, but I do it only when
it seems to be needed. And I do it manually for now; I intend to make
regzbot do it automatically sooner or later, but first I want to get a
better feeling when it's appropriate to pester people without getting
them quickly annoyed -- e.g. after what time and in which situations.
Here for example I asked why a patch didn't proceed, and got a reply
from the maintainer:
https://lore.kernel.org/regressions/cfac5f5c-83d7-e0d9-5368-07ca041ebaed@leemhuis.info/
Seems the patch still didn't get any farther, so I guess it soon time
for me to poke again.
In another thread two pokes both helped to get things rolling again, but
seems I soon need to send a third one, as it seems no one really feels
responsible:
https://lore.kernel.org/lkml/40550C00-4EE5-480F-AFD4-A2ACA01F9DBB@live.com/
Those are just two examples, I poked a few more discussions already and
it most of the time helped.
IOW: for now I'm trying the strategy "only send mails when its needed".
But this made me aware that I should teach regzbot one thing: if the
culprit is known, it should check if everyone in the SOB chain of the
change is CCed to the discussion on the regression.
> If
> I see a long list with no names on it, It's easier for me to just ignore
> :)>
> Anyway, I know that's more work for you, don't do it if you don't want
> to, but as you have individual items in your database already, maybe it
> is easy to do,
Not really, as for now regzbot doesn't assign regressions to a
subsystem. I could add that, but how to fill and maintain that
association? When the culprit is known, regzbot should be able to fill
it automatically using the SOB chain or the path of the merge. But if
it's not, it becomes something that some human would need to do
manually. I really tried hard to avoid anything that requires manual
work, as it always hard to make people do it :-/
That being said: I'm pretty sure I sooner or later will make regzbot
store a association to a subsystem.
> I don't know. Thanks again for doing this work, it's
> great to see happen.
thx again!
Ciao, Thorsten
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux regressions report for mainline [2021-11-24]
2021-11-24 9:25 Linux regressions report for mainline [2021-11-24] Regzbot (on behalf of Thorsten Leemhuis)
2021-11-24 10:01 ` Thorsten Leemhuis
@ 2021-11-24 18:13 ` Linus Torvalds
2021-11-25 11:52 ` Thorsten Leemhuis
1 sibling, 1 reply; 6+ messages in thread
From: Linus Torvalds @ 2021-11-24 18:13 UTC (permalink / raw)
To: Regzbot (on behalf of Thorsten Leemhuis)
Cc: LKML, Linux regressions mailing list
Ok, nice to see the new regression tracking bot start to show life.
Greg had one suggestion, I have another - namely about grouping of these things.
I like how you group them by "identified" and "unknown", because
that's certainly very meaningful.
But at the same time it does mean that if I look for "what are current
issues with the development kernel", it ends up being very spread out:
On Wed, Nov 24, 2021 at 1:25 AM Regzbot (on behalf of Thorsten
Leemhuis) <regressions@leemhuis.info> wrote:
>
> ========================================================
> current cycle (v5.15.. aka v5.16-rc), culprit identified
> ========================================================
...
> =========================================================================================
> previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months
> =========================================================================================
...
> ==================================================================================
> older cycles (..v5.14), culprit identified, with activity in the past three months
> ==================================================================================
...
> ====================================================
> current cycle (v5.15.. aka v5.16-rc), unkown culprit
> ====================================================
...
note how there was a lot of other stuff in between those "culprit
idenfified" and "unknown culprit" for the current kernel.
One of the things I really liked about the regression tracking you did
before was that it helped me get a sense for the state of the release,
and so I'd like to see that "current cycle" in one go.
I suspect that Greg may have a slightly similar issue - as a driver
maintainer, he cares about current cycle things (but mainly only when
they affect his subsystems), but with his stable maintainer hat on he
then cares more about the older cycles.
Greg suggested splitting out the issues one by one - to try to have
the right people on the Cc for any _particular_ issue, and while I
think that's not the solution in this case (I very much want to see
the "summary" email), it would be good to perhaps at least organize
that summary email slightly differently.
I suspect this is something we'd need to iterate on as we use this in
our workflow, but that was my initial reaction to this first report.
Linus
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Linux regressions report for mainline [2021-11-24]
2021-11-24 18:13 ` Linus Torvalds
@ 2021-11-25 11:52 ` Thorsten Leemhuis
0 siblings, 0 replies; 6+ messages in thread
From: Thorsten Leemhuis @ 2021-11-25 11:52 UTC (permalink / raw)
To: Linus Torvalds; +Cc: LKML, Linux regressions mailing list
On 24.11.21 19:13, Linus Torvalds wrote:
> Ok, nice to see the new regression tracking bot start to show life.
Yeah. :-D
Sadly one of my biggest problems with regression tracking remains:
getting aware of regression reports. I can fully understand that most
people don't care about regzbot for now, but it would really help if
everyone would CC regressions@lists.linux.dev on mails regarding
regressions (e.g. reports or any replies to them).
> Greg had one suggestion,
Still not sure how to approach his use case, but for now I started
adding the usual subsystem commit summary prefixes (e.g. "net:", "usb:",
"drm/amd") to the title of newly added regression, which might help
somewhat and won't hurt.
> I have another - namely about grouping of these things.
>
> I like how you group them by "identified" and "unknown", because
> that's certainly very meaningful.
>
> But at the same time it does mean that if I look for "what are current
> issues with the development kernel", it ends up being very spread out:
Hah, fun fact: the order you purposed was the one I initially had in
mind. But I later changed my mind, as I thought 'hey, if the culprit of
the regression is known, it should be able to fix this quickly (e.g. by
a revert, if there are no conflicts) even for regressions that made it
into proper releases".
But whatever: I'm totally fine with this and already changed the web
interface yesterday after your mail arrived, only took a minute:
https://linux-regtracking.leemhuis.info/regzbot/mainline/
Next report will use this order as well.
> I suspect that Greg may have a slightly similar issue - as a driver
> maintainer, he cares about current cycle things (but mainly only when
> they affect his subsystems), but with his stable maintainer hat on he
> then cares more about the older cycles.
>
> Greg suggested splitting out the issues one by one - to try to have
> the right people on the Cc for any _particular_ issue, and while I
> think that's not the solution in this case (I very much want to see
> the "summary" email), it would be good to perhaps at least organize
> that summary email slightly differently.
>
> I suspect this is something we'd need to iterate on as we use this in
> our workflow
Definitely. If there is something else you want to see changed or think
is odd wrt to regzbot or my work as regression tracker, just let me know.
> but that was my initial reaction to this first report.
Thx for the feedback, much appreciated.
Ciao, Thorsten
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-25 11:57 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-24 9:25 Linux regressions report for mainline [2021-11-24] Regzbot (on behalf of Thorsten Leemhuis)
2021-11-24 10:01 ` Thorsten Leemhuis
2021-11-24 11:03 ` Greg KH
2021-11-24 12:52 ` Thorsten Leemhuis
2021-11-24 18:13 ` Linus Torvalds
2021-11-25 11:52 ` Thorsten Leemhuis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).