From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info> Hi, this is regzbot, the Linux kernel regression tracking bot. Currently I'm aware of 15 regressions in linux-mainline. Find the current status below and the latest on the web: https://linux-regtracking.leemhuis.info/regzbot/mainline/ Bye bye, hope to see you soon for the next report. Regzbot (on behalf of Thorsten Leemhuis) ======================================================== current cycle (v5.15.. aka v5.16-rc), culprit identified ======================================================== Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years ----------------------------------------------------------------------------------------------- https://lore.kernel.org/regressions/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com By Niklas Schnelle, 4 days ago; 3 activities, latest 3 days ago. Introduced in 32def4120e48 (v5.16-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com ========================================================================================= previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months ========================================================================================= CONFIG_SYSFB_SIMPLEFB breaks console scrolling ---------------------------------------------- https://lore.kernel.org/lkml/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de By Harald Dunkel, 8 days ago; 1 activities, latest 8 days ago. Introduced in 8633ef82f101 (v5.15-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de ================================================================================== older cycles (..v5.14), culprit identified, with activity in the past three months ================================================================================== wireless AP (Raspberry Pi with rt2x00usb) crashes every hour or so ------------------------------------------------------------------ https://lore.kernel.org/lkml/20211118132556.GD334428@darkstar.musicnaut.iki.fi By Aaro Koskinen, 5 days ago; 5 activities, latest 1 days ago. Introduced in 03c3911d2d67 (v5.14-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/20211118132556.GD334428@darkstar.musicnaut.iki.fi Latest activity with a patch: * [PATCH 5.16] mac80211: fix rate control for retransmitted frames https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name 1 days ago, by Felix Fietkau; thread monitored. Bluetooth: Query LE tx power on startup broke Bluetooth on MacBookPro16,1 ------------------------------------------------------------------------- https://lore.kernel.org/regressions/4970a940-211b-25d6-edab-21a815313954@protonmail.com By Orlando Chamberlain, 56 days ago; 30 activities, latest 4 days ago. Introduced in 7c395ea521e6 (v5.11-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/4970a940-211b-25d6-edab-21a815313954@protonmail.com Latest activity with a patch: * Re: [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power https://lore.kernel.org/lkml/CABBYNZLjSfcG_KqTEbL6NOSvHhA5-b1t_S=3FQP4=GwW21kuzg@mail.gmail.com 18 days ago, by Luiz Augusto von Dentz Noteworthy links: * Re: [PATCH] Bluetooth: add quirk disabling query LE tx power https://lore.kernel.org/lkml/43fb97ad-69eb-95ad-d50a-b8f1113dbee6@leemhuis.info 54 days ago, by Thorsten Leemhuis; thread monitored. * [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power https://lore.kernel.org/lkml/20211001083412.3078-1-redecorating@protonmail.com 54 days ago, by Orlando Chamberlain; thread monitored. Re: rt2x00 regression --------------------- https://lore.kernel.org/regressions/87czop5j33.fsf@tynnyri.adurom.net By Kalle Valo, 54 days ago; 16 activities, latest 4 days ago. Introduced in e383c70474db (v5.2-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/87czop5j33.fsf@tynnyri.adurom.net Latest activity with a patch: * [PATCH] rt2x00: do not mark device gone on EPROTO errors during start https://lore.kernel.org/regressions/20211111141003.GA134627@wp.pl 12 days ago, by Stanislaw Gruszka Noteworthy links: * rt2x00 regression https://lore.kernel.org/linux-wireless/bff7d309-a816-6a75-51b6-5928ef4f7a8c@exuvo.se 789 days ago, by Anton Olsson; thread monitored. PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization ----------------------------------------------------------------------------------- https://lore.kernel.org/linux-pci/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com By Stéphane Graber, 5 days ago; 4 activities, latest 5 days ago. Introduced in 6dce5aa59e0b (v5.5-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com Latest activity with a patch: * Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization https://lore.kernel.org/linux-pci/CAL_JsqKrfpDtQZMMuhA_tURit6fO82FzPbKA40o6_8jWRewm8g@mail.gmail.com 5 days ago, by Rob Herring mhi: ath11k resume fails on some devices ---------------------------------------- https://lore.kernel.org/regressions/871r5p0x2u.fsf@codeaurora.org By Kalle Valo, 69 days ago; 23 activities, latest 5 days ago. Introduced in 020d3b26c07a (v5.13-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/871r5p0x2u.fsf@codeaurora.org bug: usb: gadget: FSL_UDC_CORE Corrupted request list leads to unrecoverable loop. ---------------------------------------------------------------------------------- https://lore.kernel.org/linuxppc-dev/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com By Eugene Bordenkircher, 25 days ago; 7 activities, latest 7 days ago. Introduced in f79a60b8785 (v3.4-rc4) https://linux-regtracking.leemhuis.info/regzbot/regression/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com Noteworthy links: * https://lore.kernel.org/all/CADRPPNSrhiwr8jmBb2h4cFYqHtuDKK8rL0i6Bkg7+xEyXJPATA@mail.gmail.com/ 22 days ago, by Thorsten Leemhuis * https://lore.kernel.org/all/2c275adc278477e1e512ea6ecc0c1f4dcc46969d.camel@infinera.com/ 22 days ago, by Thorsten Leemhuis Bug in Memory Layout of rx_desc for QCA6174 ------------------------------------------- https://lore.kernel.org/ath10k/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com By Francesco Magliocca, 159 days ago; 5 activities, latest 24 days ago. Introduced in e3def6f7ddf8 (v4.16-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com Noteworthy links: * Bug in Memory Layout of rx_desc for QCA6174 https://lore.kernel.org/ath10k/CAH4F6uvX=xtTnBDaj1BVHSx_FDSUbpc4TRC2DGTHBmGJSD2oEA@mail.gmail.com 25 days ago, by Francesco Magliocca; thread monitored. ==================================================== current cycle (v5.15.. aka v5.16-rc), unkown culprit ==================================================== mm: reclaim_throttle leads to stall in near-OOM conditions ---------------------------------------------------------- https://lore.kernel.org/lkml/20211124011954.7cab9bb4@mail.inbox.lv By Alexey Avramov, 0 days ago; 1 activities, latest 0 days ago. Introduced in v5.15..v5.16-rc1 https://linux-regtracking.leemhuis.info/regzbot/regression/20211124011954.7cab9bb4@mail.inbox.lv mm: LTP/memcg testcase regression induced by 8cd7c588decf..66ce520bb7c2 series ------------------------------------------------------------------------------ https://lore.kernel.org/lkml/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de By Mike Galbraith, 2 days ago; 8 activities, latest 0 days ago. Introduced in 8cd7c588decf..66ce520bb7c2 (v5.15..v5.16-rc1) https://linux-regtracking.leemhuis.info/regzbot/regression/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de ==================================================================================== previous cycle (v5.14..v5.15), unkown culprit, with activity in the past three weeks ==================================================================================== Kernel 5.15 reboots / freezes upon ifup/ifdown ---------------------------------------------- https://lore.kernel.org/stable/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de By Stefan Dietrich, 0 days ago; 5 activities, latest 0 days ago. Introduced in v5.14..v5.15 https://linux-regtracking.leemhuis.info/regzbot/regression/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de kernel 5.15.1: AMD RX 6700 XT - Fails to resume after screen blank ------------------------------------------------------------------ https://lore.kernel.org/stable/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk By Mark Boddington, 13 days ago; 7 activities, latest 11 days ago. Introduced in v5.14..v5.15 https://linux-regtracking.leemhuis.info/regzbot/regression/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk ============================================================================= older cycles (..v5.14), unkown culprit, with activity in the past three weeks ============================================================================= Ralink RT2800 kernel deference issue since kernel 5.14 ------------------------------------------------------ https://lore.kernel.org/linux-wireless/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net By Robert W, 11 days ago; 5 activities, latest 1 days ago. Introduced in v5.13..v5.14 https://linux-regtracking.leemhuis.info/regzbot/regression/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net Latest activity with a patch: * [PATCH 5.16] mac80211: fix rate control for retransmitted frames https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name 1 days ago, by Felix Fietkau; thread monitored. ==================================================================== all others with unkown culprit and activity in the past three months ==================================================================== idle power increased from ~20 to ~28 watts ------------------------------------------ https://lore.kernel.org/lkml/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org By Idzibear, 22 days ago; 3 activities, latest 22 days ago. Introduced in v5.14..v5.15 https://linux-regtracking.leemhuis.info/regzbot/regression/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org ============= End of report ============= -- P.S.: Wanna know more about regzbot or how to use it to track regressions for your subsystem? Then check out the getting started guide or the reference documentation: https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md The short version: if you see a regression report you want to see tracked, just send a reply to the report where you Cc regressions@lists.linux.dev with a line like this: #regzbot introduced: v5.13..v5.14-rc1 If you want to fix a tracked regression, just do what is expected anyway: add a 'Link:' tag with the url to the report, e.g.: Link: https://lore.kernel.org/all/30th.anniversary.repost@klaava.Helsinki.FI/
Lo! This was the first report sent by regzbot, my Linux kernel regression tracking bot, so I guess it might be a good idea to highlight a few of it's aspects: On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote: > From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info> > > Hi, this is regzbot, the Linux kernel regression tracking bot. > > Currently I'm aware of 15 regressions in linux-mainline. Find the > current status below and the latest on the web: > > https://linux-regtracking.leemhuis.info/regzbot/mainline/ > > Bye bye, hope to see you soon for the next report. > Regzbot (on behalf of Thorsten Leemhuis) From now on I plan to make regzbot send such reports on Monday mornings, e.g. usually a few hours after Linus released a new RC. Let me know what you think about the format. A few random thoughts and explanations about the current format: - next weeks report will highlight regressions that get added to regzbot over the next few days - I chose to categorize the regressions by identification status and by version line. Those where the culprit is identified come first, as they are the ones which most of the time can be solved by reverting the culprit - the entries in each section are ordered by time of last activity, which makes it easy to spot those where nothing happened recently, as they are near the end of a section. - the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ ) links to the latest five activities regzbot noticed (an activity most of the time will be a mail send in reply to the report or a related thread that regzbot monitors). I for now chose to not do that in the report, as it would make it much longer and might be something that spam filters consider suspicious. That's it from my side. Let me know what you think. Ciao, Thorsten > ======================================================== > current cycle (v5.15.. aka v5.16-rc), culprit identified > ======================================================== > > > Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years > ----------------------------------------------------------------------------------------------- > https://lore.kernel.org/regressions/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com > By Niklas Schnelle, 4 days ago; 3 activities, latest 3 days ago. > Introduced in 32def4120e48 (v5.16-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/15db9c1d11d32fb16269afceb527b5d743177ac4.camel@linux.ibm.com > > > ========================================================================================= > previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months > ========================================================================================= > > > CONFIG_SYSFB_SIMPLEFB breaks console scrolling > ---------------------------------------------- > https://lore.kernel.org/lkml/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de > By Harald Dunkel, 8 days ago; 1 activities, latest 8 days ago. > Introduced in 8633ef82f101 (v5.15-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/e50d5ad5-19fd-07ae-41e4-5a2d26a98bcf@afaics.de > > > ================================================================================== > older cycles (..v5.14), culprit identified, with activity in the past three months > ================================================================================== > > > wireless AP (Raspberry Pi with rt2x00usb) crashes every hour or so > ------------------------------------------------------------------ > https://lore.kernel.org/lkml/20211118132556.GD334428@darkstar.musicnaut.iki.fi > By Aaro Koskinen, 5 days ago; 5 activities, latest 1 days ago. > Introduced in 03c3911d2d67 (v5.14-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/20211118132556.GD334428@darkstar.musicnaut.iki.fi > > Latest activity with a patch: > * [PATCH 5.16] mac80211: fix rate control for retransmitted frames > https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name > 1 days ago, by Felix Fietkau; thread monitored. > > > Bluetooth: Query LE tx power on startup broke Bluetooth on MacBookPro16,1 > ------------------------------------------------------------------------- > https://lore.kernel.org/regressions/4970a940-211b-25d6-edab-21a815313954@protonmail.com > By Orlando Chamberlain, 56 days ago; 30 activities, latest 4 days ago. > Introduced in 7c395ea521e6 (v5.11-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/4970a940-211b-25d6-edab-21a815313954@protonmail.com > > Latest activity with a patch: > * Re: [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power > https://lore.kernel.org/lkml/CABBYNZLjSfcG_KqTEbL6NOSvHhA5-b1t_S=3FQP4=GwW21kuzg@mail.gmail.com > 18 days ago, by Luiz Augusto von Dentz > > Noteworthy links: > * Re: [PATCH] Bluetooth: add quirk disabling query LE tx power > https://lore.kernel.org/lkml/43fb97ad-69eb-95ad-d50a-b8f1113dbee6@leemhuis.info > 54 days ago, by Thorsten Leemhuis; thread monitored. > * [PATCHv2] Bluetooth: quirk disabling LE Read Transmit Power > https://lore.kernel.org/lkml/20211001083412.3078-1-redecorating@protonmail.com > 54 days ago, by Orlando Chamberlain; thread monitored. > > > Re: rt2x00 regression > --------------------- > https://lore.kernel.org/regressions/87czop5j33.fsf@tynnyri.adurom.net > By Kalle Valo, 54 days ago; 16 activities, latest 4 days ago. > Introduced in e383c70474db (v5.2-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/87czop5j33.fsf@tynnyri.adurom.net > > Latest activity with a patch: > * [PATCH] rt2x00: do not mark device gone on EPROTO errors during start > https://lore.kernel.org/regressions/20211111141003.GA134627@wp.pl > 12 days ago, by Stanislaw Gruszka > > Noteworthy links: > * rt2x00 regression > https://lore.kernel.org/linux-wireless/bff7d309-a816-6a75-51b6-5928ef4f7a8c@exuvo.se > 789 days ago, by Anton Olsson; thread monitored. > > > PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization > ----------------------------------------------------------------------------------- > https://lore.kernel.org/linux-pci/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com > By Stéphane Graber, 5 days ago; 4 activities, latest 5 days ago. > Introduced in 6dce5aa59e0b (v5.5-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/CA%2Benf=v9rY_xnZML01oEgKLmvY1NGBUUhnSJaETmXtDtXfaczA@mail.gmail.com > > Latest activity with a patch: > * Re: PCIe regression on APM Merlin (aarch64 dev platform) preventing NVME initialization > https://lore.kernel.org/linux-pci/CAL_JsqKrfpDtQZMMuhA_tURit6fO82FzPbKA40o6_8jWRewm8g@mail.gmail.com > 5 days ago, by Rob Herring > > > mhi: ath11k resume fails on some devices > ---------------------------------------- > https://lore.kernel.org/regressions/871r5p0x2u.fsf@codeaurora.org > By Kalle Valo, 69 days ago; 23 activities, latest 5 days ago. > Introduced in 020d3b26c07a (v5.13-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/871r5p0x2u.fsf@codeaurora.org > > > bug: usb: gadget: FSL_UDC_CORE Corrupted request list leads to unrecoverable loop. > ---------------------------------------------------------------------------------- > https://lore.kernel.org/linuxppc-dev/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com > By Eugene Bordenkircher, 25 days ago; 7 activities, latest 7 days ago. > Introduced in f79a60b8785 (v3.4-rc4) > https://linux-regtracking.leemhuis.info/regzbot/regression/MWHPR2201MB152074F47BF142189365627B91879@MWHPR2201MB1520.namprd22.prod.outlook.com > > Noteworthy links: > * https://lore.kernel.org/all/CADRPPNSrhiwr8jmBb2h4cFYqHtuDKK8rL0i6Bkg7+xEyXJPATA@mail.gmail.com/ > 22 days ago, by Thorsten Leemhuis > * https://lore.kernel.org/all/2c275adc278477e1e512ea6ecc0c1f4dcc46969d.camel@infinera.com/ > 22 days ago, by Thorsten Leemhuis > > > Bug in Memory Layout of rx_desc for QCA6174 > ------------------------------------------- > https://lore.kernel.org/ath10k/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com > By Francesco Magliocca, 159 days ago; 5 activities, latest 24 days ago. > Introduced in e3def6f7ddf8 (v4.16-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/CAH4F6usFu8-A6k5Z7rU9__iENcSC6Zr-NtRhh_aypR74UvN1uQ@mail.gmail.com > > Noteworthy links: > * Bug in Memory Layout of rx_desc for QCA6174 > https://lore.kernel.org/ath10k/CAH4F6uvX=xtTnBDaj1BVHSx_FDSUbpc4TRC2DGTHBmGJSD2oEA@mail.gmail.com > 25 days ago, by Francesco Magliocca; thread monitored. > > > ==================================================== > current cycle (v5.15.. aka v5.16-rc), unkown culprit > ==================================================== > > > mm: reclaim_throttle leads to stall in near-OOM conditions > ---------------------------------------------------------- > https://lore.kernel.org/lkml/20211124011954.7cab9bb4@mail.inbox.lv > By Alexey Avramov, 0 days ago; 1 activities, latest 0 days ago. > Introduced in v5.15..v5.16-rc1 > https://linux-regtracking.leemhuis.info/regzbot/regression/20211124011954.7cab9bb4@mail.inbox.lv > > > mm: LTP/memcg testcase regression induced by 8cd7c588decf..66ce520bb7c2 series > ------------------------------------------------------------------------------ > https://lore.kernel.org/lkml/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de > By Mike Galbraith, 2 days ago; 8 activities, latest 0 days ago. > Introduced in 8cd7c588decf..66ce520bb7c2 (v5.15..v5.16-rc1) > https://linux-regtracking.leemhuis.info/regzbot/regression/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de > > > ==================================================================================== > previous cycle (v5.14..v5.15), unkown culprit, with activity in the past three weeks > ==================================================================================== > > > Kernel 5.15 reboots / freezes upon ifup/ifdown > ---------------------------------------------- > https://lore.kernel.org/stable/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de > By Stefan Dietrich, 0 days ago; 5 activities, latest 0 days ago. > Introduced in v5.14..v5.15 > https://linux-regtracking.leemhuis.info/regzbot/regression/924175a188159f4e03bd69908a91e606b574139b.camel@gmx.de > > > kernel 5.15.1: AMD RX 6700 XT - Fails to resume after screen blank > ------------------------------------------------------------------ > https://lore.kernel.org/stable/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk > By Mark Boddington, 13 days ago; 7 activities, latest 11 days ago. > Introduced in v5.14..v5.15 > https://linux-regtracking.leemhuis.info/regzbot/regression/dbadfe41-24bf-5811-cf38-74973df45214@badpenguin.co.uk > > > ============================================================================= > older cycles (..v5.14), unkown culprit, with activity in the past three weeks > ============================================================================= > > > Ralink RT2800 kernel deference issue since kernel 5.14 > ------------------------------------------------------ > https://lore.kernel.org/linux-wireless/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net > By Robert W, 11 days ago; 5 activities, latest 1 days ago. > Introduced in v5.13..v5.14 > https://linux-regtracking.leemhuis.info/regzbot/regression/c07b4142fb725ed87a2cef530bae9ee7@lost-in-the-void.net > > Latest activity with a patch: > * [PATCH 5.16] mac80211: fix rate control for retransmitted frames > https://lore.kernel.org/linux-wireless/20211122204323.9787-1-nbd@nbd.name > 1 days ago, by Felix Fietkau; thread monitored. > > > ==================================================================== > all others with unkown culprit and activity in the past three months > ==================================================================== > > > idle power increased from ~20 to ~28 watts > ------------------------------------------ > https://lore.kernel.org/lkml/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org > By Idzibear, 22 days ago; 3 activities, latest 22 days ago. > Introduced in v5.14..v5.15 > https://linux-regtracking.leemhuis.info/regzbot/regression/c11d94b4-1701-4e26-efd1-42038342c4aa@kaputniks.org > > > ============= > End of report > ============= >
On Wed, Nov 24, 2021 at 11:01:52AM +0100, Thorsten Leemhuis wrote:
> Lo! This was the first report sent by regzbot, my Linux kernel
> regression tracking bot, so I guess it might be a good idea to highlight
> a few of it's aspects:
>
> On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote:
> > From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info>
> >
> > Hi, this is regzbot, the Linux kernel regression tracking bot.
> >
> > Currently I'm aware of 15 regressions in linux-mainline. Find the
> > current status below and the latest on the web:
> >
> > https://linux-regtracking.leemhuis.info/regzbot/mainline/
> >
> > Bye bye, hope to see you soon for the next report.
> > Regzbot (on behalf of Thorsten Leemhuis)
>
> >From now on I plan to make regzbot send such reports on Monday mornings,
> e.g. usually a few hours after Linus released a new RC.
>
> Let me know what you think about the format.
>
> A few random thoughts and explanations about the current format:
>
> - next weeks report will highlight regressions that get added to regzbot
> over the next few days
>
> - I chose to categorize the regressions by identification status and by
> version line. Those where the culprit is identified come first, as they
> are the ones which most of the time can be solved by reverting the culprit
>
> - the entries in each section are ordered by time of last activity,
> which makes it easy to spot those where nothing happened recently, as
> they are near the end of a section.
>
> - the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ )
> links to the latest five activities regzbot noticed (an activity most of
> the time will be a mail send in reply to the report or a related thread
> that regzbot monitors). I for now chose to not do that in the report, as
> it would make it much longer and might be something that spam filters
> consider suspicious.
>
> That's it from my side. Let me know what you think.
I like it, but as a maintainer, I find this hard to know what to do with
it.
As a maintainer, I want to be reminded of what regressions have been
reported for my subsystem, so I know what to do and who to go poke about
them.
I could dig through the list and try to see if these are relevant to me,
but it's not always obvious.
How about something like "one email per issue" as a response from the
first report, that would cc: the needed maintainers (or people from
MAINTAINERS, you should be able to get that automatically from
get_maintainer.pl) and the subsystem mailing list (again from
get_maintainers.pl), so that I am constantly reminded that "you need to
get this fixed!".
Pester me, it's my job as a maintainer to get regressions resolved. If
I see a long list with no names on it, It's easier for me to just ignore
:)
Anyway, I know that's more work for you, don't do it if you don't want
to, but as you have individual items in your database already, maybe it
is easy to do, I don't know. Thanks again for doing this work, it's
great to see happen.
thanks,
greg k-h
On 24.11.21 12:03, Greg KH wrote: > On Wed, Nov 24, 2021 at 11:01:52AM +0100, Thorsten Leemhuis wrote: >> Lo! This was the first report sent by regzbot, my Linux kernel >> regression tracking bot, so I guess it might be a good idea to highlight >> a few of it's aspects: >> >> On 24.11.21 10:25, Regzbot (on behalf of Thorsten Leemhuis) wrote: >>> From: Regzbot (for Thorsten Leemhuis) <regressions@leemhuis.info> >>> >>> Hi, this is regzbot, the Linux kernel regression tracking bot. >>> >>> Currently I'm aware of 15 regressions in linux-mainline. Find the >>> current status below and the latest on the web: >>> >>> https://linux-regtracking.leemhuis.info/regzbot/mainline/ >>> >>> Bye bye, hope to see you soon for the next report. >>> Regzbot (on behalf of Thorsten Leemhuis) >> >> >From now on I plan to make regzbot send such reports on Monday mornings, >> e.g. usually a few hours after Linus released a new RC. >> >> Let me know what you think about the format. >> >> A few random thoughts and explanations about the current format: >> >> - next weeks report will highlight regressions that get added to regzbot >> over the next few days >> >> - I chose to categorize the regressions by identification status and by >> version line. Those where the culprit is identified come first, as they >> are the ones which most of the time can be solved by reverting the culprit >> >> - the entries in each section are ordered by time of last activity, >> which makes it easy to spot those where nothing happened recently, as >> they are near the end of a section. >> >> - the webui (https://linux-regtracking.leemhuis.info/regzbot/mainline/ ) >> links to the latest five activities regzbot noticed (an activity most of >> the time will be a mail send in reply to the report or a related thread >> that regzbot monitors). I for now chose to not do that in the report, as >> it would make it much longer and might be something that spam filters >> consider suspicious. >> >> That's it from my side. Let me know what you think. > > I like it, Glad to hear. Many thx for the feedback! > but as a maintainer, I find this hard to know what to do with > it. Well, I designed this a bit more for the tree maintainer, so Linus can: (1) see when things are not moving forward as quickly as he'd like to and get involved in the discussions or revert a commit (2) decide if he wants to release another rc or the final > As a maintainer, I want to be reminded of what regressions have been > reported for my subsystem, so I know what to do and who to go poke about > them. > > I could dig through the list and try to see if these are relevant to me, > but it's not always obvious. Yeah, that true. :-/ > How about something like "one email per issue" as a response from the > first report, that would cc: the needed maintainers (or people from > MAINTAINERS, you should be able to get that automatically from > get_maintainer.pl) and the subsystem mailing list (again from > get_maintainers.pl), so that I am constantly reminded that "you need to > get this fixed!". I'll keep this idea in mind. But that would result in a lot of mails. That's something I like to avoid, as we all get enough mails already: if regzbot starts to spam people, then some people might set up a mental or procmail filter to ignore all of them. That's why I settled on a different strategy: > Pester me, it's my job as a maintainer to get regressions resolved. I'm pestering developers and maintainers already, but I do it only when it seems to be needed. And I do it manually for now; I intend to make regzbot do it automatically sooner or later, but first I want to get a better feeling when it's appropriate to pester people without getting them quickly annoyed -- e.g. after what time and in which situations. Here for example I asked why a patch didn't proceed, and got a reply from the maintainer: https://lore.kernel.org/regressions/cfac5f5c-83d7-e0d9-5368-07ca041ebaed@leemhuis.info/ Seems the patch still didn't get any farther, so I guess it soon time for me to poke again. In another thread two pokes both helped to get things rolling again, but seems I soon need to send a third one, as it seems no one really feels responsible: https://lore.kernel.org/lkml/40550C00-4EE5-480F-AFD4-A2ACA01F9DBB@live.com/ Those are just two examples, I poked a few more discussions already and it most of the time helped. IOW: for now I'm trying the strategy "only send mails when its needed". But this made me aware that I should teach regzbot one thing: if the culprit is known, it should check if everyone in the SOB chain of the change is CCed to the discussion on the regression. > If > I see a long list with no names on it, It's easier for me to just ignore > :)> > Anyway, I know that's more work for you, don't do it if you don't want > to, but as you have individual items in your database already, maybe it > is easy to do, Not really, as for now regzbot doesn't assign regressions to a subsystem. I could add that, but how to fill and maintain that association? When the culprit is known, regzbot should be able to fill it automatically using the SOB chain or the path of the merge. But if it's not, it becomes something that some human would need to do manually. I really tried hard to avoid anything that requires manual work, as it always hard to make people do it :-/ That being said: I'm pretty sure I sooner or later will make regzbot store a association to a subsystem. > I don't know. Thanks again for doing this work, it's > great to see happen. thx again! Ciao, Thorsten
Ok, nice to see the new regression tracking bot start to show life. Greg had one suggestion, I have another - namely about grouping of these things. I like how you group them by "identified" and "unknown", because that's certainly very meaningful. But at the same time it does mean that if I look for "what are current issues with the development kernel", it ends up being very spread out: On Wed, Nov 24, 2021 at 1:25 AM Regzbot (on behalf of Thorsten Leemhuis) <regressions@leemhuis.info> wrote: > > ======================================================== > current cycle (v5.15.. aka v5.16-rc), culprit identified > ======================================================== ... > ========================================================================================= > previous cycle (v5.14..v5.15), culprit identified, with activity in the past three months > ========================================================================================= ... > ================================================================================== > older cycles (..v5.14), culprit identified, with activity in the past three months > ================================================================================== ... > ==================================================== > current cycle (v5.15.. aka v5.16-rc), unkown culprit > ==================================================== ... note how there was a lot of other stuff in between those "culprit idenfified" and "unknown culprit" for the current kernel. One of the things I really liked about the regression tracking you did before was that it helped me get a sense for the state of the release, and so I'd like to see that "current cycle" in one go. I suspect that Greg may have a slightly similar issue - as a driver maintainer, he cares about current cycle things (but mainly only when they affect his subsystems), but with his stable maintainer hat on he then cares more about the older cycles. Greg suggested splitting out the issues one by one - to try to have the right people on the Cc for any _particular_ issue, and while I think that's not the solution in this case (I very much want to see the "summary" email), it would be good to perhaps at least organize that summary email slightly differently. I suspect this is something we'd need to iterate on as we use this in our workflow, but that was my initial reaction to this first report. Linus
On 24.11.21 19:13, Linus Torvalds wrote: > Ok, nice to see the new regression tracking bot start to show life. Yeah. :-D Sadly one of my biggest problems with regression tracking remains: getting aware of regression reports. I can fully understand that most people don't care about regzbot for now, but it would really help if everyone would CC regressions@lists.linux.dev on mails regarding regressions (e.g. reports or any replies to them). > Greg had one suggestion, Still not sure how to approach his use case, but for now I started adding the usual subsystem commit summary prefixes (e.g. "net:", "usb:", "drm/amd") to the title of newly added regression, which might help somewhat and won't hurt. > I have another - namely about grouping of these things. > > I like how you group them by "identified" and "unknown", because > that's certainly very meaningful. > > But at the same time it does mean that if I look for "what are current > issues with the development kernel", it ends up being very spread out: Hah, fun fact: the order you purposed was the one I initially had in mind. But I later changed my mind, as I thought 'hey, if the culprit of the regression is known, it should be able to fix this quickly (e.g. by a revert, if there are no conflicts) even for regressions that made it into proper releases". But whatever: I'm totally fine with this and already changed the web interface yesterday after your mail arrived, only took a minute: https://linux-regtracking.leemhuis.info/regzbot/mainline/ Next report will use this order as well. > I suspect that Greg may have a slightly similar issue - as a driver > maintainer, he cares about current cycle things (but mainly only when > they affect his subsystems), but with his stable maintainer hat on he > then cares more about the older cycles. > > Greg suggested splitting out the issues one by one - to try to have > the right people on the Cc for any _particular_ issue, and while I > think that's not the solution in this case (I very much want to see > the "summary" email), it would be good to perhaps at least organize > that summary email slightly differently. > > I suspect this is something we'd need to iterate on as we use this in > our workflow Definitely. If there is something else you want to see changed or think is odd wrt to regzbot or my work as regression tracker, just let me know. > but that was my initial reaction to this first report. Thx for the feedback, much appreciated. Ciao, Thorsten