From: Niklas Schnelle <schnelle@linux.ibm.com>
To: Amir Tzin <amirtz@nvidia.com>, Moshe Shemesh <moshe@nvidia.com>,
Saeed Mahameed <saeedm@nvidia.com>
Cc: linux-netdev <linux-netdev@vger.kernel.org>,
regressions@lists.linux.dev,
linux-s390 <linux-s390@vger.kernel.org>
Subject: Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years
Date: Fri, 19 Nov 2021 11:47:52 +0100 [thread overview]
Message-ID: <a627b3cbd2ed4ba5dd75e1163760730187f1416b.camel@linux.ibm.com> (raw)
Hello Amir, Moshe, and Saeed,
During testing of PCI device recovery, I found a problem in the mlx5
recovery support introduced in v5.16-rc1 by commit 32def4120e48
("net/mlx5: Read timeout values from DTOR"). It follows my analysis of
the problem.
When the device is in an error state, at least on s390 but I believe
also on other systems, it is isolated and all PCI MMIO reads return
0xff. This is detected by your driver and it will immediately attempt
to recovery the device with a mlx5_core driver specific recovery
mechanism. Since at this point no reset has been done that would take
the device out of isolation this will of course fail as it can't
communicate with the device. Under normal circumstances this reset
would happen later during the new recovery flow introduced in
4cdf2f4e24ff ("s390/pci: implement minimal PCI error recovery") once
firmware has done their side of the recovery allowing that to succeed
once the driver specific recovery has failed.
With v5.16-rc1 however the driver specific recovery gets stuck holding
locks which will block our recovery. Without our recovery mechanism
this can also be seen by "echo 1 > /sys/bus/pci/devices/<dev>/remove"
which hangs on the device lock forever.
Digging into this I tracked the problem down to
mlx5_health_wait_pci_up() hangig. I added a debug print to it and it
turns out that with the device isolated mlx5_tout_ms(dev, FW_RESET)
returns 774039849367420401 (0x6B...6B) milliseconds and we try to wait
245 million years. After reverting that commit things work again,
though of course the driver specific recovery flow will still fail
before ours can kick in and finally succeed.
Thanks,
Niklas Schnelle
#regzbot introduced: 32def4120e48
next reply other threads:[~2021-11-19 10:47 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-19 10:47 Niklas Schnelle [this message]
2021-11-19 11:38 ` Regression in v5.16-rc1: Timeout in mlx5_health_wait_pci_up() may try to wait 245 million years Thorsten Leemhuis
2021-11-19 12:17 ` Thorsten Leemhuis
2021-11-19 10:58 Niklas Schnelle
2021-11-20 16:38 ` Moshe Shemesh
2021-12-02 6:52 ` Thorsten Leemhuis
2021-12-02 10:05 ` Moshe Shemesh
2021-12-02 13:56 ` Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a627b3cbd2ed4ba5dd75e1163760730187f1416b.camel@linux.ibm.com \
--to=schnelle@linux.ibm.com \
--cc=amirtz@nvidia.com \
--cc=linux-netdev@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=moshe@nvidia.com \
--cc=regressions@lists.linux.dev \
--cc=saeedm@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).