* [PATCH V3] nvme-pci: Fixes EEH failure on ppc
@ 2018-02-15 20:05 ` wenxiong
0 siblings, 0 replies; 8+ messages in thread
From: wenxiong @ 2018-02-15 20:05 UTC (permalink / raw)
To: linux-nvme; +Cc: keith.busch, axboe, linux-kernel, wenxiong, Wen Xiong
From: Wen Xiong <wenxiong@linux.vnet.ibm.com>
With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
timer), EEH recovery stops working on ppc.
After removing whatdog timer routine, when trigger EEH on ppc, we hit
EEH in nvme_timeout(). We would like to check if pci channel is offline
or not at the beginning of nvme_timeout(), if it is already offline,
we don't need to do future nvme timeout process.
Add mrmory barrier before calling pci_channel_offline().
With the patch, EEH recovery works successfuly on ppc.
Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
[ 232.585495] EEH: PHB#3 failure detected, location: N/A
[ 232.585545] CPU: 8 PID: 4873 Comm: kworker/8:1H Not tainted
4.14.0-6.el7a.ppc64le #1
[ 232.585646] Workqueue: kblockd blk_mq_timeout_work
[ 232.585705] Call Trace:
[ 232.585743] [c000003f7a533940] [c000000000c3556c]
dump_stack+0xb0/0xf4 (unreliable)
[ 232.585823] [c000003f7a533980] [c000000000043eb0]
eeh_check_failure+0x290/0x630
[ 232.585924] [c000003f7a533a30] [c008000011063f30]
nvme_timeout+0x1f0/0x410 [nvme]
[ 232.586038] [c000003f7a533b00] [c000000000637fc8]
blk_mq_check_expired+0x118/0x1a0
[ 232.586134] [c000003f7a533b80] [c00000000063e65c]
bt_for_each+0x11c/0x200
[ 232.586191] [c000003f7a533be0] [c00000000063f1f8]
blk_mq_queue_tag_busy_iter+0x78/0x110
[ 232.586272] [c000003f7a533c30] [c0000000006367b8]
blk_mq_timeout_work+0xa8/0x1c0
[ 232.586351] [c000003f7a533c80] [c00000000015d5ec]
process_one_work+0x1bc/0x5f0
[ 232.586431] [c000003f7a533d20] [c00000000016060c]
worker_thread+0xac/0x6b0
[ 232.586485] [c000003f7a533dc0] [c00000000016a528] kthread+0x168/0x1b0
[ 232.586539] [c000003f7a533e30] [c00000000000b4e8]
ret_from_kernel_thread+0x5c/0x74
[ 232.586640] nvme nvme0: I/O 10 QID 0 timeout, reset controller
[ 232.586640] EEH: Detected error on PHB#3
[ 232.586642] EEH: This PCI device has failed 1 times in the last hour
[ 232.586642] EEH: Notify device drivers to shutdown
[ 232.586645] nvme nvme0: frozen state error detected, reset controller
[ 234.098667] EEH: Collect temporary log
[ 234.098694] PHB4 PHB#3 Diag-data (Version: 1)
[ 234.098728] brdgCtl: 00000002
[ 234.098748] RootSts: 00070020 00402000 c1010008 00100107 00000000
[ 234.098807] RootErrSts: 00000000 00000020 00000001
[ 234.098878] nFir: 0000800000000000 0030001c00000000
0000800000000000
[ 234.098937] PhbSts: 0000001800000000 0000001800000000
[ 234.098990] Lem: 0000000100000100 0000000000000000
0000000100000000
[ 234.099067] PhbErr: 000004a000000000 0000008000000000
2148000098000240 a008400000000000
[ 234.099140] RxeMrgErr: 0000000000000001 0000000000000001
0000000000000000 0000000000000000
[ 234.099250] PcieDlp: 0000000000000000 0000000000000000
8000000000000000
[ 234.099326] RegbErr: 00d0000010000000 0000000010000000
8800005800000000 0000000007011000
[ 234.099418] EEH: Reset without hotplug activity
[ 237.317675] nvme 0003:01:00.0: Refused to change power state,
currently in D3
[ 237.317740] nvme 0003:01:00.0: Using 64-bit DMA iommu bypass
[ 237.317797] nvme nvme0: Removing after probe failure status: -19
[ 361.139047689,3] PHB#0003[0:3]: Escalating freeze to fence
PESTA[0]=a440002a01000000
[ 237.617706] EEH: Notify device drivers the completion of reset
[ 237.617754] nvme nvme0: restart after slot reset
[ 237.617834] EEH: Notify device driver to resume
[ 238.777746] nvme0n1: detected capacity change from 24576000000 to 0
[ 238.777841] nvme0n2: detected capacity change from 24576000000 to 0
[ 238.777944] nvme0n3: detected capacity change from 24576000000 to 0
[ 238.778019] nvme0n4: detected capacity change from 24576000000 to 0
[ 238.778132] nvme0n5: detected capacity change from 24576000000 to 0
[ 238.778222] nvme0n6: detected capacity change from 24576000000 to 0
[ 238.778314] nvme0n7: detected capacity change from 24576000000 to 0
[ 238.778416] nvme0n8: detected capacity change from 24576000000 to 0
---
---
drivers/nvme/host/pci.c | 13 +++++++------
1 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 6fe7af0..dfba90d 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1153,12 +1153,6 @@ static bool nvme_should_reset(struct nvme_dev *dev, u32 csts)
if (!(csts & NVME_CSTS_CFS) && !nssro)
return false;
- /* If PCI error recovery process is happening, we cannot reset or
- * the recovery mechanism will surely fail.
- */
- if (pci_channel_offline(to_pci_dev(dev->dev)))
- return false;
-
return true;
}
@@ -1189,6 +1183,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
struct nvme_command cmd;
u32 csts = readl(dev->bar + NVME_REG_CSTS);
+ /* If PCI error recovery process is happening, we cannot reset or
+ * the recovery mechanism will surely fail.
+ */
+ mb();
+ if (pci_channel_offline(to_pci_dev(dev->dev)))
+ return BLK_EH_RESET_TIMER;
+
/*
* Reset immediately if the controller is failed
*/
--
1.7.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH V3] nvme-pci: Fixes EEH failure on ppc
@ 2018-02-15 20:05 ` wenxiong
0 siblings, 0 replies; 8+ messages in thread
From: wenxiong @ 2018-02-15 20:05 UTC (permalink / raw)
From: Wen Xiong <wenxiong@linux.vnet.ibm.com>
With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
timer), EEH recovery stops working on ppc.
After removing whatdog timer routine, when trigger EEH on ppc, we hit
EEH in nvme_timeout(). We would like to check if pci channel is offline
or not at the beginning of nvme_timeout(), if it is already offline,
we don't need to do future nvme timeout process.
Add mrmory barrier before calling pci_channel_offline().
With the patch, EEH recovery works successfuly on ppc.
Signed-off-by: Wen Xiong <wenxiong at linux.vnet.ibm.com>
[ 232.585495] EEH: PHB#3 failure detected, location: N/A
[ 232.585545] CPU: 8 PID: 4873 Comm: kworker/8:1H Not tainted
4.14.0-6.el7a.ppc64le #1
[ 232.585646] Workqueue: kblockd blk_mq_timeout_work
[ 232.585705] Call Trace:
[ 232.585743] [c000003f7a533940] [c000000000c3556c]
dump_stack+0xb0/0xf4 (unreliable)
[ 232.585823] [c000003f7a533980] [c000000000043eb0]
eeh_check_failure+0x290/0x630
[ 232.585924] [c000003f7a533a30] [c008000011063f30]
nvme_timeout+0x1f0/0x410 [nvme]
[ 232.586038] [c000003f7a533b00] [c000000000637fc8]
blk_mq_check_expired+0x118/0x1a0
[ 232.586134] [c000003f7a533b80] [c00000000063e65c]
bt_for_each+0x11c/0x200
[ 232.586191] [c000003f7a533be0] [c00000000063f1f8]
blk_mq_queue_tag_busy_iter+0x78/0x110
[ 232.586272] [c000003f7a533c30] [c0000000006367b8]
blk_mq_timeout_work+0xa8/0x1c0
[ 232.586351] [c000003f7a533c80] [c00000000015d5ec]
process_one_work+0x1bc/0x5f0
[ 232.586431] [c000003f7a533d20] [c00000000016060c]
worker_thread+0xac/0x6b0
[ 232.586485] [c000003f7a533dc0] [c00000000016a528] kthread+0x168/0x1b0
[ 232.586539] [c000003f7a533e30] [c00000000000b4e8]
ret_from_kernel_thread+0x5c/0x74
[ 232.586640] nvme nvme0: I/O 10 QID 0 timeout, reset controller
[ 232.586640] EEH: Detected error on PHB#3
[ 232.586642] EEH: This PCI device has failed 1 times in the last hour
[ 232.586642] EEH: Notify device drivers to shutdown
[ 232.586645] nvme nvme0: frozen state error detected, reset controller
[ 234.098667] EEH: Collect temporary log
[ 234.098694] PHB4 PHB#3 Diag-data (Version: 1)
[ 234.098728] brdgCtl: 00000002
[ 234.098748] RootSts: 00070020 00402000 c1010008 00100107 00000000
[ 234.098807] RootErrSts: 00000000 00000020 00000001
[ 234.098878] nFir: 0000800000000000 0030001c00000000
0000800000000000
[ 234.098937] PhbSts: 0000001800000000 0000001800000000
[ 234.098990] Lem: 0000000100000100 0000000000000000
0000000100000000
[ 234.099067] PhbErr: 000004a000000000 0000008000000000
2148000098000240 a008400000000000
[ 234.099140] RxeMrgErr: 0000000000000001 0000000000000001
0000000000000000 0000000000000000
[ 234.099250] PcieDlp: 0000000000000000 0000000000000000
8000000000000000
[ 234.099326] RegbErr: 00d0000010000000 0000000010000000
8800005800000000 0000000007011000
[ 234.099418] EEH: Reset without hotplug activity
[ 237.317675] nvme 0003:01:00.0: Refused to change power state,
currently in D3
[ 237.317740] nvme 0003:01:00.0: Using 64-bit DMA iommu bypass
[ 237.317797] nvme nvme0: Removing after probe failure status: -19
[ 361.139047689,3] PHB#0003[0:3]: Escalating freeze to fence
PESTA[0]=a440002a01000000
[ 237.617706] EEH: Notify device drivers the completion of reset
[ 237.617754] nvme nvme0: restart after slot reset
[ 237.617834] EEH: Notify device driver to resume
[ 238.777746] nvme0n1: detected capacity change from 24576000000 to 0
[ 238.777841] nvme0n2: detected capacity change from 24576000000 to 0
[ 238.777944] nvme0n3: detected capacity change from 24576000000 to 0
[ 238.778019] nvme0n4: detected capacity change from 24576000000 to 0
[ 238.778132] nvme0n5: detected capacity change from 24576000000 to 0
[ 238.778222] nvme0n6: detected capacity change from 24576000000 to 0
[ 238.778314] nvme0n7: detected capacity change from 24576000000 to 0
[ 238.778416] nvme0n8: detected capacity change from 24576000000 to 0
---
---
drivers/nvme/host/pci.c | 13 +++++++------
1 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 6fe7af0..dfba90d 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1153,12 +1153,6 @@ static bool nvme_should_reset(struct nvme_dev *dev, u32 csts)
if (!(csts & NVME_CSTS_CFS) && !nssro)
return false;
- /* If PCI error recovery process is happening, we cannot reset or
- * the recovery mechanism will surely fail.
- */
- if (pci_channel_offline(to_pci_dev(dev->dev)))
- return false;
-
return true;
}
@@ -1189,6 +1183,13 @@ static enum blk_eh_timer_return nvme_timeout(struct request *req, bool reserved)
struct nvme_command cmd;
u32 csts = readl(dev->bar + NVME_REG_CSTS);
+ /* If PCI error recovery process is happening, we cannot reset or
+ * the recovery mechanism will surely fail.
+ */
+ mb();
+ if (pci_channel_offline(to_pci_dev(dev->dev)))
+ return BLK_EH_RESET_TIMER;
+
/*
* Reset immediately if the controller is failed
*/
--
1.7.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH V3] nvme-pci: Fixes EEH failure on ppc
2018-02-15 20:05 ` wenxiong
@ 2018-02-28 22:31 ` wenxiong
-1 siblings, 0 replies; 8+ messages in thread
From: wenxiong @ 2018-02-28 22:31 UTC (permalink / raw)
To: wenxiong; +Cc: linux-nvme, keith.busch, axboe, linux-kernel, wenxiong
On 2018-02-15 14:05, wenxiong@linux.vnet.ibm.com wrote:
> From: Wen Xiong <wenxiong@linux.vnet.ibm.com>
>
> With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
> timer), EEH recovery stops working on ppc.
>
> After removing whatdog timer routine, when trigger EEH on ppc, we hit
> EEH in nvme_timeout(). We would like to check if pci channel is offline
> or not at the beginning of nvme_timeout(), if it is already offline,
> we don't need to do future nvme timeout process.
>
> Add mrmory barrier before calling pci_channel_offline().
>
> With the patch, EEH recovery works successfuly on ppc.
>
> Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
Hi Keith and All,
We have the newer Linux distro releases came out recently, so I got more
reports for this issue.
Test teams have verified the patch in several distro kernel
version(v4.14, v4.15).
If you have any question about the V3 patch I submitted two weeks, Let
me know.
Thanks,
Wendy
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH V3] nvme-pci: Fixes EEH failure on ppc
@ 2018-02-28 22:31 ` wenxiong
0 siblings, 0 replies; 8+ messages in thread
From: wenxiong @ 2018-02-28 22:31 UTC (permalink / raw)
On 2018-02-15 14:05, wenxiong@linux.vnet.ibm.com wrote:
> From: Wen Xiong <wenxiong at linux.vnet.ibm.com>
>
> With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
> timer), EEH recovery stops working on ppc.
>
> After removing whatdog timer routine, when trigger EEH on ppc, we hit
> EEH in nvme_timeout(). We would like to check if pci channel is offline
> or not at the beginning of nvme_timeout(), if it is already offline,
> we don't need to do future nvme timeout process.
>
> Add mrmory barrier before calling pci_channel_offline().
>
> With the patch, EEH recovery works successfuly on ppc.
>
> Signed-off-by: Wen Xiong <wenxiong at linux.vnet.ibm.com>
Hi Keith and All,
We have the newer Linux distro releases came out recently, so I got more
reports for this issue.
Test teams have verified the patch in several distro kernel
version(v4.14, v4.15).
If you have any question about the V3 patch I submitted two weeks, Let
me know.
Thanks,
Wendy
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH V3] nvme-pci: Fixes EEH failure on ppc
2018-02-28 22:31 ` wenxiong
@ 2018-02-28 23:55 ` Keith Busch
-1 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2018-02-28 23:55 UTC (permalink / raw)
To: wenxiong; +Cc: linux-nvme, axboe, linux-kernel, wenxiong
On Wed, Feb 28, 2018 at 04:31:37PM -0600, wenxiong wrote:
> On 2018-02-15 14:05, wenxiong@linux.vnet.ibm.com wrote:
> > From: Wen Xiong <wenxiong@linux.vnet.ibm.com>
> >
> > With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
> > timer), EEH recovery stops working on ppc.
> >
> > After removing whatdog timer routine, when trigger EEH on ppc, we hit
> > EEH in nvme_timeout(). We would like to check if pci channel is offline
> > or not at the beginning of nvme_timeout(), if it is already offline,
> > we don't need to do future nvme timeout process.
> >
> > Add mrmory barrier before calling pci_channel_offline().
> >
> > With the patch, EEH recovery works successfuly on ppc.
> >
> > Signed-off-by: Wen Xiong <wenxiong@linux.vnet.ibm.com>
>
> Hi Keith and All,
>
> We have the newer Linux distro releases came out recently, so I got more
> reports for this issue.
>
> Test teams have verified the patch in several distro kernel version(v4.14,
> v4.15).
>
>
> If you have any question about the V3 patch I submitted two weeks, Let me
> know.
Sorry, I'd nearly forgotten about this one.
We need a better change log. Could you help explain how this really
works? As I understand it, I would write it something like this, but
let me know if there's more to it:
Triggering PPC EEH detection and handling requires a memory mapped
read failure. The NVMe driver removed the periodic health check MMIO,
so there's no early detection mechanism to trigger the recovery.
Instead, the detection happens when the nvme driver handles an IO
timeout event. Since this takes the pci channel offline, we do not
want the driver to proceed with escalating its own recovery efforts
that may conflict with the EEH handler. This patch ensures the driver
will observe the channel was set to offline after a failed MMIO read
and resets the IO timer so the EEH handler has a chance to recover
the device.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH V3] nvme-pci: Fixes EEH failure on ppc
@ 2018-02-28 23:55 ` Keith Busch
0 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2018-02-28 23:55 UTC (permalink / raw)
On Wed, Feb 28, 2018@04:31:37PM -0600, wenxiong wrote:
> On 2018-02-15 14:05, wenxiong@linux.vnet.ibm.com wrote:
> > From: Wen Xiong <wenxiong at linux.vnet.ibm.com>
> >
> > With b2a0eb1a0ac72869c910a79d935a0b049ec78ad9(nvme-pci: Remove watchdog
> > timer), EEH recovery stops working on ppc.
> >
> > After removing whatdog timer routine, when trigger EEH on ppc, we hit
> > EEH in nvme_timeout(). We would like to check if pci channel is offline
> > or not at the beginning of nvme_timeout(), if it is already offline,
> > we don't need to do future nvme timeout process.
> >
> > Add mrmory barrier before calling pci_channel_offline().
> >
> > With the patch, EEH recovery works successfuly on ppc.
> >
> > Signed-off-by: Wen Xiong <wenxiong at linux.vnet.ibm.com>
>
> Hi Keith and All,
>
> We have the newer Linux distro releases came out recently, so I got more
> reports for this issue.
>
> Test teams have verified the patch in several distro kernel version(v4.14,
> v4.15).
>
>
> If you have any question about the V3 patch I submitted two weeks, Let me
> know.
Sorry, I'd nearly forgotten about this one.
We need a better change log. Could you help explain how this really
works? As I understand it, I would write it something like this, but
let me know if there's more to it:
Triggering PPC EEH detection and handling requires a memory mapped
read failure. The NVMe driver removed the periodic health check MMIO,
so there's no early detection mechanism to trigger the recovery.
Instead, the detection happens when the nvme driver handles an IO
timeout event. Since this takes the pci channel offline, we do not
want the driver to proceed with escalating its own recovery efforts
that may conflict with the EEH handler. This patch ensures the driver
will observe the channel was set to offline after a failed MMIO read
and resets the IO timer so the EEH handler has a chance to recover
the device.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH V3] nvme-pci: Fixes EEH failure on ppc
[not found] ` <OFB428CB19.833F7B7C-ON00258243.005E4D59-86258243.005E7EAB@notes.na.collabserv.com>
@ 2018-03-01 17:16 ` Keith Busch
0 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2018-03-01 17:16 UTC (permalink / raw)
To: Wen Xiong; +Cc: axboe, linux-kernel, linux-nvme, wenxiong
On Thu, Mar 01, 2018 at 11:12:08AM -0600, Wen Xiong wrote:
> Hi Keith,
>
> It is perfect! I go with it.
Thanks, queued up for 4.16.
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH V3] nvme-pci: Fixes EEH failure on ppc
@ 2018-03-01 17:16 ` Keith Busch
0 siblings, 0 replies; 8+ messages in thread
From: Keith Busch @ 2018-03-01 17:16 UTC (permalink / raw)
On Thu, Mar 01, 2018@11:12:08AM -0600, Wen Xiong wrote:
> Hi Keith,
>
> It is perfect! I go with it.
Thanks, queued up for 4.16.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-03-01 17:16 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-15 20:05 [PATCH V3] nvme-pci: Fixes EEH failure on ppc wenxiong
2018-02-15 20:05 ` wenxiong
2018-02-28 22:31 ` wenxiong
2018-02-28 22:31 ` wenxiong
2018-02-28 23:55 ` Keith Busch
2018-02-28 23:55 ` Keith Busch
[not found] ` <OFB428CB19.833F7B7C-ON00258243.005E4D59-86258243.005E7EAB@notes.na.collabserv.com>
2018-03-01 17:16 ` Keith Busch
2018-03-01 17:16 ` Keith Busch
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.