* [PATCH blktests] Fix block/011 to not use sysfs for device disabling
@ 2018-05-18 17:42 ` Keith Busch
0 siblings, 0 replies; 12+ messages in thread
From: Keith Busch @ 2018-05-18 17:42 UTC (permalink / raw)
The PCI sysfs interface may not be a dependable method for toggling the
PCI device state to trigger the timeouts. This patch goes directly to
the config space to make device failure occur.
The success of this test is still senstive to timing, as it may disable
IO memory when a driver is trying to bring it online. This can look like
a permanent device failure from the driver's perspective.
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
tests/block/011 | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tests/block/011 b/tests/block/011
index 62e89f7..2fc0ffb 100755
--- a/tests/block/011
+++ b/tests/block/011
@@ -21,7 +21,7 @@ DESCRIPTION="disable PCI device while doing I/O"
TIMED=1
requires() {
- _have_fio
+ _have_fio && _have_program setpci
}
device_requires() {
@@ -43,10 +43,11 @@ test_device() {
_run_fio_rand_io --filename="$TEST_DEV" --size="$size" \
--ignore_error=EIO,ENXIO,ENODEV &
+ # toggle PCI Command Register's Memory and Bus Master enabling
while kill -0 $! 2>/dev/null; do
- echo 0 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=0:6
sleep .2
- echo 1 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=6:6
sleep .2
done
--
2.14.3
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
2018-05-18 17:42 ` Keith Busch
@ 2018-05-19 11:58 ` Ming Lei
-1 siblings, 0 replies; 12+ messages in thread
From: Ming Lei @ 2018-05-19 11:58 UTC (permalink / raw)
To: Keith Busch
Cc: Omar Sandoval, linux-block, linux-nvme, Jens Axboe, Ming Lei,
Christoph Hellwig, Johannes Thumshirn
On Sat, May 19, 2018 at 1:42 AM, Keith Busch <keith.busch@intel.com> wrote:
> The PCI sysfs interface may not be a dependable method for toggling the
> PCI device state to trigger the timeouts. This patch goes directly to
> the config space to make device failure occur.
>
> The success of this test is still senstive to timing, as it may disable
> IO memory when a driver is trying to bring it online. This can look like
> a permanent device failure from the driver's perspective.
>
> Signed-off-by: Keith Busch <keith.busch@intel.com>
> ---
> tests/block/011 | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/tests/block/011 b/tests/block/011
> index 62e89f7..2fc0ffb 100755
> --- a/tests/block/011
> +++ b/tests/block/011
> @@ -21,7 +21,7 @@ DESCRIPTION="disable PCI device while doing I/O"
> TIMED=1
>
> requires() {
> - _have_fio
> + _have_fio && _have_program setpci
> }
>
> device_requires() {
> @@ -43,10 +43,11 @@ test_device() {
> _run_fio_rand_io --filename="$TEST_DEV" --size="$size" \
> --ignore_error=EIO,ENXIO,ENODEV &
>
> + # toggle PCI Command Register's Memory and Bus Master enabling
> while kill -0 $! 2>/dev/null; do
> - echo 0 > "/sys/bus/pci/devices/${pdev}/enable"
> + setpci -s "${pdev}" 4.w=0:6
> sleep .2
> - echo 1 > "/sys/bus/pci/devices/${pdev}/enable"
> + setpci -s "${pdev}" 4.w=6:6
> sleep .2
> done
Given 'echo 0 > .../${pdev}/enable' clears 'Bus Master' bit only,
I suggest this patch only clear and set 'Bus Master' bit too.
And similar issue can be triggered as before if only 'Bus Master'
is touched.
Otherwise, this patch looks fine.
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH blktests] Fix block/011 to not use sysfs for device disabling
@ 2018-05-19 11:58 ` Ming Lei
0 siblings, 0 replies; 12+ messages in thread
From: Ming Lei @ 2018-05-19 11:58 UTC (permalink / raw)
On Sat, May 19, 2018@1:42 AM, Keith Busch <keith.busch@intel.com> wrote:
> The PCI sysfs interface may not be a dependable method for toggling the
> PCI device state to trigger the timeouts. This patch goes directly to
> the config space to make device failure occur.
>
> The success of this test is still senstive to timing, as it may disable
> IO memory when a driver is trying to bring it online. This can look like
> a permanent device failure from the driver's perspective.
>
> Signed-off-by: Keith Busch <keith.busch at intel.com>
> ---
> tests/block/011 | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/tests/block/011 b/tests/block/011
> index 62e89f7..2fc0ffb 100755
> --- a/tests/block/011
> +++ b/tests/block/011
> @@ -21,7 +21,7 @@ DESCRIPTION="disable PCI device while doing I/O"
> TIMED=1
>
> requires() {
> - _have_fio
> + _have_fio && _have_program setpci
> }
>
> device_requires() {
> @@ -43,10 +43,11 @@ test_device() {
> _run_fio_rand_io --filename="$TEST_DEV" --size="$size" \
> --ignore_error=EIO,ENXIO,ENODEV &
>
> + # toggle PCI Command Register's Memory and Bus Master enabling
> while kill -0 $! 2>/dev/null; do
> - echo 0 > "/sys/bus/pci/devices/${pdev}/enable"
> + setpci -s "${pdev}" 4.w=0:6
> sleep .2
> - echo 1 > "/sys/bus/pci/devices/${pdev}/enable"
> + setpci -s "${pdev}" 4.w=6:6
> sleep .2
> done
Given 'echo 0 > .../${pdev}/enable' clears 'Bus Master' bit only,
I suggest this patch only clear and set 'Bus Master' bit too.
And similar issue can be triggered as before if only 'Bus Master'
is touched.
Otherwise, this patch looks fine.
Thanks,
Ming Lei
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
2018-05-18 17:42 ` Keith Busch
@ 2018-05-21 6:37 ` Yi Zhang
-1 siblings, 0 replies; 12+ messages in thread
From: Yi Zhang @ 2018-05-21 6:37 UTC (permalink / raw)
To: Keith Busch
Cc: Omar Sandoval, linux-block, linux-nvme, Johannes Thumshirn,
Christoph Hellwig, Jens Axboe, Ming Lei
Hi Keith
I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
Console log:
storageqe-62 login:
Kernel 4.17.0-rc5 on an x86_64
storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
[ 1058.118261] {1}[Hardware Error]: event severity: fatal
[ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
[ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
[ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
[ 1058.118267] {1}[Hardware Error]: version: 1.16
[ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
[ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
[ 1058.118271] {1}[Hardware Error]: slot: 0
[ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
[ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
[ 1058.118274] {1}[Hardware Error]: class_code: 020801
[ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
[ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Best Regards,
Yi Zhang
----- Original Message -----
From: "Keith Busch" <keith.busch@intel.com>
To: "Omar Sandoval" <osandov@osandov.com>, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org
Cc: "Johannes Thumshirn" <jthumshirn@suse.de>, "Christoph Hellwig" <hch@lst.de>, "Jens Axboe" <axboe@kernel.dk>, "Ming Lei" <ming.lei@redhat.com>, "Keith Busch" <keith.busch@intel.com>
Sent: Saturday, May 19, 2018 1:42:47 AM
Subject: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
The PCI sysfs interface may not be a dependable method for toggling the
PCI device state to trigger the timeouts. This patch goes directly to
the config space to make device failure occur.
The success of this test is still senstive to timing, as it may disable
IO memory when a driver is trying to bring it online. This can look like
a permanent device failure from the driver's perspective.
Signed-off-by: Keith Busch <keith.busch@intel.com>
---
tests/block/011 | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tests/block/011 b/tests/block/011
index 62e89f7..2fc0ffb 100755
--- a/tests/block/011
+++ b/tests/block/011
@@ -21,7 +21,7 @@ DESCRIPTION="disable PCI device while doing I/O"
TIMED=1
requires() {
- _have_fio
+ _have_fio && _have_program setpci
}
device_requires() {
@@ -43,10 +43,11 @@ test_device() {
_run_fio_rand_io --filename="$TEST_DEV" --size="$size" \
--ignore_error=EIO,ENXIO,ENODEV &
+ # toggle PCI Command Register's Memory and Bus Master enabling
while kill -0 $! 2>/dev/null; do
- echo 0 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=0:6
sleep .2
- echo 1 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=6:6
sleep .2
done
--
2.14.3
^ permalink raw reply related [flat|nested] 12+ messages in thread* [PATCH blktests] Fix block/011 to not use sysfs for device disabling
@ 2018-05-21 6:37 ` Yi Zhang
0 siblings, 0 replies; 12+ messages in thread
From: Yi Zhang @ 2018-05-21 6:37 UTC (permalink / raw)
Hi Keith
I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
Console log:
storageqe-62 login:
Kernel 4.17.0-rc5 on an x86_64
storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
[ 1058.118261] {1}[Hardware Error]: event severity: fatal
[ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
[ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
[ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
[ 1058.118267] {1}[Hardware Error]: version: 1.16
[ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
[ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
[ 1058.118271] {1}[Hardware Error]: slot: 0
[ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
[ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
[ 1058.118274] {1}[Hardware Error]: class_code: 020801
[ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
[ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Best Regards,
Yi Zhang
----- Original Message -----
From: "Keith Busch" <keith.busch@intel.com>
To: "Omar Sandoval" <osandov at osandov.com>, linux-block at vger.kernel.org, linux-nvme at lists.infradead.org
Cc: "Johannes Thumshirn" <jthumshirn at suse.de>, "Christoph Hellwig" <hch at lst.de>, "Jens Axboe" <axboe at kernel.dk>, "Ming Lei" <ming.lei at redhat.com>, "Keith Busch" <keith.busch at intel.com>
Sent: Saturday, May 19, 2018 1:42:47 AM
Subject: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
The PCI sysfs interface may not be a dependable method for toggling the
PCI device state to trigger the timeouts. This patch goes directly to
the config space to make device failure occur.
The success of this test is still senstive to timing, as it may disable
IO memory when a driver is trying to bring it online. This can look like
a permanent device failure from the driver's perspective.
Signed-off-by: Keith Busch <keith.busch at intel.com>
---
tests/block/011 | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tests/block/011 b/tests/block/011
index 62e89f7..2fc0ffb 100755
--- a/tests/block/011
+++ b/tests/block/011
@@ -21,7 +21,7 @@ DESCRIPTION="disable PCI device while doing I/O"
TIMED=1
requires() {
- _have_fio
+ _have_fio && _have_program setpci
}
device_requires() {
@@ -43,10 +43,11 @@ test_device() {
_run_fio_rand_io --filename="$TEST_DEV" --size="$size" \
--ignore_error=EIO,ENXIO,ENODEV &
+ # toggle PCI Command Register's Memory and Bus Master enabling
while kill -0 $! 2>/dev/null; do
- echo 0 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=0:6
sleep .2
- echo 1 > "/sys/bus/pci/devices/${pdev}/enable"
+ setpci -s "${pdev}" 4.w=6:6
sleep .2
done
--
2.14.3
^ permalink raw reply related [flat|nested] 12+ messages in thread* Re: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
2018-05-21 6:37 ` Yi Zhang
@ 2018-05-21 14:08 ` Keith Busch
-1 siblings, 0 replies; 12+ messages in thread
From: Keith Busch @ 2018-05-21 14:08 UTC (permalink / raw)
To: Yi Zhang
Cc: Keith Busch, Jens Axboe, linux-nvme, Ming Lei, linux-block,
Johannes Thumshirn, Omar Sandoval, Christoph Hellwig
On Mon, May 21, 2018 at 02:37:56AM -0400, Yi Zhang wrote:
> Hi Keith
> I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
>
> Console log:
> storageqe-62 login:
> Kernel 4.17.0-rc5 on an x86_64
>
> storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
> [ 1058.118261] {1}[Hardware Error]: event severity: fatal
> [ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
> [ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
> [ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
> [ 1058.118267] {1}[Hardware Error]: version: 1.16
> [ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
> [ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
> [ 1058.118271] {1}[Hardware Error]: slot: 0
> [ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
> [ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
> [ 1058.118274] {1}[Hardware Error]: class_code: 020801
> [ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
> [ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Thanks for the notice. The test may be going to far with the config
registers it's touching. Let me see if we just do the BME bit as Ming
suggested fixes this.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH blktests] Fix block/011 to not use sysfs for device disabling
@ 2018-05-21 14:08 ` Keith Busch
0 siblings, 0 replies; 12+ messages in thread
From: Keith Busch @ 2018-05-21 14:08 UTC (permalink / raw)
On Mon, May 21, 2018@02:37:56AM -0400, Yi Zhang wrote:
> Hi Keith
> I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
>
> Console log:
> storageqe-62 login:
> Kernel 4.17.0-rc5 on an x86_64
>
> storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
> [ 1058.118261] {1}[Hardware Error]: event severity: fatal
> [ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
> [ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
> [ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
> [ 1058.118267] {1}[Hardware Error]: version: 1.16
> [ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
> [ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
> [ 1058.118271] {1}[Hardware Error]: slot: 0
> [ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
> [ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
> [ 1058.118274] {1}[Hardware Error]: class_code: 020801
> [ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
> [ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Thanks for the notice. The test may be going to far with the config
registers it's touching. Let me see if we just do the BME bit as Ming
suggested fixes this.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
2018-05-21 14:08 ` Keith Busch
@ 2018-05-29 19:54 ` Omar Sandoval
-1 siblings, 0 replies; 12+ messages in thread
From: Omar Sandoval @ 2018-05-29 19:54 UTC (permalink / raw)
To: Keith Busch
Cc: Yi Zhang, Keith Busch, Jens Axboe, linux-nvme, Ming Lei,
linux-block, Johannes Thumshirn, Christoph Hellwig
On Mon, May 21, 2018 at 08:08:21AM -0600, Keith Busch wrote:
> On Mon, May 21, 2018 at 02:37:56AM -0400, Yi Zhang wrote:
> > Hi Keith
> > I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
> >
> > Console log:
> > storageqe-62 login:
> > Kernel 4.17.0-rc5 on an x86_64
> >
> > storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
> > [ 1058.118261] {1}[Hardware Error]: event severity: fatal
> > [ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
> > [ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
> > [ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
> > [ 1058.118267] {1}[Hardware Error]: version: 1.16
> > [ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
> > [ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
> > [ 1058.118271] {1}[Hardware Error]: slot: 0
> > [ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
> > [ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
> > [ 1058.118274] {1}[Hardware Error]: class_code: 020801
> > [ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
> > [ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
> Thanks for the notice. The test may be going to far with the config
> registers it's touching. Let me see if we just do the BME bit as Ming
> suggested fixes this.
What's the plan for this test? Do you have a v2 coming?
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH blktests] Fix block/011 to not use sysfs for device disabling
@ 2018-05-29 19:54 ` Omar Sandoval
0 siblings, 0 replies; 12+ messages in thread
From: Omar Sandoval @ 2018-05-29 19:54 UTC (permalink / raw)
On Mon, May 21, 2018@08:08:21AM -0600, Keith Busch wrote:
> On Mon, May 21, 2018@02:37:56AM -0400, Yi Zhang wrote:
> > Hi Keith
> > I tried this patch on my R730 Server, but it lead to system hang after setpci, could you help check it, thanks.
> >
> > Console log:
> > storageqe-62 login:
> > Kernel 4.17.0-rc5 on an x86_64
> >
> > storageqe-62 login: [ 1058.118258] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 3
> > [ 1058.118261] {1}[Hardware Error]: event severity: fatal
> > [ 1058.118262] {1}[Hardware Error]: Error 0, type: fatal
> > [ 1058.118265] {1}[Hardware Error]: section_type: PCIe error
> > [ 1058.118266] {1}[Hardware Error]: port_type: 0, PCIe end point
> > [ 1058.118267] {1}[Hardware Error]: version: 1.16
> > [ 1058.118269] {1}[Hardware Error]: command: 0x0400, status: 0x0010
> > [ 1058.118270] {1}[Hardware Error]: device_id: 0000:85:00.0
> > [ 1058.118271] {1}[Hardware Error]: slot: 0
> > [ 1058.118271] {1}[Hardware Error]: secondary_bus: 0x00
> > [ 1058.118273] {1}[Hardware Error]: vendor_id: 0x144d, device_id: 0xa821
> > [ 1058.118274] {1}[Hardware Error]: class_code: 020801
> > [ 1058.118275] Kernel panic - not syncing: Fatal hardware error!
> > [ 1058.118301] Kernel Offset: 0x14800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
> Thanks for the notice. The test may be going to far with the config
> registers it's touching. Let me see if we just do the BME bit as Ming
> suggested fixes this.
What's the plan for this test? Do you have a v2 coming?
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [PATCH blktests] Fix block/011 to not use sysfs for device disabling
2018-05-29 19:54 ` Omar Sandoval
@ 2018-06-04 19:38 ` Keith Busch
-1 siblings, 0 replies; 12+ messages in thread
From: Keith Busch @ 2018-06-04 19:38 UTC (permalink / raw)
To: Omar Sandoval
Cc: Jens Axboe, linux-block, Yi Zhang, linux-nvme, Ming Lei,
Keith Busch, Johannes Thumshirn, Christoph Hellwig
On Tue, May 29, 2018 at 12:54:28PM -0700, Omar Sandoval wrote:
> What's the plan for this test? Do you have a v2 coming?
Sorry for the delay. I've been out on holiday, but I'm catching up
quickly and will send a v2 shortly.
Thanks,
Keith
^ permalink raw reply [flat|nested] 12+ messages in thread