linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
@ 2019-12-27 14:08 Dakshaja Uppalapati
  2019-12-27 15:03 ` Eduard Hasenleithner
  0 siblings, 1 reply; 6+ messages in thread
From: Dakshaja Uppalapati @ 2019-12-27 14:08 UTC (permalink / raw)
  To: linux-nvme, kbusch, eduard; +Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

Hi,

I am trying to setup and run NVMF with tot linux kernel installed(5.5.0-rc3) on
both target and host. Attached is the target configuration using ramdisk device.

On host, I tried to format the LUN discovered from target machine by using
#mkfs.ext3 /dev/nvme0n1.

The following error is seen in dmesg:
[  219.507989] blk_update_request: I/O error, dev nvme0c0n1, sector 0 op 0x3:(DISCARD) flags 0x4000800 phys_seg 1 prio class 0

The above error is seen from kernel-5.5-rc1 onwards. Bisect points to the 
following commit:

530436c45ef2e446c12538a400e465929a0b3ade is the first bad commit 
commit 530436c45ef2e446c12538a400e465929a0b3ade
Author: Eduard Hasenleithner <eduard@hasenleithner.at>
Date:   Tue Nov 12 21:55:01 2019 +0100

    nvme: Discard workaround for non-conformant devices

    Users observe IOMMU related errors when performing discard on nvme from
    non-compliant nvme devices reading beyond the end of the DMA mapped
    ranges to discard.

    Two different variants of this behavior have been observed: SM22XX
    controllers round up the read size to a multiple of 512 bytes, and Phison
    E12 unconditionally reads the maximum discard size allowed by the spec
    (256 segments or 4kB).

    Make nvme_setup_discard unconditionally allocate the maximum DSM buffer
    so the driver DMA maps a memory range that will always succeed.

    Link: https://bugzilla.kernel.org/show_bug.cgi?id=202665 many
    Signed-off-by: Eduard Hasenleithner <eduard@hasenleithner.at>
    [changelog, use existing define, kernel coding style]
    Signed-off-by: Keith Busch <kbusch@kernel.org>

Please let me know if any info is needed further.

Thanks,
Dakshaja

[-- Attachment #2: nvmf.conf --]
[-- Type: application/octet-stream, Size: 1026 bytes --]

{
  "hosts": [],
  "ports": [
    {
      "addr": {
        "adrfam": "ipv4",
        "traddr": "",
        "treq": "not specified",
        "trsvcid": "4420",
        "trtype": "rdma"
      },
      "portid": 2,
      "referrals": [],
      "subsystems": []
    },
    {
      "addr": {
        "adrfam": "ipv4",
        "traddr": "10.0.2.1",
        "treq": "not specified",
        "trsvcid": "4420",
        "trtype": "rdma"
      },
      "portid": 1,
      "referrals": [],
      "subsystems": [
        "nvme-ram"
      ]
    }
  ],
  "subsystems": [
    {
      "allowed_hosts": [],
      "attr": {
        "allow_any_host": "1",
        "serial": "1850eae32a76fb",
        "version": "1.3"
      },
      "namespaces": [
        {
          "device": {
            "nguid": "00000000-0000-0000-0000-000000000000",
            "path": "/dev/ram0",
            "uuid": "af09ebe1-db1a-4eb6-97ef-bef00f42d59c"
          },
          "enable": 1,
          "nsid": 1
        }
      ],
      "nqn": "nvme-ram"
    }
  ]
}

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
  2019-12-27 14:08 nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above Dakshaja Uppalapati
@ 2019-12-27 15:03 ` Eduard Hasenleithner
  2019-12-28  9:01   ` Sagi Grimberg
  0 siblings, 1 reply; 6+ messages in thread
From: Eduard Hasenleithner @ 2019-12-27 15:03 UTC (permalink / raw)
  To: Dakshaja Uppalapati, linux-nvme, kbusch
  Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan

On 27.12.19 15:08, Dakshaja Uppalapati wrote:
> I am trying to setup and run NVMF with tot linux kernel installed(5.5.0-rc3) on
> both target and host. Attached is the target configuration using ramdisk device.

> The following error is seen in dmesg:
> [  219.507989] blk_update_request: I/O error, dev nvme0c0n1, sector 0 op 0x3:(DISCARD) flags 0x4000800 phys_seg 1 prio class 0

> Please let me know if any info is needed further.

How big is the ramdisk?
Are there further error messages in the kernel log?
My current speculation is that it might fail in blk_cloned_rq_check_limits,
but then there should be an additional error message,
 e.g. "blk_cloned_rq_check_limits: over max size limit".

Eduard




_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
  2019-12-27 15:03 ` Eduard Hasenleithner
@ 2019-12-28  9:01   ` Sagi Grimberg
  2019-12-30 10:52     ` Dakshaja Uppalapati
  0 siblings, 1 reply; 6+ messages in thread
From: Sagi Grimberg @ 2019-12-28  9:01 UTC (permalink / raw)
  To: Eduard Hasenleithner, Dakshaja Uppalapati, linux-nvme, kbusch
  Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan


>> I am trying to setup and run NVMF with tot linux kernel installed(5.5.0-rc3) on
>> both target and host. Attached is the target configuration using ramdisk device.
> 
>> The following error is seen in dmesg:
>> [  219.507989] blk_update_request: I/O error, dev nvme0c0n1, sector 0 op 0x3:(DISCARD) flags 0x4000800 phys_seg 1 prio class 0
> 
>> Please let me know if any info is needed further.
> 
> How big is the ramdisk?
> Are there further error messages in the kernel log?
> My current speculation is that it might fail in blk_cloned_rq_check_limits,
> but then there should be an additional error message,
>   e.g. "blk_cloned_rq_check_limits: over max size limit".

Its probably the target not expecting that dsm ranges payload doesn't
match the command sgl desc...

Does this untested patch help?
--
iff --git a/drivers/nvme/target/io-cmd-bdev.c 
b/drivers/nvme/target/io-cmd-bdev.c
index b6fca0e421ef..945dffc82503 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -280,8 +280,11 @@ static void nvmet_bdev_execute_discard(struct 
nvmet_req *req)

  static void nvmet_bdev_execute_dsm(struct nvmet_req *req)
  {
-       if (!nvmet_check_data_len(req, nvmet_dsm_len(req)))
+       if (unlikely(nvmet_dsm_len(req) > req->transfer_len)) {
+               req->error_loc = offsetof(struct nvme_common_command, dptr);
+               nvmet_req_complete(req, NVME_SC_SGL_INVALID_DATA | 
NVME_SC_DNR);
                 return;
+       }
--

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* RE: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
  2019-12-28  9:01   ` Sagi Grimberg
@ 2019-12-30 10:52     ` Dakshaja Uppalapati
  2020-01-24  8:51       ` Dakshaja Uppalapati
  0 siblings, 1 reply; 6+ messages in thread
From: Dakshaja Uppalapati @ 2019-12-30 10:52 UTC (permalink / raw)
  To: Sagi Grimberg, Eduard Hasenleithner, linux-nvme, kbusch
  Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan



>> How big is the ramdisk?
It's 200MB and also tried with 1GB issue is seen.
>> Are there further error messages in the kernel log?
No.

> Its probably the target not expecting that dsm ranges payload doesn't match the command sgl desc...

> Does this untested patch help?
Yes, with the given patch error is not seen.

Thanks for the help.

Regards,
Dakshaja.

-----Original Message-----
From: Sagi Grimberg <sagi@grimberg.me> 
Sent: Saturday, December 28, 2019 2:31 PM
To: Eduard Hasenleithner <eduard@hasenleithner.at>; Dakshaja Uppalapati <dakshaja@chelsio.com>; linux-nvme@lists.infradead.org; kbusch@kernel.org
Cc: Potnuri Bharat Teja <bharat@chelsio.com>; Nirranjan Kirubaharan <nirranjan@chelsio.com>
Subject: Re: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.


>> I am trying to setup and run NVMF with tot linux kernel 
>> installed(5.5.0-rc3) on both target and host. Attached is the target configuration using ramdisk device.
> 
>> The following error is seen in dmesg:
>> [  219.507989] blk_update_request: I/O error, dev nvme0c0n1, sector 0 
>> op 0x3:(DISCARD) flags 0x4000800 phys_seg 1 prio class 0
> 
>> Please let me know if any info is needed further.
> 
> How big is the ramdisk?
> Are there further error messages in the kernel log?
> My current speculation is that it might fail in 
> blk_cloned_rq_check_limits, but then there should be an additional error message,
>   e.g. "blk_cloned_rq_check_limits: over max size limit".

Its probably the target not expecting that dsm ranges payload doesn't match the command sgl desc...

Does this untested patch help?
--
iff --git a/drivers/nvme/target/io-cmd-bdev.c
b/drivers/nvme/target/io-cmd-bdev.c
index b6fca0e421ef..945dffc82503 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -280,8 +280,11 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)

  static void nvmet_bdev_execute_dsm(struct nvmet_req *req)
  {
-       if (!nvmet_check_data_len(req, nvmet_dsm_len(req)))
+       if (unlikely(nvmet_dsm_len(req) > req->transfer_len)) {
+               req->error_loc = offsetof(struct nvme_common_command, dptr);
+               nvmet_req_complete(req, NVME_SC_SGL_INVALID_DATA |
NVME_SC_DNR);
                 return;
+       }
--
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* RE: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
  2019-12-30 10:52     ` Dakshaja Uppalapati
@ 2020-01-24  8:51       ` Dakshaja Uppalapati
  2020-01-24 17:40         ` Sagi Grimberg
  0 siblings, 1 reply; 6+ messages in thread
From: Dakshaja Uppalapati @ 2020-01-24  8:51 UTC (permalink / raw)
  To: Sagi Grimberg, Eduard Hasenleithner, linux-nvme, kbusch
  Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan

Hi Sagi,

Can you please push the patch.

Thanks & Regards,
Dakshaja
-----Original Message-----
From: Dakshaja Uppalapati 
Sent: Monday, December 30, 2019 4:23 PM
To: Sagi Grimberg <sagi@grimberg.me>; Eduard Hasenleithner <eduard@hasenleithner.at>; linux-nvme@lists.infradead.org; kbusch@kernel.org
Cc: Potnuri Bharat Teja <bharat@chelsio.com>; Nirranjan Kirubaharan <nirranjan@chelsio.com>
Subject: RE: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.



>> How big is the ramdisk?
It's 200MB and also tried with 1GB issue is seen.
>> Are there further error messages in the kernel log?
No.

> Its probably the target not expecting that dsm ranges payload doesn't match the command sgl desc...

> Does this untested patch help?
Yes, with the given patch error is not seen.

Thanks for the help.

Regards,
Dakshaja.

-----Original Message-----
From: Sagi Grimberg <sagi@grimberg.me> 
Sent: Saturday, December 28, 2019 2:31 PM
To: Eduard Hasenleithner <eduard@hasenleithner.at>; Dakshaja Uppalapati <dakshaja@chelsio.com>; linux-nvme@lists.infradead.org; kbusch@kernel.org
Cc: Potnuri Bharat Teja <bharat@chelsio.com>; Nirranjan Kirubaharan <nirranjan@chelsio.com>
Subject: Re: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.


>> I am trying to setup and run NVMF with tot linux kernel 
>> installed(5.5.0-rc3) on both target and host. Attached is the target configuration using ramdisk device.
> 
>> The following error is seen in dmesg:
>> [  219.507989] blk_update_request: I/O error, dev nvme0c0n1, sector 0 
>> op 0x3:(DISCARD) flags 0x4000800 phys_seg 1 prio class 0
> 
>> Please let me know if any info is needed further.
> 
> How big is the ramdisk?
> Are there further error messages in the kernel log?
> My current speculation is that it might fail in 
> blk_cloned_rq_check_limits, but then there should be an additional error message,
>   e.g. "blk_cloned_rq_check_limits: over max size limit".

Its probably the target not expecting that dsm ranges payload doesn't match the command sgl desc...

Does this untested patch help?
--
iff --git a/drivers/nvme/target/io-cmd-bdev.c
b/drivers/nvme/target/io-cmd-bdev.c
index b6fca0e421ef..945dffc82503 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -280,8 +280,11 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)

  static void nvmet_bdev_execute_dsm(struct nvmet_req *req)
  {
-       if (!nvmet_check_data_len(req, nvmet_dsm_len(req)))
+       if (unlikely(nvmet_dsm_len(req) > req->transfer_len)) {
+               req->error_loc = offsetof(struct nvme_common_command, dptr);
+               nvmet_req_complete(req, NVME_SC_SGL_INVALID_DATA |
NVME_SC_DNR);
                 return;
+       }
--
_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above.
  2020-01-24  8:51       ` Dakshaja Uppalapati
@ 2020-01-24 17:40         ` Sagi Grimberg
  0 siblings, 0 replies; 6+ messages in thread
From: Sagi Grimberg @ 2020-01-24 17:40 UTC (permalink / raw)
  To: Dakshaja Uppalapati, Eduard Hasenleithner, linux-nvme, kbusch
  Cc: Potnuri Bharat Teja, Nirranjan Kirubaharan


> Hi Sagi,
> 
> Can you please push the patch.

Yes, sent a patch, thanks for the reminer

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2020-01-24 17:40 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-27 14:08 nvme blk_update_request IO error observed on formatting device with kernels 5.5-rc1 and above Dakshaja Uppalapati
2019-12-27 15:03 ` Eduard Hasenleithner
2019-12-28  9:01   ` Sagi Grimberg
2019-12-30 10:52     ` Dakshaja Uppalapati
2020-01-24  8:51       ` Dakshaja Uppalapati
2020-01-24 17:40         ` Sagi Grimberg

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).