[SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

[SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

[lullajd]

[-- Attachment #1: Type: text/plain, Size: 2871 bytes --]

Hi Shuhei, 
Thankyou for responding.

What NVMe-oF target are you using? SPDK NVMe-oF tatrget?
JL===> Its a private target implementation which has been tested thoroughly with linux kernel based initiator. The failover (when one controller becomes unavailable) also on this private target implementation works smoothly with linux kernel based initiator. 

The bdev conf you provided looks good.  
We need multiple namespaces to have the same NSID, EUI64, NGUID, and UUID to configure shared namespace.  
JL=====> Yes, is there anything I need to put in the bdev conf file to let the check in bdev_nvme_compare_ns()  pass ?
There seems to some issue with bdev_nvme_compare_ns().
I added some prints to check why this check is returning 0 to its callers
nvme_bdev_add_ns
nvme_ctrlr_populate_namespace
nvme_ctrlr_populate_namespaces
nvme_ctrlr_create_done
..
..



static bool
bdev_nvme_compare_ns(struct spdk_nvme_ns *ns1, struct spdk_nvme_ns *ns2)
{
        const struct spdk_nvme_ns_data *nsdata1, *nsdata2;
        const struct spdk_uuid *uuid1, *uuid2;

        nsdata1 = spdk_nvme_ns_get_data(ns1);
        nsdata2 = spdk_nvme_ns_get_data(ns2);
        uuid1 = spdk_nvme_ns_get_uuid(ns1);
        uuid2 = spdk_nvme_ns_get_uuid(ns2);
        if (uuid1 == NULL)
                printf("%s uuid1 is NULL\n", __func__);
        if (uuid2 == NULL)
                printf("%s uuid2 is NULL\n", __func__);
        printf("%s (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):%d\n",
                        __func__,
              (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0));
        printf("%s (nsdata1->eui64 == nsdata2->eui64):%d\n", __func__,
                                (nsdata1->eui64 == nsdata2->eui64));
        unsigned char myexp;
        myexp = memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
                nsdata1->eui64 == nsdata2->eui64 &&
                uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
        printf("%s myexp:%d\n", __func__, myexp);

        return memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
               nsdata1->eui64 == nsdata2->eui64 &&
               uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
}




its printing the following:
bdev_nvme_compare_ns uuid1 is NULL
bdev_nvme_compare_ns uuid2 is NULL
bdev_nvme_compare_ns (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):1
bdev_nvme_compare_ns (nsdata1->eui64 == nsdata2->eui64):1
bdev_nvme_compare_ns myexp:0
[2021-11-22 07:03:58.083035] bdev_nvme.c:2569:nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.



If you are using SPDK NVMe-oF target, https://review.spdk.io/gerrit/c/spdk/spdk/+/10197/13 may give you a hint.  
Thanks, Shuhei

[SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

[@ 2021-11-22 12:06 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 5655 bytes --]

Hi,

I get the value of your feedback now.
We are working on multipath toward SPDK22.01 now. I want to include your feedback and request. But there are still many patches under review now. So it may need a month or so to be ready for your testing.

Please wait shortly.

Thanks for your patience,
Shuhei
________________________________
From: 松本周平 / MATSUMOTO，SHUUHEI <shuhei.matsumoto.xt(a)hitachi.com>
Sent: Monday, November 22, 2021 5:55:46 PM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: Re: [!][SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

Hi,

Thanks for your further investigation.

You found a new bug in the current SPDK NVMe-oF initiator (NVMe bdev module).

We should not try configuring multipath if  the bit 0 of the NMIC (Namespace Multipath I/O and Namespace Sharing Capabilities) is zero.

Linux NVMe-oF initiator creates failover configuration automatically if the bit 0 of the NMIC is zero.

Linux NVMe-oF initiator

  *   for allnamespaces if the bit 0 of the NMIC is 1 and namespaces are identical, create multipath configuration, or create failover configuration otherwise

SPDK NVMe-oF initiator

  *   if the multipath parameter is "failover" or the multipath parameter is omitted, create failover configuration automatically
  *   if the multipath parameter is "multipath", create multipath configuration automatically. However it requires all namespaces have the bit 0 of the NMIC to be 1 and all namespaces are identical. Otherwise, it fails.

I guess your storage target sets the bit 0 of the NMIC to zero.

I think you posted similar questions on Slack.

So The issue we should fix is to succeed failover between SPDK NVMe-oF initiator and your proprietary NVMe-oF target when the multipath parameter is omitted.

You posted a nice log on slack. It looks that SPDK failed to failover due to conflicts.
I will read more and reply on Slack.

Please give me a time to do.

Thanks,
Shuhei

________________________________
差出人: lullajd(a)yahoo.com <lullajd(a)yahoo.com>
送信日時: 2021年11月22日 16:24
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: [!][SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

Hi Shuhei,
Thankyou for responding.

What NVMe-oF target are you using? SPDK NVMe-oF tatrget?
JL===> Its a private target implementation which has been tested thoroughly with linux kernel based initiator. The failover (when one controller becomes unavailable) also on this private target implementation works smoothly with linux kernel based initiator.

The bdev conf you provided looks good.
We need multiple namespaces to have the same NSID, EUI64, NGUID, and UUID to configure shared namespace.
JL=====> Yes, is there anything I need to put in the bdev conf file to let the check in bdev_nvme_compare_ns()  pass ?
There seems to some issue with bdev_nvme_compare_ns().
I added some prints to check why this check is returning 0 to its callers
nvme_bdev_add_ns
nvme_ctrlr_populate_namespace
nvme_ctrlr_populate_namespaces
nvme_ctrlr_create_done
..
..



static bool
bdev_nvme_compare_ns(struct spdk_nvme_ns *ns1, struct spdk_nvme_ns *ns2)
{
        const struct spdk_nvme_ns_data *nsdata1, *nsdata2;
        const struct spdk_uuid *uuid1, *uuid2;

        nsdata1 = spdk_nvme_ns_get_data(ns1);
        nsdata2 = spdk_nvme_ns_get_data(ns2);
        uuid1 = spdk_nvme_ns_get_uuid(ns1);
        uuid2 = spdk_nvme_ns_get_uuid(ns2);
        if (uuid1 == NULL)
                printf("%s uuid1 is NULL\n", __func__);
        if (uuid2 == NULL)
                printf("%s uuid2 is NULL\n", __func__);
        printf("%s (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):%d\n",
                        __func__,
              (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0));
        printf("%s (nsdata1->eui64 == nsdata2->eui64):%d\n", __func__,
                                (nsdata1->eui64 == nsdata2->eui64));
        unsigned char myexp;
        myexp = memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
                nsdata1->eui64 == nsdata2->eui64 &&
                uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
        printf("%s myexp:%d\n", __func__, myexp);

        return memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
               nsdata1->eui64 == nsdata2->eui64 &&
               uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
}




its printing the following:
bdev_nvme_compare_ns uuid1 is NULL
bdev_nvme_compare_ns uuid2 is NULL
bdev_nvme_compare_ns (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):1
bdev_nvme_compare_ns (nsdata1->eui64 == nsdata2->eui64):1
bdev_nvme_compare_ns myexp:0
[2021-11-22 07:03:58.083035] bdev_nvme.c:2569:nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.



If you are using SPDK NVMe-oF target, https://secure-web.cisco.com/15dcv2VdFmba5julQd4z64OXO9n9GEVXszHB6zYz2w80-Hv2PZuanNpbbxiXfFhUYxgAZJGHHiV9M7oW9rh1JiqTD8FevRb4pM1oZZc_qtM4wDrUj1D0hjoryC4zVenI1_oUtGHhSg6b7KhCSAmyebOKqZaw-bkeXJnpAs09AL1VRakD-TVa0SsxvcEXpAPSVSsIOJyyQRe-F66I4qKEeAbU6mlyGGmLim7a5KQyblSC_xBOEuCgZzS5ZGEG9OwAxE2NEHEQiD11LC9J4bj4BCi7MR-Yxe3qMCpvYBB0Zi_XPotygnNh6u1OHq5NSDVRL/https%3A%2F%2Freview.spdk.io%2Fgerrit%2Fc%2Fspdk%2Fspdk%2F%2B%2F10197%2F13 may give you a hint.
Thanks, Shuhei
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

[@ 2021-11-22 8:55 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 5052 bytes --]

Hi,

Thanks for your further investigation.

You found a new bug in the current SPDK NVMe-oF initiator (NVMe bdev module).

We should not try configuring multipath if  the bit 0 of the NMIC (Namespace Multipath I/O and Namespace Sharing Capabilities) is zero.

Linux NVMe-oF initiator creates failover configuration automatically if the bit 0 of the NMIC is zero.

Linux NVMe-oF initiator

  *   for allnamespaces if the bit 0 of the NMIC is 1 and namespaces are identical, create multipath configuration, or create failover configuration otherwise

SPDK NVMe-oF initiator

  *   if the multipath parameter is "failover" or the multipath parameter is omitted, create failover configuration automatically
  *   if the multipath parameter is "multipath", create multipath configuration automatically. However it requires all namespaces have the bit 0 of the NMIC to be 1 and all namespaces are identical. Otherwise, it fails.

I guess your storage target sets the bit 0 of the NMIC to zero.

I think you posted similar questions on Slack.

So The issue we should fix is to succeed failover between SPDK NVMe-oF initiator and your proprietary NVMe-oF target when the multipath parameter is omitted.

You posted a nice log on slack. It looks that SPDK failed to failover due to conflicts.
I will read more and reply on Slack.

Please give me a time to do.

Thanks,
Shuhei

________________________________
差出人: lullajd(a)yahoo.com <lullajd(a)yahoo.com>
送信日時: 2021年11月22日 16:24
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: [!][SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

Hi Shuhei,
Thankyou for responding.

What NVMe-oF target are you using? SPDK NVMe-oF tatrget?
JL===> Its a private target implementation which has been tested thoroughly with linux kernel based initiator. The failover (when one controller becomes unavailable) also on this private target implementation works smoothly with linux kernel based initiator.

The bdev conf you provided looks good.
We need multiple namespaces to have the same NSID, EUI64, NGUID, and UUID to configure shared namespace.
JL=====> Yes, is there anything I need to put in the bdev conf file to let the check in bdev_nvme_compare_ns()  pass ?
There seems to some issue with bdev_nvme_compare_ns().
I added some prints to check why this check is returning 0 to its callers
nvme_bdev_add_ns
nvme_ctrlr_populate_namespace
nvme_ctrlr_populate_namespaces
nvme_ctrlr_create_done
..
..



static bool
bdev_nvme_compare_ns(struct spdk_nvme_ns *ns1, struct spdk_nvme_ns *ns2)
{
        const struct spdk_nvme_ns_data *nsdata1, *nsdata2;
        const struct spdk_uuid *uuid1, *uuid2;

        nsdata1 = spdk_nvme_ns_get_data(ns1);
        nsdata2 = spdk_nvme_ns_get_data(ns2);
        uuid1 = spdk_nvme_ns_get_uuid(ns1);
        uuid2 = spdk_nvme_ns_get_uuid(ns2);
        if (uuid1 == NULL)
                printf("%s uuid1 is NULL\n", __func__);
        if (uuid2 == NULL)
                printf("%s uuid2 is NULL\n", __func__);
        printf("%s (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):%d\n",
                        __func__,
              (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0));
        printf("%s (nsdata1->eui64 == nsdata2->eui64):%d\n", __func__,
                                (nsdata1->eui64 == nsdata2->eui64));
        unsigned char myexp;
        myexp = memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
                nsdata1->eui64 == nsdata2->eui64 &&
                uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
        printf("%s myexp:%d\n", __func__, myexp);

        return memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0 &&
               nsdata1->eui64 == nsdata2->eui64 &&
               uuid1 != NULL && uuid2 != NULL && spdk_uuid_compare(uuid1, uuid2) == 0;
}




its printing the following:
bdev_nvme_compare_ns uuid1 is NULL
bdev_nvme_compare_ns uuid2 is NULL
bdev_nvme_compare_ns (memcmp(nsdata1->nguid, nsdata2->nguid, sizeof(nsdata1->nguid)) == 0):1
bdev_nvme_compare_ns (nsdata1->eui64 == nsdata2->eui64):1
bdev_nvme_compare_ns myexp:0
[2021-11-22 07:03:58.083035] bdev_nvme.c:2569:nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.



If you are using SPDK NVMe-oF target, https://secure-web.cisco.com/15dcv2VdFmba5julQd4z64OXO9n9GEVXszHB6zYz2w80-Hv2PZuanNpbbxiXfFhUYxgAZJGHHiV9M7oW9rh1JiqTD8FevRb4pM1oZZc_qtM4wDrUj1D0hjoryC4zVenI1_oUtGHhSg6b7KhCSAmyebOKqZaw-bkeXJnpAs09AL1VRakD-TVa0SsxvcEXpAPSVSsIOJyyQRe-F66I4qKEeAbU6mlyGGmLim7a5KQyblSC_xBOEuCgZzS5ZGEG9OwAxE2NEHEQiD11LC9J4bj4BCi7MR-Yxe3qMCpvYBB0Zi_XPotygnNh6u1OHq5NSDVRL/https%3A%2F%2Freview.spdk.io%2Fgerrit%2Fc%2Fspdk%2Fspdk%2F%2B%2F10197%2F13 may give you a hint.
Thanks, Shuhei
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

[@ 2021-11-22 0:59 UTC (permalink / raw)]

[-- Attachment #1: Type: text/plain, Size: 4580 bytes --]

Hi,

What NVMe-oF target are you using? SPDK NVMe-oF tatrget?
The bdev conf you provided looks good.

We need multiple namespaces to have the same NSID, EUI64, NGUID, and UUID to configure shared namespace.

If you are using SPDK NVMe-oF target, https://review.spdk.io/gerrit/c/spdk/spdk/+/10197/13
may give you a hint.

Thanks,
Shuhei


________________________________
差出人: lullajd(a)yahoo.com <lullajd(a)yahoo.com>
送信日時: 2021年11月22日 8:06
宛先: spdk(a)lists.01.org <spdk(a)lists.01.org>
件名: [SPDK] nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.

Hi,

This is with v21.10-rc1-122-g64fa301f6

When I try to configure multipath and run bdevperf, I see that nvme_bdev_add_ns() is complaining that "Namespaces are not identical". Could anybody please suggest what is wrong in the conf file ?

Here is my nvme discover o/p at initiator:
root(a)myVM:~/user/spdk# nvme discover -t tcp -a 10.1.109.152 -q nqn.2015-09.com.Xyz:15.4.7.3

Discovery Log Number of Records 2, Generation counter 1
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2015-09.com.xyz:93111e6e-663a-4ad0-9028-a3d75d90e5d3
traddr:  15.161.1.2
sectype: none
=====Discovery Log Entry 1======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2015-09.com.xyz:93111e6e-663a-4ad0-9028-a3d75d90e5d3
traddr:  15.162.2.2
sectype: none

here is my dev_new.conf
root(a)myVM:~/user/spdk# cat ../bdev_new.conf
{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
        "method": "bdev_nvme_set_options", "params": {
            "timeout_us": 255000000,
            "bdev_retry_count": -1,
            "action_on_timeout": "none"
        }
        },
        {
        "method": "bdev_nvme_attach_controller", "params": {
            "name": "Nvme0",
            "trtype": "tcp",
            "traddr": "15.161.1.2",
            "trsvcid": "4420",
            "subnqn": "nqn.2015-09.com.xyz:93111e6e-663a-4ad0-9028-a3d75d90e5d3",
            "adrfam": "IPv4",
            "hostnqn": "nqn.2015-09.com.Xyz:15.4.7.3",
            "multipath": "multipath"
        }
        },
        {
        "method": "bdev_nvme_attach_controller", "params": {
            "name": "Nvme0",
            "trtype": "tcp",
            "traddr": "15.162.2.2",
            "trsvcid": "4420",
            "subnqn": "nqn.2015-09.com.xyz:93111e6e-663a-4ad0-9028-a3d75d90e5d3",
            "adrfam": "IPv4",
            "multipath": "multipath",
            "hostnqn": "nqn.2015-09.com.Xyz:15.4.7.3"
        }
        }
      ]
    }
  ]
}


here is the bdevperf run showing "Namespaces are not identical".
root(a)myVM:~/user/spdk# ./test/bdev/bdevperf/bdevperf -q 1 -o 4096 -w randwrite -c /home/localadmin/user/bdev_new.conf -t 60 -k 9999999
[2021-11-21 22:56:23.465497] Starting SPDK v22.01-pre git sha1 64fa301f6 / DPDK 21.08.0 initialization...
[2021-11-21 22:56:23.465680] [ DPDK EAL parameters: [2021-11-21 22:56:23.466282] bdevperf [2021-11-21 22:56:23.466548] --no-shconf [2021-11-21 22:56:23.466602] -c 0x1 [2021-11-21 22:56:23.467107] --log-level=lib.eal:6 [2021-11-21 22:56:23.467165] --log-level=lib.cryptodev:5 [2021-11-21 22:56:23.467690] --log-level=user1:6 [2021-11-21 22:56:23.467745] --iova-mode=pa [2021-11-21 22:56:23.467994] --base-virtaddr=0x200000000000 [2021-11-21 22:56:23.468136] --match-allocations [2021-11-21 22:56:23.468172] --file-prefix=spdk_pid12913 [2021-11-21 22:56:23.468262] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2021-11-21 22:56:23.600199] app.c: 543:spdk_app_start: *NOTICE*: Total cores available: 1
[2021-11-21 22:56:23.877499] reactor.c: 943:reactor_run: *NOTICE*: Reactor started on core 0
[2021-11-21 22:56:23.886093] accel_engine.c:1012:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.
[2021-11-21 22:56:24.102809] bdev_nvme.c:2555:nvme_bdev_add_ns: *ERROR*: Namespaces are not identical.
Running I/O for 60 seconds...
 Job: Nvme0n1 (Core Mask 0x1)
         Nvme0n1             :   14367.88 IOPS      56.12 MiB/s
 =============================================================
 Total                       :   14367.88 IOPS      56.12 MiB/s
root(a)myVM:~/user/spdk#
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org