Linux-NVME Archive on lore.kernel.org
 help / color / Atom feed
From: Mark Ruijter <MRuijter@onestopsystems.com>
To: Sagi Grimberg <sagi@grimberg.me>,
	James Smart <james.smart@broadcom.com>,
	 Max Gurtovoy <maxg@mellanox.com>,
	Chaitanya Kulkarni <chaitanya.kulkarni@wdc.com>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Subject: Re: [PATCH] nvmet: make ctrl-id configurable
Date: Fri, 8 Nov 2019 16:15:34 +0000
Message-ID: <3D590371-4C4F-4355-B36E-7795FB218FC6@onestopsystems.com> (raw)
In-Reply-To: <802cb171-34de-65cc-75c5-83ef7c200dc4@grimberg.me>

[-- Attachment #1: Type: text/plain, Size: 2591 bytes --]


I have setup a cluster using two nodes with a dual ported nvme drive and tested exporting the drive from both nodes.

The configuration on both nodes, with the exception of the ip address, looks like this:
    {
      "addr": {
        "adrfam": "ipv4", 
        "traddr": "192.168.1.11", 
        "treq": "not specified", 
        "trsvcid": "4420", 
        "trtype": "tcp"
      }, 
      "portid": 322, 
      "referrals": [], 
      "subsystems": [
        "clvol"
      ]
    },

  "subsystems": [
    {
      "allowed_hosts": [], 
      "attr": {
        "allow_any_host": "1", 
        "model": "clpool/clvol", 
        "serial": "e94430920f6103af", 
        "version": "1.3"
      }, 
      "namespaces": [
        {
          "device": {
            "nguid": "00000000-0000-0000-0000-000000000000", 
            "path": "/dev/clpool/clvol", 
            "uuid": "99451596-9675-4382-bff8-b78ee34de567"
          }, 
          "enable": 1, 
          "nsid": 1
        }
      ], 
      "nqn": "clvol"
    },

When I now connect the initiator to the two cluster nodes without the patch the result will be a controller id collision.
And the initiator refuses to connect to the second node that you try to connect.
--
[ 7895.052302] nvme nvme0: new ctrl: NQN "clvol", addr 192.168.8.10:4420
[ 7895.053297] nvme0n1: detected capacity change from 0 to 107374182400
[ 7898.188321] nvme nvme1: Duplicate cntlid 1 with nvme0, rejecting
--

With the attached patch I am able to force the second node to use a higher offset when enumerating controller ids:
nodea:
/sys/kernel/config/nvmet/subsystems/clvol # cat cntlid_min 
1
nodeb:/sys/kernel/config/nvmet/subsystems/clvol # cat cntlid_min 
32

In real life the number used for the offsets could be derived from something like the Pacemaker node number.
And the Pacemaker + DLM could be used for nvme persistent reservations.

The controller id collision now no longer occurs.
And the initiator can connect, and two paths will be shown:

root@r11i1:~# nvme list-subsys
nvme-subsys0 - NQN=clvol
\
 +- nvme0 tcp traddr=192.168.1.10 trsvcid=4420
 +- nvme1 tcp traddr=192.168.1.11 trsvcid=4420

Does this all make sense?

Thanks,

Mark



Op 06-11-19 17:51 heeft Sagi Grimberg <sagi@grimberg.me> geschreven:

    
    >> Hi Max & Chaitanya,
    >>
    >> Does the NVME specification even allow duplicate controller id's?
    > Not within a single subsystem.
    > But across multiple subsystems there can be.
    
    Yes, this patch is broken..
    


[-- Attachment #2: cntlid_min.patch --]
[-- Type: application/octet-stream, Size: 2611 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

  reply index

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-03 18:13 Chaitanya Kulkarni
2019-11-03 18:15 ` Chaitanya Kulkarni
2019-11-04  9:36 ` Max Gurtovoy
2019-11-04 10:45   ` Mark Ruijter
2019-11-04 14:04     ` Max Gurtovoy
2019-11-04 16:36       ` Mark Ruijter
2019-11-05 22:38         ` James Smart
2019-11-06 16:51           ` Sagi Grimberg
2019-11-08 16:15             ` Mark Ruijter [this message]
2019-11-12 19:01               ` Sagi Grimberg
2019-11-12 21:19                 ` Mark Ruijter
2019-11-13  0:44                   ` James Smart
2019-11-13 23:40                 ` Chaitanya Kulkarni
2019-11-14 15:55                   ` Mark Ruijter
2019-11-05 21:15     ` Chaitanya Kulkarni

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3D590371-4C4F-4355-B36E-7795FB218FC6@onestopsystems.com \
    --to=mruijter@onestopsystems.com \
    --cc=chaitanya.kulkarni@wdc.com \
    --cc=james.smart@broadcom.com \
    --cc=linux-nvme@lists.infradead.org \
    --cc=maxg@mellanox.com \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NVME Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nvme/0 linux-nvme/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvme linux-nvme/ https://lore.kernel.org/linux-nvme \
		linux-nvme@lists.infradead.org
	public-inbox-index linux-nvme

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.infradead.lists.linux-nvme


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git