I have setup a cluster using two nodes with a dual ported nvme drive and tested exporting the drive from both nodes.

The configuration on both nodes, with the exception of the ip address, looks like this:
    {
      "addr": {
        "adrfam": "ipv4", 
        "traddr": "192.168.1.11", 
        "treq": "not specified", 
        "trsvcid": "4420", 
        "trtype": "tcp"
      }, 
      "portid": 322, 
      "referrals": [], 
      "subsystems": [
        "clvol"
      ]
    },

  "subsystems": [
    {
      "allowed_hosts": [], 
      "attr": {
        "allow_any_host": "1", 
        "model": "clpool/clvol", 
        "serial": "e94430920f6103af", 
        "version": "1.3"
      }, 
      "namespaces": [
        {
          "device": {
            "nguid": "00000000-0000-0000-0000-000000000000", 
            "path": "/dev/clpool/clvol", 
            "uuid": "99451596-9675-4382-bff8-b78ee34de567"
          }, 
          "enable": 1, 
          "nsid": 1
        }
      ], 
      "nqn": "clvol"
    },

When I now connect the initiator to the two cluster nodes without the patch the result will be a controller id collision.
And the initiator refuses to connect to the second node that you try to connect.
--
[ 7895.052302] nvme nvme0: new ctrl: NQN "clvol", addr 192.168.8.10:4420
[ 7895.053297] nvme0n1: detected capacity change from 0 to 107374182400
[ 7898.188321] nvme nvme1: Duplicate cntlid 1 with nvme0, rejecting
--

With the attached patch I am able to force the second node to use a higher offset when enumerating controller ids:
nodea:
/sys/kernel/config/nvmet/subsystems/clvol # cat cntlid_min 
1
nodeb:/sys/kernel/config/nvmet/subsystems/clvol # cat cntlid_min 
32

In real life the number used for the offsets could be derived from something like the Pacemaker node number.
And the Pacemaker + DLM could be used for nvme persistent reservations.

The controller id collision now no longer occurs.
And the initiator can connect, and two paths will be shown:

root@r11i1:~# nvme list-subsys
nvme-subsys0 - NQN=clvol
\
 +- nvme0 tcp traddr=192.168.1.10 trsvcid=4420
 +- nvme1 tcp traddr=192.168.1.11 trsvcid=4420

Does this all make sense?

Thanks,

Mark



Op 06-11-19 17:51 heeft Sagi Grimberg <sagi@grimberg.me> geschreven:

    
    >> Hi Max & Chaitanya,
    >>
    >> Does the NVME specification even allow duplicate controller id's?
    > Not within a single subsystem.
    > But across multiple subsystems there can be.
    
    Yes, this patch is broken..