NVMf initiator persistent across boots

NVMf initiator persistent across boots

[Narasimhan V]

Hi All,

Could we have NVMf initiator setup persistent across boots ?

Is this a possibility ? If yes, I would like suggestions on how this 
could be implemented.

-- 
Regards
Narasimhan V

NVMf initiator persistent across boots

[Sagi Grimberg]

Hi,

> Could we have NVMf initiator setup persistent across boots ?
> 
> Is this a possibility ? If yes, I would like suggestions on how this 
> could be implemented.

We could add a systemctl one-shot add-on that would
issue nvme connect-all for each line available in
the discovery.conf file.

Something like:
--
[Unit]
Description=NVMf auto discovery service
After=systemd-modules-load.service network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nvme connect-all
StandardOutput=journal

[Install]
WantedBy=multi-user.target
--

Where discovery.conf would have the relevant parameters
like:
--
--traddr=<traddr1> --trsvcid=<trsvcid1>
--traddr=<traddr2> --trsvcid=<trsvcid2>
--

One thing we may want to have is to make sure
that nvme relevant modules are loaded at boot time
so we may want nvme-cli to either manually load them
or the installation would add them to systemd-modules-load.service
(i.e. add relevant files to /etc/modules-load.d/)

Thoughts?

NVMf initiator persistent across boots

[Johannes Thumshirn]

[+Cc James]

On Wed, Aug 30, 2017@12:37:47PM +0300, Sagi Grimberg wrote:
[...]
> Something like:
> --
> [Unit]
> Description=NVMf auto discovery service
> After=systemd-modules-load.service network-online.target
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/nvme connect-all
> StandardOutput=journal
> 
> [Install]
> WantedBy=multi-user.target
> --
> 
> Where discovery.conf would have the relevant parameters
> like:
> --
> --traddr=<traddr1> --trsvcid=<trsvcid1>
> --traddr=<traddr2> --trsvcid=<trsvcid2>
> --
> 
> One thing we may want to have is to make sure
> that nvme relevant modules are loaded at boot time
> so we may want nvme-cli to either manually load them
> or the installation would add them to systemd-modules-load.service
> (i.e. add relevant files to /etc/modules-load.d/)
> 
> Thoughts?

That's exactly what I wanted to do for SUSE distros as well, so I'm fine with
it.

James Smart has a bunch of scripts and udev rules for FC-NVMe autoconnecting
and I asked him earlier today to sumbit them here as well. 

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

NVMf initiator persistent across boots

[Max Gurtovoy]

Hi all,

Sorry for the delay, but I wanted to know if we had a progress for this 
topic. I want to add functionality that gives persistency for fabric 
devices not only across boots but also across adapter/device removals.
My idea is to add a flag --persist to "nvme connect" command. This 
command will add the created ctrls to a file 
(/etc/nvme-fabrics/persistent_ctrls for example).
And I want to add a udev rule that in case of ctrl removal, we'll run a 
script that try to periodicly connect to the target.

thoughts ?

On 8/30/2017 12:45 PM, Johannes Thumshirn wrote:
> [+Cc James]
> 
> On Wed, Aug 30, 2017@12:37:47PM +0300, Sagi Grimberg wrote:
> [...]
>> Something like:
>> --
>> [Unit]
>> Description=NVMf auto discovery service
>> After=systemd-modules-load.service network-online.target
>>
>> [Service]
>> Type=oneshot
>> ExecStart=/usr/bin/nvme connect-all
>> StandardOutput=journal
>>
>> [Install]
>> WantedBy=multi-user.target
>> --
>>
>> Where discovery.conf would have the relevant parameters
>> like:
>> --
>> --traddr=<traddr1> --trsvcid=<trsvcid1>
>> --traddr=<traddr2> --trsvcid=<trsvcid2>
>> --
>>
>> One thing we may want to have is to make sure
>> that nvme relevant modules are loaded at boot time
>> so we may want nvme-cli to either manually load them
>> or the installation would add them to systemd-modules-load.service
>> (i.e. add relevant files to /etc/modules-load.d/)
>>
>> Thoughts?
> 
> That's exactly what I wanted to do for SUSE distros as well, so I'm fine with
> it.
> 
> James Smart has a bunch of scripts and udev rules for FC-NVMe autoconnecting
> and I asked him earlier today to sumbit them here as well.

Can you share these rules so we can create something generic ?

> 
> Byte,
> 	Johannes
> 

Cheers,
Max.

NVMf initiator persistent across boots

[Sagi Grimberg]

> Hi all,
> 
> Sorry for the delay, but I wanted to know if we had a progress for this 
> topic. I want to add functionality that gives persistency for fabric 
> devices not only across boots but also across adapter/device removals.
> My idea is to add a flag --persist to "nvme connect" command. This 
> command will add the created ctrls to a file 
> (/etc/nvme-fabrics/persistent_ctrls for example).
> And I want to add a udev rule that in case of ctrl removal, we'll run a 
> script that try to periodicly connect to the target.
Why not just add a periodic task to run nvme connect-all?
--
[Unit]
Description=NVMf auto discovery service
After=systemd-modules-load.service network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nvme connect-all
StandardOutput=journal

[Timer]
OnUnitActiveSec=1min

[Install]
WantedBy=multi-user.target timers.target
--

That would simply run nvme connect-all once every say minute.
The only problem is that it relies on the kernel to fail
duplicate subsystems. We could enforce that in nvme-cli for that
matter though (we can compare against sysfs address and subsysnqn).

Johannes is probably one to know better than me if this is the
correct way to go...

NVMf initiator persistent across boots

[Max Gurtovoy]

On 3/5/2018 9:45 PM, Sagi Grimberg wrote:
> 
>> Hi all,
>>
>> Sorry for the delay, but I wanted to know if we had a progress for 
>> this topic. I want to add functionality that gives persistency for 
>> fabric devices not only across boots but also across adapter/device 
>> removals.
>> My idea is to add a flag --persist to "nvme connect" command. This 
>> command will add the created ctrls to a file 
>> (/etc/nvme-fabrics/persistent_ctrls for example).
>> And I want to add a udev rule that in case of ctrl removal, we'll run 
>> a script that try to periodicly connect to the target.
> Why not just add a periodic task to run nvme connect-all?
> -- 
> [Unit]
> Description=NVMf auto discovery service
> After=systemd-modules-load.service network-online.target
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/nvme connect-all
> StandardOutput=journal
> 
> [Timer]
> OnUnitActiveSec=1min
> 
> [Install]
> WantedBy=multi-user.target timers.target
> -- 
> 
> That would simply run nvme connect-all once every say minute.
> The only problem is that it relies on the kernel to fail
> duplicate subsystems. We could enforce that in nvme-cli for that
> matter though (we can compare against sysfs address and subsysnqn).

Yes, we can also add flag to nvme discover command to add the parameters 
to the discovery file in case they are not exist.


> 
> Johannes is probably one to know better than me if this is the
> correct way to go...

Johannes, any comment ?
 From what I tried, we need to create a .timer and .service files for 
systemd...

NVMf initiator persistent across boots

[Johannes Thumshirn]

On Wed, Mar 07, 2018@02:47:52PM +0200, Max Gurtovoy wrote:
> > [Unit]
> > Description=NVMf auto discovery service
> > After=systemd-modules-load.service network-online.target
> > 
> > [Service]
> > Type=oneshot
> > ExecStart=/usr/bin/nvme connect-all
> > StandardOutput=journal
> > 
> > [Timer]
> > OnUnitActiveSec=1min
> > 
> > [Install]
> > WantedBy=multi-user.target timers.target
> > -- 
> > 
> > That would simply run nvme connect-all once every say minute.
> > The only problem is that it relies on the kernel to fail
> > duplicate subsystems. We could enforce that in nvme-cli for that
> > matter though (we can compare against sysfs address and subsysnqn).
> 
> Yes, we can also add flag to nvme discover command to add the parameters to
> the discovery file in case they are not exist.
> 
> 
> > 
> > Johannes is probably one to know better than me if this is the
> > correct way to go...
> 
> Johannes, any comment ?
> From what I tried, we need to create a .timer and .service files for
> systemd...

Sorry I was on FTO for some days.

One thing that I feel is missing in this whole "let's just call nvme
connect-all and we're done" discussion is, we currently can't really
specifiy how many connections to the target we want to initiate.

I usually run several nvme connect calls with different --host-traddr
arguments to connect from multiple HCAs to the target.

I tried to hack the ability to specify a list of host traddrs for nvme
connect and connect-all and then just loop the connect cann in
nvme-cli but this somehow feels wrong.

For the systemd service + timer units above, they look good on the
first sight, but I'm not sure I like the unconditional "polling" of
the connect-all call.

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

NVMf initiator persistent across boots

[Max Gurtovoy]

On 3/8/2018 10:39 AM, Johannes Thumshirn wrote:
> On Wed, Mar 07, 2018@02:47:52PM +0200, Max Gurtovoy wrote:
>>> [Unit]
>>> Description=NVMf auto discovery service
>>> After=systemd-modules-load.service network-online.target
>>>
>>> [Service]
>>> Type=oneshot
>>> ExecStart=/usr/bin/nvme connect-all
>>> StandardOutput=journal
>>>
>>> [Timer]
>>> OnUnitActiveSec=1min
>>>
>>> [Install]
>>> WantedBy=multi-user.target timers.target
>>> -- 
>>>
>>> That would simply run nvme connect-all once every say minute.
>>> The only problem is that it relies on the kernel to fail
>>> duplicate subsystems. We could enforce that in nvme-cli for that
>>> matter though (we can compare against sysfs address and subsysnqn).
>>
>> Yes, we can also add flag to nvme discover command to add the parameters to
>> the discovery file in case they are not exist.
>>
>>
>>>
>>> Johannes is probably one to know better than me if this is the
>>> correct way to go...
>>
>> Johannes, any comment ?
>>  From what I tried, we need to create a .timer and .service files for
>> systemd...
> 
> Sorry I was on FTO for some days.
> 
> One thing that I feel is missing in this whole "let's just call nvme
> connect-all and we're done" discussion is, we currently can't really
> specifiy how many connections to the target we want to initiate.
> 
> I usually run several nvme connect calls with different --host-traddr
> arguments to connect from multiple HCAs to the target.
> 
> I tried to hack the ability to specify a list of host traddrs for nvme
> connect and connect-all and then just loop the connect cann in
> nvme-cli but this somehow feels wrong.

Why not adding -w <host-traddr> to the /etc/nvme/discovery.conf file lines ?
I guess connect-all will use it too, right ? (need to check it).

> 
> For the systemd service + timer units above, they look good on the
> first sight, but I'm not sure I like the unconditional "polling" of
> the connect-all call.

Yes this is no the perfect solution but I guess future discovery_manager 
should solve it, but we need to find solution till then.

I tought about adding new commands/flags to add/remove parameters to 
/etc/nvme/discovery.conf.
how about adding:
nvme persist-add -t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12 (will add 
to discovery file "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12")
nvme persist-remove -t rdma -a 11.11.11.11 -s 4420 -w 11.11.12.12 (will 
remove "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12" from discovery file).

Or it's better to add flags to nvme discover ?

> 
> Byte,
> 	Johannes
> 

-Max.

NVMf initiator persistent across boots

[Johannes Thumshirn]

On Thu, Mar 08, 2018@12:03:18PM +0200, Max Gurtovoy wrote:
> 
> 
> On 3/8/2018 10:39 AM, Johannes Thumshirn wrote:
> > On Wed, Mar 07, 2018@02:47:52PM +0200, Max Gurtovoy wrote:
> > > > [Unit]
> > > > Description=NVMf auto discovery service
> > > > After=systemd-modules-load.service network-online.target
> > > > 
> > > > [Service]
> > > > Type=oneshot
> > > > ExecStart=/usr/bin/nvme connect-all
> > > > StandardOutput=journal
> > > > 
> > > > [Timer]
> > > > OnUnitActiveSec=1min
> > > > 
> > > > [Install]
> > > > WantedBy=multi-user.target timers.target
> > > > -- 
> > > > 
> > > > That would simply run nvme connect-all once every say minute.
> > > > The only problem is that it relies on the kernel to fail
> > > > duplicate subsystems. We could enforce that in nvme-cli for that
> > > > matter though (we can compare against sysfs address and subsysnqn).
> > > 
> > > Yes, we can also add flag to nvme discover command to add the parameters to
> > > the discovery file in case they are not exist.
> > > 
> > > 
> > > > 
> > > > Johannes is probably one to know better than me if this is the
> > > > correct way to go...
> > > 
> > > Johannes, any comment ?
> > >  From what I tried, we need to create a .timer and .service files for
> > > systemd...
> > 
> > Sorry I was on FTO for some days.
> > 
> > One thing that I feel is missing in this whole "let's just call nvme
> > connect-all and we're done" discussion is, we currently can't really
> > specifiy how many connections to the target we want to initiate.
> > 
> > I usually run several nvme connect calls with different --host-traddr
> > arguments to connect from multiple HCAs to the target.
> > 
> > I tried to hack the ability to specify a list of host traddrs for nvme
> > connect and connect-all and then just loop the connect cann in
> > nvme-cli but this somehow feels wrong.
> 
> Why not adding -w <host-traddr> to the /etc/nvme/discovery.conf file lines ?
> I guess connect-all will use it too, right ? (need to check it).

I meant multiple -w arguments. My usual connect is:
nvme connect -t rdma -a 1.1.1.1 -n nvme-test -s 4420 -w 1.1.1.2
nvme connect -t rdma -a 1.1.1.1 -n nvme-test -s 4420 -w 1.1.1.3
to connect from both HCAs to the target.

> 
> > 
> > For the systemd service + timer units above, they look good on the
> > first sight, but I'm not sure I like the unconditional "polling" of
> > the connect-all call.
> 
> Yes this is no the perfect solution but I guess future discovery_manager
> should solve it, but we need to find solution till then.
> 
> I tought about adding new commands/flags to add/remove parameters to
> /etc/nvme/discovery.conf.
> how about adding:
> nvme persist-add -t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12 (will add to
> discovery file "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12")
> nvme persist-remove -t rdma -a 11.11.11.11 -s 4420 -w 11.11.12.12 (will
> remove "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12" from discovery file).
> 
> Or it's better to add flags to nvme discover ?

Good question. I'd be fine with both. The only thing that should be
considered as well is, if you already need the nvmf connection in the
initrd, you'll have to rebuild the initrd after each change of
/etc/nvme/discovery.conf and thus every nvme
persist-add/persist-remove (or discover --persistent) call.

Byte,
	Johannes

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

NVMf initiator persistent across boots

[Max Gurtovoy]

On 3/8/2018 12:13 PM, Johannes Thumshirn wrote:
> On Thu, Mar 08, 2018@12:03:18PM +0200, Max Gurtovoy wrote:
>>
>>
>> On 3/8/2018 10:39 AM, Johannes Thumshirn wrote:
>>> On Wed, Mar 07, 2018@02:47:52PM +0200, Max Gurtovoy wrote:
>>>>> [Unit]
>>>>> Description=NVMf auto discovery service
>>>>> After=systemd-modules-load.service network-online.target
>>>>>
>>>>> [Service]
>>>>> Type=oneshot
>>>>> ExecStart=/usr/bin/nvme connect-all
>>>>> StandardOutput=journal
>>>>>
>>>>> [Timer]
>>>>> OnUnitActiveSec=1min
>>>>>
>>>>> [Install]
>>>>> WantedBy=multi-user.target timers.target
>>>>> -- 
>>>>>
>>>>> That would simply run nvme connect-all once every say minute.
>>>>> The only problem is that it relies on the kernel to fail
>>>>> duplicate subsystems. We could enforce that in nvme-cli for that
>>>>> matter though (we can compare against sysfs address and subsysnqn).
>>>>
>>>> Yes, we can also add flag to nvme discover command to add the parameters to
>>>> the discovery file in case they are not exist.
>>>>
>>>>
>>>>>
>>>>> Johannes is probably one to know better than me if this is the
>>>>> correct way to go...
>>>>
>>>> Johannes, any comment ?
>>>>   From what I tried, we need to create a .timer and .service files for
>>>> systemd...
>>>
>>> Sorry I was on FTO for some days.
>>>
>>> One thing that I feel is missing in this whole "let's just call nvme
>>> connect-all and we're done" discussion is, we currently can't really
>>> specifiy how many connections to the target we want to initiate.
>>>
>>> I usually run several nvme connect calls with different --host-traddr
>>> arguments to connect from multiple HCAs to the target.
>>>
>>> I tried to hack the ability to specify a list of host traddrs for nvme
>>> connect and connect-all and then just loop the connect cann in
>>> nvme-cli but this somehow feels wrong.
>>
>> Why not adding -w <host-traddr> to the /etc/nvme/discovery.conf file lines ?
>> I guess connect-all will use it too, right ? (need to check it).
> 
> I meant multiple -w arguments. My usual connect is:
> nvme connect -t rdma -a 1.1.1.1 -n nvme-test -s 4420 -w 1.1.1.2
> nvme connect -t rdma -a 1.1.1.1 -n nvme-test -s 4420 -w 1.1.1.3
> to connect from both HCAs to the target.

Yes, I got it.
IMO if you will add:
-t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.2
-t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.3

to the discovery file it should be fine.
I think we'll need to fix few things in nvmecli and in the driver too.
I'm checking this.

> 
>>
>>>
>>> For the systemd service + timer units above, they look good on the
>>> first sight, but I'm not sure I like the unconditional "polling" of
>>> the connect-all call.
>>
>> Yes this is no the perfect solution but I guess future discovery_manager
>> should solve it, but we need to find solution till then.
>>
>> I tought about adding new commands/flags to add/remove parameters to
>> /etc/nvme/discovery.conf.
>> how about adding:
>> nvme persist-add -t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12 (will add to
>> discovery file "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12")
>> nvme persist-remove -t rdma -a 11.11.11.11 -s 4420 -w 11.11.12.12 (will
>> remove "-t rdma -a 11.11.11.11 -s 4420 -w 11.12.12.12" from discovery file).
>>
>> Or it's better to add flags to nvme discover ?
> 
> Good question. I'd be fine with both. The only thing that should be
> considered as well is, if you already need the nvmf connection in the
> initrd, you'll have to rebuild the initrd after each change of
> /etc/nvme/discovery.conf and thus every nvme
> persist-add/persist-remove (or discover --persistent) call.

Let's assume that we don't need it for now, otherwise it will be 
unpleasent :)

> 
> Byte,
> 	Johannes
> 

NVMf initiator persistent across boots

[Johannes Thumshirn]

On Thu, Mar 08, 2018@03:15:42PM +0200, Max Gurtovoy wrote:
> Yes, I got it.
> IMO if you will add:
> -t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.2
> -t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.3
> 
> to the discovery file it should be fine.
> I think we'll need to fix few things in nvmecli and in the driver too.
> I'm checking this.

Hmm interesting. This probably was too obvious for me to try...

> > Good question. I'd be fine with both. The only thing that should be
> > considered as well is, if you already need the nvmf connection in the
> > initrd, you'll have to rebuild the initrd after each change of
> > /etc/nvme/discovery.conf and thus every nvme
> > persist-add/persist-remove (or discover --persistent) call.
> 
> Let's assume that we don't need it for now, otherwise it will be unpleasent
> :)

Given the number of "Boot from SAN failed with XXX" Bugs I have seen
the last years I doubt we won't need it ;-).

But yes we can think about this in step two I guess.

-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

NVMf initiator persistent across boots

[Max Gurtovoy]

On 3/8/2018 3:26 PM, Johannes Thumshirn wrote:
> On Thu, Mar 08, 2018@03:15:42PM +0200, Max Gurtovoy wrote:
>> Yes, I got it.
>> IMO if you will add:
>> -t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.2
>> -t rdma -a 1.1.1.1 -s 4420 -w 1.1.1.3
>>
>> to the discovery file it should be fine.
>> I think we'll need to fix few things in nvmecli and in the driver too.
>> I'm checking this.
> 
> Hmm interesting. This probably was too obvious for me to try...
> 

worked for me with:

diff --git a/fabrics.c b/fabrics.c
index 4bdaaf2..774d5dc 100644
--- a/fabrics.c
+++ b/fabrics.c
@@ -614,6 +614,14 @@ static int connect_ctrl(struct 
nvmf_disc_rsp_page_entry *e)
                 p += len;
         }

+       if (cfg.host_traddr) {
+               len = sprintf(p, ",host_traddr=%s", cfg.host_traddr);
+               if (len < 0)
+                       return -EINVAL;
+               p+= len;
+       }
+
+
         switch (e->trtype) {
         case NVMF_TRTYPE_LOOP: /* loop */
                 len = sprintf(p, ",transport=loop");
@@ -659,11 +667,6 @@ static int connect_ctrl(struct 
nvmf_disc_rsp_page_entry *e)
                                 return -EINVAL;
                         p += len;

-                       len = sprintf(p, ",host_traddr=%s", 
cfg.host_traddr);
-                       if (len < 0)
-                               return -EINVAL;
-                       p+= len;
-
                         len = sprintf(p, ",traddr=%.*s",
                                       space_strip_len(NVMF_TRADDR_SIZE, 
e->traddr),
                                       e->traddr);


thoughts ?


>>> Good question. I'd be fine with both. The only thing that should be
>>> considered as well is, if you already need the nvmf connection in the
>>> initrd, you'll have to rebuild the initrd after each change of
>>> /etc/nvme/discovery.conf and thus every nvme
>>> persist-add/persist-remove (or discover --persistent) call.
>>
>> Let's assume that we don't need it for now, otherwise it will be unpleasent
>> :)
> 
> Given the number of "Boot from SAN failed with XXX" Bugs I have seen
> the last years I doubt we won't need it ;-).
> 
> But yes we can think about this in step two I guess.
> 

NVMf initiator persistent across boots

[Johannes Thumshirn]

On Thu, Mar 08, 2018@07:04:17PM +0200, Max Gurtovoy wrote:
> diff --git a/fabrics.c b/fabrics.c
> index 4bdaaf2..774d5dc 100644
> --- a/fabrics.c
> +++ b/fabrics.c
> @@ -614,6 +614,14 @@ static int connect_ctrl(struct nvmf_disc_rsp_page_entry
> *e)
>                 p += len;
>         }
> 
> +       if (cfg.host_traddr) {
> +               len = sprintf(p, ",host_traddr=%s", cfg.host_traddr);
> +               if (len < 0)
> +                       return -EINVAL;
> +               p+= len;
> +       }
> +
> +
>         switch (e->trtype) {
>         case NVMF_TRTYPE_LOOP: /* loop */
>                 len = sprintf(p, ",transport=loop");
> @@ -659,11 +667,6 @@ static int connect_ctrl(struct nvmf_disc_rsp_page_entry
> *e)
>                                 return -EINVAL;
>                         p += len;
> 
> -                       len = sprintf(p, ",host_traddr=%s",
> cfg.host_traddr);
> -                       if (len < 0)
> -                               return -EINVAL;
> -                       p+= len;
> -
>                         len = sprintf(p, ",traddr=%.*s",
>                                       space_strip_len(NVMF_TRADDR_SIZE,
> e->traddr),
>                                       e->traddr);
> 
> 
> thoughts ?

Looks good to me.
-- 
Johannes Thumshirn                                          Storage
jthumshirn at suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Jane Smithard, Graham Norton
HRB 21284 (AG N?rnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

NVMf initiator persistent across boots

[Hannes Reinecke]

On 03/05/2018 08:45 PM, Sagi Grimberg wrote:
> 
>> Hi all,
>>
>> Sorry for the delay, but I wanted to know if we had a progress for
>> this topic. I want to add functionality that gives persistency for
>> fabric devices not only across boots but also across adapter/device
>> removals.
>> My idea is to add a flag --persist to "nvme connect" command. This
>> command will add the created ctrls to a file
>> (/etc/nvme-fabrics/persistent_ctrls for example).
>> And I want to add a udev rule that in case of ctrl removal, we'll run
>> a script that try to periodicly connect to the target.
> Why not just add a periodic task to run nvme connect-all?
> -- 
> [Unit]
> Description=NVMf auto discovery service
> After=systemd-modules-load.service network-online.target
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/nvme connect-all
> StandardOutput=journal
> 
> [Timer]
> OnUnitActiveSec=1min
> 
> [Install]
> WantedBy=multi-user.target timers.target
> -- 
> 
> That would simply run nvme connect-all once every say minute.
> The only problem is that it relies on the kernel to fail
> duplicate subsystems. We could enforce that in nvme-cli for that
> matter though (we can compare against sysfs address and subsysnqn).
> 
> Johannes is probably one to know better than me if this is the
> correct way to go...
> 
In principle we could; but we should first decide what the correct
strategy here is.
FC has the same issue, and tries to overcome this by adding custom
uevents.

Wasn't there a session proposed at LSF/MM?
I guess that's the right forum to discuss it, as it'll involve all fabrics.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		   Teamlead Storage & Networking
hare at suse.de			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: F. Imend?rffer, J. Smithard, J. Guild, D. Upmanyu, G. Norton
HRB 21284 (AG N?rnberg)

NVMf initiator persistent across boots

[Max Gurtovoy]

On 3/7/2018 5:48 PM, Hannes Reinecke wrote:
> On 03/05/2018 08:45 PM, Sagi Grimberg wrote:
>>
>>> Hi all,
>>>
>>> Sorry for the delay, but I wanted to know if we had a progress for
>>> this topic. I want to add functionality that gives persistency for
>>> fabric devices not only across boots but also across adapter/device
>>> removals.
>>> My idea is to add a flag --persist to "nvme connect" command. This
>>> command will add the created ctrls to a file
>>> (/etc/nvme-fabrics/persistent_ctrls for example).
>>> And I want to add a udev rule that in case of ctrl removal, we'll run
>>> a script that try to periodicly connect to the target.
>> Why not just add a periodic task to run nvme connect-all?
>> -- 
>> [Unit]
>> Description=NVMf auto discovery service
>> After=systemd-modules-load.service network-online.target
>>
>> [Service]
>> Type=oneshot
>> ExecStart=/usr/bin/nvme connect-all
>> StandardOutput=journal
>>
>> [Timer]
>> OnUnitActiveSec=1min
>>
>> [Install]
>> WantedBy=multi-user.target timers.target
>> -- 
>>
>> That would simply run nvme connect-all once every say minute.
>> The only problem is that it relies on the kernel to fail
>> duplicate subsystems. We could enforce that in nvme-cli for that
>> matter though (we can compare against sysfs address and subsysnqn).
>>
>> Johannes is probably one to know better than me if this is the
>> correct way to go...
>>
> In principle we could; but we should first decide what the correct
> strategy here is.
> FC has the same issue, and tries to overcome this by adding custom
> uevents.
> 
> Wasn't there a session proposed at LSF/MM?
> I guess that's the right forum to discuss it, as it'll involve all fabrics.

Yes, I'll catch up since I probably wouldn't make it to LSF this year.
Meanwhile I created the following timer/service for systemd:

timer:
------------
[Unit]
Description=NVMf auto discovery timer

[Timer]
OnUnitActiveSec=60
Unit=nvme_fabrics_persistant.service

[Install]
WantedBy=multi-user.target timers.target

service:
----------
[Unit]
Description=NVMf auto discovery service
Requires=network.target
After=systemd-modules-load.service network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nvme connect-all
StandardOutput=journal

[Install]
WantedBy=multi-user.target timers.target


I works for me but the problem is that we trigger nvme discover every 60 
seconds...
But I guess until we'll have final definition for discovery for the 
fabrics we can live with that.

> 
> Cheers,
> 
> Hannes
>