[SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

[SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]



On 12/20/19, 1:05 PM, "jasonm(a)ami.com" <jasonm(a)ami.com> wrote:

    I'm trying to see logical volumes created by SPDK_A on SPDK_B, but the only way I can see them is if I reboot the NODE_B system running SPDK_B.  What needs to be done in order to sync the logical volumes on a namespace between two SPDKs running on separate hosts (namespace sharing)?  Is this even possible?

Hi Jason,

That's not possible currently.  When SPDK_A creates the logical volume, there is no notification mechanism to tell SPDK_B that a new logical volumes exists and/or that it should try to reread logical volume metadata to detect the new logical volume.

It's not exactly clear why you need to do a full reboot, rather than just restarting SPDK_B.  But the SPDK logical volume store implementation is not currently designed to be used like this from two different hosts.  Coordination of the dirty bit and the on-disk used masks (for md pages, clusters and blobids) would need to be considered.

Regards,

-Jim
  
    
    My system setup is described below.  I have confirmed namespace sharing works with the Linux kernel driver (not LVS just namespace).  SPDK version is 19.04.00,  Kernel 4.15.0-29,  Distro Ubuntu 18.04.1 LTS.
    
    DUAL PORT CHASSIS
    - NODE_A
    --- SPDK_A (create volume store, create volumes)
    
    - NODE_B
    --- SPDK_B (must be rebooted to see logical volumes created)
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    

[SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 3484 bytes --]

On Sat, Dec 21, 2019, 00:49 Jason Messer <JasonM(a)ami.com> wrote:

> If I restart SPDK_B (killall spdk_tgt and then restart it), I still do not
> see the logical volumes created by SPDK_A.  I don't understand why an
> actual reboot is needed as opposed to simply restarting the SPDK.
>

That's likely some issue with your dual-port drives.


> Regarding the design, what other approach can SPDK be used in a system
> design with two hosts sharing an nvme namespace on a dual port drive for
> high availability and fail over?
>

There's no design here. It just needs to be done, and public mailing list
is hardly a place to ask for design ideas. You might want to consult
distributed system textbooks like Tannenbaum's classic.

Regards,
Andrey


> -----Original Message-----
> From: Andrey Kuzmin [mailto:andrey.v.kuzmin(a)gmail.com]
> Sent: Friday, December 20, 2019 4:22 PM
> To: Storage Performance Development Kit
> Subject: [SPDK] Re: Lvol sync issues running SPDK on separate hosts
> connected via dual port drives
>
> On Fri, Dec 20, 2019, 23:04 <jasonm(a)ami.com> wrote:
>
> > I'm trying to see logical volumes created by SPDK_A on SPDK_B, but the
> > only way I can see them is if I reboot the NODE_B system running SPDK_B.
> > What needs to be done in order to sync the logical volumes on a
> > namespace between two SPDKs running on separate hosts (namespace
> > sharing)?  Is this even possible?
> >
>
> Likely not, at least not the way you're trying to achieve it.
>
> In your setup, SPDK_A persists lvol config on media, but there's no way
> for SPDK_B to learn that any changes occurred other than by regularly
> scanning known blobstore metadata location(s). Since, in your settings,
> such a scan occurs just at power-on when SPDK_B loads blobstore, you see
> lvols only when SPDK_B reboots.
>
> In practice, this is hardly a realistic approach to building distributed
> system.
>
> Regards,
> Andrey
>
> >
> > My system setup is described below.  I have confirmed namespace
> > sharing works with the Linux kernel driver (not LVS just namespace).
> > SPDK version is 19.04.00,  Kernel 4.15.0-29,  Distro Ubuntu 18.04.1 LTS.
> >
> > DUAL PORT CHASSIS
> > - NODE_A
> > --- SPDK_A (create volume store, create volumes)
> >
> > - NODE_B
> > --- SPDK_B (must be rebooted to see logical volumes created)
> > _______________________________________________
> > SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to
> > spdk-leave(a)lists.01.org
> >
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>
> Please consider the environment before printing this email.
>
> The information contained in this message may be confidential and
> proprietary to American Megatrends (AMI).  This communication is intended
> to be read only by the individual or entity to whom it is addressed or by
> their designee. If the reader of this message is not the intended
> recipient, you are on notice that any distribution of this message, in any
> form, is strictly prohibited.  Please promptly notify the sender by reply
> e-mail or by telephone at 770-246-8600, and then delete or destroy all
> copies of the transmission.
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>

[SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

[Jason Messer]

[-- Attachment #1: Type: text/plain, Size: 2834 bytes --]

If I restart SPDK_B (killall spdk_tgt and then restart it), I still do not see the logical volumes created by SPDK_A.  I don't understand why an actual reboot is needed as opposed to simply restarting the SPDK.

Regarding the design, what other approach can SPDK be used in a system design with two hosts sharing an nvme namespace on a dual port drive for high availability and fail over?

-----Original Message-----
From: Andrey Kuzmin [mailto:andrey.v.kuzmin(a)gmail.com] 
Sent: Friday, December 20, 2019 4:22 PM
To: Storage Performance Development Kit
Subject: [SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

On Fri, Dec 20, 2019, 23:04 <jasonm(a)ami.com> wrote:

> I'm trying to see logical volumes created by SPDK_A on SPDK_B, but the 
> only way I can see them is if I reboot the NODE_B system running SPDK_B.
> What needs to be done in order to sync the logical volumes on a 
> namespace between two SPDKs running on separate hosts (namespace 
> sharing)?  Is this even possible?
>

Likely not, at least not the way you're trying to achieve it.

In your setup, SPDK_A persists lvol config on media, but there's no way for SPDK_B to learn that any changes occurred other than by regularly scanning known blobstore metadata location(s). Since, in your settings, such a scan occurs just at power-on when SPDK_B loads blobstore, you see lvols only when SPDK_B reboots.

In practice, this is hardly a realistic approach to building distributed system.

Regards,
Andrey

>
> My system setup is described below.  I have confirmed namespace 
> sharing works with the Linux kernel driver (not LVS just namespace).  
> SPDK version is 19.04.00,  Kernel 4.15.0-29,  Distro Ubuntu 18.04.1 LTS.
>
> DUAL PORT CHASSIS
> - NODE_A
> --- SPDK_A (create volume store, create volumes)
>
> - NODE_B
> --- SPDK_B (must be rebooted to see logical volumes created) 
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to 
> spdk-leave(a)lists.01.org
>
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org

Please consider the environment before printing this email.

The information contained in this message may be confidential and proprietary to American Megatrends (AMI).  This communication is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited.  Please promptly notify the sender by reply e-mail or by telephone at 770-246-8600, and then delete or destroy all copies of the transmission.

[SPDK] Re: Lvol sync issues running SPDK on separate hosts connected via dual port drives

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 1425 bytes --]

On Fri, Dec 20, 2019, 23:04 <jasonm(a)ami.com> wrote:

> I'm trying to see logical volumes created by SPDK_A on SPDK_B, but the
> only way I can see them is if I reboot the NODE_B system running SPDK_B.
> What needs to be done in order to sync the logical volumes on a namespace
> between two SPDKs running on separate hosts (namespace sharing)?  Is this
> even possible?
>

Likely not, at least not the way you're trying to achieve it.

In your setup, SPDK_A persists lvol config on media, but there's no way for
SPDK_B to learn that any changes occurred other than by regularly scanning
known blobstore metadata location(s). Since, in your settings, such a scan
occurs just at power-on when SPDK_B loads blobstore, you see lvols only
when SPDK_B reboots.

In practice, this is hardly a realistic approach to building distributed
system.

Regards,
Andrey

>
> My system setup is described below.  I have confirmed namespace sharing
> works with the Linux kernel driver (not LVS just namespace).  SPDK version
> is 19.04.00,  Kernel 4.15.0-29,  Distro Ubuntu 18.04.1 LTS.
>
> DUAL PORT CHASSIS
> - NODE_A
> --- SPDK_A (create volume store, create volumes)
>
> - NODE_B
> --- SPDK_B (must be rebooted to see logical volumes created)
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>