[linux-lvm] LVM + Multipathing

[linux-lvm] LVM + Multipathing

[Ritesh Raj Sarraf]

Hi,

I have a question regarding the combination of LVM + Multipathing in a SAN
environment.

I have a LUN mapped to a host using iSCSI. I have two two paths to the LUN. This
setup, using multipathing, provides me failover functionality.

My question is how does LVM react to path failures ?
In a failover environment, multipathing can take upto 120 seconds to detect that
the active path to the LUN is no more available and then switch to the
secondary path. During this 120 seconds, how does LVM react ? Does it really
sense that the device is offlined ? Or that information is never passed to LVM
and it just allows I/O to happen leaving that responsibility to the
multipathing layer ?

Thanks,
Ritesh
-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Re: [linux-lvm] LVM + Multipathing

[Luca Berra]

On Sat, Feb 17, 2007 at 09:52:27PM +0530, Ritesh Raj Sarraf wrote:
>Hi,
>
>I have a question regarding the combination of LVM + Multipathing in a SAN
>environment.
>
>I have a LUN mapped to a host using iSCSI. I have two two paths to the LUN. This
>setup, using multipathing, provides me failover functionality.
>
>My question is how does LVM react to path failures ?
>In a failover environment, multipathing can take upto 120 seconds to detect that
>the active path to the LUN is no more available and then switch to the
>secondary path. During this 120 seconds, how does LVM react ? Does it really
>sense that the device is offlined ? Or that information is never passed to LVM
>and it just allows I/O to happen leaving that responsibility to the
>multipathing layer ?

i believe LVM does not react at all.

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: LVM + Multipathing

[Ritesh Raj Sarraf]

Luca Berra wrote:

> On Sat, Feb 17, 2007 at 09:52:27PM +0530, Ritesh Raj Sarraf wrote:
>>Hi,
>>
>>I have a question regarding the combination of LVM + Multipathing in a SAN
>>environment.
>>
>>I have a LUN mapped to a host using iSCSI. I have two two paths to the LUN.
>>This setup, using multipathing, provides me failover functionality.
>>
>>My question is how does LVM react to path failures ?
>>In a failover environment, multipathing can take upto 120 seconds to detect
>>that the active path to the LUN is no more available and then switch to the
>>secondary path. During this 120 seconds, how does LVM react ? Does it really
>>sense that the device is offlined ? Or that information is never passed to LVM
>>and it just allows I/O to happen leaving that responsibility to the
>>multipathing layer ?
> 
> i believe LVM does not react at all.
> 

I brought this question because I'm seeing Filesystem READONLY issues when doing
a takeover/giveback.

My understanding is that a filesystem usually goes read-only, only when it
senses errors in the drive. Now I believe LVM might the culprit because beneath
the filesystem and above the block device, LVM is the only layer.

Thanks,
Ritesh
-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Re: [linux-lvm] Re: LVM + Multipathing

[Luca Berra]

On Tue, Feb 20, 2007 at 06:00:25PM +0530, Ritesh Raj Sarraf wrote:
>Luca Berra wrote:
>
>> On Sat, Feb 17, 2007 at 09:52:27PM +0530, Ritesh Raj Sarraf wrote:
>>>Hi,
>>>
>>>I have a question regarding the combination of LVM + Multipathing in a SAN
>>>environment.
>>>
>>>I have a LUN mapped to a host using iSCSI. I have two two paths to the LUN.
>>>This setup, using multipathing, provides me failover functionality.
>>>
>>>My question is how does LVM react to path failures ?
>>>In a failover environment, multipathing can take upto 120 seconds to detect
>>>that the active path to the LUN is no more available and then switch to the
>>>secondary path. During this 120 seconds, how does LVM react ? Does it really
>>>sense that the device is offlined ? Or that information is never passed to LVM
>>>and it just allows I/O to happen leaving that responsibility to the
>>>multipathing layer ?
>> 
>> i believe LVM does not react at all.
>> 
>
>I brought this question because I'm seeing Filesystem READONLY issues when doing
>a takeover/giveback.
>
>My understanding is that a filesystem usually goes read-only, only when it
>senses errors in the drive. Now I believe LVM might the culprit because beneath
>the filesystem and above the block device, LVM is the only layer.
to clarify my point above:
i meant that LVM (actually device-mapper) does not do anything special.
it just passes the IO requests from the fs layer to the
underlying block device and if the underlyng block device returns an IO
error then it is passed back to the fs. which will cause ext2 to remount
the filesystem readonly.

L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

[linux-lvm] Re: Re: LVM + Multipathing

[Ritesh Raj Sarraf]

Luca Berra wrote:

> i meant that LVM (actually device-mapper) does not do anything special.
> it just passes the IO requests from the fs layer to the
> underlying block device and if the underlyng block device returns an IO
> error then it is passed back to the fs. which will cause ext2 to remount
> the filesystem readonly.

Hi,

Thanks for clarifying.

In my case, the block device is hidden by the multipathing layer. 
It is something like:
(Block Device(Multipathing(LVM(Filesystem) ) ) )

Now during takeover/giveback, the multipathing layer is intelligent enough to
wait till <120 seconds before declaring that the path has really gone offline
and no more paths are available.
If within the 120 seconds time span, the takeover succeeds, the path is back
online and everything works fine in a non-LVM setup.

It is only in an LVM setup that a takeover/giveback ends up with the host OS
having a filesystem read-only problem.
Now if I go with your explanation, I shouldn't have had the filesystem read-only
problem since the I/O is being passed on to the multipathing layer which is
intelligent enough to wait for N seconds before really sending an I/O Error.

Are there any timeout options in LVM to allow it to wait for N seconds before
sending out an error ?
(I understand that LVM might not be involved, but just wondering).


Thanks,
Ritesh
-- 
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
"Stealing logic from one person is plagiarism, stealing from many is research."
"The great are those who achieve the impossible, the petty are those who
cannot - rrs"

Re: [linux-lvm] Re: LVM + Multipathing

[Matt P]

Your problem might be related to this bug
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=213921

On 2/20/07, Luca Berra <bluca@comedia.it> wrote:
> On Tue, Feb 20, 2007 at 06:00:25PM +0530, Ritesh Raj Sarraf wrote:
> >Luca Berra wrote:
> >
> >> On Sat, Feb 17, 2007 at 09:52:27PM +0530, Ritesh Raj Sarraf wrote:
> >>>Hi,
> >>>
> >>>I have a question regarding the combination of LVM + Multipathing in a SAN
> >>>environment.
> >>>
> >>>I have a LUN mapped to a host using iSCSI. I have two two paths to the LUN.
> >>>This setup, using multipathing, provides me failover functionality.
> >>>
> >>>My question is how does LVM react to path failures ?
> >>>In a failover environment, multipathing can take upto 120 seconds to detect
> >>>that the active path to the LUN is no more available and then switch to the
> >>>secondary path. During this 120 seconds, how does LVM react ? Does it really
> >>>sense that the device is offlined ? Or that information is never passed to LVM
> >>>and it just allows I/O to happen leaving that responsibility to the
> >>>multipathing layer ?
> >>
> >> i believe LVM does not react at all.
> >>
> >
> >I brought this question because I'm seeing Filesystem READONLY issues when doing
> >a takeover/giveback.
> >
> >My understanding is that a filesystem usually goes read-only, only when it
> >senses errors in the drive. Now I believe LVM might the culprit because beneath
> >the filesystem and above the block device, LVM is the only layer.
> to clarify my point above:
> i meant that LVM (actually device-mapper) does not do anything special.
> it just passes the IO requests from the fs layer to the
> underlying block device and if the underlyng block device returns an IO
> error then it is passed back to the fs. which will cause ext2 to remount
> the filesystem readonly.
>
> L.
>
> --
> Luca Berra -- bluca@comedia.it
>         Communication Media & Services S.r.l.
>  /"\
>  \ /     ASCII RIBBON CAMPAIGN
>   X        AGAINST HTML MAIL
>  / \
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

Re: [linux-lvm] Re: LVM + Multipathing

[Christian.Rohrmeier]

Using device-mapper-multipath to create a device alias (/dev/mapper/<...>)
which you would then use as a PV in your LVM setup. Device-mapper-multipath
can indeed handle multi-pathingsuch that a failure of one one of the path
would not lead to any IO issues and LVm would just keep on ticking.




                                                                           
             "Matt P"                                                      
             <slarty.tj@gmail.                                             
             com>                                                       To 
             Sent by:                  linux-lvm@redhat.com                
             linux-lvm-bounces                                          cc 
             @redhat.com                                                   
                                                                   Subject 
                                       Re: [linux-lvm] Re: LVM +           
             20.02.2007 18:03          Multipathing                        
                                                                           
                                                                           
             Please respond to                                             
                LVM general                                                
              discussion and                                               
                development                                                
             <linux-lvm@redhat                                             
                   .com>                                                   
                                                                           
                                                                           




Your problem might be related to this bug
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=213921

On 2/20/07, Luca Berra <bluca@comedia.it> wrote:
> On Tue, Feb 20, 2007 at 06:00:25PM +0530, Ritesh Raj Sarraf wrote:
> >Luca Berra wrote:
> >
> >> On Sat, Feb 17, 2007 at 09:52:27PM +0530, Ritesh Raj Sarraf wrote:
> >>>Hi,
> >>>
> >>>I have a question regarding the combination of LVM + Multipathing in a
SAN
> >>>environment.
> >>>
> >>>I have a LUN mapped to a host using iSCSI. I have two two paths to the
LUN.
> >>>This setup, using multipathing, provides me failover functionality.
> >>>
> >>>My question is how does LVM react to path failures ?
> >>>In a failover environment, multipathing can take upto 120 seconds to
detect
> >>>that the active path to the LUN is no more available and then switch
to the
> >>>secondary path. During this 120 seconds, how does LVM react ? Does it
really
> >>>sense that the device is offlined ? Or that information is never
passed to LVM
> >>>and it just allows I/O to happen leaving that responsibility to the
> >>>multipathing layer ?
> >>
> >> i believe LVM does not react at all.
> >>
> >
> >I brought this question because I'm seeing Filesystem READONLY issues
when doing
> >a takeover/giveback.
> >
> >My understanding is that a filesystem usually goes read-only, only when
it
> >senses errors in the drive. Now I believe LVM might the culprit because
beneath
> >the filesystem and above the block device, LVM is the only layer.
> to clarify my point above:
> i meant that LVM (actually device-mapper) does not do anything special.
> it just passes the IO requests from the fs layer to the
> underlying block device and if the underlyng block device returns an IO
> error then it is passed back to the fs. which will cause ext2 to remount
> the filesystem readonly.
>
> L.
>
> --
> Luca Berra -- bluca@comedia.it
>         Communication Media & Services S.r.l.
>  /"\
>  \ /     ASCII RIBBON CAMPAIGN
>   X        AGAINST HTML MAIL
>  / \
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Re: Re: LVM + Multipathing

[Dave Wysochanski]

On Tue, 2007-02-20 at 20:19 +0530, Ritesh Raj Sarraf wrote:
> Luca Berra wrote:
> 
> > i meant that LVM (actually device-mapper) does not do anything special.
> > it just passes the IO requests from the fs layer to the
> > underlying block device and if the underlyng block device returns an IO
> > error then it is passed back to the fs. which will cause ext2 to remount
> > the filesystem readonly.
> 
> Hi,
> 
> Thanks for clarifying.
> 
> In my case, the block device is hidden by the multipathing layer. 
> It is something like:
> (Block Device(Multipathing(LVM(Filesystem) ) ) )
> 
> Now during takeover/giveback, the multipathing layer is intelligent enough to
> wait till <120 seconds before declaring that the path has really gone offline
> and no more paths are available.
> If within the 120 seconds time span, the takeover succeeds, the path is back
> online and everything works fine in a non-LVM setup.
> 
> It is only in an LVM setup that a takeover/giveback ends up with the host OS
> having a filesystem read-only problem.

Interesting.

> Now if I go with your explanation, I shouldn't have had the filesystem read-only
> problem since the I/O is being passed on to the multipathing layer which is
> intelligent enough to wait for N seconds before really sending an I/O Error.
> 
> Are there any timeout options in LVM to allow it to wait for N seconds before
> sending out an error ?
> (I understand that LVM might not be involved, but just wondering).
> 

There are no LVM timeouts like you are suggesting.  LVM just does I/O to
devices like any other application.  The timeout settings you're looking
for should be in the layers below LVM.

What does your /etc/multipath.conf file look like?  Do you have
"no_path_retry" set and/or "queue_if_no_path"?  Both of these settings
will affect how multipath deals with a "no paths available" situation.

There are also settings below multipath, in the low-level driver(s).
Since you are using iscsi, I assume you've got open-iscsi?
You can set node.session.timeo.replacement_timeout to a lower value (it
is 120 by default in /etc/iscsi/iscsid.conf) if you want faster path
failure detection and faster failover.

In a lot of cases, people set the low level driver timeouts smaller (<
30 sec) to allow for quicker failure detection and failover, but a
higher or infinite value for the "no paths available" situation.

You might try open-iscsi.org and/or dm-devel lists.


> 
> Thanks,
> Ritesh