Re: multipath-tools: scsi_id based path priorities and multiple prioritizers

Re: multipath-tools: scsi_id based path priorities and multiple prioritizers

[Christophe Varoqui]

> 
> So, we poked up with code a bit, and wrote up a custom prioritizer,
> called sg_id. (patch for the latest multipath-tools available here:
> http://viktor.ee/multipath-tools-patches/sg_id_prio.patch)
> 
Would the existing weightedpath work for you ?

> And so we’ve played another couple of our hours with multipath-tools
> code allowing it to accept multiple prioritizers in prio
> configuration. (patch here
> http://viktor.ee/multipath-tools-patches/multiprio.patch) 
> 
> In this case, prioritizers should be separated by coma, semicolon or
> space, and the end priority would be a sum of priorities given by all
> of the specified prioritizers. (a single prioritizer value is also
> accepted of course.)
> 
> As an example:
> 
>         prio                  "sg_id, alua"
> 
>         prio_args             "prio_sg_id(default)=0
> prio_sg_id(^[0-2]:0)=100"
> 

> All comments are kindly welcome!

I personnaly find the multi-prioritizer feature interesting.

Let's hear other maintainers' comments ...

Best regards,
Christophe Varoqui
www.opensvc.com


--
dm-devel mailing list
dm-devel@redhat.com
https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipath-tools: scsi_id based path priorities and multiple prioritizers

[Viktor Larionov]

[-- Attachment #1.1.1: Type: text/plain, Size: 2690 bytes --]

Hi Christophe!


Viktor Larionov
IT osakonna juhataja
IT-osakond
Salva Kindlustuse AS
Tel: (+372) 683 0630 | GSM: (+372) 566 86811 | Viktor.Larionov@salva.ee | www.salva.ee
(SMX)338844

On 18.05.2013 16:40, Christophe Varoqui wrote:

>> So, we poked up with code a bit, and wrote up a custom prioritizer,
>> called sg_id. (patch for the latest multipath-tools available here:
>> http://viktor.ee/multipath-tools-patches/sg_id_prio.patch [1])
>
> Would the existing weightedpath work for you ?
>

It's a shame I missed weighted path while googling for a solution in 
the
first place, thanks for the hint. I checked into that, and I should say
it's very alike with what we have done in sg_id_prio patch. I've 
briefly
checked into weighted path and I don't quite like the concept of the
design.
Weighted path as suggested wants you to pass the arguments via the same
"prio" conf param which is used to indicate the prioritizer(s) to be
used. I don't see much sense in that, keeping in mind that you have a
"prio_args" param designed initially for that purpose.
That is also the reason why weighted path needs to hack deep into the
libmultipath architecture. (while the modular nature of libmultipath 
sees
that the prioritizer is a separate shared lib and has nothing much to
hack in the main core). From my point that's not quite right.
When it comes to enabling multiple prioritizers in a row (our second
patch) - weighted path approach makes it impossible, as it passes its 
own
arguments on the prio line.

So answering your question - I think that weighted path would not solve
our problem, mainly due to its architectural approach and difficulties
  with multi prioritizer setup when dealing with weighted path.

I also agree with Kiyoshi comment to weighted path back from 2008. The
whole story of assigning a static priority based on hbtl or node name
pattern from my oppinion is useless on it's own. It's a great extra, 
for
use cases as ours, where you just need to tweak a little the priority
given by a more intelligent prioritizer. (that's why multiprio)

On the other hand, weighted path has a couple of neat features like
setting a static priority not only based on hbtl, but also on a device
node name. sg_io_prio supports hbtl only as this is less likely to
change. (though not totally excluded of course). If disregard this 
point,
and the architectural difference both modules are identical.

I personally think that either weighted path should implement a more
intelligent way of operation (utilize prio_args and not mess with the
basic architecture of libmultipath) or our prio_sg_id should support
device node names (which is a half an hour work we can do).

Cheers!
vik

[-- Attachment #1.1.2: Type: text/html, Size: 4600 bytes --]

[-- Attachment #1.2: 5VfDWTiyBUOwvJT.gif --]
[-- Type: image/gif, Size: 6207 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: multipath-tools: scsi_id based path priorities and multiple prioritizers

[Hannes Reinecke]

On 05/18/2013 01:54 PM, Viktor Larionov wrote:
> Hi everybody!
> 
>  
> 
> First of all, thanks for all the hard work you guys have been doing
> developing dm. It’s an amazing piece of work you have done!
> 
> While working with dm-multipath we have bumped into some limitations
> which we felt bit uncomfortable with, and seems like managed to
> change. I’d thought I share the experience on that with others, in
> hope that this would help somebody.
> 
>  
> 
> Long story short – our servers are connected to our SAN with both fc
> and iscsi links. (same targets, same wwid’s are exported both
> through fc and iscsi)
> 
> Pretty much a standard installation – two independent controllers on
> the storage side (fc and iscsi each), dual port fc controllers on
> the server side + iscsi.
> 
> All this leaves us with approximate of 6 paths per device. (2 fc,
> and 4 iscsi – 1 fc, and 2 iscsi per storage controller)
> 
>  
> 
> Now if we use ALUA, which is standard for our infra (IBM Storewize
> V3700), the picture looks pretty much like this:
> 
>  
> 
> alessandra viktor.larionov # multipath -ll www-2-mysql
> 
> www-2-mysql (360050763008080581000000000000029) dm-37 IBM,2145
> 
> size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
> 
> |-+- policy='round-robin 0' prio=50 status=active
> 
> | |- 2:0:0:9  sdak 66:64   active ready running
> 
> | |- 3:0:0:9  sdcf 69:48   active ready running
> 
> | `- 4:0:0:9  sdcy 70:96   active ready running
> 
> `-+- policy='round-robin 0' prio=10 status=enabled
> 
>   |- 1:0:0:9  sdl  8:176   active ready running
> 
>   |- 5:0:0:9  sdcb 68:240  active ready running
> 
>   `- 6:0:0:9  sdct 70:16   active ready running
> 
>  
> 
> Where sdak and sdl are fiber links and the rest of those are iscsi.
> Priorities come from alua which correspond to san controller
> preference at this particular moment.
> 
> What we don’t like about this setup is that fc and iscsi links end
> up with the same prioriy in the same group. The idea behind having
> iscsi links on machines having fc at all, is redundancy to fc failures.
> 
> But we surely don’t want to operate iscsi links the times when
> either primary or backup fc are fully operational.
> 
>  
> 
> So this led us to the idea, of somehow telling the prioritizer to be
> more granular and separate fc and iscsi controller priorities. After
> doing some several hour googling, I found out that we are not the
> only ones with such a story, and there has been no solution to the
> point. (take this one for example
> http://www.redhat.com/archives/dm-devel/2008-August/msg00083.html)
> In fact prio_callout which could possibly solve this kind of thing,
> is deprecated.
> 
>  
> 
> It’s true that there’s no easy or trivial way to determine if a path
> behing an sg is fiber or iscsi (or something else). But thinking on
> this issue, we thought that we actully can satisfy if we could just
> assign a custom priority based on a scsi_id of the device. The idea
> behind it is simple – say in our case we have an IBM ServeRAID
> controller, which is SCSI host 0, Emulex Light Pulse which is SCSI
> host 1 and 2 (for each port respectively and all of the rest is
> iSCSI. So if we could give static priorities based on this
> information this could do the trick.
> 
>  
> 
> So, we poked up with code a bit, and wrote up a custom prioritizer,
> called sg_id. (patch for the latest multipath-tools available here:
> http://viktor.ee/multipath-tools-patches/sg_id_prio.patch)
> 
> Usage is very simple: in /etc/multipath.conf: prio „sg_id“, and
> priorities are passed through prio_args as regexes: e.g. a prio_args of
> 
> prio_sg_id(default)=0 prio_sg_id(^[0-2]:0)=40 prio_sg_id(^5:[2-3]:)=30
> 
> will give prio 40 for everything on SCSI hosts 0, 1 and 2, channel
> 0. 30 on scsi_host 5 channels 2 and 3, and everything else will get 0.
> 
>  
> 
> Using sg_id in the upper example we will have sdl and sdak in the
> first group, and all othe other stuff in the second. Which is ok,
> but not quite.
> 
> The problem with this approach for us is that ALUA gives us valuable
> information on our storage priorities (which controller is primary
> and which is secondary for that particular lun at this particular
> moment), and we’re not quite ready to sacrifice this information
> even for sg_id prios. If there only would be a way to use multiple
> prioritizers.
> 
> And so we’ve played another couple of our hours with multipath-tools
> code allowing it to accept multiple prioritizers in prio
> configuration. (patch here
> http://viktor.ee/multipath-tools-patches/multiprio.patch)
> 
> In this case, prioritizers should be separated by coma, semicolon or
> space, and the end priority would be a sum of priorities given by
> all of the specified prioritizers. (a single prioritizer value is
> also accepted of course.)
> 
> As an example:
> 
>         prio                  "sg_id, alua"
> 
>         prio_args             "prio_sg_id(default)=0
> prio_sg_id(^[0-2]:0)=100"
> 
>  
> 
> So combining the two of above with the same example we get:
> 
>  
> 
> alessandra multipath-tools-0.4.9 # multipath -r www-2-mysql
> 
> reload: www-2-mysql (360050763008080581000000000000029) undef IBM,2145
> 
> size=10G features='1 queue_if_no_path' hwhandler='0' wp=undef
> 
> |-+- policy='round-robin 0' prio=150 status=undef
> 
> | `- 2:0:0:9  sdak 66:64   active ready running
> 
> |-+- policy='round-robin 0' prio=110 status=undef
> 
> | `- 1:0:0:9  sdl  8:176   active ready running
> 
> |-+- policy='round-robin 0' prio=50 status=undef
> 
> | |- 3:0:0:9  sdcf 69:48   active ready running
> 
> | `- 4:0:0:9  sdcy 70:96   active ready running
> 
> `-+- policy='round-robin 0' prio=10 status=undef
> 
>   |- 5:0:0:9  sdcb 68:240  active ready running
> 
>   `- 6:0:0:9  sdct 70:16   active ready running
> 
>  
> 
> Exactly what we needed: primary FC link with 150, secondary 110, and
> then follow primary and secondary ISCSI links with 50 and 10
> respectively.
> 
> All in all this one seems to have solved our problem, and well maybe
> can help anybody elses too.
> 
Actually, I like the idea with the stackable prioritizers.
Not sure about the 'sg_id' thing; that's still too much to configure.
We should be identifying the transport, and base some priorities
based on the transport.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

multipath-tools: scsi_id based path priorities and multiple prioritizers

[Viktor Larionov]

[-- Attachment #1.1.1: Type: text/plain, Size: 5903 bytes --]

Hi everybody!

 

First of all, thanks for all the hard work you guys have been doing
developing dm. It's an amazing piece of work you have done!

While working with dm-multipath we have bumped into some limitations which
we felt bit uncomfortable with, and seems like managed to change. I'd
thought I share the experience on that with others, in hope that this would
help somebody.

 

Long story short - our servers are connected to our SAN with both fc and
iscsi links. (same targets, same wwid's are exported both through fc and
iscsi)

Pretty much a standard installation - two independent controllers on the
storage side (fc and iscsi each), dual port fc controllers on the server
side + iscsi.

All this leaves us with approximate of 6 paths per device. (2 fc, and 4
iscsi - 1 fc, and 2 iscsi per storage controller)

 

Now if we use ALUA, which is standard for our infra (IBM Storewize V3700),
the picture looks pretty much like this:

 

alessandra viktor.larionov # multipath -ll www-2-mysql

www-2-mysql (360050763008080581000000000000029) dm-37 IBM,2145

size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| |- 2:0:0:9  sdak 66:64   active ready running

| |- 3:0:0:9  sdcf 69:48   active ready running

| `- 4:0:0:9  sdcy 70:96   active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

  |- 1:0:0:9  sdl  8:176   active ready running

  |- 5:0:0:9  sdcb 68:240  active ready running

  `- 6:0:0:9  sdct 70:16   active ready running

 

Where sdak and sdl are fiber links and the rest of those are iscsi.
Priorities come from alua which correspond to san controller preference at
this particular moment.

What we don't like about this setup is that fc and iscsi links end up with
the same prioriy in the same group. The idea behind having iscsi links on
machines having fc at all, is redundancy to fc failures.

But we surely don't want to operate iscsi links the times when either
primary or backup fc are fully operational.

 

So this led us to the idea, of somehow telling the prioritizer to be more
granular and separate fc and iscsi controller priorities. After doing some
several hour googling, I found out that we are not the only ones with such a
story, and there has been no solution to the point. (take this one for
example http://www.redhat.com/archives/dm-devel/2008-August/msg00083.html)
In fact prio_callout which could possibly solve this kind of thing, is
deprecated.

 

It's true that there's no easy or trivial way to determine if a path behing
an sg is fiber or iscsi (or something else). But thinking on this issue, we
thought that we actully can satisfy if we could just assign a custom
priority based on a scsi_id of the device. The idea behind it is simple -
say in our case we have an IBM ServeRAID controller, which is SCSI host 0,
Emulex Light Pulse which is SCSI host 1 and 2 (for each port respectively
and all of the rest is iSCSI. So if we could give static priorities based on
this information this could do the trick.

 

So, we poked up with code a bit, and wrote up a custom prioritizer, called
sg_id. (patch for the latest multipath-tools available here:
http://viktor.ee/multipath-tools-patches/sg_id_prio.patch)

Usage is very simple: in /etc/multipath.conf: prio "sg_id", and priorities
are passed through prio_args as regexes: e.g. a prio_args of

prio_sg_id(default)=0 prio_sg_id(^[0-2]:0)=40 prio_sg_id(^5:[2-3]:)=30

will give prio 40 for everything on SCSI hosts 0, 1 and 2, channel 0. 30 on
scsi_host 5 channels 2 and 3, and everything else will get 0.

 

Using sg_id in the upper example we will have sdl and sdak in the first
group, and all othe other stuff in the second. Which is ok, but not quite.

The problem with this approach for us is that ALUA gives us valuable
information on our storage priorities (which controller is primary and which
is secondary for that particular lun at this particular moment), and we're
not quite ready to sacrifice this information even for sg_id prios. If there
only would be a way to use multiple prioritizers.

And so we've played another couple of our hours with multipath-tools code
allowing it to accept multiple prioritizers in prio configuration. (patch
here http://viktor.ee/multipath-tools-patches/multiprio.patch) 

In this case, prioritizers should be separated by coma, semicolon or space,
and the end priority would be a sum of priorities given by all of the
specified prioritizers. (a single prioritizer value is also accepted of
course.)

As an example:

        prio                  "sg_id, alua"

        prio_args             "prio_sg_id(default)=0
prio_sg_id(^[0-2]:0)=100"

 

So combining the two of above with the same example we get:

 

alessandra multipath-tools-0.4.9 # multipath -r www-2-mysql

reload: www-2-mysql (360050763008080581000000000000029) undef IBM,2145

size=10G features='1 queue_if_no_path' hwhandler='0' wp=undef

|-+- policy='round-robin 0' prio=150 status=undef

| `- 2:0:0:9  sdak 66:64   active ready running

|-+- policy='round-robin 0' prio=110 status=undef

| `- 1:0:0:9  sdl  8:176   active ready running

|-+- policy='round-robin 0' prio=50 status=undef

| |- 3:0:0:9  sdcf 69:48   active ready running

| `- 4:0:0:9  sdcy 70:96   active ready running

`-+- policy='round-robin 0' prio=10 status=undef

  |- 5:0:0:9  sdcb 68:240  active ready running

  `- 6:0:0:9  sdct 70:16   active ready running

 

Exactly what we needed: primary FC link with 150, secondary 110, and then
follow primary and secondary ISCSI links with 50 and 10 respectively.

All in all this one seems to have solved our problem, and well maybe can
help anybody elses too.

 

All comments are kindly welcome!

 

Cheers,

Viktor

Viktor Larionov
IT osakonna juhataja
IT-osakond
Salva Kindlustuse AS
Tel: (+372) 683 0630 | GSM: (+372) 566 86811 | Viktor.Larionov@salva.ee | www.salva.ee
(SMX)338844

[-- Attachment #1.1.2: Type: text/html, Size: 11219 bytes --]

[-- Attachment #1.2: kqIy6NcXsxCukGN.gif --]
[-- Type: image/gif, Size: 6207 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]