* configure MD3000
@ 2009-03-12 16:53 Thomas Witzel
2009-03-12 18:33 ` Chandra Seetharaman
2009-03-12 19:07 ` Thomas Witzel
0 siblings, 2 replies; 10+ messages in thread
From: Thomas Witzel @ 2009-03-12 16:53 UTC (permalink / raw)
To: dm-devel
Hello all,
I have a Dell MD3000 attached to an NX1950. It seems the MD3000 is
connected through a SAS 5/E card and uses the mptsas driver. My kernel
version is 2.6.27.
multipath -ll gives me:
sdj: checker msg is "directio checker reports path is down"
360022190009773680000214a495047cedm-5 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# sdd 8:48 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ #:#:#:# sdj 8:144 [failed][faulty]
360022190009773680000000000000000dm-0 ,
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
\_ #:#:#:# sdg 8:96 [failed][ready]
sde: checker msg is "directio checker reports path is down"
3600221900096f27d00000ab449504839dm-4 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# sdk 8:160 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ #:#:#:# sde 8:64 [failed][faulty]
sdc: checker msg is "directio checker reports path is down"
3600221900096f27d00000ab3495047fddm-2 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# sdi 8:128 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ #:#:#:# sdc 8:32 [failed][faulty]
sdh: checker msg is "directio checker reports path is down"
360022190009773680000214749504780dm-6 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# sdb 8:16 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ #:#:#:# sdh 8:112 [failed][faulty]
sdl: checker msg is "directio checker reports path is down"
360022190009773680000214c49504806dm-3 ,
[size=1.1T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][active]
\_ #:#:#:# sdf 8:80 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ #:#:#:# sdl 8:176 [failed][faulty]
3600221900096f27d0000000000000000dm-1 ,
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][enabled]
\_ #:#:#:# sdm 8:192 [failed][ready]
How do I configure this to work correctly ? I'm sorry if that is a
redundant question, but I googled for about 8 hours now and couldn't
find anything conclusive. Also is there a manual/tutorial for this
somewhere ?
Thank you,
Thomas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: configure MD3000
2009-03-12 16:53 configure MD3000 Thomas Witzel
@ 2009-03-12 18:33 ` Chandra Seetharaman
2009-03-12 19:07 ` Thomas Witzel
1 sibling, 0 replies; 10+ messages in thread
From: Chandra Seetharaman @ 2009-03-12 18:33 UTC (permalink / raw)
To: device-mapper development
To my knowledge DELL DB3000 is a rdac based storage. Hence, you need to
add a special device stanza to /etc/multipath.conf file.
--------
devices {
device {
vendor "DELL"
product "MD3000"
hardware_handler "1 rdac"
path_checker rdac
failback immediate
path_grouping_policy group_by_prio
no_path_retry queue
prio_callout "/sbin/mpath_prio_tpc /dev/%n"
}
}
---------
Make the change, run "multipath -F" followed by "multipath" for the
change to affect your configuration.
What is your distro (RHEL or SLES) and what is the version ?
For SLES you need to replace the prio_callout line by
prio rdac
chandra
On Thu, 2009-03-12 at 12:53 -0400, Thomas Witzel wrote:
> Hello all,
>
> I have a Dell MD3000 attached to an NX1950. It seems the MD3000 is
> connected through a SAS 5/E card and uses the mptsas driver. My kernel
> version is 2.6.27.
> multipath -ll gives me:
>
> sdj: checker msg is "directio checker reports path is down"
> 360022190009773680000214a495047cedm-5 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][active]
> \_ #:#:#:# sdd 8:48 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ #:#:#:# sdj 8:144 [failed][faulty]
> 360022190009773680000000000000000dm-0 ,
> [size=20M][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][enabled]
> \_ #:#:#:# sdg 8:96 [failed][ready]
> sde: checker msg is "directio checker reports path is down"
> 3600221900096f27d00000ab449504839dm-4 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][active]
> \_ #:#:#:# sdk 8:160 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ #:#:#:# sde 8:64 [failed][faulty]
> sdc: checker msg is "directio checker reports path is down"
> 3600221900096f27d00000ab3495047fddm-2 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][active]
> \_ #:#:#:# sdi 8:128 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ #:#:#:# sdc 8:32 [failed][faulty]
> sdh: checker msg is "directio checker reports path is down"
> 360022190009773680000214749504780dm-6 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][active]
> \_ #:#:#:# sdb 8:16 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ #:#:#:# sdh 8:112 [failed][faulty]
> sdl: checker msg is "directio checker reports path is down"
> 360022190009773680000214c49504806dm-3 ,
> [size=1.1T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][active]
> \_ #:#:#:# sdf 8:80 [active][ready]
> \_ round-robin 0 [prio=0][enabled]
> \_ #:#:#:# sdl 8:176 [failed][faulty]
> 3600221900096f27d0000000000000000dm-1 ,
> [size=20M][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][enabled]
> \_ #:#:#:# sdm 8:192 [failed][ready]
>
>
> How do I configure this to work correctly ? I'm sorry if that is a
> redundant question, but I googled for about 8 hours now and couldn't
> find anything conclusive. Also is there a manual/tutorial for this
> somewhere ?
>
> Thank you,
> Thomas
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: configure MD3000
2009-03-12 16:53 configure MD3000 Thomas Witzel
2009-03-12 18:33 ` Chandra Seetharaman
@ 2009-03-12 19:07 ` Thomas Witzel
2009-03-12 19:18 ` Chandra Seetharaman
1 sibling, 1 reply; 10+ messages in thread
From: Thomas Witzel @ 2009-03-12 19:07 UTC (permalink / raw)
To: dm-devel
Thanks Chandra,
sorry for the out of order reply, still had the digest mode on. I
setup the config similar to what you said
devices {
device {
vendor "DELL"
product "MD3000"
hardware_handler "1 rdac"
path_checker rdac
path_grouping_policy group_by_prio
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
failback immediate
getuid_callout "/sbin/scsi_id -g -u /block/%n"
}
}
and I'm still getting:
sdh: checker msg is "directio checker reports path is down"
create: 360022190009773680000214749504780 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdb 8:16 [undef][ready]
\_ round-robin 0 [prio=0][undef]
\_ #:#:#:# sdh 8:112 [undef][faulty]
sdc: checker msg is "directio checker reports path is down"
create: 3600221900096f27d00000ab3495047fd ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][undef]
\_ #:#:#:# sdc 8:32 [undef][faulty]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdi 8:128 [undef][ready]
sdj: checker msg is "directio checker reports path is down"
create: 360022190009773680000214a495047ce ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdd 8:48 [undef][ready]
\_ round-robin 0 [prio=0][undef]
\_ #:#:#:# sdj 8:144 [undef][faulty]
sde: checker msg is "directio checker reports path is down"
create: 3600221900096f27d00000ab449504839 ,
[size=2.0T][features=0][hwhandler=0]
\_ round-robin 0 [prio=0][undef]
\_ #:#:#:# sde 8:64 [undef][faulty]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdk 8:160 [undef][ready]
sdl: checker msg is "directio checker reports path is down"
create: 360022190009773680000214c49504806 ,
[size=1.1T][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdf 8:80 [undef][ready]
\_ round-robin 0 [prio=0][undef]
\_ #:#:#:# sdl 8:176 [undef][faulty]
create: 360022190009773680000000000000000 ,
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdg 8:96 [undef][ready]
create: 3600221900096f27d0000000000000000 ,
[size=20M][features=0][hwhandler=0]
\_ round-robin 0 [prio=1][undef]
\_ #:#:#:# sdm 8:192 [undef][ready]
libdevmapper: libdm-common.c(312): Created
/dev/mapper/360022190009773680000000000000000
libdevmapper: libdm-common.c(312): Created
/dev/mapper/3600221900096f27d0000000000000000
Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
necessary I would consider switching the distribution, but right now
I'd prefer to stick with what all the other machines have.
Thanks, Thomas
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:07 ` Thomas Witzel
@ 2009-03-12 19:18 ` Chandra Seetharaman
2009-03-12 19:25 ` Thomas Witzel
0 siblings, 1 reply; 10+ messages in thread
From: Chandra Seetharaman @ 2009-03-12 19:18 UTC (permalink / raw)
To: device-mapper development
No, you don't have to change the distro.
Do you see the disks ? (the #:#:#:# makes me think that the underlying
block devices are not configured properly). IOW, do you the sd devices
under /sys/block ? Are you able to read/write from/to the devices ?
chandra
On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
> Thanks Chandra,
>
> sorry for the out of order reply, still had the digest mode on. I
> setup the config similar to what you said
>
> devices {
> device {
> vendor "DELL"
> product "MD3000"
> hardware_handler "1 rdac"
> path_checker rdac
> path_grouping_policy group_by_prio
> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
> failback immediate
> getuid_callout "/sbin/scsi_id -g -u /block/%n"
> }
> }
>
>
> and I'm still getting:
> sdh: checker msg is "directio checker reports path is down"
> create: 360022190009773680000214749504780 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdb 8:16 [undef][ready]
> \_ round-robin 0 [prio=0][undef]
> \_ #:#:#:# sdh 8:112 [undef][faulty]
> sdc: checker msg is "directio checker reports path is down"
> create: 3600221900096f27d00000ab3495047fd ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][undef]
> \_ #:#:#:# sdc 8:32 [undef][faulty]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdi 8:128 [undef][ready]
> sdj: checker msg is "directio checker reports path is down"
> create: 360022190009773680000214a495047ce ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdd 8:48 [undef][ready]
> \_ round-robin 0 [prio=0][undef]
> \_ #:#:#:# sdj 8:144 [undef][faulty]
> sde: checker msg is "directio checker reports path is down"
> create: 3600221900096f27d00000ab449504839 ,
> [size=2.0T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=0][undef]
> \_ #:#:#:# sde 8:64 [undef][faulty]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdk 8:160 [undef][ready]
> sdl: checker msg is "directio checker reports path is down"
> create: 360022190009773680000214c49504806 ,
> [size=1.1T][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdf 8:80 [undef][ready]
> \_ round-robin 0 [prio=0][undef]
> \_ #:#:#:# sdl 8:176 [undef][faulty]
> create: 360022190009773680000000000000000 ,
> [size=20M][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdg 8:96 [undef][ready]
> create: 3600221900096f27d0000000000000000 ,
> [size=20M][features=0][hwhandler=0]
> \_ round-robin 0 [prio=1][undef]
> \_ #:#:#:# sdm 8:192 [undef][ready]
> libdevmapper: libdm-common.c(312): Created
> /dev/mapper/360022190009773680000000000000000
> libdevmapper: libdm-common.c(312): Created
> /dev/mapper/3600221900096f27d0000000000000000
>
> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
> necessary I would consider switching the distribution, but right now
> I'd prefer to stick with what all the other machines have.
>
> Thanks, Thomas
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:18 ` Chandra Seetharaman
@ 2009-03-12 19:25 ` Thomas Witzel
2009-03-12 19:31 ` Chandra Seetharaman
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Witzel @ 2009-03-12 19:25 UTC (permalink / raw)
To: sekharan, device-mapper development
My /sys/block looks like this:
ls /sys/block/
dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
sdc sdf sdi sdl
dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
sdd sdg sdj sdm
dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
sde sdh sdk sr0
My /proc/partitions looks like this:
major minor #blocks name
8 0 71041024 sda
8 1 68099503 sda1
8 2 1 sda2
8 5 2939863 sda5
8 16 2134896640 sdb
8 17 2134893883 sdb1
8 32 2134896640 sdc
8 48 2134896640 sdd
8 49 2134893883 sdd1
8 64 2134896640 sde
8 80 1222794240 sdf
8 96 20480 sdg
8 112 2134896640 sdh
8 128 2134896640 sdi
8 129 2134893883 sdi1
8 144 2134896640 sdj
8 160 2134896640 sdk
8 161 2134893883 sdk1
8 176 1222794240 sdl
8 192 20480 sdm
254 0 2134896640 dm-0
254 1 2134896640 dm-1
254 2 2134896640 dm-2
254 3 2134896640 dm-3
254 4 1222794240 dm-4
254 5 20480 dm-5
254 6 20480 dm-6
All the sd's with a partition are working fine, I even created a test
file system at some point. I did load scsi_dh_rdac for testing at some
point, and that put the following into the syslog:
[10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
[10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
[10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
[10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
[10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
[10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
[10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
[10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
[10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
[10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
[10170.096088] rdac: device handler registered
Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
Buffer I/O error on device dm-5, logical block 5
[ 8996.087539] Buffer I/O error on device dm-5, logical block 6
[ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
[ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
[ 8996.087601] Buffer I/O error on device dm-5, logical block 12
[ 8996.087606] Buffer I/O error on device dm-5, logical block 13
[ 8996.087609] Buffer I/O error on device dm-5, logical block 14
[ 8996.087612] Buffer I/O error on device dm-5, logical block 15
[ 8996.087615] Buffer I/O error on device dm-5, logical block 16
[ 8996.087618] Buffer I/O error on device dm-5, logical block 17
type messages.
On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
<sekharan@us.ibm.com> wrote:
> No, you don't have to change the distro.
>
> Do you see the disks ? (the #:#:#:# makes me think that the underlying
> block devices are not configured properly). IOW, do you the sd devices
> under /sys/block ? Are you able to read/write from/to the devices ?
>
> chandra
>
> On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
>> Thanks Chandra,
>>
>> sorry for the out of order reply, still had the digest mode on. I
>> setup the config similar to what you said
>>
>> devices {
>> device {
>> vendor "DELL"
>> product "MD3000"
>> hardware_handler "1 rdac"
>> path_checker rdac
>> path_grouping_policy group_by_prio
>> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
>> failback immediate
>> getuid_callout "/sbin/scsi_id -g -u /block/%n"
>> }
>> }
>>
>>
>> and I'm still getting:
>> sdh: checker msg is "directio checker reports path is down"
>> create: 360022190009773680000214749504780 ,
>> [size=2.0T][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdb 8:16 [undef][ready]
>> \_ round-robin 0 [prio=0][undef]
>> \_ #:#:#:# sdh 8:112 [undef][faulty]
>> sdc: checker msg is "directio checker reports path is down"
>> create: 3600221900096f27d00000ab3495047fd ,
>> [size=2.0T][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=0][undef]
>> \_ #:#:#:# sdc 8:32 [undef][faulty]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdi 8:128 [undef][ready]
>> sdj: checker msg is "directio checker reports path is down"
>> create: 360022190009773680000214a495047ce ,
>> [size=2.0T][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdd 8:48 [undef][ready]
>> \_ round-robin 0 [prio=0][undef]
>> \_ #:#:#:# sdj 8:144 [undef][faulty]
>> sde: checker msg is "directio checker reports path is down"
>> create: 3600221900096f27d00000ab449504839 ,
>> [size=2.0T][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=0][undef]
>> \_ #:#:#:# sde 8:64 [undef][faulty]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdk 8:160 [undef][ready]
>> sdl: checker msg is "directio checker reports path is down"
>> create: 360022190009773680000214c49504806 ,
>> [size=1.1T][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdf 8:80 [undef][ready]
>> \_ round-robin 0 [prio=0][undef]
>> \_ #:#:#:# sdl 8:176 [undef][faulty]
>> create: 360022190009773680000000000000000 ,
>> [size=20M][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdg 8:96 [undef][ready]
>> create: 3600221900096f27d0000000000000000 ,
>> [size=20M][features=0][hwhandler=0]
>> \_ round-robin 0 [prio=1][undef]
>> \_ #:#:#:# sdm 8:192 [undef][ready]
>> libdevmapper: libdm-common.c(312): Created
>> /dev/mapper/360022190009773680000000000000000
>> libdevmapper: libdm-common.c(312): Created
>> /dev/mapper/3600221900096f27d0000000000000000
>>
>> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
>> necessary I would consider switching the distribution, but right now
>> I'd prefer to stick with what all the other machines have.
>>
>> Thanks, Thomas
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:25 ` Thomas Witzel
@ 2009-03-12 19:31 ` Chandra Seetharaman
2009-03-12 19:38 ` Thomas Witzel
0 siblings, 1 reply; 10+ messages in thread
From: Chandra Seetharaman @ 2009-03-12 19:31 UTC (permalink / raw)
To: device-mapper development
Are you able to read/write from/to the scsi device ?
If yes, Did you do "multipath -F" followed by "multipath".
Is the multipathd running ?
chandra
On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote:
> My /sys/block looks like this:
> ls /sys/block/
> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
> sdc sdf sdi sdl
> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
> sdd sdg sdj sdm
> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
> sde sdh sdk sr0
>
> My /proc/partitions looks like this:
>
> major minor #blocks name
>
> 8 0 71041024 sda
> 8 1 68099503 sda1
> 8 2 1 sda2
> 8 5 2939863 sda5
> 8 16 2134896640 sdb
> 8 17 2134893883 sdb1
> 8 32 2134896640 sdc
> 8 48 2134896640 sdd
> 8 49 2134893883 sdd1
> 8 64 2134896640 sde
> 8 80 1222794240 sdf
> 8 96 20480 sdg
> 8 112 2134896640 sdh
> 8 128 2134896640 sdi
> 8 129 2134893883 sdi1
> 8 144 2134896640 sdj
> 8 160 2134896640 sdk
> 8 161 2134893883 sdk1
> 8 176 1222794240 sdl
> 8 192 20480 sdm
> 254 0 2134896640 dm-0
> 254 1 2134896640 dm-1
> 254 2 2134896640 dm-2
> 254 3 2134896640 dm-3
> 254 4 1222794240 dm-4
> 254 5 20480 dm-5
> 254 6 20480 dm-6
>
> All the sd's with a partition are working fine, I even created a test
> file system at some point. I did load scsi_dh_rdac for testing at some
> point, and that put the following into the syslog:
> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
> [10170.096088] rdac: device handler registered
>
> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
> Buffer I/O error on device dm-5, logical block 5
> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6
> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12
> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13
> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14
> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15
> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16
> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17
>
> type messages.
>
> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
> <sekharan@us.ibm.com> wrote:
> > No, you don't have to change the distro.
> >
> > Do you see the disks ? (the #:#:#:# makes me think that the underlying
> > block devices are not configured properly). IOW, do you the sd devices
> > under /sys/block ? Are you able to read/write from/to the devices ?
> >
> > chandra
> >
> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
> >> Thanks Chandra,
> >>
> >> sorry for the out of order reply, still had the digest mode on. I
> >> setup the config similar to what you said
> >>
> >> devices {
> >> device {
> >> vendor "DELL"
> >> product "MD3000"
> >> hardware_handler "1 rdac"
> >> path_checker rdac
> >> path_grouping_policy group_by_prio
> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
> >> failback immediate
> >> getuid_callout "/sbin/scsi_id -g -u /block/%n"
> >> }
> >> }
> >>
> >>
> >> and I'm still getting:
> >> sdh: checker msg is "directio checker reports path is down"
> >> create: 360022190009773680000214749504780 ,
> >> [size=2.0T][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdb 8:16 [undef][ready]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ #:#:#:# sdh 8:112 [undef][faulty]
> >> sdc: checker msg is "directio checker reports path is down"
> >> create: 3600221900096f27d00000ab3495047fd ,
> >> [size=2.0T][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ #:#:#:# sdc 8:32 [undef][faulty]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdi 8:128 [undef][ready]
> >> sdj: checker msg is "directio checker reports path is down"
> >> create: 360022190009773680000214a495047ce ,
> >> [size=2.0T][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdd 8:48 [undef][ready]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ #:#:#:# sdj 8:144 [undef][faulty]
> >> sde: checker msg is "directio checker reports path is down"
> >> create: 3600221900096f27d00000ab449504839 ,
> >> [size=2.0T][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ #:#:#:# sde 8:64 [undef][faulty]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdk 8:160 [undef][ready]
> >> sdl: checker msg is "directio checker reports path is down"
> >> create: 360022190009773680000214c49504806 ,
> >> [size=1.1T][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdf 8:80 [undef][ready]
> >> \_ round-robin 0 [prio=0][undef]
> >> \_ #:#:#:# sdl 8:176 [undef][faulty]
> >> create: 360022190009773680000000000000000 ,
> >> [size=20M][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdg 8:96 [undef][ready]
> >> create: 3600221900096f27d0000000000000000 ,
> >> [size=20M][features=0][hwhandler=0]
> >> \_ round-robin 0 [prio=1][undef]
> >> \_ #:#:#:# sdm 8:192 [undef][ready]
> >> libdevmapper: libdm-common.c(312): Created
> >> /dev/mapper/360022190009773680000000000000000
> >> libdevmapper: libdm-common.c(312): Created
> >> /dev/mapper/3600221900096f27d0000000000000000
> >>
> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
> >> necessary I would consider switching the distribution, but right now
> >> I'd prefer to stick with what all the other machines have.
> >>
> >> Thanks, Thomas
> >>
> >> --
> >> dm-devel mailing list
> >> dm-devel@redhat.com
> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:31 ` Chandra Seetharaman
@ 2009-03-12 19:38 ` Thomas Witzel
2009-03-12 19:40 ` Thomas Witzel
2009-03-12 22:06 ` Chandra Seetharaman
0 siblings, 2 replies; 10+ messages in thread
From: Thomas Witzel @ 2009-03-12 19:38 UTC (permalink / raw)
To: sekharan, device-mapper development
How do you want me to test whether I can read/write ? I can no longer
mount the /dev/sd*1 partitions because now they are busy (thats since
I started multipath). Yes, I did execute multipath -F and multipath
repeatedly, the daemon is running.
On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman
<sekharan@us.ibm.com> wrote:
> Are you able to read/write from/to the scsi device ?
>
> If yes, Did you do "multipath -F" followed by "multipath".
>
> Is the multipathd running ?
>
> chandra
> On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote:
>> My /sys/block looks like this:
>> ls /sys/block/
>> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
>> sdc sdf sdi sdl
>> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
>> sdd sdg sdj sdm
>> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
>> sde sdh sdk sr0
>>
>> My /proc/partitions looks like this:
>>
>> major minor #blocks name
>>
>> 8 0 71041024 sda
>> 8 1 68099503 sda1
>> 8 2 1 sda2
>> 8 5 2939863 sda5
>> 8 16 2134896640 sdb
>> 8 17 2134893883 sdb1
>> 8 32 2134896640 sdc
>> 8 48 2134896640 sdd
>> 8 49 2134893883 sdd1
>> 8 64 2134896640 sde
>> 8 80 1222794240 sdf
>> 8 96 20480 sdg
>> 8 112 2134896640 sdh
>> 8 128 2134896640 sdi
>> 8 129 2134893883 sdi1
>> 8 144 2134896640 sdj
>> 8 160 2134896640 sdk
>> 8 161 2134893883 sdk1
>> 8 176 1222794240 sdl
>> 8 192 20480 sdm
>> 254 0 2134896640 dm-0
>> 254 1 2134896640 dm-1
>> 254 2 2134896640 dm-2
>> 254 3 2134896640 dm-3
>> 254 4 1222794240 dm-4
>> 254 5 20480 dm-5
>> 254 6 20480 dm-6
>>
>> All the sd's with a partition are working fine, I even created a test
>> file system at some point. I did load scsi_dh_rdac for testing at some
>> point, and that put the following into the syslog:
>> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
>> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
>> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
>> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
>> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
>> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
>> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
>> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
>> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
>> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
>> [10170.096088] rdac: device handler registered
>>
>> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
>> Buffer I/O error on device dm-5, logical block 5
>> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6
>> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
>> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
>> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12
>> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13
>> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14
>> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15
>> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16
>> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17
>>
>> type messages.
>>
>> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
>> <sekharan@us.ibm.com> wrote:
>> > No, you don't have to change the distro.
>> >
>> > Do you see the disks ? (the #:#:#:# makes me think that the underlying
>> > block devices are not configured properly). IOW, do you the sd devices
>> > under /sys/block ? Are you able to read/write from/to the devices ?
>> >
>> > chandra
>> >
>> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
>> >> Thanks Chandra,
>> >>
>> >> sorry for the out of order reply, still had the digest mode on. I
>> >> setup the config similar to what you said
>> >>
>> >> devices {
>> >> device {
>> >> vendor "DELL"
>> >> product "MD3000"
>> >> hardware_handler "1 rdac"
>> >> path_checker rdac
>> >> path_grouping_policy group_by_prio
>> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
>> >> failback immediate
>> >> getuid_callout "/sbin/scsi_id -g -u /block/%n"
>> >> }
>> >> }
>> >>
>> >>
>> >> and I'm still getting:
>> >> sdh: checker msg is "directio checker reports path is down"
>> >> create: 360022190009773680000214749504780 ,
>> >> [size=2.0T][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdb 8:16 [undef][ready]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ #:#:#:# sdh 8:112 [undef][faulty]
>> >> sdc: checker msg is "directio checker reports path is down"
>> >> create: 3600221900096f27d00000ab3495047fd ,
>> >> [size=2.0T][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ #:#:#:# sdc 8:32 [undef][faulty]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdi 8:128 [undef][ready]
>> >> sdj: checker msg is "directio checker reports path is down"
>> >> create: 360022190009773680000214a495047ce ,
>> >> [size=2.0T][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdd 8:48 [undef][ready]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ #:#:#:# sdj 8:144 [undef][faulty]
>> >> sde: checker msg is "directio checker reports path is down"
>> >> create: 3600221900096f27d00000ab449504839 ,
>> >> [size=2.0T][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ #:#:#:# sde 8:64 [undef][faulty]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdk 8:160 [undef][ready]
>> >> sdl: checker msg is "directio checker reports path is down"
>> >> create: 360022190009773680000214c49504806 ,
>> >> [size=1.1T][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdf 8:80 [undef][ready]
>> >> \_ round-robin 0 [prio=0][undef]
>> >> \_ #:#:#:# sdl 8:176 [undef][faulty]
>> >> create: 360022190009773680000000000000000 ,
>> >> [size=20M][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdg 8:96 [undef][ready]
>> >> create: 3600221900096f27d0000000000000000 ,
>> >> [size=20M][features=0][hwhandler=0]
>> >> \_ round-robin 0 [prio=1][undef]
>> >> \_ #:#:#:# sdm 8:192 [undef][ready]
>> >> libdevmapper: libdm-common.c(312): Created
>> >> /dev/mapper/360022190009773680000000000000000
>> >> libdevmapper: libdm-common.c(312): Created
>> >> /dev/mapper/3600221900096f27d0000000000000000
>> >>
>> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
>> >> necessary I would consider switching the distribution, but right now
>> >> I'd prefer to stick with what all the other machines have.
>> >>
>> >> Thanks, Thomas
>> >>
>> >> --
>> >> dm-devel mailing list
>> >> dm-devel@redhat.com
>> >> https://www.redhat.com/mailman/listinfo/dm-devel
>> >
>> > --
>> > dm-devel mailing list
>> > dm-devel@redhat.com
>> > https://www.redhat.com/mailman/listinfo/dm-devel
>> >
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:38 ` Thomas Witzel
@ 2009-03-12 19:40 ` Thomas Witzel
2009-03-12 22:07 ` Chandra Seetharaman
2009-03-12 22:06 ` Chandra Seetharaman
1 sibling, 1 reply; 10+ messages in thread
From: Thomas Witzel @ 2009-03-12 19:40 UTC (permalink / raw)
To: sekharan, device-mapper development
What sort of bothers me, is that although we specified rdac, it says
hwhandler=0 while on the web there seems to be output examples where
it does say rdac in the multipath output.
Also googling convinced me that some people have
drivers/md/md-multipath-rdac.c or something like that in their kernel,
while my ubuntu kernel and the vanilla git kernel do not have this. I
have the feeling this is an important component of this.
On Thu, Mar 12, 2009 at 3:38 PM, Thomas Witzel <witzel.thomas@gmail.com> wrote:
> How do you want me to test whether I can read/write ? I can no longer
> mount the /dev/sd*1 partitions because now they are busy (thats since
> I started multipath). Yes, I did execute multipath -F and multipath
> repeatedly, the daemon is running.
>
> On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman
> <sekharan@us.ibm.com> wrote:
>> Are you able to read/write from/to the scsi device ?
>>
>> If yes, Did you do "multipath -F" followed by "multipath".
>>
>> Is the multipathd running ?
>>
>> chandra
>> On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote:
>>> My /sys/block looks like this:
>>> ls /sys/block/
>>> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
>>> sdc sdf sdi sdl
>>> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
>>> sdd sdg sdj sdm
>>> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
>>> sde sdh sdk sr0
>>>
>>> My /proc/partitions looks like this:
>>>
>>> major minor #blocks name
>>>
>>> 8 0 71041024 sda
>>> 8 1 68099503 sda1
>>> 8 2 1 sda2
>>> 8 5 2939863 sda5
>>> 8 16 2134896640 sdb
>>> 8 17 2134893883 sdb1
>>> 8 32 2134896640 sdc
>>> 8 48 2134896640 sdd
>>> 8 49 2134893883 sdd1
>>> 8 64 2134896640 sde
>>> 8 80 1222794240 sdf
>>> 8 96 20480 sdg
>>> 8 112 2134896640 sdh
>>> 8 128 2134896640 sdi
>>> 8 129 2134893883 sdi1
>>> 8 144 2134896640 sdj
>>> 8 160 2134896640 sdk
>>> 8 161 2134893883 sdk1
>>> 8 176 1222794240 sdl
>>> 8 192 20480 sdm
>>> 254 0 2134896640 dm-0
>>> 254 1 2134896640 dm-1
>>> 254 2 2134896640 dm-2
>>> 254 3 2134896640 dm-3
>>> 254 4 1222794240 dm-4
>>> 254 5 20480 dm-5
>>> 254 6 20480 dm-6
>>>
>>> All the sd's with a partition are working fine, I even created a test
>>> file system at some point. I did load scsi_dh_rdac for testing at some
>>> point, and that put the following into the syslog:
>>> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
>>> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
>>> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
>>> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
>>> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
>>> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
>>> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
>>> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
>>> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
>>> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
>>> [10170.096088] rdac: device handler registered
>>>
>>> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
>>> Buffer I/O error on device dm-5, logical block 5
>>> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6
>>> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
>>> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
>>> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12
>>> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13
>>> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14
>>> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15
>>> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16
>>> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17
>>>
>>> type messages.
>>>
>>> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
>>> <sekharan@us.ibm.com> wrote:
>>> > No, you don't have to change the distro.
>>> >
>>> > Do you see the disks ? (the #:#:#:# makes me think that the underlying
>>> > block devices are not configured properly). IOW, do you the sd devices
>>> > under /sys/block ? Are you able to read/write from/to the devices ?
>>> >
>>> > chandra
>>> >
>>> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
>>> >> Thanks Chandra,
>>> >>
>>> >> sorry for the out of order reply, still had the digest mode on. I
>>> >> setup the config similar to what you said
>>> >>
>>> >> devices {
>>> >> device {
>>> >> vendor "DELL"
>>> >> product "MD3000"
>>> >> hardware_handler "1 rdac"
>>> >> path_checker rdac
>>> >> path_grouping_policy group_by_prio
>>> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
>>> >> failback immediate
>>> >> getuid_callout "/sbin/scsi_id -g -u /block/%n"
>>> >> }
>>> >> }
>>> >>
>>> >>
>>> >> and I'm still getting:
>>> >> sdh: checker msg is "directio checker reports path is down"
>>> >> create: 360022190009773680000214749504780 ,
>>> >> [size=2.0T][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdb 8:16 [undef][ready]
>>> >> \_ round-robin 0 [prio=0][undef]
>>> >> \_ #:#:#:# sdh 8:112 [undef][faulty]
>>> >> sdc: checker msg is "directio checker reports path is down"
>>> >> create: 3600221900096f27d00000ab3495047fd ,
>>> >> [size=2.0T][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=0][undef]
>>> >> \_ #:#:#:# sdc 8:32 [undef][faulty]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdi 8:128 [undef][ready]
>>> >> sdj: checker msg is "directio checker reports path is down"
>>> >> create: 360022190009773680000214a495047ce ,
>>> >> [size=2.0T][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdd 8:48 [undef][ready]
>>> >> \_ round-robin 0 [prio=0][undef]
>>> >> \_ #:#:#:# sdj 8:144 [undef][faulty]
>>> >> sde: checker msg is "directio checker reports path is down"
>>> >> create: 3600221900096f27d00000ab449504839 ,
>>> >> [size=2.0T][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=0][undef]
>>> >> \_ #:#:#:# sde 8:64 [undef][faulty]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdk 8:160 [undef][ready]
>>> >> sdl: checker msg is "directio checker reports path is down"
>>> >> create: 360022190009773680000214c49504806 ,
>>> >> [size=1.1T][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdf 8:80 [undef][ready]
>>> >> \_ round-robin 0 [prio=0][undef]
>>> >> \_ #:#:#:# sdl 8:176 [undef][faulty]
>>> >> create: 360022190009773680000000000000000 ,
>>> >> [size=20M][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdg 8:96 [undef][ready]
>>> >> create: 3600221900096f27d0000000000000000 ,
>>> >> [size=20M][features=0][hwhandler=0]
>>> >> \_ round-robin 0 [prio=1][undef]
>>> >> \_ #:#:#:# sdm 8:192 [undef][ready]
>>> >> libdevmapper: libdm-common.c(312): Created
>>> >> /dev/mapper/360022190009773680000000000000000
>>> >> libdevmapper: libdm-common.c(312): Created
>>> >> /dev/mapper/3600221900096f27d0000000000000000
>>> >>
>>> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
>>> >> necessary I would consider switching the distribution, but right now
>>> >> I'd prefer to stick with what all the other machines have.
>>> >>
>>> >> Thanks, Thomas
>>> >>
>>> >> --
>>> >> dm-devel mailing list
>>> >> dm-devel@redhat.com
>>> >> https://www.redhat.com/mailman/listinfo/dm-devel
>>> >
>>> > --
>>> > dm-devel mailing list
>>> > dm-devel@redhat.com
>>> > https://www.redhat.com/mailman/listinfo/dm-devel
>>> >
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:38 ` Thomas Witzel
2009-03-12 19:40 ` Thomas Witzel
@ 2009-03-12 22:06 ` Chandra Seetharaman
1 sibling, 0 replies; 10+ messages in thread
From: Chandra Seetharaman @ 2009-03-12 22:06 UTC (permalink / raw)
To: Thomas Witzel; +Cc: device-mapper development
On Thu, 2009-03-12 at 15:38 -0400, Thomas Witzel wrote:
> How do you want me to test whether I can read/write ? I can no longer
> mount the /dev/sd*1 partitions because now they are busy (thats since
you should be able to do
dd if=/dev/sdb1 of=/dev/null count=1000 iflag=direct
etc.,
> I started multipath). Yes, I did execute multipath -F and multipath
> repeatedly, the daemon is running.
In your multipath -ll output is bizarre:
- As you mentioned (in the other email) there is no rdac hw handler
- it is still using the directio path checker (it should be using
rdac path checker).
which makes me think that your device is not showing "DELL" "MD3000" as
vendor and product.
Can you send the o/p of lsscsi ?
>
> On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman
> <sekharan@us.ibm.com> wrote:
> > Are you able to read/write from/to the scsi device ?
> >
> > If yes, Did you do "multipath -F" followed by "multipath".
> >
> > Is the multipathd running ?
> >
> > chandra
> > On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote:
> >> My /sys/block looks like this:
> >> ls /sys/block/
> >> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
> >> sdc sdf sdi sdl
> >> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
> >> sdd sdg sdj sdm
> >> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
> >> sde sdh sdk sr0
> >>
> >> My /proc/partitions looks like this:
> >>
> >> major minor #blocks name
> >>
> >> 8 0 71041024 sda
> >> 8 1 68099503 sda1
> >> 8 2 1 sda2
> >> 8 5 2939863 sda5
> >> 8 16 2134896640 sdb
> >> 8 17 2134893883 sdb1
> >> 8 32 2134896640 sdc
> >> 8 48 2134896640 sdd
> >> 8 49 2134893883 sdd1
> >> 8 64 2134896640 sde
> >> 8 80 1222794240 sdf
> >> 8 96 20480 sdg
> >> 8 112 2134896640 sdh
> >> 8 128 2134896640 sdi
> >> 8 129 2134893883 sdi1
> >> 8 144 2134896640 sdj
> >> 8 160 2134896640 sdk
> >> 8 161 2134893883 sdk1
> >> 8 176 1222794240 sdl
> >> 8 192 20480 sdm
> >> 254 0 2134896640 dm-0
> >> 254 1 2134896640 dm-1
> >> 254 2 2134896640 dm-2
> >> 254 3 2134896640 dm-3
> >> 254 4 1222794240 dm-4
> >> 254 5 20480 dm-5
> >> 254 6 20480 dm-6
> >>
> >> All the sd's with a partition are working fine, I even created a test
> >> file system at some point. I did load scsi_dh_rdac for testing at some
> >> point, and that put the following into the syslog:
> >> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
> >> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
> >> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
> >> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
> >> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
> >> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
> >> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
> >> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
> >> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
> >> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
> >> [10170.096088] rdac: device handler registered
> >>
> >> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
> >> Buffer I/O error on device dm-5, logical block 5
> >> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6
> >> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
> >> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
> >> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12
> >> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13
> >> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14
> >> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15
> >> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16
> >> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17
> >>
> >> type messages.
> >>
> >> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
> >> <sekharan@us.ibm.com> wrote:
> >> > No, you don't have to change the distro.
> >> >
> >> > Do you see the disks ? (the #:#:#:# makes me think that the underlying
> >> > block devices are not configured properly). IOW, do you the sd devices
> >> > under /sys/block ? Are you able to read/write from/to the devices ?
> >> >
> >> > chandra
> >> >
> >> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
> >> >> Thanks Chandra,
> >> >>
> >> >> sorry for the out of order reply, still had the digest mode on. I
> >> >> setup the config similar to what you said
> >> >>
> >> >> devices {
> >> >> device {
> >> >> vendor "DELL"
> >> >> product "MD3000"
> >> >> hardware_handler "1 rdac"
> >> >> path_checker rdac
> >> >> path_grouping_policy group_by_prio
> >> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
> >> >> failback immediate
> >> >> getuid_callout "/sbin/scsi_id -g -u /block/%n"
> >> >> }
> >> >> }
> >> >>
> >> >>
> >> >> and I'm still getting:
> >> >> sdh: checker msg is "directio checker reports path is down"
> >> >> create: 360022190009773680000214749504780 ,
> >> >> [size=2.0T][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdb 8:16 [undef][ready]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ #:#:#:# sdh 8:112 [undef][faulty]
> >> >> sdc: checker msg is "directio checker reports path is down"
> >> >> create: 3600221900096f27d00000ab3495047fd ,
> >> >> [size=2.0T][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ #:#:#:# sdc 8:32 [undef][faulty]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdi 8:128 [undef][ready]
> >> >> sdj: checker msg is "directio checker reports path is down"
> >> >> create: 360022190009773680000214a495047ce ,
> >> >> [size=2.0T][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdd 8:48 [undef][ready]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ #:#:#:# sdj 8:144 [undef][faulty]
> >> >> sde: checker msg is "directio checker reports path is down"
> >> >> create: 3600221900096f27d00000ab449504839 ,
> >> >> [size=2.0T][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ #:#:#:# sde 8:64 [undef][faulty]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdk 8:160 [undef][ready]
> >> >> sdl: checker msg is "directio checker reports path is down"
> >> >> create: 360022190009773680000214c49504806 ,
> >> >> [size=1.1T][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdf 8:80 [undef][ready]
> >> >> \_ round-robin 0 [prio=0][undef]
> >> >> \_ #:#:#:# sdl 8:176 [undef][faulty]
> >> >> create: 360022190009773680000000000000000 ,
> >> >> [size=20M][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdg 8:96 [undef][ready]
> >> >> create: 3600221900096f27d0000000000000000 ,
> >> >> [size=20M][features=0][hwhandler=0]
> >> >> \_ round-robin 0 [prio=1][undef]
> >> >> \_ #:#:#:# sdm 8:192 [undef][ready]
> >> >> libdevmapper: libdm-common.c(312): Created
> >> >> /dev/mapper/360022190009773680000000000000000
> >> >> libdevmapper: libdm-common.c(312): Created
> >> >> /dev/mapper/3600221900096f27d0000000000000000
> >> >>
> >> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
> >> >> necessary I would consider switching the distribution, but right now
> >> >> I'd prefer to stick with what all the other machines have.
> >> >>
> >> >> Thanks, Thomas
> >> >>
> >> >> --
> >> >> dm-devel mailing list
> >> >> dm-devel@redhat.com
> >> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >> >
> >> > --
> >> > dm-devel mailing list
> >> > dm-devel@redhat.com
> >> > https://www.redhat.com/mailman/listinfo/dm-devel
> >> >
> >>
> >> --
> >> dm-devel mailing list
> >> dm-devel@redhat.com
> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000
2009-03-12 19:40 ` Thomas Witzel
@ 2009-03-12 22:07 ` Chandra Seetharaman
0 siblings, 0 replies; 10+ messages in thread
From: Chandra Seetharaman @ 2009-03-12 22:07 UTC (permalink / raw)
To: Thomas Witzel; +Cc: device-mapper development
On Thu, 2009-03-12 at 15:40 -0400, Thomas Witzel wrote:
> What sort of bothers me, is that although we specified rdac, it says
> hwhandler=0 while on the web there seems to be output examples where
> it does say rdac in the multipath output.
> Also googling convinced me that some people have
> drivers/md/md-multipath-rdac.c or something like that in their kernel,
> while my ubuntu kernel and the vanilla git kernel do not have this. I
> have the feeling this is an important component of this.
From 2.6.27 onwards, rdac has moved to scsi layer as scsi_dh_rdac. You
can do a "lsmod | grep rdac" to see if that module is inserted.
>
> On Thu, Mar 12, 2009 at 3:38 PM, Thomas Witzel <witzel.thomas@gmail.com> wrote:
> > How do you want me to test whether I can read/write ? I can no longer
> > mount the /dev/sd*1 partitions because now they are busy (thats since
> > I started multipath). Yes, I did execute multipath -F and multipath
> > repeatedly, the daemon is running.
> >
> > On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman
> > <sekharan@us.ibm.com> wrote:
> >> Are you able to read/write from/to the scsi device ?
> >>
> >> If yes, Did you do "multipath -F" followed by "multipath".
> >>
> >> Is the multipathd running ?
> >>
> >> chandra
> >> On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote:
> >>> My /sys/block looks like this:
> >>> ls /sys/block/
> >>> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9
> >>> sdc sdf sdi sdl
> >>> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda
> >>> sdd sdg sdj sdm
> >>> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb
> >>> sde sdh sdk sr0
> >>>
> >>> My /proc/partitions looks like this:
> >>>
> >>> major minor #blocks name
> >>>
> >>> 8 0 71041024 sda
> >>> 8 1 68099503 sda1
> >>> 8 2 1 sda2
> >>> 8 5 2939863 sda5
> >>> 8 16 2134896640 sdb
> >>> 8 17 2134893883 sdb1
> >>> 8 32 2134896640 sdc
> >>> 8 48 2134896640 sdd
> >>> 8 49 2134893883 sdd1
> >>> 8 64 2134896640 sde
> >>> 8 80 1222794240 sdf
> >>> 8 96 20480 sdg
> >>> 8 112 2134896640 sdh
> >>> 8 128 2134896640 sdi
> >>> 8 129 2134893883 sdi1
> >>> 8 144 2134896640 sdj
> >>> 8 160 2134896640 sdk
> >>> 8 161 2134893883 sdk1
> >>> 8 176 1222794240 sdl
> >>> 8 192 20480 sdm
> >>> 254 0 2134896640 dm-0
> >>> 254 1 2134896640 dm-1
> >>> 254 2 2134896640 dm-2
> >>> 254 3 2134896640 dm-3
> >>> 254 4 1222794240 dm-4
> >>> 254 5 20480 dm-5
> >>> 254 6 20480 dm-6
> >>>
> >>> All the sd's with a partition are working fine, I even created a test
> >>> file system at some point. I did load scsi_dh_rdac for testing at some
> >>> point, and that put the following into the syslog:
> >>> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned)
> >>> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned)
> >>> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned)
> >>> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned)
> >>> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned)
> >>> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned)
> >>> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned)
> >>> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned)
> >>> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned)
> >>> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned)
> >>> [10170.096088] rdac: device handler registered
> >>>
> >>> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508]
> >>> Buffer I/O error on device dm-5, logical block 5
> >>> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6
> >>> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112
> >>> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112
> >>> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12
> >>> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13
> >>> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14
> >>> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15
> >>> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16
> >>> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17
> >>>
> >>> type messages.
> >>>
> >>> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman
> >>> <sekharan@us.ibm.com> wrote:
> >>> > No, you don't have to change the distro.
> >>> >
> >>> > Do you see the disks ? (the #:#:#:# makes me think that the underlying
> >>> > block devices are not configured properly). IOW, do you the sd devices
> >>> > under /sys/block ? Are you able to read/write from/to the devices ?
> >>> >
> >>> > chandra
> >>> >
> >>> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote:
> >>> >> Thanks Chandra,
> >>> >>
> >>> >> sorry for the out of order reply, still had the digest mode on. I
> >>> >> setup the config similar to what you said
> >>> >>
> >>> >> devices {
> >>> >> device {
> >>> >> vendor "DELL"
> >>> >> product "MD3000"
> >>> >> hardware_handler "1 rdac"
> >>> >> path_checker rdac
> >>> >> path_grouping_policy group_by_prio
> >>> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n"
> >>> >> failback immediate
> >>> >> getuid_callout "/sbin/scsi_id -g -u /block/%n"
> >>> >> }
> >>> >> }
> >>> >>
> >>> >>
> >>> >> and I'm still getting:
> >>> >> sdh: checker msg is "directio checker reports path is down"
> >>> >> create: 360022190009773680000214749504780 ,
> >>> >> [size=2.0T][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdb 8:16 [undef][ready]
> >>> >> \_ round-robin 0 [prio=0][undef]
> >>> >> \_ #:#:#:# sdh 8:112 [undef][faulty]
> >>> >> sdc: checker msg is "directio checker reports path is down"
> >>> >> create: 3600221900096f27d00000ab3495047fd ,
> >>> >> [size=2.0T][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=0][undef]
> >>> >> \_ #:#:#:# sdc 8:32 [undef][faulty]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdi 8:128 [undef][ready]
> >>> >> sdj: checker msg is "directio checker reports path is down"
> >>> >> create: 360022190009773680000214a495047ce ,
> >>> >> [size=2.0T][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdd 8:48 [undef][ready]
> >>> >> \_ round-robin 0 [prio=0][undef]
> >>> >> \_ #:#:#:# sdj 8:144 [undef][faulty]
> >>> >> sde: checker msg is "directio checker reports path is down"
> >>> >> create: 3600221900096f27d00000ab449504839 ,
> >>> >> [size=2.0T][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=0][undef]
> >>> >> \_ #:#:#:# sde 8:64 [undef][faulty]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdk 8:160 [undef][ready]
> >>> >> sdl: checker msg is "directio checker reports path is down"
> >>> >> create: 360022190009773680000214c49504806 ,
> >>> >> [size=1.1T][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdf 8:80 [undef][ready]
> >>> >> \_ round-robin 0 [prio=0][undef]
> >>> >> \_ #:#:#:# sdl 8:176 [undef][faulty]
> >>> >> create: 360022190009773680000000000000000 ,
> >>> >> [size=20M][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdg 8:96 [undef][ready]
> >>> >> create: 3600221900096f27d0000000000000000 ,
> >>> >> [size=20M][features=0][hwhandler=0]
> >>> >> \_ round-robin 0 [prio=1][undef]
> >>> >> \_ #:#:#:# sdm 8:192 [undef][ready]
> >>> >> libdevmapper: libdm-common.c(312): Created
> >>> >> /dev/mapper/360022190009773680000000000000000
> >>> >> libdevmapper: libdm-common.c(312): Created
> >>> >> /dev/mapper/3600221900096f27d0000000000000000
> >>> >>
> >>> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely
> >>> >> necessary I would consider switching the distribution, but right now
> >>> >> I'd prefer to stick with what all the other machines have.
> >>> >>
> >>> >> Thanks, Thomas
> >>> >>
> >>> >> --
> >>> >> dm-devel mailing list
> >>> >> dm-devel@redhat.com
> >>> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >>> >
> >>> > --
> >>> > dm-devel mailing list
> >>> > dm-devel@redhat.com
> >>> > https://www.redhat.com/mailman/listinfo/dm-devel
> >>> >
> >>>
> >>> --
> >>> dm-devel mailing list
> >>> dm-devel@redhat.com
> >>> https://www.redhat.com/mailman/listinfo/dm-devel
> >>
> >> --
> >> dm-devel mailing list
> >> dm-devel@redhat.com
> >> https://www.redhat.com/mailman/listinfo/dm-devel
> >>
> >
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-03-12 22:07 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-03-12 16:53 configure MD3000 Thomas Witzel
2009-03-12 18:33 ` Chandra Seetharaman
2009-03-12 19:07 ` Thomas Witzel
2009-03-12 19:18 ` Chandra Seetharaman
2009-03-12 19:25 ` Thomas Witzel
2009-03-12 19:31 ` Chandra Seetharaman
2009-03-12 19:38 ` Thomas Witzel
2009-03-12 19:40 ` Thomas Witzel
2009-03-12 22:07 ` Chandra Seetharaman
2009-03-12 22:06 ` Chandra Seetharaman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.