* configure MD3000 @ 2009-03-12 16:53 Thomas Witzel 2009-03-12 18:33 ` Chandra Seetharaman 2009-03-12 19:07 ` Thomas Witzel 0 siblings, 2 replies; 10+ messages in thread From: Thomas Witzel @ 2009-03-12 16:53 UTC (permalink / raw) To: dm-devel Hello all, I have a Dell MD3000 attached to an NX1950. It seems the MD3000 is connected through a SAS 5/E card and uses the mptsas driver. My kernel version is 2.6.27. multipath -ll gives me: sdj: checker msg is "directio checker reports path is down" 360022190009773680000214a495047cedm-5 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][active] \_ #:#:#:# sdd 8:48 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# sdj 8:144 [failed][faulty] 360022190009773680000000000000000dm-0 , [size=20M][features=0][hwhandler=0] \_ round-robin 0 [prio=1][enabled] \_ #:#:#:# sdg 8:96 [failed][ready] sde: checker msg is "directio checker reports path is down" 3600221900096f27d00000ab449504839dm-4 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][active] \_ #:#:#:# sdk 8:160 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# sde 8:64 [failed][faulty] sdc: checker msg is "directio checker reports path is down" 3600221900096f27d00000ab3495047fddm-2 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][active] \_ #:#:#:# sdi 8:128 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# sdc 8:32 [failed][faulty] sdh: checker msg is "directio checker reports path is down" 360022190009773680000214749504780dm-6 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][active] \_ #:#:#:# sdb 8:16 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# sdh 8:112 [failed][faulty] sdl: checker msg is "directio checker reports path is down" 360022190009773680000214c49504806dm-3 , [size=1.1T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][active] \_ #:#:#:# sdf 8:80 [active][ready] \_ round-robin 0 [prio=0][enabled] \_ #:#:#:# sdl 8:176 [failed][faulty] 3600221900096f27d0000000000000000dm-1 , [size=20M][features=0][hwhandler=0] \_ round-robin 0 [prio=1][enabled] \_ #:#:#:# sdm 8:192 [failed][ready] How do I configure this to work correctly ? I'm sorry if that is a redundant question, but I googled for about 8 hours now and couldn't find anything conclusive. Also is there a manual/tutorial for this somewhere ? Thank you, Thomas ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: configure MD3000 2009-03-12 16:53 configure MD3000 Thomas Witzel @ 2009-03-12 18:33 ` Chandra Seetharaman 2009-03-12 19:07 ` Thomas Witzel 1 sibling, 0 replies; 10+ messages in thread From: Chandra Seetharaman @ 2009-03-12 18:33 UTC (permalink / raw) To: device-mapper development To my knowledge DELL DB3000 is a rdac based storage. Hence, you need to add a special device stanza to /etc/multipath.conf file. -------- devices { device { vendor "DELL" product "MD3000" hardware_handler "1 rdac" path_checker rdac failback immediate path_grouping_policy group_by_prio no_path_retry queue prio_callout "/sbin/mpath_prio_tpc /dev/%n" } } --------- Make the change, run "multipath -F" followed by "multipath" for the change to affect your configuration. What is your distro (RHEL or SLES) and what is the version ? For SLES you need to replace the prio_callout line by prio rdac chandra On Thu, 2009-03-12 at 12:53 -0400, Thomas Witzel wrote: > Hello all, > > I have a Dell MD3000 attached to an NX1950. It seems the MD3000 is > connected through a SAS 5/E card and uses the mptsas driver. My kernel > version is 2.6.27. > multipath -ll gives me: > > sdj: checker msg is "directio checker reports path is down" > 360022190009773680000214a495047cedm-5 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][active] > \_ #:#:#:# sdd 8:48 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ #:#:#:# sdj 8:144 [failed][faulty] > 360022190009773680000000000000000dm-0 , > [size=20M][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][enabled] > \_ #:#:#:# sdg 8:96 [failed][ready] > sde: checker msg is "directio checker reports path is down" > 3600221900096f27d00000ab449504839dm-4 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][active] > \_ #:#:#:# sdk 8:160 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ #:#:#:# sde 8:64 [failed][faulty] > sdc: checker msg is "directio checker reports path is down" > 3600221900096f27d00000ab3495047fddm-2 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][active] > \_ #:#:#:# sdi 8:128 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ #:#:#:# sdc 8:32 [failed][faulty] > sdh: checker msg is "directio checker reports path is down" > 360022190009773680000214749504780dm-6 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][active] > \_ #:#:#:# sdb 8:16 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ #:#:#:# sdh 8:112 [failed][faulty] > sdl: checker msg is "directio checker reports path is down" > 360022190009773680000214c49504806dm-3 , > [size=1.1T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][active] > \_ #:#:#:# sdf 8:80 [active][ready] > \_ round-robin 0 [prio=0][enabled] > \_ #:#:#:# sdl 8:176 [failed][faulty] > 3600221900096f27d0000000000000000dm-1 , > [size=20M][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][enabled] > \_ #:#:#:# sdm 8:192 [failed][ready] > > > How do I configure this to work correctly ? I'm sorry if that is a > redundant question, but I googled for about 8 hours now and couldn't > find anything conclusive. Also is there a manual/tutorial for this > somewhere ? > > Thank you, > Thomas > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: configure MD3000 2009-03-12 16:53 configure MD3000 Thomas Witzel 2009-03-12 18:33 ` Chandra Seetharaman @ 2009-03-12 19:07 ` Thomas Witzel 2009-03-12 19:18 ` Chandra Seetharaman 1 sibling, 1 reply; 10+ messages in thread From: Thomas Witzel @ 2009-03-12 19:07 UTC (permalink / raw) To: dm-devel Thanks Chandra, sorry for the out of order reply, still had the digest mode on. I setup the config similar to what you said devices { device { vendor "DELL" product "MD3000" hardware_handler "1 rdac" path_checker rdac path_grouping_policy group_by_prio prio_callout "/sbin/mpath_prio_rdac /dev/%n" failback immediate getuid_callout "/sbin/scsi_id -g -u /block/%n" } } and I'm still getting: sdh: checker msg is "directio checker reports path is down" create: 360022190009773680000214749504780 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdb 8:16 [undef][ready] \_ round-robin 0 [prio=0][undef] \_ #:#:#:# sdh 8:112 [undef][faulty] sdc: checker msg is "directio checker reports path is down" create: 3600221900096f27d00000ab3495047fd , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=0][undef] \_ #:#:#:# sdc 8:32 [undef][faulty] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdi 8:128 [undef][ready] sdj: checker msg is "directio checker reports path is down" create: 360022190009773680000214a495047ce , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdd 8:48 [undef][ready] \_ round-robin 0 [prio=0][undef] \_ #:#:#:# sdj 8:144 [undef][faulty] sde: checker msg is "directio checker reports path is down" create: 3600221900096f27d00000ab449504839 , [size=2.0T][features=0][hwhandler=0] \_ round-robin 0 [prio=0][undef] \_ #:#:#:# sde 8:64 [undef][faulty] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdk 8:160 [undef][ready] sdl: checker msg is "directio checker reports path is down" create: 360022190009773680000214c49504806 , [size=1.1T][features=0][hwhandler=0] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdf 8:80 [undef][ready] \_ round-robin 0 [prio=0][undef] \_ #:#:#:# sdl 8:176 [undef][faulty] create: 360022190009773680000000000000000 , [size=20M][features=0][hwhandler=0] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdg 8:96 [undef][ready] create: 3600221900096f27d0000000000000000 , [size=20M][features=0][hwhandler=0] \_ round-robin 0 [prio=1][undef] \_ #:#:#:# sdm 8:192 [undef][ready] libdevmapper: libdm-common.c(312): Created /dev/mapper/360022190009773680000000000000000 libdevmapper: libdm-common.c(312): Created /dev/mapper/3600221900096f27d0000000000000000 Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely necessary I would consider switching the distribution, but right now I'd prefer to stick with what all the other machines have. Thanks, Thomas ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:07 ` Thomas Witzel @ 2009-03-12 19:18 ` Chandra Seetharaman 2009-03-12 19:25 ` Thomas Witzel 0 siblings, 1 reply; 10+ messages in thread From: Chandra Seetharaman @ 2009-03-12 19:18 UTC (permalink / raw) To: device-mapper development No, you don't have to change the distro. Do you see the disks ? (the #:#:#:# makes me think that the underlying block devices are not configured properly). IOW, do you the sd devices under /sys/block ? Are you able to read/write from/to the devices ? chandra On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: > Thanks Chandra, > > sorry for the out of order reply, still had the digest mode on. I > setup the config similar to what you said > > devices { > device { > vendor "DELL" > product "MD3000" > hardware_handler "1 rdac" > path_checker rdac > path_grouping_policy group_by_prio > prio_callout "/sbin/mpath_prio_rdac /dev/%n" > failback immediate > getuid_callout "/sbin/scsi_id -g -u /block/%n" > } > } > > > and I'm still getting: > sdh: checker msg is "directio checker reports path is down" > create: 360022190009773680000214749504780 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdb 8:16 [undef][ready] > \_ round-robin 0 [prio=0][undef] > \_ #:#:#:# sdh 8:112 [undef][faulty] > sdc: checker msg is "directio checker reports path is down" > create: 3600221900096f27d00000ab3495047fd , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][undef] > \_ #:#:#:# sdc 8:32 [undef][faulty] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdi 8:128 [undef][ready] > sdj: checker msg is "directio checker reports path is down" > create: 360022190009773680000214a495047ce , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdd 8:48 [undef][ready] > \_ round-robin 0 [prio=0][undef] > \_ #:#:#:# sdj 8:144 [undef][faulty] > sde: checker msg is "directio checker reports path is down" > create: 3600221900096f27d00000ab449504839 , > [size=2.0T][features=0][hwhandler=0] > \_ round-robin 0 [prio=0][undef] > \_ #:#:#:# sde 8:64 [undef][faulty] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdk 8:160 [undef][ready] > sdl: checker msg is "directio checker reports path is down" > create: 360022190009773680000214c49504806 , > [size=1.1T][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdf 8:80 [undef][ready] > \_ round-robin 0 [prio=0][undef] > \_ #:#:#:# sdl 8:176 [undef][faulty] > create: 360022190009773680000000000000000 , > [size=20M][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdg 8:96 [undef][ready] > create: 3600221900096f27d0000000000000000 , > [size=20M][features=0][hwhandler=0] > \_ round-robin 0 [prio=1][undef] > \_ #:#:#:# sdm 8:192 [undef][ready] > libdevmapper: libdm-common.c(312): Created > /dev/mapper/360022190009773680000000000000000 > libdevmapper: libdm-common.c(312): Created > /dev/mapper/3600221900096f27d0000000000000000 > > Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely > necessary I would consider switching the distribution, but right now > I'd prefer to stick with what all the other machines have. > > Thanks, Thomas > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:18 ` Chandra Seetharaman @ 2009-03-12 19:25 ` Thomas Witzel 2009-03-12 19:31 ` Chandra Seetharaman 0 siblings, 1 reply; 10+ messages in thread From: Thomas Witzel @ 2009-03-12 19:25 UTC (permalink / raw) To: sekharan, device-mapper development My /sys/block looks like this: ls /sys/block/ dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 sdc sdf sdi sdl dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda sdd sdg sdj sdm dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb sde sdh sdk sr0 My /proc/partitions looks like this: major minor #blocks name 8 0 71041024 sda 8 1 68099503 sda1 8 2 1 sda2 8 5 2939863 sda5 8 16 2134896640 sdb 8 17 2134893883 sdb1 8 32 2134896640 sdc 8 48 2134896640 sdd 8 49 2134893883 sdd1 8 64 2134896640 sde 8 80 1222794240 sdf 8 96 20480 sdg 8 112 2134896640 sdh 8 128 2134896640 sdi 8 129 2134893883 sdi1 8 144 2134896640 sdj 8 160 2134896640 sdk 8 161 2134893883 sdk1 8 176 1222794240 sdl 8 192 20480 sdm 254 0 2134896640 dm-0 254 1 2134896640 dm-1 254 2 2134896640 dm-2 254 3 2134896640 dm-3 254 4 1222794240 dm-4 254 5 20480 dm-5 254 6 20480 dm-6 All the sd's with a partition are working fine, I even created a test file system at some point. I did load scsi_dh_rdac for testing at some point, and that put the following into the syslog: [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) [10170.096088] rdac: device handler registered Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] Buffer I/O error on device dm-5, logical block 5 [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 type messages. On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman <sekharan@us.ibm.com> wrote: > No, you don't have to change the distro. > > Do you see the disks ? (the #:#:#:# makes me think that the underlying > block devices are not configured properly). IOW, do you the sd devices > under /sys/block ? Are you able to read/write from/to the devices ? > > chandra > > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: >> Thanks Chandra, >> >> sorry for the out of order reply, still had the digest mode on. I >> setup the config similar to what you said >> >> devices { >> device { >> vendor "DELL" >> product "MD3000" >> hardware_handler "1 rdac" >> path_checker rdac >> path_grouping_policy group_by_prio >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" >> failback immediate >> getuid_callout "/sbin/scsi_id -g -u /block/%n" >> } >> } >> >> >> and I'm still getting: >> sdh: checker msg is "directio checker reports path is down" >> create: 360022190009773680000214749504780 , >> [size=2.0T][features=0][hwhandler=0] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdb 8:16 [undef][ready] >> \_ round-robin 0 [prio=0][undef] >> \_ #:#:#:# sdh 8:112 [undef][faulty] >> sdc: checker msg is "directio checker reports path is down" >> create: 3600221900096f27d00000ab3495047fd , >> [size=2.0T][features=0][hwhandler=0] >> \_ round-robin 0 [prio=0][undef] >> \_ #:#:#:# sdc 8:32 [undef][faulty] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdi 8:128 [undef][ready] >> sdj: checker msg is "directio checker reports path is down" >> create: 360022190009773680000214a495047ce , >> [size=2.0T][features=0][hwhandler=0] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdd 8:48 [undef][ready] >> \_ round-robin 0 [prio=0][undef] >> \_ #:#:#:# sdj 8:144 [undef][faulty] >> sde: checker msg is "directio checker reports path is down" >> create: 3600221900096f27d00000ab449504839 , >> [size=2.0T][features=0][hwhandler=0] >> \_ round-robin 0 [prio=0][undef] >> \_ #:#:#:# sde 8:64 [undef][faulty] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdk 8:160 [undef][ready] >> sdl: checker msg is "directio checker reports path is down" >> create: 360022190009773680000214c49504806 , >> [size=1.1T][features=0][hwhandler=0] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdf 8:80 [undef][ready] >> \_ round-robin 0 [prio=0][undef] >> \_ #:#:#:# sdl 8:176 [undef][faulty] >> create: 360022190009773680000000000000000 , >> [size=20M][features=0][hwhandler=0] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdg 8:96 [undef][ready] >> create: 3600221900096f27d0000000000000000 , >> [size=20M][features=0][hwhandler=0] >> \_ round-robin 0 [prio=1][undef] >> \_ #:#:#:# sdm 8:192 [undef][ready] >> libdevmapper: libdm-common.c(312): Created >> /dev/mapper/360022190009773680000000000000000 >> libdevmapper: libdm-common.c(312): Created >> /dev/mapper/3600221900096f27d0000000000000000 >> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely >> necessary I would consider switching the distribution, but right now >> I'd prefer to stick with what all the other machines have. >> >> Thanks, Thomas >> >> -- >> dm-devel mailing list >> dm-devel@redhat.com >> https://www.redhat.com/mailman/listinfo/dm-devel > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:25 ` Thomas Witzel @ 2009-03-12 19:31 ` Chandra Seetharaman 2009-03-12 19:38 ` Thomas Witzel 0 siblings, 1 reply; 10+ messages in thread From: Chandra Seetharaman @ 2009-03-12 19:31 UTC (permalink / raw) To: device-mapper development Are you able to read/write from/to the scsi device ? If yes, Did you do "multipath -F" followed by "multipath". Is the multipathd running ? chandra On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote: > My /sys/block looks like this: > ls /sys/block/ > dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 > sdc sdf sdi sdl > dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda > sdd sdg sdj sdm > dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb > sde sdh sdk sr0 > > My /proc/partitions looks like this: > > major minor #blocks name > > 8 0 71041024 sda > 8 1 68099503 sda1 > 8 2 1 sda2 > 8 5 2939863 sda5 > 8 16 2134896640 sdb > 8 17 2134893883 sdb1 > 8 32 2134896640 sdc > 8 48 2134896640 sdd > 8 49 2134893883 sdd1 > 8 64 2134896640 sde > 8 80 1222794240 sdf > 8 96 20480 sdg > 8 112 2134896640 sdh > 8 128 2134896640 sdi > 8 129 2134893883 sdi1 > 8 144 2134896640 sdj > 8 160 2134896640 sdk > 8 161 2134893883 sdk1 > 8 176 1222794240 sdl > 8 192 20480 sdm > 254 0 2134896640 dm-0 > 254 1 2134896640 dm-1 > 254 2 2134896640 dm-2 > 254 3 2134896640 dm-3 > 254 4 1222794240 dm-4 > 254 5 20480 dm-5 > 254 6 20480 dm-6 > > All the sd's with a partition are working fine, I even created a test > file system at some point. I did load scsi_dh_rdac for testing at some > point, and that put the following into the syslog: > [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) > [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) > [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) > [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) > [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) > [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) > [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) > [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) > [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) > [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) > [10170.096088] rdac: device handler registered > > Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] > Buffer I/O error on device dm-5, logical block 5 > [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 > [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 > [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 > [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 > [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 > [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 > [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 > [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 > [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 > > type messages. > > On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman > <sekharan@us.ibm.com> wrote: > > No, you don't have to change the distro. > > > > Do you see the disks ? (the #:#:#:# makes me think that the underlying > > block devices are not configured properly). IOW, do you the sd devices > > under /sys/block ? Are you able to read/write from/to the devices ? > > > > chandra > > > > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: > >> Thanks Chandra, > >> > >> sorry for the out of order reply, still had the digest mode on. I > >> setup the config similar to what you said > >> > >> devices { > >> device { > >> vendor "DELL" > >> product "MD3000" > >> hardware_handler "1 rdac" > >> path_checker rdac > >> path_grouping_policy group_by_prio > >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" > >> failback immediate > >> getuid_callout "/sbin/scsi_id -g -u /block/%n" > >> } > >> } > >> > >> > >> and I'm still getting: > >> sdh: checker msg is "directio checker reports path is down" > >> create: 360022190009773680000214749504780 , > >> [size=2.0T][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdb 8:16 [undef][ready] > >> \_ round-robin 0 [prio=0][undef] > >> \_ #:#:#:# sdh 8:112 [undef][faulty] > >> sdc: checker msg is "directio checker reports path is down" > >> create: 3600221900096f27d00000ab3495047fd , > >> [size=2.0T][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=0][undef] > >> \_ #:#:#:# sdc 8:32 [undef][faulty] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdi 8:128 [undef][ready] > >> sdj: checker msg is "directio checker reports path is down" > >> create: 360022190009773680000214a495047ce , > >> [size=2.0T][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdd 8:48 [undef][ready] > >> \_ round-robin 0 [prio=0][undef] > >> \_ #:#:#:# sdj 8:144 [undef][faulty] > >> sde: checker msg is "directio checker reports path is down" > >> create: 3600221900096f27d00000ab449504839 , > >> [size=2.0T][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=0][undef] > >> \_ #:#:#:# sde 8:64 [undef][faulty] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdk 8:160 [undef][ready] > >> sdl: checker msg is "directio checker reports path is down" > >> create: 360022190009773680000214c49504806 , > >> [size=1.1T][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdf 8:80 [undef][ready] > >> \_ round-robin 0 [prio=0][undef] > >> \_ #:#:#:# sdl 8:176 [undef][faulty] > >> create: 360022190009773680000000000000000 , > >> [size=20M][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdg 8:96 [undef][ready] > >> create: 3600221900096f27d0000000000000000 , > >> [size=20M][features=0][hwhandler=0] > >> \_ round-robin 0 [prio=1][undef] > >> \_ #:#:#:# sdm 8:192 [undef][ready] > >> libdevmapper: libdm-common.c(312): Created > >> /dev/mapper/360022190009773680000000000000000 > >> libdevmapper: libdm-common.c(312): Created > >> /dev/mapper/3600221900096f27d0000000000000000 > >> > >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely > >> necessary I would consider switching the distribution, but right now > >> I'd prefer to stick with what all the other machines have. > >> > >> Thanks, Thomas > >> > >> -- > >> dm-devel mailing list > >> dm-devel@redhat.com > >> https://www.redhat.com/mailman/listinfo/dm-devel > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:31 ` Chandra Seetharaman @ 2009-03-12 19:38 ` Thomas Witzel 2009-03-12 19:40 ` Thomas Witzel 2009-03-12 22:06 ` Chandra Seetharaman 0 siblings, 2 replies; 10+ messages in thread From: Thomas Witzel @ 2009-03-12 19:38 UTC (permalink / raw) To: sekharan, device-mapper development How do you want me to test whether I can read/write ? I can no longer mount the /dev/sd*1 partitions because now they are busy (thats since I started multipath). Yes, I did execute multipath -F and multipath repeatedly, the daemon is running. On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman <sekharan@us.ibm.com> wrote: > Are you able to read/write from/to the scsi device ? > > If yes, Did you do "multipath -F" followed by "multipath". > > Is the multipathd running ? > > chandra > On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote: >> My /sys/block looks like this: >> ls /sys/block/ >> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 >> sdc sdf sdi sdl >> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda >> sdd sdg sdj sdm >> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb >> sde sdh sdk sr0 >> >> My /proc/partitions looks like this: >> >> major minor #blocks name >> >> 8 0 71041024 sda >> 8 1 68099503 sda1 >> 8 2 1 sda2 >> 8 5 2939863 sda5 >> 8 16 2134896640 sdb >> 8 17 2134893883 sdb1 >> 8 32 2134896640 sdc >> 8 48 2134896640 sdd >> 8 49 2134893883 sdd1 >> 8 64 2134896640 sde >> 8 80 1222794240 sdf >> 8 96 20480 sdg >> 8 112 2134896640 sdh >> 8 128 2134896640 sdi >> 8 129 2134893883 sdi1 >> 8 144 2134896640 sdj >> 8 160 2134896640 sdk >> 8 161 2134893883 sdk1 >> 8 176 1222794240 sdl >> 8 192 20480 sdm >> 254 0 2134896640 dm-0 >> 254 1 2134896640 dm-1 >> 254 2 2134896640 dm-2 >> 254 3 2134896640 dm-3 >> 254 4 1222794240 dm-4 >> 254 5 20480 dm-5 >> 254 6 20480 dm-6 >> >> All the sd's with a partition are working fine, I even created a test >> file system at some point. I did load scsi_dh_rdac for testing at some >> point, and that put the following into the syslog: >> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) >> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) >> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) >> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) >> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) >> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) >> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) >> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) >> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) >> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) >> [10170.096088] rdac: device handler registered >> >> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] >> Buffer I/O error on device dm-5, logical block 5 >> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 >> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 >> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 >> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 >> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 >> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 >> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 >> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 >> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 >> >> type messages. >> >> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman >> <sekharan@us.ibm.com> wrote: >> > No, you don't have to change the distro. >> > >> > Do you see the disks ? (the #:#:#:# makes me think that the underlying >> > block devices are not configured properly). IOW, do you the sd devices >> > under /sys/block ? Are you able to read/write from/to the devices ? >> > >> > chandra >> > >> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: >> >> Thanks Chandra, >> >> >> >> sorry for the out of order reply, still had the digest mode on. I >> >> setup the config similar to what you said >> >> >> >> devices { >> >> device { >> >> vendor "DELL" >> >> product "MD3000" >> >> hardware_handler "1 rdac" >> >> path_checker rdac >> >> path_grouping_policy group_by_prio >> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" >> >> failback immediate >> >> getuid_callout "/sbin/scsi_id -g -u /block/%n" >> >> } >> >> } >> >> >> >> >> >> and I'm still getting: >> >> sdh: checker msg is "directio checker reports path is down" >> >> create: 360022190009773680000214749504780 , >> >> [size=2.0T][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdb 8:16 [undef][ready] >> >> \_ round-robin 0 [prio=0][undef] >> >> \_ #:#:#:# sdh 8:112 [undef][faulty] >> >> sdc: checker msg is "directio checker reports path is down" >> >> create: 3600221900096f27d00000ab3495047fd , >> >> [size=2.0T][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=0][undef] >> >> \_ #:#:#:# sdc 8:32 [undef][faulty] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdi 8:128 [undef][ready] >> >> sdj: checker msg is "directio checker reports path is down" >> >> create: 360022190009773680000214a495047ce , >> >> [size=2.0T][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdd 8:48 [undef][ready] >> >> \_ round-robin 0 [prio=0][undef] >> >> \_ #:#:#:# sdj 8:144 [undef][faulty] >> >> sde: checker msg is "directio checker reports path is down" >> >> create: 3600221900096f27d00000ab449504839 , >> >> [size=2.0T][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=0][undef] >> >> \_ #:#:#:# sde 8:64 [undef][faulty] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdk 8:160 [undef][ready] >> >> sdl: checker msg is "directio checker reports path is down" >> >> create: 360022190009773680000214c49504806 , >> >> [size=1.1T][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdf 8:80 [undef][ready] >> >> \_ round-robin 0 [prio=0][undef] >> >> \_ #:#:#:# sdl 8:176 [undef][faulty] >> >> create: 360022190009773680000000000000000 , >> >> [size=20M][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdg 8:96 [undef][ready] >> >> create: 3600221900096f27d0000000000000000 , >> >> [size=20M][features=0][hwhandler=0] >> >> \_ round-robin 0 [prio=1][undef] >> >> \_ #:#:#:# sdm 8:192 [undef][ready] >> >> libdevmapper: libdm-common.c(312): Created >> >> /dev/mapper/360022190009773680000000000000000 >> >> libdevmapper: libdm-common.c(312): Created >> >> /dev/mapper/3600221900096f27d0000000000000000 >> >> >> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely >> >> necessary I would consider switching the distribution, but right now >> >> I'd prefer to stick with what all the other machines have. >> >> >> >> Thanks, Thomas >> >> >> >> -- >> >> dm-devel mailing list >> >> dm-devel@redhat.com >> >> https://www.redhat.com/mailman/listinfo/dm-devel >> > >> > -- >> > dm-devel mailing list >> > dm-devel@redhat.com >> > https://www.redhat.com/mailman/listinfo/dm-devel >> > >> >> -- >> dm-devel mailing list >> dm-devel@redhat.com >> https://www.redhat.com/mailman/listinfo/dm-devel > > -- > dm-devel mailing list > dm-devel@redhat.com > https://www.redhat.com/mailman/listinfo/dm-devel > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:38 ` Thomas Witzel @ 2009-03-12 19:40 ` Thomas Witzel 2009-03-12 22:07 ` Chandra Seetharaman 2009-03-12 22:06 ` Chandra Seetharaman 1 sibling, 1 reply; 10+ messages in thread From: Thomas Witzel @ 2009-03-12 19:40 UTC (permalink / raw) To: sekharan, device-mapper development What sort of bothers me, is that although we specified rdac, it says hwhandler=0 while on the web there seems to be output examples where it does say rdac in the multipath output. Also googling convinced me that some people have drivers/md/md-multipath-rdac.c or something like that in their kernel, while my ubuntu kernel and the vanilla git kernel do not have this. I have the feeling this is an important component of this. On Thu, Mar 12, 2009 at 3:38 PM, Thomas Witzel <witzel.thomas@gmail.com> wrote: > How do you want me to test whether I can read/write ? I can no longer > mount the /dev/sd*1 partitions because now they are busy (thats since > I started multipath). Yes, I did execute multipath -F and multipath > repeatedly, the daemon is running. > > On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman > <sekharan@us.ibm.com> wrote: >> Are you able to read/write from/to the scsi device ? >> >> If yes, Did you do "multipath -F" followed by "multipath". >> >> Is the multipathd running ? >> >> chandra >> On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote: >>> My /sys/block looks like this: >>> ls /sys/block/ >>> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 >>> sdc sdf sdi sdl >>> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda >>> sdd sdg sdj sdm >>> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb >>> sde sdh sdk sr0 >>> >>> My /proc/partitions looks like this: >>> >>> major minor #blocks name >>> >>> 8 0 71041024 sda >>> 8 1 68099503 sda1 >>> 8 2 1 sda2 >>> 8 5 2939863 sda5 >>> 8 16 2134896640 sdb >>> 8 17 2134893883 sdb1 >>> 8 32 2134896640 sdc >>> 8 48 2134896640 sdd >>> 8 49 2134893883 sdd1 >>> 8 64 2134896640 sde >>> 8 80 1222794240 sdf >>> 8 96 20480 sdg >>> 8 112 2134896640 sdh >>> 8 128 2134896640 sdi >>> 8 129 2134893883 sdi1 >>> 8 144 2134896640 sdj >>> 8 160 2134896640 sdk >>> 8 161 2134893883 sdk1 >>> 8 176 1222794240 sdl >>> 8 192 20480 sdm >>> 254 0 2134896640 dm-0 >>> 254 1 2134896640 dm-1 >>> 254 2 2134896640 dm-2 >>> 254 3 2134896640 dm-3 >>> 254 4 1222794240 dm-4 >>> 254 5 20480 dm-5 >>> 254 6 20480 dm-6 >>> >>> All the sd's with a partition are working fine, I even created a test >>> file system at some point. I did load scsi_dh_rdac for testing at some >>> point, and that put the following into the syslog: >>> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) >>> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) >>> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) >>> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) >>> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) >>> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) >>> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) >>> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) >>> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) >>> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) >>> [10170.096088] rdac: device handler registered >>> >>> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] >>> Buffer I/O error on device dm-5, logical block 5 >>> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 >>> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 >>> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 >>> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 >>> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 >>> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 >>> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 >>> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 >>> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 >>> >>> type messages. >>> >>> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman >>> <sekharan@us.ibm.com> wrote: >>> > No, you don't have to change the distro. >>> > >>> > Do you see the disks ? (the #:#:#:# makes me think that the underlying >>> > block devices are not configured properly). IOW, do you the sd devices >>> > under /sys/block ? Are you able to read/write from/to the devices ? >>> > >>> > chandra >>> > >>> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: >>> >> Thanks Chandra, >>> >> >>> >> sorry for the out of order reply, still had the digest mode on. I >>> >> setup the config similar to what you said >>> >> >>> >> devices { >>> >> device { >>> >> vendor "DELL" >>> >> product "MD3000" >>> >> hardware_handler "1 rdac" >>> >> path_checker rdac >>> >> path_grouping_policy group_by_prio >>> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" >>> >> failback immediate >>> >> getuid_callout "/sbin/scsi_id -g -u /block/%n" >>> >> } >>> >> } >>> >> >>> >> >>> >> and I'm still getting: >>> >> sdh: checker msg is "directio checker reports path is down" >>> >> create: 360022190009773680000214749504780 , >>> >> [size=2.0T][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdb 8:16 [undef][ready] >>> >> \_ round-robin 0 [prio=0][undef] >>> >> \_ #:#:#:# sdh 8:112 [undef][faulty] >>> >> sdc: checker msg is "directio checker reports path is down" >>> >> create: 3600221900096f27d00000ab3495047fd , >>> >> [size=2.0T][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=0][undef] >>> >> \_ #:#:#:# sdc 8:32 [undef][faulty] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdi 8:128 [undef][ready] >>> >> sdj: checker msg is "directio checker reports path is down" >>> >> create: 360022190009773680000214a495047ce , >>> >> [size=2.0T][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdd 8:48 [undef][ready] >>> >> \_ round-robin 0 [prio=0][undef] >>> >> \_ #:#:#:# sdj 8:144 [undef][faulty] >>> >> sde: checker msg is "directio checker reports path is down" >>> >> create: 3600221900096f27d00000ab449504839 , >>> >> [size=2.0T][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=0][undef] >>> >> \_ #:#:#:# sde 8:64 [undef][faulty] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdk 8:160 [undef][ready] >>> >> sdl: checker msg is "directio checker reports path is down" >>> >> create: 360022190009773680000214c49504806 , >>> >> [size=1.1T][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdf 8:80 [undef][ready] >>> >> \_ round-robin 0 [prio=0][undef] >>> >> \_ #:#:#:# sdl 8:176 [undef][faulty] >>> >> create: 360022190009773680000000000000000 , >>> >> [size=20M][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdg 8:96 [undef][ready] >>> >> create: 3600221900096f27d0000000000000000 , >>> >> [size=20M][features=0][hwhandler=0] >>> >> \_ round-robin 0 [prio=1][undef] >>> >> \_ #:#:#:# sdm 8:192 [undef][ready] >>> >> libdevmapper: libdm-common.c(312): Created >>> >> /dev/mapper/360022190009773680000000000000000 >>> >> libdevmapper: libdm-common.c(312): Created >>> >> /dev/mapper/3600221900096f27d0000000000000000 >>> >> >>> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely >>> >> necessary I would consider switching the distribution, but right now >>> >> I'd prefer to stick with what all the other machines have. >>> >> >>> >> Thanks, Thomas >>> >> >>> >> -- >>> >> dm-devel mailing list >>> >> dm-devel@redhat.com >>> >> https://www.redhat.com/mailman/listinfo/dm-devel >>> > >>> > -- >>> > dm-devel mailing list >>> > dm-devel@redhat.com >>> > https://www.redhat.com/mailman/listinfo/dm-devel >>> > >>> >>> -- >>> dm-devel mailing list >>> dm-devel@redhat.com >>> https://www.redhat.com/mailman/listinfo/dm-devel >> >> -- >> dm-devel mailing list >> dm-devel@redhat.com >> https://www.redhat.com/mailman/listinfo/dm-devel >> > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:40 ` Thomas Witzel @ 2009-03-12 22:07 ` Chandra Seetharaman 0 siblings, 0 replies; 10+ messages in thread From: Chandra Seetharaman @ 2009-03-12 22:07 UTC (permalink / raw) To: Thomas Witzel; +Cc: device-mapper development On Thu, 2009-03-12 at 15:40 -0400, Thomas Witzel wrote: > What sort of bothers me, is that although we specified rdac, it says > hwhandler=0 while on the web there seems to be output examples where > it does say rdac in the multipath output. > Also googling convinced me that some people have > drivers/md/md-multipath-rdac.c or something like that in their kernel, > while my ubuntu kernel and the vanilla git kernel do not have this. I > have the feeling this is an important component of this. From 2.6.27 onwards, rdac has moved to scsi layer as scsi_dh_rdac. You can do a "lsmod | grep rdac" to see if that module is inserted. > > On Thu, Mar 12, 2009 at 3:38 PM, Thomas Witzel <witzel.thomas@gmail.com> wrote: > > How do you want me to test whether I can read/write ? I can no longer > > mount the /dev/sd*1 partitions because now they are busy (thats since > > I started multipath). Yes, I did execute multipath -F and multipath > > repeatedly, the daemon is running. > > > > On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman > > <sekharan@us.ibm.com> wrote: > >> Are you able to read/write from/to the scsi device ? > >> > >> If yes, Did you do "multipath -F" followed by "multipath". > >> > >> Is the multipathd running ? > >> > >> chandra > >> On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote: > >>> My /sys/block looks like this: > >>> ls /sys/block/ > >>> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 > >>> sdc sdf sdi sdl > >>> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda > >>> sdd sdg sdj sdm > >>> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb > >>> sde sdh sdk sr0 > >>> > >>> My /proc/partitions looks like this: > >>> > >>> major minor #blocks name > >>> > >>> 8 0 71041024 sda > >>> 8 1 68099503 sda1 > >>> 8 2 1 sda2 > >>> 8 5 2939863 sda5 > >>> 8 16 2134896640 sdb > >>> 8 17 2134893883 sdb1 > >>> 8 32 2134896640 sdc > >>> 8 48 2134896640 sdd > >>> 8 49 2134893883 sdd1 > >>> 8 64 2134896640 sde > >>> 8 80 1222794240 sdf > >>> 8 96 20480 sdg > >>> 8 112 2134896640 sdh > >>> 8 128 2134896640 sdi > >>> 8 129 2134893883 sdi1 > >>> 8 144 2134896640 sdj > >>> 8 160 2134896640 sdk > >>> 8 161 2134893883 sdk1 > >>> 8 176 1222794240 sdl > >>> 8 192 20480 sdm > >>> 254 0 2134896640 dm-0 > >>> 254 1 2134896640 dm-1 > >>> 254 2 2134896640 dm-2 > >>> 254 3 2134896640 dm-3 > >>> 254 4 1222794240 dm-4 > >>> 254 5 20480 dm-5 > >>> 254 6 20480 dm-6 > >>> > >>> All the sd's with a partition are working fine, I even created a test > >>> file system at some point. I did load scsi_dh_rdac for testing at some > >>> point, and that put the following into the syslog: > >>> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) > >>> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) > >>> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) > >>> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) > >>> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) > >>> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) > >>> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) > >>> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) > >>> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) > >>> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) > >>> [10170.096088] rdac: device handler registered > >>> > >>> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] > >>> Buffer I/O error on device dm-5, logical block 5 > >>> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 > >>> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 > >>> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 > >>> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 > >>> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 > >>> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 > >>> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 > >>> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 > >>> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 > >>> > >>> type messages. > >>> > >>> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman > >>> <sekharan@us.ibm.com> wrote: > >>> > No, you don't have to change the distro. > >>> > > >>> > Do you see the disks ? (the #:#:#:# makes me think that the underlying > >>> > block devices are not configured properly). IOW, do you the sd devices > >>> > under /sys/block ? Are you able to read/write from/to the devices ? > >>> > > >>> > chandra > >>> > > >>> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: > >>> >> Thanks Chandra, > >>> >> > >>> >> sorry for the out of order reply, still had the digest mode on. I > >>> >> setup the config similar to what you said > >>> >> > >>> >> devices { > >>> >> device { > >>> >> vendor "DELL" > >>> >> product "MD3000" > >>> >> hardware_handler "1 rdac" > >>> >> path_checker rdac > >>> >> path_grouping_policy group_by_prio > >>> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" > >>> >> failback immediate > >>> >> getuid_callout "/sbin/scsi_id -g -u /block/%n" > >>> >> } > >>> >> } > >>> >> > >>> >> > >>> >> and I'm still getting: > >>> >> sdh: checker msg is "directio checker reports path is down" > >>> >> create: 360022190009773680000214749504780 , > >>> >> [size=2.0T][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdb 8:16 [undef][ready] > >>> >> \_ round-robin 0 [prio=0][undef] > >>> >> \_ #:#:#:# sdh 8:112 [undef][faulty] > >>> >> sdc: checker msg is "directio checker reports path is down" > >>> >> create: 3600221900096f27d00000ab3495047fd , > >>> >> [size=2.0T][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=0][undef] > >>> >> \_ #:#:#:# sdc 8:32 [undef][faulty] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdi 8:128 [undef][ready] > >>> >> sdj: checker msg is "directio checker reports path is down" > >>> >> create: 360022190009773680000214a495047ce , > >>> >> [size=2.0T][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdd 8:48 [undef][ready] > >>> >> \_ round-robin 0 [prio=0][undef] > >>> >> \_ #:#:#:# sdj 8:144 [undef][faulty] > >>> >> sde: checker msg is "directio checker reports path is down" > >>> >> create: 3600221900096f27d00000ab449504839 , > >>> >> [size=2.0T][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=0][undef] > >>> >> \_ #:#:#:# sde 8:64 [undef][faulty] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdk 8:160 [undef][ready] > >>> >> sdl: checker msg is "directio checker reports path is down" > >>> >> create: 360022190009773680000214c49504806 , > >>> >> [size=1.1T][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdf 8:80 [undef][ready] > >>> >> \_ round-robin 0 [prio=0][undef] > >>> >> \_ #:#:#:# sdl 8:176 [undef][faulty] > >>> >> create: 360022190009773680000000000000000 , > >>> >> [size=20M][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdg 8:96 [undef][ready] > >>> >> create: 3600221900096f27d0000000000000000 , > >>> >> [size=20M][features=0][hwhandler=0] > >>> >> \_ round-robin 0 [prio=1][undef] > >>> >> \_ #:#:#:# sdm 8:192 [undef][ready] > >>> >> libdevmapper: libdm-common.c(312): Created > >>> >> /dev/mapper/360022190009773680000000000000000 > >>> >> libdevmapper: libdm-common.c(312): Created > >>> >> /dev/mapper/3600221900096f27d0000000000000000 > >>> >> > >>> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely > >>> >> necessary I would consider switching the distribution, but right now > >>> >> I'd prefer to stick with what all the other machines have. > >>> >> > >>> >> Thanks, Thomas > >>> >> > >>> >> -- > >>> >> dm-devel mailing list > >>> >> dm-devel@redhat.com > >>> >> https://www.redhat.com/mailman/listinfo/dm-devel > >>> > > >>> > -- > >>> > dm-devel mailing list > >>> > dm-devel@redhat.com > >>> > https://www.redhat.com/mailman/listinfo/dm-devel > >>> > > >>> > >>> -- > >>> dm-devel mailing list > >>> dm-devel@redhat.com > >>> https://www.redhat.com/mailman/listinfo/dm-devel > >> > >> -- > >> dm-devel mailing list > >> dm-devel@redhat.com > >> https://www.redhat.com/mailman/listinfo/dm-devel > >> > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Re: configure MD3000 2009-03-12 19:38 ` Thomas Witzel 2009-03-12 19:40 ` Thomas Witzel @ 2009-03-12 22:06 ` Chandra Seetharaman 1 sibling, 0 replies; 10+ messages in thread From: Chandra Seetharaman @ 2009-03-12 22:06 UTC (permalink / raw) To: Thomas Witzel; +Cc: device-mapper development On Thu, 2009-03-12 at 15:38 -0400, Thomas Witzel wrote: > How do you want me to test whether I can read/write ? I can no longer > mount the /dev/sd*1 partitions because now they are busy (thats since you should be able to do dd if=/dev/sdb1 of=/dev/null count=1000 iflag=direct etc., > I started multipath). Yes, I did execute multipath -F and multipath > repeatedly, the daemon is running. In your multipath -ll output is bizarre: - As you mentioned (in the other email) there is no rdac hw handler - it is still using the directio path checker (it should be using rdac path checker). which makes me think that your device is not showing "DELL" "MD3000" as vendor and product. Can you send the o/p of lsscsi ? > > On Thu, Mar 12, 2009 at 3:31 PM, Chandra Seetharaman > <sekharan@us.ibm.com> wrote: > > Are you able to read/write from/to the scsi device ? > > > > If yes, Did you do "multipath -F" followed by "multipath". > > > > Is the multipathd running ? > > > > chandra > > On Thu, 2009-03-12 at 15:25 -0400, Thomas Witzel wrote: > >> My /sys/block looks like this: > >> ls /sys/block/ > >> dm-0 dm-3 dm-6 loop2 loop5 ram0 ram11 ram14 ram3 ram6 ram9 > >> sdc sdf sdi sdl > >> dm-1 dm-4 loop0 loop3 loop6 ram1 ram12 ram15 ram4 ram7 sda > >> sdd sdg sdj sdm > >> dm-2 dm-5 loop1 loop4 loop7 ram10 ram13 ram2 ram5 ram8 sdb > >> sde sdh sdk sr0 > >> > >> My /proc/partitions looks like this: > >> > >> major minor #blocks name > >> > >> 8 0 71041024 sda > >> 8 1 68099503 sda1 > >> 8 2 1 sda2 > >> 8 5 2939863 sda5 > >> 8 16 2134896640 sdb > >> 8 17 2134893883 sdb1 > >> 8 32 2134896640 sdc > >> 8 48 2134896640 sdd > >> 8 49 2134893883 sdd1 > >> 8 64 2134896640 sde > >> 8 80 1222794240 sdf > >> 8 96 20480 sdg > >> 8 112 2134896640 sdh > >> 8 128 2134896640 sdi > >> 8 129 2134893883 sdi1 > >> 8 144 2134896640 sdj > >> 8 160 2134896640 sdk > >> 8 161 2134893883 sdk1 > >> 8 176 1222794240 sdl > >> 8 192 20480 sdm > >> 254 0 2134896640 dm-0 > >> 254 1 2134896640 dm-1 > >> 254 2 2134896640 dm-2 > >> 254 3 2134896640 dm-3 > >> 254 4 1222794240 dm-4 > >> 254 5 20480 dm-5 > >> 254 6 20480 dm-6 > >> > >> All the sd's with a partition are working fine, I even created a test > >> file system at some point. I did load scsi_dh_rdac for testing at some > >> point, and that put the following into the syslog: > >> [10170.087743] sd 1:0:0:0: rdac: LUN 0 (owned) > >> [10170.089151] sd 1:0:0:1: rdac: LUN 1 (unowned) > >> [10170.089941] sd 1:0:0:2: rdac: LUN 2 (owned) > >> [10170.090719] sd 1:0:0:3: rdac: LUN 3 (unowned) > >> [10170.092112] sd 1:0:0:4: rdac: LUN 4 (owned) > >> [10170.092964] sd 1:0:1:0: rdac: LUN 0 (unowned) > >> [10170.093733] sd 1:0:1:1: rdac: LUN 1 (owned) > >> [10170.094507] sd 1:0:1:2: rdac: LUN 2 (unowned) > >> [10170.095261] sd 1:0:1:3: rdac: LUN 3 (owned) > >> [10170.096081] sd 1:0:1:4: rdac: LUN 4 (unowned) > >> [10170.096088] rdac: device handler registered > >> > >> Also loading the scsi_dh_rdac module seems to stop the [ 8996.087508] > >> Buffer I/O error on device dm-5, logical block 5 > >> [ 8996.087539] Buffer I/O error on device dm-5, logical block 6 > >> [ 8996.087548] Buffer I/O error on device dm-5, logical block 5112 > >> [ 8996.087553] Buffer I/O error on device dm-5, logical block 5112 > >> [ 8996.087601] Buffer I/O error on device dm-5, logical block 12 > >> [ 8996.087606] Buffer I/O error on device dm-5, logical block 13 > >> [ 8996.087609] Buffer I/O error on device dm-5, logical block 14 > >> [ 8996.087612] Buffer I/O error on device dm-5, logical block 15 > >> [ 8996.087615] Buffer I/O error on device dm-5, logical block 16 > >> [ 8996.087618] Buffer I/O error on device dm-5, logical block 17 > >> > >> type messages. > >> > >> On Thu, Mar 12, 2009 at 3:18 PM, Chandra Seetharaman > >> <sekharan@us.ibm.com> wrote: > >> > No, you don't have to change the distro. > >> > > >> > Do you see the disks ? (the #:#:#:# makes me think that the underlying > >> > block devices are not configured properly). IOW, do you the sd devices > >> > under /sys/block ? Are you able to read/write from/to the devices ? > >> > > >> > chandra > >> > > >> > On Thu, 2009-03-12 at 15:07 -0400, Thomas Witzel wrote: > >> >> Thanks Chandra, > >> >> > >> >> sorry for the out of order reply, still had the digest mode on. I > >> >> setup the config similar to what you said > >> >> > >> >> devices { > >> >> device { > >> >> vendor "DELL" > >> >> product "MD3000" > >> >> hardware_handler "1 rdac" > >> >> path_checker rdac > >> >> path_grouping_policy group_by_prio > >> >> prio_callout "/sbin/mpath_prio_rdac /dev/%n" > >> >> failback immediate > >> >> getuid_callout "/sbin/scsi_id -g -u /block/%n" > >> >> } > >> >> } > >> >> > >> >> > >> >> and I'm still getting: > >> >> sdh: checker msg is "directio checker reports path is down" > >> >> create: 360022190009773680000214749504780 , > >> >> [size=2.0T][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdb 8:16 [undef][ready] > >> >> \_ round-robin 0 [prio=0][undef] > >> >> \_ #:#:#:# sdh 8:112 [undef][faulty] > >> >> sdc: checker msg is "directio checker reports path is down" > >> >> create: 3600221900096f27d00000ab3495047fd , > >> >> [size=2.0T][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=0][undef] > >> >> \_ #:#:#:# sdc 8:32 [undef][faulty] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdi 8:128 [undef][ready] > >> >> sdj: checker msg is "directio checker reports path is down" > >> >> create: 360022190009773680000214a495047ce , > >> >> [size=2.0T][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdd 8:48 [undef][ready] > >> >> \_ round-robin 0 [prio=0][undef] > >> >> \_ #:#:#:# sdj 8:144 [undef][faulty] > >> >> sde: checker msg is "directio checker reports path is down" > >> >> create: 3600221900096f27d00000ab449504839 , > >> >> [size=2.0T][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=0][undef] > >> >> \_ #:#:#:# sde 8:64 [undef][faulty] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdk 8:160 [undef][ready] > >> >> sdl: checker msg is "directio checker reports path is down" > >> >> create: 360022190009773680000214c49504806 , > >> >> [size=1.1T][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdf 8:80 [undef][ready] > >> >> \_ round-robin 0 [prio=0][undef] > >> >> \_ #:#:#:# sdl 8:176 [undef][faulty] > >> >> create: 360022190009773680000000000000000 , > >> >> [size=20M][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdg 8:96 [undef][ready] > >> >> create: 3600221900096f27d0000000000000000 , > >> >> [size=20M][features=0][hwhandler=0] > >> >> \_ round-robin 0 [prio=1][undef] > >> >> \_ #:#:#:# sdm 8:192 [undef][ready] > >> >> libdevmapper: libdm-common.c(312): Created > >> >> /dev/mapper/360022190009773680000000000000000 > >> >> libdevmapper: libdm-common.c(312): Created > >> >> /dev/mapper/3600221900096f27d0000000000000000 > >> >> > >> >> Now this is not on RH or SLES, its an ubuntu 8.4 system. If absolutely > >> >> necessary I would consider switching the distribution, but right now > >> >> I'd prefer to stick with what all the other machines have. > >> >> > >> >> Thanks, Thomas > >> >> > >> >> -- > >> >> dm-devel mailing list > >> >> dm-devel@redhat.com > >> >> https://www.redhat.com/mailman/listinfo/dm-devel > >> > > >> > -- > >> > dm-devel mailing list > >> > dm-devel@redhat.com > >> > https://www.redhat.com/mailman/listinfo/dm-devel > >> > > >> > >> -- > >> dm-devel mailing list > >> dm-devel@redhat.com > >> https://www.redhat.com/mailman/listinfo/dm-devel > > > > -- > > dm-devel mailing list > > dm-devel@redhat.com > > https://www.redhat.com/mailman/listinfo/dm-devel > > ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2009-03-12 22:07 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-03-12 16:53 configure MD3000 Thomas Witzel 2009-03-12 18:33 ` Chandra Seetharaman 2009-03-12 19:07 ` Thomas Witzel 2009-03-12 19:18 ` Chandra Seetharaman 2009-03-12 19:25 ` Thomas Witzel 2009-03-12 19:31 ` Chandra Seetharaman 2009-03-12 19:38 ` Thomas Witzel 2009-03-12 19:40 ` Thomas Witzel 2009-03-12 22:07 ` Chandra Seetharaman 2009-03-12 22:06 ` Chandra Seetharaman
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.