/dev/md2 stopped after changing SAS controller

/dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

Hi,

I've been having very bad performance with an LSISAS2008 controller
attached to 8 WD Caviar Black 1TB disks.

So I swapped it out for an Apaptec RAID 6805 with the same disks and now
my /dev/md2 won't start.

	May 13 17:14:37 zenon kernel: md: sdh3 does not have a valid v1.2 superblock, no
	t importing!
	May 13 17:14:37 zenon kernel: md: md_import_device returned -22
	May 13 17:14:37 zenon kernel: md: sda3 does not have a valid v1.2 superblock, no
	t importing!
	May 13 17:14:37 zenon kernel: md: md_import_device returned -22
	May 13 17:14:37 zenon kernel: md: sdf3 does not have a valid v1.2 superblock, no
	t importing!

	Etc...

Both controllers are used without any configuration. I just use the
separate disks for soft raid.

Each disk is configured thus:

	Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
	255 heads, 63 sectors/track, 121601 cylinders
	Units = cylinders of 16065 * 512 = 8225280 bytes
	Sector size (logical/physical): 512 bytes / 512 bytes
	I/O size (minimum/optimal): 512 bytes / 512 bytes
	Disk identifier: 0x05022e04

	   Device Boot      Start         End      Blocks   Id  System
	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect

And my raid config used to be:

	Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] 
	md2 : active raid6 sdc3[0] sdd3[7] sdf3[6] sdb3[5] sdh3[4] sda3[3] sdg3[2] sde3[1]
		  5017138176 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
		  bitmap: 0/7 pages [0KB], 65536KB chunk

	md1 : active raid6 sdc2[0] sdd2[7] sdf2[8] sdb2[5] sdg2[4] sdh2[3] sda2[2] sde2[1]
		  841863168 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
		  bitmap: 2/2 pages [8KB], 65536KB chunk

	md0 : active raid1 sdc1[0] sdd1[7] sdf1[6] sdh1[5] sde1[4] sdb1[3] sda1[2] sdg1[1]
		  256896 blocks [8/8] [UUUUUUUU]
		  bitmap: 0/1 pages [0KB], 65536KB chunk

	unused devices: <none>


Going back to the LSISAS2008 controller makes /dev/md2 come back.

Any idea why the Adaptec wont let me use /dev/md2? Going into its
bios configuration menu I see a JBOD mode but it seems each disk has to
be "initialized" in order to be used in that mode.

Meanwhile the disks are used in "legacy" mode:

	May 13 17:09:57 zenon kernel: scsi 0:0:0:0: Direct-Access     Adaptec  6805 Legacy      V1.0 PQ: 0 ANSI: 2


Thanks,

Re: /dev/md2 stopped after changing SAS controller

[NeilBrown]

On Fri, 13 May 2011 22:29:01 +0200 Louis-David Mitterrand
<vindex+lists-linux-raid@apartia.org> wrote:

> Hi,
> 
> I've been having very bad performance with an LSISAS2008 controller
> attached to 8 WD Caviar Black 1TB disks.
> 
> So I swapped it out for an Apaptec RAID 6805 with the same disks and now
> my /dev/md2 won't start.
> 
> 	May 13 17:14:37 zenon kernel: md: sdh3 does not have a valid v1.2 superblock, no
> 	t importing!
> 	May 13 17:14:37 zenon kernel: md: md_import_device returned -22
> 	May 13 17:14:37 zenon kernel: md: sda3 does not have a valid v1.2 superblock, no
> 	t importing!
> 	May 13 17:14:37 zenon kernel: md: md_import_device returned -22
> 	May 13 17:14:37 zenon kernel: md: sdf3 does not have a valid v1.2 superblock, no
> 	t importing!
> 
> 	Etc...
> 
> Both controllers are used without any configuration. I just use the
> separate disks for soft raid.
> 
> Each disk is configured thus:
> 
> 	Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
> 	255 heads, 63 sectors/track, 121601 cylinders
> 	Units = cylinders of 16065 * 512 = 8225280 bytes
> 	Sector size (logical/physical): 512 bytes / 512 bytes
> 	I/O size (minimum/optimal): 512 bytes / 512 bytes
> 	Disk identifier: 0x05022e04
> 
> 	   Device Boot      Start         End      Blocks   Id  System
> 	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
> 	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
> 	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
> 

Is this the config reported with the old controller or with the new
controller?

Because my guess is that the new controller makes the devices look a little
bit smaller.  That would cause the kernel to reject them, but quite possibly
allow mdadm to think they look OK.

It would also explain why the first 2 partitions work fine and only the last
one is a problem.

If this were the case I would expect a message like:
         "%s: p%d size %llu extends beyond EOD

to appear during boot-up.


NeilBrown



> And my raid config used to be:
> 
> 	Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] 
> 	md2 : active raid6 sdc3[0] sdd3[7] sdf3[6] sdb3[5] sdh3[4] sda3[3] sdg3[2] sde3[1]
> 		  5017138176 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
> 		  bitmap: 0/7 pages [0KB], 65536KB chunk
> 
> 	md1 : active raid6 sdc2[0] sdd2[7] sdf2[8] sdb2[5] sdg2[4] sdh2[3] sda2[2] sde2[1]
> 		  841863168 blocks super 1.2 level 6, 512k chunk, algorithm 2 [8/8] [UUUUUUUU]
> 		  bitmap: 2/2 pages [8KB], 65536KB chunk
> 
> 	md0 : active raid1 sdc1[0] sdd1[7] sdf1[6] sdh1[5] sde1[4] sdb1[3] sda1[2] sdg1[1]
> 		  256896 blocks [8/8] [UUUUUUUU]
> 		  bitmap: 0/1 pages [0KB], 65536KB chunk
> 
> 	unused devices: <none>
> 
> 
> Going back to the LSISAS2008 controller makes /dev/md2 come back.
> 
> Any idea why the Adaptec wont let me use /dev/md2? Going into its
> bios configuration menu I see a JBOD mode but it seems each disk has to
> be "initialized" in order to be used in that mode.
> 
> Meanwhile the disks are used in "legacy" mode:
> 
> 	May 13 17:09:57 zenon kernel: scsi 0:0:0:0: Direct-Access     Adaptec  6805 Legacy      V1.0 PQ: 0 ANSI: 2
> 
> 
> Thanks,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: /dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

On Sat, May 14, 2011 at 07:33:47AM +1000, NeilBrown wrote:
> On Fri, 13 May 2011 22:29:01 +0200 Louis-David Mitterrand
> <vindex+lists-linux-raid@apartia.org> wrote:
> > Each disk is configured thus:
> > 
> > 	Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
> > 	255 heads, 63 sectors/track, 121601 cylinders
> > 	Units = cylinders of 16065 * 512 = 8225280 bytes
> > 	Sector size (logical/physical): 512 bytes / 512 bytes
> > 	I/O size (minimum/optimal): 512 bytes / 512 bytes
> > 	Disk identifier: 0x05022e04
> > 
> > 	   Device Boot      Start         End      Blocks   Id  System
> > 	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
> > 	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
> > 	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
> > 
> 
> Is this the config reported with the old controller or with the new
> controller?

Sorry for the confusion, the above was with the old (LSI) controller.
This is the disk geometry with the new (Adaptec) controller:

	Disk /dev/sda: 1000.1 GB, 1000104157184 bytes
	255 heads, 63 sectors/track, 121589 cylinders
	Units = cylinders of 16065 * 512 = 8225280 bytes
	Sector size (logical/physical): 512 bytes / 512 bytes
	I/O size (minimum/optimal): 512 bytes / 512 bytes
	Disk identifier: 0x6397e8f6

	   Device Boot      Start         End      Blocks   Id  System
	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect

Ah! 121589 vs 121601 cylinders...

> Because my guess is that the new controller makes the devices look a little
> bit smaller.  That would cause the kernel to reject them, but quite possibly
> allow mdadm to think they look OK.
> 
> It would also explain why the first 2 partitions work fine and only the last
> one is a problem.
> 
> If this were the case I would expect a message like:
>          "%s: p%d size %llu extends beyond EOD

Spot on:

	May 14 10:15:46 zenon kernel: sda: p3 size 1672382565 extends beyond EOD, trunca
ted

Is there any solution other than backuping my /dev/md2 somewhere with
the old controller and re-creating a (slightly) smaller one with the new
controller?

Thanks,

Re: /dev/md2 stopped after changing SAS controller

[CoolCold]

On Sat, May 14, 2011 at 12:21 PM, Louis-David Mitterrand
<vindex+lists-linux-raid@apartia.org> wrote:
> On Sat, May 14, 2011 at 07:33:47AM +1000, NeilBrown wrote:
>> On Fri, 13 May 2011 22:29:01 +0200 Louis-David Mitterrand
>> <vindex+lists-linux-raid@apartia.org> wrote:
>> > Each disk is configured thus:
>> >
>> >     Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
>> >     255 heads, 63 sectors/track, 121601 cylinders
>> >     Units = cylinders of 16065 * 512 = 8225280 bytes
>> >     Sector size (logical/physical): 512 bytes / 512 bytes
>> >     I/O size (minimum/optimal): 512 bytes / 512 bytes
>> >     Disk identifier: 0x05022e04
>> >
>> >        Device Boot      Start         End      Blocks   Id  System
>> >     /dev/sda1               1          32      257008+  fd  Linux raid autodetect
>> >     /dev/sda2              33       17500   140311710   fd  Linux raid autodetect
>> >     /dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
>> >
>>
>> Is this the config reported with the old controller or with the new
>> controller?
>
> Sorry for the confusion, the above was with the old (LSI) controller.
> This is the disk geometry with the new (Adaptec) controller:
>
>        Disk /dev/sda: 1000.1 GB, 1000104157184 bytes
>        255 heads, 63 sectors/track, 121589 cylinders
>        Units = cylinders of 16065 * 512 = 8225280 bytes
>        Sector size (logical/physical): 512 bytes / 512 bytes
>        I/O size (minimum/optimal): 512 bytes / 512 bytes
>        Disk identifier: 0x6397e8f6
>
>           Device Boot      Start         End      Blocks   Id  System
>        /dev/sda1               1          32      257008+  fd  Linux raid autodetect
>        /dev/sda2              33       17500   140311710   fd  Linux raid autodetect
>        /dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
>
> Ah! 121589 vs 121601 cylinders...
>
>> Because my guess is that the new controller makes the devices look a little
>> bit smaller.  That would cause the kernel to reject them, but quite possibly
>> allow mdadm to think they look OK.
>>
>> It would also explain why the first 2 partitions work fine and only the last
>> one is a problem.
>>
>> If this were the case I would expect a message like:
>>          "%s: p%d size %llu extends beyond EOD
>
> Spot on:
>
>        May 14 10:15:46 zenon kernel: sda: p3 size 1672382565 extends beyond EOD, trunca
> ted
>
> Is there any solution other than backuping my /dev/md2 somewhere with
> the old controller and re-creating a (slightly) smaller one with the new
> controller?
This is just one more live example for partitions vs disks setups.

>
> Thanks,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Best regards,
[COOLCOLD-RIPN]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: /dev/md2 stopped after changing SAS controller

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

On Sat, 14 May 2011 12:27:45 +0400
CoolCold <coolthecold@gmail.com> wrote:

> This is just one more live example for partitions vs disks setups.

Well, they did use partitions, but what do you do when the new RAID controller
bites off a whopping 100 million bytes from the end of your disk? I use
partitions too, and do leave some space at the end for cases like this, but
what I leave is just 8 MB (enough for the Gigabyte motherboards HPA), not
100+. Would you suggest setting aside 500MB, 1GB just in case?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: /dev/md2 stopped after changing SAS controller

[CoolCold]

On Sat, May 14, 2011 at 12:36 PM, Roman Mamedov <rm@romanrm.ru> wrote:
> On Sat, 14 May 2011 12:27:45 +0400
> CoolCold <coolthecold@gmail.com> wrote:
>
>> This is just one more live example for partitions vs disks setups.
>
> Well, they did use partitions, but what do you do when the new RAID controller
> bites off a whopping 100 million bytes from the end of your disk? I use
> partitions too, and do leave some space at the end for cases like this, but
> what I leave is just 8 MB (enough for the Gigabyte motherboards HPA), not
> 100+. Would you suggest setting aside 500MB, 1GB just in case?
I've always (well, after switching to partitions) used something near
80mb, for the last servers using ~ 160mb
root@kappa2:~# parted /dev/cciss/c0d1 print free
Model: Compaq Smart Array (cpqarray)
Disk /dev/cciss/c0d1: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      16.4kB  1000GB  1000GB  primary
        1000GB  1000GB  168MB            Free Space


>
> --
> With respect,
> Roman
>



-- 
Best regards,
[COOLCOLD-RIPN]

Re: /dev/md2 stopped after changing SAS controller

[NeilBrown]

On Sat, 14 May 2011 10:21:59 +0200 Louis-David Mitterrand
<vindex+lists-linux-raid@apartia.org> wrote:

> On Sat, May 14, 2011 at 07:33:47AM +1000, NeilBrown wrote:
> > On Fri, 13 May 2011 22:29:01 +0200 Louis-David Mitterrand
> > <vindex+lists-linux-raid@apartia.org> wrote:
> > > Each disk is configured thus:
> > > 
> > > 	Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
> > > 	255 heads, 63 sectors/track, 121601 cylinders
> > > 	Units = cylinders of 16065 * 512 = 8225280 bytes
> > > 	Sector size (logical/physical): 512 bytes / 512 bytes
> > > 	I/O size (minimum/optimal): 512 bytes / 512 bytes
> > > 	Disk identifier: 0x05022e04
> > > 
> > > 	   Device Boot      Start         End      Blocks   Id  System
> > > 	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
> > > 	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
> > > 	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
> > > 
> > 
> > Is this the config reported with the old controller or with the new
> > controller?
> 
> Sorry for the confusion, the above was with the old (LSI) controller.
> This is the disk geometry with the new (Adaptec) controller:
> 
> 	Disk /dev/sda: 1000.1 GB, 1000104157184 bytes
> 	255 heads, 63 sectors/track, 121589 cylinders
> 	Units = cylinders of 16065 * 512 = 8225280 bytes
> 	Sector size (logical/physical): 512 bytes / 512 bytes
> 	I/O size (minimum/optimal): 512 bytes / 512 bytes
> 	Disk identifier: 0x6397e8f6
> 
> 	   Device Boot      Start         End      Blocks   Id  System
> 	/dev/sda1               1          32      257008+  fd  Linux raid autodetect
> 	/dev/sda2              33       17500   140311710   fd  Linux raid autodetect
> 	/dev/sda3           17501      121601   836191282+  fd  Linux raid autodetect
> 
> Ah! 121589 vs 121601 cylinders...
> 
> > Because my guess is that the new controller makes the devices look a little
> > bit smaller.  That would cause the kernel to reject them, but quite possibly
> > allow mdadm to think they look OK.
> > 
> > It would also explain why the first 2 partitions work fine and only the last
> > one is a problem.
> > 
> > If this were the case I would expect a message like:
> >          "%s: p%d size %llu extends beyond EOD
> 
> Spot on:
> 
> 	May 14 10:15:46 zenon kernel: sda: p3 size 1672382565 extends beyond EOD, trunca
> ted
> 
> Is there any solution other than backuping my /dev/md2 somewhere with
> the old controller and re-creating a (slightly) smaller one with the new
> controller?

If the filesystem is ext3, then with the old controller you could
resize2fs the filesystem to be several megabytes smaller - aim to over-shrink
rather than under-shrink.  Say 2 Meg per devices, so 12Mmeg ... so maybe
20Meg for safety... or even double that again.

Then use mdadm to make the  array smaller (mdadm --grow --size=whatever).
Then 'fsck -f' the filesystem to make sure it is still OK.  If you messed up
you can still make the size bigger again with mdadm.  The 'size' is the
per-device size, so 1/6 of the array size, but a multiple of the chunk size.

Then switch back to the new controller and assemble with --update=devicesize
so:
   mdadm -A /dev/md2 --update=devicesize /dev/sd[abcdefg]3

and it should all work.

If you don't have ext3, then you probably need to backup and restore - I
don't think other filesystems support shrinking, but I could be wrong.

NeilBrown

Re: /dev/md2 stopped after changing SAS controller

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Sat, 14 May 2011 21:58:21 +1000
NeilBrown <neilb@suse.de> wrote:

> If you don't have ext3, then you probably need to backup and restore - I
> don't think other filesystems support shrinking, but I could be wrong.

Well, ext4 does support shrinking as well.

And btrfs, it has both online shrinking and growing.

XFS, JFS can only be grown, not shrinked.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: /dev/md2 stopped after changing SAS controller

[David Brown]

On 14/05/11 14:06, Roman Mamedov wrote:
> On Sat, 14 May 2011 21:58:21 +1000
> NeilBrown<neilb@suse.de>  wrote:
>
>> If you don't have ext3, then you probably need to backup and restore - I
>> don't think other filesystems support shrinking, but I could be wrong.
>
> Well, ext4 does support shrinking as well.
>
> And btrfs, it has both online shrinking and growing.
>
> XFS, JFS can only be grown, not shrinked.
>

reiserfs3 can also be shrunk - you can even do it while mounted, if you 
want.  (resize2fs will grow a mounted extX system, but it must be 
unmounted for a shrink.)

Re: /dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

On Sat, May 14, 2011 at 09:58:21PM +1000, NeilBrown wrote:
> 
> If the filesystem is ext3, then with the old controller you could
> resize2fs the filesystem to be several megabytes smaller - aim to over-shrink
> rather than under-shrink.  Say 2 Meg per devices, so 12Mmeg ... so maybe
> 20Meg for safety... or even double that again.
> 
> Then use mdadm to make the  array smaller (mdadm --grow --size=whatever).
> Then 'fsck -f' the filesystem to make sure it is still OK.  If you messed up
> you can still make the size bigger again with mdadm.  The 'size' is the
> per-device size, so 1/6 of the array size, but a multiple of the chunk size.
> 
> Then switch back to the new controller and assemble with --update=devicesize
> so:
>    mdadm -A /dev/md2 --update=devicesize /dev/sd[abcdefg]3
> 
> and it should all work.
> 
> If you don't have ext3, then you probably need to backup and restore - I
> don't think other filesystems support shrinking, but I could be wrong.

Thanks for your help.

Alas /dev/md2 is xfs so dump/restore it will be...

For next time, what is the prudent disk space to reserve at the end for
these greedy controllers?

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/13/2011 3:29 PM, Louis-David Mitterrand wrote:

> I've been having very bad performance with an LSISAS2008 controller
> attached to 8 WD Caviar Black 1TB disks.

What is the nature of this 'very bad performance'?  Did you recently
upgrade kernel/driver and run into a regression?  If not, either the HBA
has developed a circuit defect, or the problem lies elsewhere, either a
backplane/cabling issue, or a drive going south.

Do you have any log entries showing problems with the LSI HBA?  Drive
errors?  What does smartctl tell you about each drive?

Usually when HBAs fail, they fail, period.  They don't typically show
continuously degrading performance until totally failing.

I'll be interested to know if swapping the LSI for the Adaptec fixes the
'very bad performance' issue.

-- 
Stan

Re: /dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

On Sat, May 14, 2011 at 10:03:44AM -0500, Stan Hoeppner wrote:
> On 5/13/2011 3:29 PM, Louis-David Mitterrand wrote:
> 
> > I've been having very bad performance with an LSISAS2008 controller
> > attached to 8 WD Caviar Black 1TB disks.
> 
> What is the nature of this 'very bad performance'?  Did you recently
> upgrade kernel/driver and run into a regression?  If not, either the HBA
> has developed a circuit defect, or the problem lies elsewhere, either a
> backplane/cabling issue, or a drive going south.

The performance problem can be clearly attributed to that model of the
LSI card because with the Adaptec card and the same cables and disks the
problem disappears.

The LSI card has always performed very poorly with all kernel versions I
tried, including the very latest, and the with most recent firmware
revisions from LSI. 

After bricking the card while trying to change its firmware to an "IT"
version (Initiatior Target i.e. non-raid) I had Dell send me a new one
on warranty and the performance problem remained.

> Do you have any log entries showing problems with the LSI HBA?  Drive
> errors?  What does smartctl tell you about each drive?

No error. Just plain sloth.

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/14/2011 10:18 AM, Louis-David Mitterrand wrote:
> On Sat, May 14, 2011 at 10:03:44AM -0500, Stan Hoeppner wrote:
>> On 5/13/2011 3:29 PM, Louis-David Mitterrand wrote:
>>
>>> I've been having very bad performance with an LSISAS2008 controller
>>> attached to 8 WD Caviar Black 1TB disks.
>>
>> What is the nature of this 'very bad performance'?  Did you recently
>> upgrade kernel/driver and run into a regression?  If not, either the HBA
>> has developed a circuit defect, or the problem lies elsewhere, either a
>> backplane/cabling issue, or a drive going south.
> 
> The performance problem can be clearly attributed to that model of the
> LSI card because with the Adaptec card and the same cables and disks the
> problem disappears.

How have you ascertained this already?  As of 2 hours ago you still
don't have your filesystem(s) back up and running after the HBA swap.  I
would assume a backup restore of a few TB would take more than a few
hours...

> The LSI card has always performed very poorly with all kernel versions I
> tried, including the very latest, and the with most recent firmware
> revisions from LSI.

LSI HBAs are usually held in pretty high regard.  Which specific part#
are we talking about?  Is this an OEM Dell HBA?

> After bricking the card while trying to change its firmware to an "IT"
> version (Initiatior Target i.e. non-raid) I had Dell send me a new one
> on warranty and the performance problem remained.

You still haven't described the nature of the performance problem.  This
is a technical mailing list after all.

>> Do you have any log entries showing problems with the LSI HBA?  Drive
>> errors?  What does smartctl tell you about each drive?
> 
> No error. Just plain sloth.

Interesting technical description.  Now we have yet another post in the
Interwebs archives stating a certain piece of hardware is junk without
any real explanation as to the nature of the problem...

-- 
Stan

Re: /dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

On Sat, May 14, 2011 at 01:19:44PM -0500, Stan Hoeppner wrote:
> On 5/14/2011 10:18 AM, Louis-David Mitterrand wrote:
> > On Sat, May 14, 2011 at 10:03:44AM -0500, Stan Hoeppner wrote:
> >> On 5/13/2011 3:29 PM, Louis-David Mitterrand wrote:
> >>
> >>> I've been having very bad performance with an LSISAS2008 controller
> >>> attached to 8 WD Caviar Black 1TB disks.
> >>
> >> What is the nature of this 'very bad performance'?  Did you recently
> >> upgrade kernel/driver and run into a regression?  If not, either the HBA
> >> has developed a circuit defect, or the problem lies elsewhere, either a
> >> backplane/cabling issue, or a drive going south.
> > 
> > The performance problem can be clearly attributed to that model of the
> > LSI card because with the Adaptec card and the same cables and disks the
> > problem disappears.
> 
> How have you ascertained this already?  As of 2 hours ago you still
> don't have your filesystem(s) back up and running after the HBA swap.  I
> would assume a backup restore of a few TB would take more than a few
> hours...

My system partiton is fine, thank you very much. Only my data partition
is affected by the geometry inconsistency. And the performance
difference was quickly obvious: 22 seconds to untar kernel sources vs 20
minutes.

> > The LSI card has always performed very poorly with all kernel versions I
> > tried, including the very latest, and the with most recent firmware
> > revisions from LSI.
> 
> LSI HBAs are usually held in pretty high regard.  Which specific part#
> are we talking about?  Is this an OEM Dell HBA?

In high regard by whom I wonder. I've found them generally crappy. My
last one (MegaRAID SAS 1078) wouldn't let the disks appear to the OS
unless I created a raid0 vdisk for each of them. How lame is that? And
the LSISAS2008 shipped with a firmware so crappy that a simple lilo run
would bump 2 out of my 8 disk raid6. Dont even get me started on Dell
with their overpriced parts and bloated 600MB OMSA suite just to do a
simple firmware upgrade.

And all that for what? Just to provide some lameass raid features I
don't need or use. I loathe hardware raid, would never trust it with my
data. As far as I'm concerned all these useless "features" stand in the
way of performance and simplicity.

But it seems finding a simple SAS/Sata expander with no raid and that
just gives me my fucking disks whole is too much to ask. They are either
out of stock, EOL, wrong plugs or not available in my area.

> > After bricking the card while trying to change its firmware to an "IT"
> > version (Initiatior Target i.e. non-raid) I had Dell send me a new one
> > on warranty and the performance problem remained.
> 
> You still haven't described the nature of the performance problem.  This
> is a technical mailing list after all.

Try 2 orders of magnitude time difference when rm'ing a kernel tree. No
errors.

> >> Do you have any log entries showing problems with the LSI HBA?  Drive
> >> errors?  What does smartctl tell you about each drive?
> > 
> > No error. Just plain sloth.
> 
> Interesting technical description.  Now we have yet another post in the
> Interwebs archives stating a certain piece of hardware is junk without
> any real explanation as to the nature of the problem...

It's junk and bloated with useless features.

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/14/2011 2:15 PM, Louis-David Mitterrand wrote:

> My system partiton is fine, thank you very much. Only my data partition
> is affected by the geometry inconsistency. And the performance
> difference was quickly obvious: 22 seconds to untar kernel sources vs 20
> minutes.

If there are no errors then that's not a controller issue but a problem
with barrier and cache configuration.  This exact issue came up on the
XFS list not 2 months ago.  Having barriers enabled on a BBWC card was
the OP's problem.

And, BTW, you should enable the XFS delaylog mount option if you haven't
already.  This will drop big metadata operation run time by another
order of magnitude.

If you want a pure SAS/SATA HBA without fakeRAID, this may be an option
worth looking at:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358
http://www.supermicro.com/manuals/other/AOC-SASLP-MV8.pdf

Simple JBOD only HBA, no fakeRAID.  Uses a Marvell 88SE6480 chip, 8
SAS/SATA ports via two SFF8087.  $110 USD.  I've heard minor rumblings
WRT the mvsas driver though I don't recall specifics.  The board itself
is good quality, as with most things SuperMicro.

-- 
Stan

Re: /dev/md2 stopped after changing SAS controller

[Mikael Abrahamsson]

On Sat, 14 May 2011, Stan Hoeppner wrote:

>
> http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358
> http://www.supermicro.com/manuals/other/AOC-SASLP-MV8.pdf
>
> Simple JBOD only HBA, no fakeRAID.  Uses a Marvell 88SE6480 chip, 8
> SAS/SATA ports via two SFF8087.  $110 USD.  I've heard minor rumblings
> WRT the mvsas driver though I don't recall specifics.  The board itself
> is good quality, as with most things SuperMicro.

MINOR!??? I've had this card for 1.5 years, it still not usable (as far as 
I can discern) as of 2.6.38. Do NOT buy it. Please stop recommending this 
card for Linux use unless it's to your enemies.

Buy a 1068E (3081E) based card, they've been working fine for a long time.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: /dev/md2 stopped after changing SAS controller

[Joe Landman]

On 05/15/2011 03:26 AM, Mikael Abrahamsson wrote:
> On Sat, 14 May 2011, Stan Hoeppner wrote:
>
>>
>> http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358
>> http://www.supermicro.com/manuals/other/AOC-SASLP-MV8.pdf
>>
>> Simple JBOD only HBA, no fakeRAID. Uses a Marvell 88SE6480 chip, 8
>> SAS/SATA ports via two SFF8087. $110 USD. I've heard minor rumblings
>> WRT the mvsas driver though I don't recall specifics. The board itself
>> is good quality, as with most things SuperMicro.
>
> MINOR!??? I've had this card for 1.5 years, it still not usable (as far
> as I can discern) as of 2.6.38. Do NOT buy it. Please stop recommending
> this card for Linux use unless it's to your enemies.

+1 to Mikael

We have a number of these.  They are pretty much useless.  We *do not 
recommend them for any purpose whatsoever*.  We got them to test, and 
they were and are wastes of money.

Again, this is why making sure that the bona fides of those making 
recommendations don't begin and end with a google search ...

>
> Buy a 1068E (3081E) based card, they've been working fine for a long time.
>

+1 on that.  We like the 3081E, and the 9211-8i for these purposes. 
Both work quite well.

Again, ignore these cards, they will absorb your money and your time.



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

Re: /dev/md2 stopped after changing SAS controller

[Louis-David Mitterrand]

On Sun, May 15, 2011 at 10:29:11AM -0400, Joe Landman wrote:
> 
> +1 to Mikael
> 
> We have a number of these.  They are pretty much useless.  We *do
> not recommend them for any purpose whatsoever*.  We got them to
> test, and they were and are wastes of money.
> 
> Again, this is why making sure that the bona fides of those making
> recommendations don't begin and end with a google search ...
> 
> >
> >Buy a 1068E (3081E) based card, they've been working fine for a long time.
> >
> 
> +1 on that.  We like the 3081E, and the 9211-8i for these purposes.
> Both work quite well.
> 
> Again, ignore these cards, they will absorb your money and your time.

Someone pointed me to ARECA non-raid controllers: 

http://www.areca.com.tw/products/sasnoneraid3g.htm

Does anyone have good or bad experiences with these?

Thanks,

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/15/2011 2:26 AM, Mikael Abrahamsson wrote:
> On Sat, 14 May 2011, Stan Hoeppner wrote:

>> http://www.newegg.com/Product/Product.aspx?Item=N82E16816101358
>> http://www.supermicro.com/manuals/other/AOC-SASLP-MV8.pdf
>>
>> Simple JBOD only HBA, no fakeRAID.  Uses a Marvell 88SE6480 chip, 8
>> SAS/SATA ports via two SFF8087.  $110 USD.  I've heard minor rumblings
>> WRT the mvsas driver though I don't recall specifics.  The board itself
>> is good quality, as with most things SuperMicro.
> 
> MINOR!??? I've had this card for 1.5 years, it still not usable (as far
> as I can discern) as of 2.6.38. Do NOT buy it. Please stop recommending
> this card for Linux use unless it's to your enemies.

Now you get the idea Mikal.  ;) If the OP is so convinced that LSI
sucks, with ample evidence to the contrary, let him try something that
*everyone* knows sucks.

-- 
Stan

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/15/2011 9:29 AM, Joe Landman wrote:

> Again, this is why making sure that the bona fides of those making
> recommendations don't begin and end with a google search ...

It would benefit some list members to read entire threads before making
comments such as this, demonstrating a total lack of clue, pettiness,
and a grudge holding personality.

Someone must have wounded your Id deeply for you to take a swipe at that
person in every post you make to this list...

-- 
Stan

Re: /dev/md2 stopped after changing SAS controller

[Stan Hoeppner]

On 5/15/2011 9:37 AM, Louis-David Mitterrand wrote:

> Someone pointed me to ARECA non-raid controllers: 
> 
> http://www.areca.com.tw/products/sasnoneraid3g.htm
> 
> Does anyone have good or bad experiences with these?

http://www.newegg.com/Product/Product.aspx?Item=N82E16816151061

Read the 3 reviews.  Probably best to stay away from the Areca JBOD
HBAs.  All 4 use a Marvell chip.  Many NewEgg reviews are valuable, as
in this case.

Lest anyone denigrate the validity of quoting NewEgg due to their
"perceived customer base", note that today's NewEgg ships plenty of
mid/upper range gear (see links below) into corporations and
universities, including blade chassis/blades, FC and converged switches,
FC disk arrays, LTO libraries, etc.  They now have a business division
that competes to a degree with the likes of CDW et al.

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&Description=hewlett%20packard&bop=And&Order=PRICED&PageSize=100

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&Description=ibm&bop=And&Order=PRICED&PageSize=100

-- 
Stan

RE: /dev/md2 stopped after changing SAS controller

[Leslie Rhorer]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
> owner@vger.kernel.org] On Behalf Of Stan Hoeppner
> Sent: Monday, May 16, 2011 1:13 AM
> To: linux-raid@vger.kernel.org
> Subject: Re: /dev/md2 stopped after changing SAS controller
> 
> On 5/15/2011 9:37 AM, Louis-David Mitterrand wrote:
> 
> > Someone pointed me to ARECA non-raid controllers:
> >
> > http://www.areca.com.tw/products/sasnoneraid3g.htm
> >
> > Does anyone have good or bad experiences with these?
> 
> http://www.newegg.com/Product/Product.aspx?Item=N82E16816151061
> 
> Read the 3 reviews.  Probably best to stay away from the Areca JBOD
> HBAs.  All 4 use a Marvell chip.  Many NewEgg reviews are valuable, as
> in this case.
> 
> Lest anyone denigrate the validity of quoting NewEgg due to their
> "perceived customer base", note that today's NewEgg ships plenty of
> mid/upper range gear (see links below) into corporations and
> universities, including blade chassis/blades, FC and converged switches,
> FC disk arrays, LTO libraries, etc.  They now have a business division
> that competes to a degree with the likes of CDW et al.

	Well, yes, but the reviews are sometimes a different matter.  The
best I can advise (and this is true for any review on any site) is to look
closely at the tone and clarity of each review.  Many are clearly written by
fools who haven't a clue.  Others are better written, but the level of
expertise behind the review may not be as high as one might first think.
Basically, no matter where one might read or see it, I recommend one take
all reviews with a grain of salt.

	That said, I have seen some useful reviews on NewEgg.