Direct disk access on IBM Server

Direct disk access on IBM Server

[David Brown]

I have recently got an IBM x3650 M3 server, which has a "Serveraid 
M5014" raid controller.  Booting from a Linux CD (system rescue CD) and 
running lspci identifies this raid controller as:

LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)

The controller works fine for hardware raid - I can open its bios setup 
utility, and set up a RAID5 (or whatever) with the disks I have.  The OS 
then just sees a single virtual disk.

But I would like direct access to the sata drives - I want to set up 
mdadm raid, under my own control.  As far as I can see, there is no way 
to put this controller into "JBOD" or "direct access" mode of any sort.

Does anyone here have experience with this card, or can give me any hints?

The only idea I have at the moment is to put each disk within its own 
single-disk RAID 0 set, but then I don't get sata hot-swap 
functionality, SMART, hddtemp, etc.

Thanks for any clues or hints,

David

Re: Direct disk access on IBM Server

[Mathias Burén]

On 19 April 2011 14:21, David Brown <david@westcontrol.com> wrote:
> I have recently got an IBM x3650 M3 server, which has a "Serveraid M5014"
> raid controller.  Booting from a Linux CD (system rescue CD) and running
> lspci identifies this raid controller as:
>
> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>
> The controller works fine for hardware raid - I can open its bios setup
> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
> then just sees a single virtual disk.
>
> But I would like direct access to the sata drives - I want to set up mdadm
> raid, under my own control.  As far as I can see, there is no way to put
> this controller into "JBOD" or "direct access" mode of any sort.
>
> Does anyone here have experience with this card, or can give me any hints?
>
> The only idea I have at the moment is to put each disk within its own
> single-disk RAID 0 set, but then I don't get sata hot-swap functionality,
> SMART, hddtemp, etc.
>
> Thanks for any clues or hints,
>
> David
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Regarding SMART, could you try (after loading the appropriate
megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
is a number between 0 and 31 IIRC (depending on the HDD in the array).

Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[David Brown]

On 19/04/2011 15:25, Mathias Burén wrote:
> On 19 April 2011 14:21, David Brown<david@westcontrol.com>  wrote:
>> I have recently got an IBM x3650 M3 server, which has a "Serveraid M5014"
>> raid controller.  Booting from a Linux CD (system rescue CD) and running
>> lspci identifies this raid controller as:
>>
>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>
>> The controller works fine for hardware raid - I can open its bios setup
>> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
>> then just sees a single virtual disk.
>>
>> But I would like direct access to the sata drives - I want to set up mdadm
>> raid, under my own control.  As far as I can see, there is no way to put
>> this controller into "JBOD" or "direct access" mode of any sort.
>>
>> Does anyone here have experience with this card, or can give me any hints?
>>
>> The only idea I have at the moment is to put each disk within its own
>> single-disk RAID 0 set, but then I don't get sata hot-swap functionality,
>> SMART, hddtemp, etc.
>>
>> Thanks for any clues or hints,
>>
>> David
>>
>
> Regarding SMART, could you try (after loading the appropriate
> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
> is a number between 0 and 31 IIRC (depending on the HDD in the array).
>

Are you sure about the syntax for that command?  Trying that with "0" 
for $I just gives me "megaraid,0: No such file or directory".  As far as 
I can see, the megaraid module is loaded (lsmod shows "megaraid_sas" in 
the modules list).

Thanks anyway,

David


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[Mathias Burén]

On 19 April 2011 15:04, David Brown <david@westcontrol.com> wrote:
> On 19/04/2011 15:25, Mathias Burén wrote:
>>
>> On 19 April 2011 14:21, David Brown<david@westcontrol.com>  wrote:
>>>
>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid M5014"
>>> raid controller.  Booting from a Linux CD (system rescue CD) and running
>>> lspci identifies this raid controller as:
>>>
>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>
>>> The controller works fine for hardware raid - I can open its bios setup
>>> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
>>> then just sees a single virtual disk.
>>>
>>> But I would like direct access to the sata drives - I want to set up
>>> mdadm
>>> raid, under my own control.  As far as I can see, there is no way to put
>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>
>>> Does anyone here have experience with this card, or can give me any
>>> hints?
>>>
>>> The only idea I have at the moment is to put each disk within its own
>>> single-disk RAID 0 set, but then I don't get sata hot-swap functionality,
>>> SMART, hddtemp, etc.
>>>
>>> Thanks for any clues or hints,
>>>
>>> David
>>>
>>
>> Regarding SMART, could you try (after loading the appropriate
>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
>> is a number between 0 and 31 IIRC (depending on the HDD in the array).
>>
>
> Are you sure about the syntax for that command?  Trying that with "0" for $I
> just gives me "megaraid,0: No such file or directory".  As far as I can see,
> the megaraid module is loaded (lsmod shows "megaraid_sas" in the modules
> list).
>
> Thanks anyway,
>
> David
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

From the smartctl man page:

         Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
controllers, use syntax such as:
              smartctl -a -d megaraid,2 /dev/sda
              smartctl -a -d megaraid,0 /dev/sdb
              where  in the argument megaraid,N, the integer N is the
physical disk number within the MegaRAID controller.  This interface
will also work for Dell PERC controllers.  The follow‐
              ing /dev/XXX entry must exist:
              For PERC2/3/4 controllers: /dev/megadev0
              For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node

Maybe you can experiment some with that. Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[David Brown]

On 19/04/2011 16:07, Mathias Burén wrote:
> On 19 April 2011 15:04, David Brown<david@westcontrol.com>  wrote:
>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>
>>> On 19 April 2011 14:21, David Brown<david@westcontrol.com>    wrote:
>>>>
>>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid M5014"
>>>> raid controller.  Booting from a Linux CD (system rescue CD) and running
>>>> lspci identifies this raid controller as:
>>>>
>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>
>>>> The controller works fine for hardware raid - I can open its bios setup
>>>> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
>>>> then just sees a single virtual disk.
>>>>
>>>> But I would like direct access to the sata drives - I want to set up
>>>> mdadm
>>>> raid, under my own control.  As far as I can see, there is no way to put
>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>
>>>> Does anyone here have experience with this card, or can give me any
>>>> hints?
>>>>
>>>> The only idea I have at the moment is to put each disk within its own
>>>> single-disk RAID 0 set, but then I don't get sata hot-swap functionality,
>>>> SMART, hddtemp, etc.
>>>>
>>>> Thanks for any clues or hints,
>>>>
>>>> David
>>>>
>>>
>>> Regarding SMART, could you try (after loading the appropriate
>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
>>> is a number between 0 and 31 IIRC (depending on the HDD in the array).
>>>
>>
>> Are you sure about the syntax for that command?  Trying that with "0" for $I
>> just gives me "megaraid,0: No such file or directory".  As far as I can see,
>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the modules
>> list).
>>
>> Thanks anyway,
>>
>> David
>>
>>
>
>  From the smartctl man page:
>
>           Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
> controllers, use syntax such as:
>                smartctl -a -d megaraid,2 /dev/sda
>                smartctl -a -d megaraid,0 /dev/sdb
>                where  in the argument megaraid,N, the integer N is the
> physical disk number within the MegaRAID controller.  This interface
> will also work for Dell PERC controllers.  The follow‐
>                ing /dev/XXX entry must exist:
>                For PERC2/3/4 controllers: /dev/megadev0
>                For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node
>
> Maybe you can experiment some with that. Regards,
> M

Ah, it was "smartctl" - you first wrote "hdparm".  I should have thought 
of smartctl myself.

Still no luck as yet - the smartctl on my test installation (Centos 5.6) 
doesn't seem to support megaraid devices.  But maybe that's just because 
it is an older version of smartctl - Centos 5.6 is not exactly "bleeding 
edge".


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[Mathias Burén]

On 19 April 2011 16:12, David Brown <david@westcontrol.com> wrote:
> On 19/04/2011 16:07, Mathias Burén wrote:
>>
>> On 19 April 2011 15:04, David Brown<david@westcontrol.com>  wrote:
>>>
>>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>>
>>>> On 19 April 2011 14:21, David Brown<david@westcontrol.com>    wrote:
>>>>>
>>>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid
>>>>> M5014"
>>>>> raid controller.  Booting from a Linux CD (system rescue CD) and
>>>>> running
>>>>> lspci identifies this raid controller as:
>>>>>
>>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>>
>>>>> The controller works fine for hardware raid - I can open its bios setup
>>>>> utility, and set up a RAID5 (or whatever) with the disks I have.  The
>>>>> OS
>>>>> then just sees a single virtual disk.
>>>>>
>>>>> But I would like direct access to the sata drives - I want to set up
>>>>> mdadm
>>>>> raid, under my own control.  As far as I can see, there is no way to
>>>>> put
>>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>>
>>>>> Does anyone here have experience with this card, or can give me any
>>>>> hints?
>>>>>
>>>>> The only idea I have at the moment is to put each disk within its own
>>>>> single-disk RAID 0 set, but then I don't get sata hot-swap
>>>>> functionality,
>>>>> SMART, hddtemp, etc.
>>>>>
>>>>> Thanks for any clues or hints,
>>>>>
>>>>> David
>>>>>
>>>>
>>>> Regarding SMART, could you try (after loading the appropriate
>>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
>>>> is a number between 0 and 31 IIRC (depending on the HDD in the array).
>>>>
>>>
>>> Are you sure about the syntax for that command?  Trying that with "0" for
>>> $I
>>> just gives me "megaraid,0: No such file or directory".  As far as I can
>>> see,
>>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the modules
>>> list).
>>>
>>> Thanks anyway,
>>>
>>> David
>>>
>>>
>>
>>  From the smartctl man page:
>>
>>          Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
>> controllers, use syntax such as:
>>               smartctl -a -d megaraid,2 /dev/sda
>>               smartctl -a -d megaraid,0 /dev/sdb
>>               where  in the argument megaraid,N, the integer N is the
>> physical disk number within the MegaRAID controller.  This interface
>> will also work for Dell PERC controllers.  The follow‐
>>               ing /dev/XXX entry must exist:
>>               For PERC2/3/4 controllers: /dev/megadev0
>>               For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node
>>
>> Maybe you can experiment some with that. Regards,
>> M
>
> Ah, it was "smartctl" - you first wrote "hdparm".  I should have thought of
> smartctl myself.
>
> Still no luck as yet - the smartctl on my test installation (Centos 5.6)
> doesn't seem to support megaraid devices.  But maybe that's just because it
> is an older version of smartctl - Centos 5.6 is not exactly "bleeding edge".
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Opps, sorry! I blame the medicine. You could try compile a version of
smartctl yourself if you have the proper packages installed.

Regards,
M
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[Stan Hoeppner]

David Brown put forth on 4/19/2011 8:21 AM:
> I have recently got an IBM x3650 M3 server, which has a "Serveraid
> M5014" raid controller.  Booting from a Linux CD (system rescue CD) and
> running lspci identifies this raid controller as:
> 
> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
> 
> The controller works fine for hardware raid - I can open its bios setup
> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
> then just sees a single virtual disk.
> 
> But I would like direct access to the sata drives - I want to set up
> mdadm raid, under my own control.  As far as I can see, there is no way
> to put this controller into "JBOD" or "direct access" mode of any sort.

FYI, the ServeRAID 5014 is a good quality real hardware RAID card w/

PCIe 2.0 x8 interface
800 MHz LSI PowerPC RAID on Chip ASIC
256MB DDRII cache RAM
8 x 6Gb/s SAS ports via 2 SFF8087
32 drives maximum, 16 drives max per virtual disk (RAID group)

If your card has the RAID6 feature key installed, mdraid will likely
gain you little, if anything, over the card's inbuilt features.

> Does anyone here have experience with this card, or can give me any hints?

I can point you to the documentation for the 5014:

http://www.redbooks.ibm.com/technotes/tips0054.pdf

ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_doc_sraidmr_1st-ed_5014-5015_quick-install.pdf

ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_doc_sraidmr_1st-ed_5014-5015_user-guide.pdf


Dave Chinner of Red Hat, one of the lead XFS developers, runs an 8 core
test rig with a 512MB LSI card nearly identical to this ServeRAID 5014,
with something like 16 drives, using mdraid.  The method he had to use
to get mdraid working was putting each drive in its own RAID0 group
(virtual disk).  He claimed the onboard cache RAM was still enabled for
writes, I'm not sure about reads.

For high load parallel XFS operations--think severe multiuser
multithreaded test workloads designed to hammer XFS and force bugs to
surface--he stated mdraid has a performance advantage vs the onboard
RAID ASIC.  IIRC at lower disk counts and/or medium and lower load,
there's little or no performance difference.  I.e. the workload has to
saturate the 800Mhz LSI RAID ASIC before you notice performance tailing off.

Join the XFS mailing list and ask Dave.  I'm sure he'd be glad to give
you pointers even if it's a bit off topic.

http://oss.sgi.com/mailman/listinfo/xfs

Actually, I think it's an open list, so you should be able to simply
send mail to xfs@oss.sgi.com and ask to be CC'd as you're not subscribed.

Hope this information is helpful.

-- 
Stan

Re: Direct disk access on IBM Server

[David Brown]

On 19/04/2011 17:41, Mathias Burén wrote:
> On 19 April 2011 16:12, David Brown<david@westcontrol.com>  wrote:
>> On 19/04/2011 16:07, Mathias Burén wrote:
>>>
>>> On 19 April 2011 15:04, David Brown<david@westcontrol.com>    wrote:
>>>>
>>>> On 19/04/2011 15:25, Mathias Burén wrote:
>>>>>
>>>>> On 19 April 2011 14:21, David Brown<david@westcontrol.com>      wrote:
>>>>>>
>>>>>> I have recently got an IBM x3650 M3 server, which has a "Serveraid
>>>>>> M5014"
>>>>>> raid controller.  Booting from a Linux CD (system rescue CD) and
>>>>>> running
>>>>>> lspci identifies this raid controller as:
>>>>>>
>>>>>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>>>>>
>>>>>> The controller works fine for hardware raid - I can open its bios setup
>>>>>> utility, and set up a RAID5 (or whatever) with the disks I have.  The
>>>>>> OS
>>>>>> then just sees a single virtual disk.
>>>>>>
>>>>>> But I would like direct access to the sata drives - I want to set up
>>>>>> mdadm
>>>>>> raid, under my own control.  As far as I can see, there is no way to
>>>>>> put
>>>>>> this controller into "JBOD" or "direct access" mode of any sort.
>>>>>>
>>>>>> Does anyone here have experience with this card, or can give me any
>>>>>> hints?
>>>>>>
>>>>>> The only idea I have at the moment is to put each disk within its own
>>>>>> single-disk RAID 0 set, but then I don't get sata hot-swap
>>>>>> functionality,
>>>>>> SMART, hddtemp, etc.
>>>>>>
>>>>>> Thanks for any clues or hints,
>>>>>>
>>>>>> David
>>>>>>
>>>>>
>>>>> Regarding SMART, could you try (after loading the appropriate
>>>>> megaraid/megasas module) hdparm -a -d megaraid,$I /dev/sda , where $I
>>>>> is a number between 0 and 31 IIRC (depending on the HDD in the array).
>>>>>
>>>>
>>>> Are you sure about the syntax for that command?  Trying that with "0" for
>>>> $I
>>>> just gives me "megaraid,0: No such file or directory".  As far as I can
>>>> see,
>>>> the megaraid module is loaded (lsmod shows "megaraid_sas" in the modules
>>>> list).
>>>>
>>>> Thanks anyway,
>>>>
>>>> David
>>>>
>>>>
>>>
>>>   From the smartctl man page:
>>>
>>>           Under Linux , to look at SCSI/SAS disks behind LSI MegaRAID
>>> controllers, use syntax such as:
>>>                smartctl -a -d megaraid,2 /dev/sda
>>>                smartctl -a -d megaraid,0 /dev/sdb
>>>                where  in the argument megaraid,N, the integer N is the
>>> physical disk number within the MegaRAID controller.  This interface
>>> will also work for Dell PERC controllers.  The follow‐
>>>                ing /dev/XXX entry must exist:
>>>                For PERC2/3/4 controllers: /dev/megadev0
>>>                For PERC5/6 controllers: /dev/megaraid_sas_ioctl_node
>>>
>>> Maybe you can experiment some with that. Regards,
>>> M
>>
>> Ah, it was "smartctl" - you first wrote "hdparm".  I should have thought of
>> smartctl myself.
>>
>> Still no luck as yet - the smartctl on my test installation (Centos 5.6)
>> doesn't seem to support megaraid devices.  But maybe that's just because it
>> is an older version of smartctl - Centos 5.6 is not exactly "bleeding edge".
>>
>
> Opps, sorry! I blame the medicine. You could try compile a version of
> smartctl yourself if you have the proper packages installed.
>

The CentOs installation is only temporary.  I was doing some testing 
with IBM's software, which will only work on certain very specific Linux 
versions, such as various RHEL versions.  Since I like to use Debian on 
servers (it's what I'm used to), I didn't fancy buying RHEL just to try 
out the software - CentOS was a simple and convenient way to test the 
software.  Once I get my "real" installation in place, I can make sure I 
have up-to-date tools.

Thanks for your help and pointers,

David

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[David Brown]

On 19/04/2011 22:08, Stan Hoeppner wrote:
> David Brown put forth on 4/19/2011 8:21 AM:
>> I have recently got an IBM x3650 M3 server, which has a "Serveraid
>> M5014" raid controller.  Booting from a Linux CD (system rescue CD) and
>> running lspci identifies this raid controller as:
>>
>> LSI Logic/Symbus Logic Megaraid SAS 2108 [Liberator] (rev 03)
>>
>> The controller works fine for hardware raid - I can open its bios setup
>> utility, and set up a RAID5 (or whatever) with the disks I have.  The OS
>> then just sees a single virtual disk.
>>
>> But I would like direct access to the sata drives - I want to set up
>> mdadm raid, under my own control.  As far as I can see, there is no way
>> to put this controller into "JBOD" or "direct access" mode of any sort.
>
> FYI, the ServeRAID 5014 is a good quality real hardware RAID card w/
>
> PCIe 2.0 x8 interface
> 800 MHz LSI PowerPC RAID on Chip ASIC
> 256MB DDRII cache RAM
> 8 x 6Gb/s SAS ports via 2 SFF8087
> 32 drives maximum, 16 drives max per virtual disk (RAID group)
>
> If your card has the RAID6 feature key installed, mdraid will likely
> gain you little, if anything, over the card's inbuilt features.
>

Yes, I can see the card is good, and the test I ran with a 3 disk raid 5 
seems good so far.  I am not yet sure that whether I will go for mdadm 
raid or hardware raid - there are pros and cons to both solutions.

>> Does anyone here have experience with this card, or can give me any hints?
>
> I can point you to the documentation for the 5014:
>
> http://www.redbooks.ibm.com/technotes/tips0054.pdf
>
> ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_doc_sraidmr_1st-ed_5014-5015_quick-install.pdf
>
> ftp://ftp.software.ibm.com/systems/support/system_x_pdf/ibm_doc_sraidmr_1st-ed_5014-5015_user-guide.pdf
>

I had most of the IBM information already, but thanks anyway.

>
> Dave Chinner of Red Hat, one of the lead XFS developers, runs an 8 core
> test rig with a 512MB LSI card nearly identical to this ServeRAID 5014,
> with something like 16 drives, using mdraid.  The method he had to use
> to get mdraid working was putting each drive in its own RAID0 group
> (virtual disk).  He claimed the onboard cache RAM was still enabled for
> writes, I'm not sure about reads.
>

Yes, I've seen this idea on the net, and I also heard it from an IBM 
support technician.  It is certainly a possibility.

> For high load parallel XFS operations--think severe multiuser
> multithreaded test workloads designed to hammer XFS and force bugs to
> surface--he stated mdraid has a performance advantage vs the onboard
> RAID ASIC.  IIRC at lower disk counts and/or medium and lower load,
> there's little or no performance difference.  I.e. the workload has to
> saturate the 800Mhz LSI RAID ASIC before you notice performance tailing off.
>

I am not actually expecting a very large performance difference, nor it 
that going to be a critical factor.

> Join the XFS mailing list and ask Dave.  I'm sure he'd be glad to give
> you pointers even if it's a bit off topic.
>
> http://oss.sgi.com/mailman/listinfo/xfs
>
> Actually, I think it's an open list, so you should be able to simply
> send mail to xfs@oss.sgi.com and ask to be CC'd as you're not subscribed.
>
> Hope this information is helpful.
>

Yes, your information was helpful - thanks.  I have a few more questions 
which you might be able to help me with, if you have the time.  I'd also 
like to list some of the pros and cons of the hardware raid solution 
compared to md raid - I'd appreciate any comments you (or anyone else, 
of course) has on them.  I have no previous experience with hardware 
raid, so I'm learning a bit here.

For this particular server, I have 4 disks.  There will be virtual 
servers (using openvz) on it handling general file serving and an email 
server.  Performance requirements are not critical.  But I'm trying to 
make my comments more general, in the hope that they will be of interest 
and help to other people too.


First off, when I ran "lspci" on a system rescue cd, the card was 
identified as a "LSI Megaraid SAS 2108".  But running "lspci" on CentOS 
(with an older kernel), it was identified as a "MegaRAID SAS 9260". 
Looking at the LSI website, and comparing the pictures to the physical 
card, it seems very much that it is an LSI MegaRAID SAS 9260-8i card. 
Armed with this knowledge, I've come a lot further - LSI seems to have 
plenty of support for all sorts of systems (not just very specific RHEL, 
Suse, and Windows versions), and lots of information.  I am /much/ 
happier with LSI's software than I was with IBM's - the MegaCli command 
line program looks like it will give me the control I want from a 
command line interface, rather than using the card's BIOS screen.  I 
haven't yet tried fiddling with any settings using MegaCli, but the info 
dumps work.  MegaCli is pretty unfriendly and the documentation is not 
great (it could do with some examples), but it's much better than a bios 
screen.


Pros for hardware raid:

+ It can have battery backup (I don't have one at the moment - I have an 
excellent UPS for the whole system).
+ Rebuilds will be handled automatically by just adding new disks
+ The card supports online resizing and reshaping
+ It looks like the card supports caching with an SSD
+ The card supports snapshots of the virtual drives

Cons for hardware raid:

- The disks are tied to the controller, so if the machine or its 
controller fails, the data may not be recoverable (that's what external 
backups are for!).
- If a drive is used for a particular raid level, it is /all/ used at 
that level.  Thus no mixing of raid10 and raid5 on the same disk.
- It needs the MegaCli or other non-standard software for administration 
at run-time.
- Testing and experimentation is limited, because you can't fake an 
error (other than drive removal) and you can't fake drive size changes.


Pros for software raid:

+ It's flexible (such as raid1 for /boot, raid10 for swap, and raid5 for 
data - all within the same set of disks).
+ It uses standard software (any live CD or USB will work, as will any 
distribution).
+ You can put the disks in any Linux machine to recover the data if the 
main machine dies.
+ You can use standard disk administration software (smartctl, hddtemp, 
hdparm, etc.)
+ You can build layered raids, such as with one-disk mirrors at the 
bottom and top, for extra safety during risky operations.  You can also 
use external drives for such operations - they are slower, but easy to 
add for temporary changes.
+ You have more choices for raid levels (raid10,far is particularly 
useful, and you can have raid6 without an extra license key).


Cons for software raid:

- Adding replacement disks involves a few more changes, such as 
partitioning the disks and adding the right partitions to the right arrays.


I don't think there will be significant performance differences,
especially not for the number of drives I am using.


I have one question about the hardware raid that I don't know about.  I 
will have filesystems (some ext4, some xfs) on top of LVM on top of the 
raid.  With md raid, the filesystem knows about the layout, so xfs 
arranges its allocation groups to fit with the stripes of the raid. 
Will this automatic detection work as well with hardware raid?


Anyway, now it's time to play a little with MegaCli and see how I get 
on.  It seems to have options to put drives in "JBOD" mode - maybe that 
would give me direct access to the disk like a normal SATA drive?


mvh.,

David

Re: Direct disk access on IBM Server

[Rudy Zijlstra]

On 04/20/2011 01:24 PM, David Brown wrote:
> On 19/04/2011 22:08, Stan Hoeppner wrote:
>> David Brown put forth on 4/19/2011 8:21 AM:
>
> Pros for hardware raid:
>
> + It can have battery backup (I don't have one at the moment - I have 
> an excellent UPS for the whole system).
> + Rebuilds will be handled automatically by just adding new disks
> + The card supports online resizing and reshaping
> + It looks like the card supports caching with an SSD
> + The card supports snapshots of the virtual drives

I would add: no hassle to get boot loader installed on several disks, or 
on the raid. No limitation on which raid level used for booting
(this is the main reason i use LSI raid cards for the system. MD raid is 
used for the big data raids)
>
> Cons for hardware raid:
>
> - The disks are tied to the controller, so if the machine or its 
> controller fails, the data may not be recoverable (that's what 
> external backups are for!).
I have been using LSI for many years.
= The critical point is the controller, not the machine. I've moved 
controller & disks between machines several times and no problems
= if the controller fails, you can replace with same or later 
generation. It will recognize the disks and give you full access to your 
data. I've done this twice. Once from U160 to later U320 raid, once 
replacement with same generation card. The replacement of U160 to U320 
was not because of controller failure, i was upgrading the system.

> - If a drive is used for a particular raid level, it is /all/ used at 
> that level.  Thus no mixing of raid10 and raid5 on the same disk.
> - It needs the MegaCli or other non-standard software for 
> administration at run-time.
> - Testing and experimentation is limited, because you can't fake an 
> error (other than drive removal) and you can't fake drive size changes.
>
>
> Pros for software raid:
>
> + It's flexible (such as raid1 for /boot, raid10 for swap, and raid5 
> for data - all within the same set of disks).
> + It uses standard software (any live CD or USB will work, as will any 
> distribution).
> + You can put the disks in any Linux machine to recover the data if 
> the main machine dies.
> + You can use standard disk administration software (smartctl, 
> hddtemp, hdparm, etc.)
> + You can build layered raids, such as with one-disk mirrors at the 
> bottom and top, for extra safety during risky operations.  You can 
> also use external drives for such operations - they are slower, but 
> easy to add for temporary changes.
> + You have more choices for raid levels (raid10,far is particularly 
> useful, and you can have raid6 without an extra license key).
>
>
> Cons for software raid:
>
> - Adding replacement disks involves a few more changes, such as 
> partitioning the disks and adding the right partitions to the right 
> arrays.
>
With respect to layered RAID:
- several raid cards support RAID10.
- you can do MD raid in top of HW raid

Cheers,


Rudy

Re: Direct disk access on IBM Server

[David Brown]

On 20/04/2011 13:40, Rudy Zijlstra wrote:
> On 04/20/2011 01:24 PM, David Brown wrote:
>> On 19/04/2011 22:08, Stan Hoeppner wrote:
>>> David Brown put forth on 4/19/2011 8:21 AM:
>>
>> Pros for hardware raid:
>>
>> + It can have battery backup (I don't have one at the moment - I have
>> an excellent UPS for the whole system).
>> + Rebuilds will be handled automatically by just adding new disks
>> + The card supports online resizing and reshaping
>> + It looks like the card supports caching with an SSD
>> + The card supports snapshots of the virtual drives
>
> I would add: no hassle to get boot loader installed on several disks, or
> on the raid. No limitation on which raid level used for booting
> (this is the main reason i use LSI raid cards for the system. MD raid is
> used for the big data raids)

It's true that boot loaders and software raid can be an awkward 
combination.  However, it's not /that/ hard to do once you know the 
trick.  Put a small partition on each drive that will have the 
bootloader, and tie these together with a multi-drive raid1 set with 
metadata format 0.90 (which has the metadata at the end, rather than the 
beginning).  Use that for a /boot partition.  Once you've got your base 
system setup, and grub installed on the first disk, you can manually 
install the grub first stage loader to the MBR of each of the disks. 
You only need to do this once at the first installation (unless you have 
grub updates that change the first stage bootloader).

When booting, as far as grub is concerned the /boot partition is a 
normal partition.  It doesn't matter that it is part of a raid1 array - 
grub sees a normal partition and reads its various configuration files 
and proceeds with the boot.  Once you've actually booted, the partition 
is mounted as read-write raid1, and any updates (new kernels, etc.) are 
written to all disks.

Yes, it's a few extra steps.  But they are not too hard, and there are 
plenty of hits with google showing the details.  And if you are using 
LVM, you are probably already used to the idea of having a separate 
/boot partition.  I have done this successfully with a three-disk 
machine - it would happily boot from any of the disks.


>>
>> Cons for hardware raid:
>>
>> - The disks are tied to the controller, so if the machine or its
>> controller fails, the data may not be recoverable (that's what
>> external backups are for!).
> I have been using LSI for many years.
> = The critical point is the controller, not the machine. I've moved
> controller & disks between machines several times and no problems
> = if the controller fails, you can replace with same or later
> generation. It will recognize the disks and give you full access to your
> data. I've done this twice. Once from U160 to later U320 raid, once
> replacement with same generation card. The replacement of U160 to U320
> was not because of controller failure, i was upgrading the system.
>

Okay, that's good to know.  LSI raid controllers are not hard to get, so 
I am not afraid of being able to find a replacement.  What I was worried 
about is how much setup information is stored on the disks, and how much 
is stored in the card itself.  If a replacement controller can identify 
the disks automatically and restore the array, then that's one less 
thing to worry about.

>> - If a drive is used for a particular raid level, it is /all/ used at
>> that level. Thus no mixing of raid10 and raid5 on the same disk.
>> - It needs the MegaCli or other non-standard software for
>> administration at run-time.
>> - Testing and experimentation is limited, because you can't fake an
>> error (other than drive removal) and you can't fake drive size changes.
>>
>>
>> Pros for software raid:
>>
>> + It's flexible (such as raid1 for /boot, raid10 for swap, and raid5
>> for data - all within the same set of disks).
>> + It uses standard software (any live CD or USB will work, as will any
>> distribution).
>> + You can put the disks in any Linux machine to recover the data if
>> the main machine dies.
>> + You can use standard disk administration software (smartctl,
>> hddtemp, hdparm, etc.)
>> + You can build layered raids, such as with one-disk mirrors at the
>> bottom and top, for extra safety during risky operations. You can also
>> use external drives for such operations - they are slower, but easy to
>> add for temporary changes.
>> + You have more choices for raid levels (raid10,far is particularly
>> useful, and you can have raid6 without an extra license key).
>>
>>
>> Cons for software raid:
>>
>> - Adding replacement disks involves a few more changes, such as
>> partitioning the disks and adding the right partitions to the right
>> arrays.
>>
> With respect to layered RAID:
> - several raid cards support RAID10.
> - you can do MD raid in top of HW raid
>

Yes, the raid card I have can do RAID10.  But it can't do Linux md style 
raid10,far - I haven't heard of hardware raid cards that support this. 
For most uses, raid10,far is significantly faster than standard raid10 
(though as yet mdadm doesn't support reshaping and resizing on raid10,far).

It is certainly possible to do MD raid on top of HW raid.  As an 
example, it would be possible to put a raid1 mirror on top of a hardware 
raid, and mirror it with a big external drive for extra safety during 
risky operations (such as drive rebuilds on the main array).  And if I 
had lots of disks and wanted more redundancy, then it would be possible 
to use the hardware raid to make a set of raid1 pairs, and use md raid5 
on top of them (I don't have enough disks for that).

It is not possible to put an MD raid /under/ the HW raid.  I started 
another thread recently ("Growing layered raids") with an example of 
putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow 
for safer expansion.

Whether it is worth the effort doing something like this is another 
question, of course :-)

Re: Direct disk access on IBM Server

[Ryan Wagoner]

On Wed, Apr 20, 2011 at 7:24 AM, David Brown <david@westcontrol.com> wrote:
> Pros for hardware raid:
>
> + It can have battery backup (I don't have one at the moment - I have an
> excellent UPS for the whole system).
> + Rebuilds will be handled automatically by just adding new disks
> + The card supports online resizing and reshaping
> + It looks like the card supports caching with an SSD
> + The card supports snapshots of the virtual drives

Unless the card allows you to enable the write cache without the
backup battery you will not see the real performance of the card. This
option is normally locked out to prevent data loss in case the system
crashes, power is pulled, etc. An external UPS will not protect in all
cases if you enable the write back cache without battery.

I have systems with both software raid and hardware raid. If the
budget allows I tend to go the hardware raid route. When buying a
Dell, HP, etc server with hardware raid you end up with a complete
package. The enclosure, card, and server monitoring software make it
easy to manage and visually see which drive has failed.

Ryan

Re: Direct disk access on IBM Server

[Stan Hoeppner]

David Brown put forth on 4/20/2011 6:24 AM:

> For this particular server, I have 4 disks.

Seems like a lot of brain activity going on here for such a small array.
;)

> First off, when I ran "lspci" on a system rescue cd, the card was
> identified as a "LSI Megaraid SAS 2108".  But running "lspci" on CentOS
> (with an older kernel), it was identified as a "MegaRAID SAS 9260".

This is simply differences in kernels/drivers' device ID tables.
Nothing to worry about AFAIK.

> I don't think there will be significant performance differences,
> especially not for the number of drives I am using.

Correct assumption.

> I have one question about the hardware raid that I don't know about.  I
> will have filesystems (some ext4, some xfs) on top of LVM on top of the
> raid.  With md raid, the filesystem knows about the layout, so xfs
> arranges its allocation groups to fit with the stripes of the raid. Will
> this automatic detection work as well with hardware raid?

See:

Very important infor for virtual machines:
http://xfs.org/index.php/XFS_FAQ#Q:_Which_settings_are_best_with_virtualization_like_VMware.2C_XEN.2C_qemu.3F

Hardware RAID write cache, data safety info
http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_write_cache.3F

Hardware controller settings:
http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_RAID_controller_need_.3F

Calculate correct mkfs.xfs parameters:
http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance

General XFS tuning advice:
http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E

> Anyway, now it's time to play a little with MegaCli and see how I get
> on.  It seems to have options to put drives in "JBOD" mode - maybe that
> would give me direct access to the disk like a normal SATA drive?

IIRC, using JBOD mode for all the drives will disable the hardware
cache, and many/most/all other advanced features of the controller,
turning the RAID card literally into a plain SAS/SATA HBA.  I believe
this is why Dave chose the RAID0 per drive option.  Check your docs to
confirm.

In parting, carefully read about filesystem data consistency issues WRT
virtual machine environments.  It may prove more important for you than
any filesystem tuning.

-- 
Stan

Re: Direct disk access on IBM Server

[Stan Hoeppner]

David Brown put forth on 4/20/2011 7:21 AM:

> It's true that boot loaders and software raid can be an awkward
> combination.
...
> Yes, it's a few extra steps.

More than a few. :)  With an LSI RAID card, I simply create a drive
count X RAID5/6/10 array, set to initialize in the background, reboot
the machine with my Linux install disk, create my partitions, install
the OS ... done.  And I never have to worry about the bootloader
configuration.

> Okay, that's good to know.  LSI raid controllers are not hard to get, so

And they're the best cards overall, by far, which is why all the tier 1s
OEM them, including IBM, Dell, HP, etc.

> I am not afraid of being able to find a replacement.  What I was worried
> about is how much setup information is stored on the disks, and how much
> is stored in the card itself.

This information is duplicated in the card NVRAM/FLASH and on all the
drives--been this way with most RAID cards for well over a decade.
Mylex and AMI both started doing this in the mid/late '90s.  Both are
now divisions of LSI, both being acquired in the early 2000s.  FYI the
LSI "MegaRAID" brand was that of AMI's motherboard and RAID card products.

> Yes, the raid card I have can do RAID10.  But it can't do Linux md style
> raid10,far - I haven't heard of hardware raid cards that support this.

What difference does this make?  You already stated you're not concerned
with performance.  The mdraid far layout isn't going to give you any
noticeable gain with real world use anyway, only benchmarks, if that.

Some advice:  determine how much disk space you need out of what you
have.  If it's less than the capacity of two of your 4 drives, use
hardware RAID10 and don't look back.  If you need the capacity of 3,
then use hardware RAID 5.  You've got a nice hardware RAID card, so use it.

> For most uses, raid10,far is significantly faster than standard raid10

Again, what difference does this make?  You already stated performance
isn't a requirement.  You're simply vacillating out loud at this point.

> It is certainly possible to do MD raid on top of HW raid.  As an
> example, it would be possible to put a raid1 mirror on top of a hardware
> raid, and mirror it with a big external drive for extra safety during
> risky operations (such as drive rebuilds on the main array).  And if I
> had lots of disks and wanted more redundancy, then it would be possible
> to use the hardware raid to make a set of raid1 pairs, and use md raid5
> on top of them (I don't have enough disks for that).

With 4 drives, you could create two hardware RAID 0 arrays and mirror
the resulting devices with mdraid, or vice versa.  And you'd gain
nothing but unnecessary complexity.

What is your goal David?  To vacillate, mentally masturbate this for
weeks with no payoff?  Or build the array and use it?

> It is not possible to put an MD raid /under/ the HW raid.  I started
> another thread recently ("Growing layered raids") with an example of
> putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow
> for safer expansion.

I think the above answers my question.  As you appear averse to using a
good hardware RAID card as intended, I'll send you my shipping address
and take this problem off your hands.  Then all you have to vacillate
about is what mdraid level to use with your now mobo connected drives.

-- 
Stan

Re: Direct disk access on IBM Server

[David Brown]

On 21/04/11 05:50, Ryan Wagoner wrote:
> On Wed, Apr 20, 2011 at 7:24 AM, David Brown<david@westcontrol.com>  wrote:
>> Pros for hardware raid:
>>
>> + It can have battery backup (I don't have one at the moment - I have an
>> excellent UPS for the whole system).
>> + Rebuilds will be handled automatically by just adding new disks
>> + The card supports online resizing and reshaping
>> + It looks like the card supports caching with an SSD
>> + The card supports snapshots of the virtual drives
>
> Unless the card allows you to enable the write cache without the
> backup battery you will not see the real performance of the card. This
> option is normally locked out to prevent data loss in case the system
> crashes, power is pulled, etc. An external UPS will not protect in all
> cases if you enable the write back cache without battery.
>

The write cache is allowed without the battery, but disabled by default.

I agree that even with an UPS there is still a risk - I guess that's why 
it is disabled.  But the risk is pretty minimal for me - there is no 
realistic chance of the power plug being pulled, for example.  Other 
risks do exist - the power supply on the server may fail, the UPS may 
fail, etc.  In the end there are /always/ risks - even with a battery on 
the raid card, it won't protect against everything.  All I can do is 
reduce the risks to a realistic minimum while keeping good performance 
and features (and sensible cost!), and make sure I've got good backups 
of everything.

> I have systems with both software raid and hardware raid. If the
> budget allows I tend to go the hardware raid route. When buying a
> Dell, HP, etc server with hardware raid you end up with a complete
> package. The enclosure, card, and server monitoring software make it
> easy to manage and visually see which drive has failed.
>
> Ryan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Direct disk access on IBM Server

[David Brown]

On 21/04/11 06:10, Stan Hoeppner wrote:
> David Brown put forth on 4/20/2011 6:24 AM:
>
>> For this particular server, I have 4 disks.
>
> Seems like a lot of brain activity going on here for such a small array.
> ;)
>

I prefer to do my thinking and learning before committing too much - 
it's always annoying to have everything installed and /almost/ perfect, 
and then think "if only I'd set up the disks a little differently"!

And since it's my first hardware raid card (I don't count fakeraid on 
desktop motherboards), I have been learning a fair bit here.

>> First off, when I ran "lspci" on a system rescue cd, the card was
>> identified as a "LSI Megaraid SAS 2108".  But running "lspci" on CentOS
>> (with an older kernel), it was identified as a "MegaRAID SAS 9260".
>
> This is simply differences in kernels/drivers' device ID tables.
> Nothing to worry about AFAIK.
>

That was my thoughts.  I get the impression that the "SAS 2108" is the 
raid ASIC, while the "SAS 9260" is the name of a card.  That turned out 
to be more helpful in identifying the card on LSI's website.

>> I don't think there will be significant performance differences,
>> especially not for the number of drives I am using.
>
> Correct assumption.
>
>> I have one question about the hardware raid that I don't know about.  I
>> will have filesystems (some ext4, some xfs) on top of LVM on top of the
>> raid.  With md raid, the filesystem knows about the layout, so xfs
>> arranges its allocation groups to fit with the stripes of the raid. Will
>> this automatic detection work as well with hardware raid?
>
> See:
>
> Very important infor for virtual machines:
> http://xfs.org/index.php/XFS_FAQ#Q:_Which_settings_are_best_with_virtualization_like_VMware.2C_XEN.2C_qemu.3F
>
> Hardware RAID write cache, data safety info
> http://xfs.org/index.php/XFS_FAQ#Q._Should_barriers_be_enabled_with_storage_which_has_a_persistent_write_cache.3F
>
> Hardware controller settings:
> http://xfs.org/index.php/XFS_FAQ#Q._Which_settings_does_my_RAID_controller_need_.3F
>
> Calculate correct mkfs.xfs parameters:
> http://xfs.org/index.php/XFS_FAQ#Q:_How_to_calculate_the_correct_sunit.2Cswidth_values_for_optimal_performance
>
> General XFS tuning advice:
> http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E
>

I guess I should have looked at the FAQ before asking - after all, 
that's what the FAQ is for.  Many thanks for the links.

>> Anyway, now it's time to play a little with MegaCli and see how I get
>> on.  It seems to have options to put drives in "JBOD" mode - maybe that
>> would give me direct access to the disk like a normal SATA drive?
>
> IIRC, using JBOD mode for all the drives will disable the hardware
> cache, and many/most/all other advanced features of the controller,
> turning the RAID card literally into a plain SAS/SATA HBA.  I believe
> this is why Dave chose the RAID0 per drive option.  Check your docs to
> confirm.
>

My original thought was that plain old SATA is what I know and am used 
to, and I know how to work with it for md raid, hot plugging, etc.  So 
JBOD was what I was looking for.

However, having gathered a fair amount of information and done some 
testing, I am leaning heavily towards using the hardware raid card for 
hardware raid.  As you say, I've done a fair amount of thinking for a 
small array - I like to know what my options are and their pros and 
cons.  Having established that, the actual /implementation/ choice will 
be whatever gives me the functionality I need with the least effort (now 
and for future maintenance) - it looks like a hardware raid5 is the 
choice here.

> In parting, carefully read about filesystem data consistency issues WRT
> virtual machine environments.  It may prove more important for you than
> any filesystem tuning.
>

Yes, I am aware of such issues - I have read about them before (and they 
are relevant for the VirtualBox systems I use on desktops).  However, on 
the server I use openvz, which is a "lightweight" virtualisation - more 
like a glorified chroot than full virtualisation.  The host handles the 
filesystems - the guests just see a restricted part of the filesystem, 
rather than virtual drives.  So all data consistency issues are simple 
host issues.  I still need to make sure I understand about barriers, 
raid card caches, etc. (reading the xfs faq), but at least there are no 
special problems with virtual disks.

Thanks,

David

Re: Direct disk access on IBM Server

[David Brown]

On 21/04/11 08:24, Stan Hoeppner wrote:
> David Brown put forth on 4/20/2011 7:21 AM:
>
>> It's true that boot loaders and software raid can be an awkward
>> combination.
> ...
>> Yes, it's a few extra steps.
>
> More than a few. :)  With an LSI RAID card, I simply create a drive
> count X RAID5/6/10 array, set to initialize in the background, reboot
> the machine with my Linux install disk, create my partitions, install
> the OS ... done.  And I never have to worry about the bootloader
> configuration.
>
>> Okay, that's good to know.  LSI raid controllers are not hard to get, so
>
> And they're the best cards overall, by far, which is why all the tier 1s
> OEM them, including IBM, Dell, HP, etc.
>

That's also good to know.

>> I am not afraid of being able to find a replacement.  What I was worried
>> about is how much setup information is stored on the disks, and how much
>> is stored in the card itself.
>
> This information is duplicated in the card NVRAM/FLASH and on all the
> drives--been this way with most RAID cards for well over a decade.
> Mylex and AMI both started doing this in the mid/late '90s.  Both are
> now divisions of LSI, both being acquired in the early 2000s.  FYI the
> LSI "MegaRAID" brand was that of AMI's motherboard and RAID card products.
>

OK.

>> Yes, the raid card I have can do RAID10.  But it can't do Linux md style
>> raid10,far - I haven't heard of hardware raid cards that support this.
>
> What difference does this make?  You already stated you're not concerned
> with performance.  The mdraid far layout isn't going to give you any
> noticeable gain with real world use anyway, only benchmarks, if that.
>

I'm trying first to learn here (and you and the others on this thread 
have been very helpful), and establish my options.  I'm not looking for 
the fastest possible system - it's not performance critical.

But on the other hand, if I can get a performance boost for free, I'd 
take it.  That's the case with md raid10,far - for the same set of 
disks, using the "far" layout rather than a standard layout will give 
you faster performance on most workloads for the same cost, capacity and 
redundancy.  It's most relevant on 2 or 3 disks systems, I think.

> Some advice:  determine how much disk space you need out of what you
> have.  If it's less than the capacity of two of your 4 drives, use
> hardware RAID10 and don't look back.  If you need the capacity of 3,
> then use hardware RAID 5.  You've got a nice hardware RAID card, so use it.
>

I'm leaning heavily towards taking that advice.

>> For most uses, raid10,far is significantly faster than standard raid10
>
> Again, what difference does this make?  You already stated performance
> isn't a requirement.  You're simply vacillating out loud at this point.
>
>> It is certainly possible to do MD raid on top of HW raid.  As an
>> example, it would be possible to put a raid1 mirror on top of a hardware
>> raid, and mirror it with a big external drive for extra safety during
>> risky operations (such as drive rebuilds on the main array).  And if I
>> had lots of disks and wanted more redundancy, then it would be possible
>> to use the hardware raid to make a set of raid1 pairs, and use md raid5
>> on top of them (I don't have enough disks for that).
>
> With 4 drives, you could create two hardware RAID 0 arrays and mirror
> the resulting devices with mdraid, or vice versa.  And you'd gain
> nothing but unnecessary complexity.
>
> What is your goal David?  To vacillate, mentally masturbate this for
> weeks with no payoff?  Or build the array and use it?
>

My goal here is to understand my options before deciding.  I've had a 
bit of space between getting the machine and actually having the time to 
put it into service, so I've tested a bit and thought a bit and 
discussed a bit on this mailing list.  I'll probably go for hardware 
raid5 - which I could have done in the beginning.  But now I know more 
about why that's the sensible choice.

>> It is not possible to put an MD raid /under/ the HW raid.  I started
>> another thread recently ("Growing layered raids") with an example of
>> putting a raid 5 on top of a set of single-disk raid1 "mirrors" to allow
>> for safer expansion.
>
> I think the above answers my question.  As you appear averse to using a
> good hardware RAID card as intended, I'll send you my shipping address
> and take this problem off your hands.  Then all you have to vacillate
> about is what mdraid level to use with your now mobo connected drives.
>

Maybe I've been wandering a bit much with vague thoughts and ideas, and 
thinking too much about flexibility and expansions.  Realistically, when 
I need more disk space I can just add more disks to the array - and when 
that's not enough, it's probably time for a new server anyway.

You've given me a lot of good practical advice, which I plan to take. 
Many thanks,

David

Re: Direct disk access on IBM Server

[Majed B.]

Hello,

I've setup a bunch of software RAID boxes and I've had my main one
running at a friend's place with about 16 disks and 9TB of data.

One time 2 disks had bad sectors and things took a bad turn. It was a
RAID5 array, so I had to salvage everything I could rather than losing
everything we had there (no backups). It took me about 3 weeks of
working daily for about 6 hours a day in cloning disks, recovering
files and validating checksums. It was not fun at all.

If you have critical data, I'd suggest you add a battery and an
sd-card/flash card to your hardware RAID array. At least if the power
goes off, whatever is in the cache will be written to the card and
later on to the disks when power is restored.

A UPS will not help you if your power supply decides to commit seppuku.

If you keep daily backups, or any form of backups, go ahead with
software RAID and keep yourself free from vendor lock.

Regards,

On Thu, Apr 21, 2011 at 2:36 PM, David Brown <david.brown@hesbynett.no> wrote:
> On 21/04/11 08:24, Stan Hoeppner wrote:
>>
>> David Brown put forth on 4/20/2011 7:21 AM:
>>
>>> It's true that boot loaders and software raid can be an awkward
>>> combination.
>>
>> ...
>>>
>>> Yes, it's a few extra steps.
-- 
       Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[David Brown]

On 23/04/11 16:05, Majed B. wrote:
> Hello,
>
> I've setup a bunch of software RAID boxes and I've had my main one
> running at a friend's place with about 16 disks and 9TB of data.
>
> One time 2 disks had bad sectors and things took a bad turn. It was a
> RAID5 array, so I had to salvage everything I could rather than losing
> everything we had there (no backups). It took me about 3 weeks of
> working daily for about 6 hours a day in cloning disks, recovering
> files and validating checksums. It was not fun at all.
>

There's good reason for recommending raid6 for such large arrays!

I really hope Neil (or someone else) gets the chance to implement the 
features in the md roadmap - once md supports bad block lists, bad 
blocks on a disk will no longer mean the whole disk is removed.  So in 
situations like yours, unless you have bad sectors at the same place on 
two drives, all your data will be intact - and the huge majority will 
still be redundant, giving you time to do a simple disk replacement.

> If you have critical data, I'd suggest you add a battery and an
> sd-card/flash card to your hardware RAID array. At least if the power
> goes off, whatever is in the cache will be written to the card and
> later on to the disks when power is restored.
>
> A UPS will not help you if your power supply decides to commit seppuku.
>

That's true, of course.  But protecting data is always a matter of 
minimising the risks and consequences of the more likely failures.  My 
power supply /could/ fail, but I don't see it as a likely case.  With a 
good UPS and some redundancy in the disks, by far the most likely cause 
of failure is human error (someone deleting the wrong file, for 
instance).  Still, I will be checking the price of a backup battery for 
the server.

> If you keep daily backups, or any form of backups, go ahead with
> software RAID and keep yourself free from vendor lock.
>

I do have daily backups - raid doesn't prevent /all/ data loss!

mvh.,

David


> Regards,
>
> On Thu, Apr 21, 2011 at 2:36 PM, David Brown<david.brown@hesbynett.no>  wrote:
>> On 21/04/11 08:24, Stan Hoeppner wrote:
>>>
>>> David Brown put forth on 4/20/2011 7:21 AM:
>>>
>>>> It's true that boot loaders and software raid can be an awkward
>>>> combination.
>>>
>>> ...
>>>>
>>>> Yes, it's a few extra steps.

Re: Direct disk access on IBM Server

[Drew]

Hi David,

> My goal here is to understand my options before deciding.  I've had a bit of
> space between getting the machine and actually having the time to put it
> into service, so I've tested a bit and thought a bit and discussed a bit on
> this mailing list.  I'll probably go for hardware raid5 - which I could have
> done in the beginning.  But now I know more about why that's the sensible
> choice.

I'm going to jump in a bit late but as an owner of several of these
little beasts and say this. You have a skookum RAID card on there so
use it. :) I'd also *strongly* recommend you jump for the BBU
(Battery) if you haven't already.

This hasn't been mentioned as a consideration but one more point of
going in favor with the HW RAID is this: You're away on vacation and a
drive dies. Your backup admin notices and calls an IBM monkey out to
replace the drive. He goes, finds the failed drive indicated by the
little warning light, and replaces it. You get back to find a zero
dollar service order on your desk. :-)

I don't know about your shop but that's how it'd play out in mine.
mdadm is a great product no doubt, but to work effectively with it
everyone involved has to understand it. If you're a one man show like
I am, with vendors who don't really touch linux (not as much money in
it vs M$), making your linux systems as foolproof as possible is
important, especially when the bosses are somewhat biased against it.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie

"This started out as a hobby and spun horribly out of control."
-Unknown
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Direct disk access on IBM Server

[David Brown]

On 24/04/11 14:48, Drew wrote:
> Hi David,
>
>> My goal here is to understand my options before deciding.  I've had a bit of
>> space between getting the machine and actually having the time to put it
>> into service, so I've tested a bit and thought a bit and discussed a bit on
>> this mailing list.  I'll probably go for hardware raid5 - which I could have
>> done in the beginning.  But now I know more about why that's the sensible
>> choice.
>
> I'm going to jump in a bit late but as an owner of several of these
> little beasts and say this. You have a skookum RAID card on there so
> use it. :) I'd also *strongly* recommend you jump for the BBU
> (Battery) if you haven't already.
>
> This hasn't been mentioned as a consideration but one more point of
> going in favor with the HW RAID is this: You're away on vacation and a
> drive dies. Your backup admin notices and calls an IBM monkey out to
> replace the drive. He goes, finds the failed drive indicated by the
> little warning light, and replaces it. You get back to find a zero
> dollar service order on your desk. :-)
>

Yes, I understand the point about ease of maintenance for others - 
that's already an issue for us, as I'm the only one who really 
understands the systems we have.  Of course, buying a hot-spare drive 
will involve even less effort for the backup admin, and I'll probably do 
that.

> I don't know about your shop but that's how it'd play out in mine.
> mdadm is a great product no doubt, but to work effectively with it
> everyone involved has to understand it. If you're a one man show like
> I am, with vendors who don't really touch linux (not as much money in
> it vs M$), making your linux systems as foolproof as possible is
> important, especially when the bosses are somewhat biased against it.
>

My vendor here does use Linux, but not in the way /I/ do.  It's a 
disadvantage of the flexibility and choice in the Linux world.  (I have 
no problem with the boss regarding Linux - it would cost more money just 
to figure out what MS software and client licenses we would need than it 
cost us to buy the Linux hardware.)

I've set up the system now with hardware raid5.  I've used MegaCli to 
monitor and test the system, and grow it online by adding another drive. 
  There is no doubt that the hardware raid card works, and it's fast, 
and the little lights on the drives are a great idea.

But there is also no doubt that the MegaCli command line tool is 
absolutely terrible in comparison to mdadm.  It is downright 
user-unfriendly.  I haven't bothered with the LSI gui tools - they are 
useless on a server, where there is no place for X.  The "webbios" bios 
setup is okay, but only available when rebooting the machine - it's fine 
for the initial setup.  So the only real tool that will work over a ssh 
session while the system is running, is MegaCli.  Since it is statically 
linked it works with any system, which is nice.  But the interface is 
painfully unclear, sadistically inconsistent, and badly documented.  If 
an emergency situation arose, I could explain mdadm and dictate commands 
over the phone to a backup admin - there's not a chance I could do that 
with MegaCli.

Overall, with the server I have which came with the LSI card and no 
clear way to get direct access to the disks, then using the hardware 
raid is my best option.  But for future purchases I would think very 
hard before spending money on the same solution (for the same usage) - 
I'd be inclined to spend the money on more disks than on the raid card.

Re: Direct disk access on IBM Server

[Rudy Zijlstra]

On 04/24/2011 10:00 PM, David Brown wrote:
>
>
> But there is also no doubt that the MegaCli command line tool is 
> absolutely terrible in comparison to mdadm.  It is downright 
> user-unfriendly.  I haven't bothered with the LSI gui tools - they are 
> useless on a server, where there is no place for X.  The "webbios" 
> bios setup is okay, but only available when rebooting the machine - 
> it's fine for the initial setup.  So the only real tool that will work 
> over a ssh session while the system is running, is MegaCli.  Since it 
> is statically linked it works with any system, which is nice.  But the 
> interface is painfully unclear, sadistically inconsistent, and badly 
> documented.  If an emergency situation arose, I could explain mdadm 
> and dictate commands over the phone to a backup admin - there's not a 
> chance I could do that with MegaCli.
>
You can run the GUI on  a different machine then the raid is on. The SW 
contains a "server" app and the X client side. The server app needs to 
run on the actual server, the GUI can run on any network connected machine.

Cheers,


Rudy

Re: Direct disk access on IBM Server

[David Brown]

On 24/04/11 22:25, Rudy Zijlstra wrote:
> On 04/24/2011 10:00 PM, David Brown wrote:
>>
>>
>> But there is also no doubt that the MegaCli command line tool is
>> absolutely terrible in comparison to mdadm. It is downright
>> user-unfriendly. I haven't bothered with the LSI gui tools - they are
>> useless on a server, where there is no place for X. The "webbios" bios
>> setup is okay, but only available when rebooting the machine - it's
>> fine for the initial setup. So the only real tool that will work over
>> a ssh session while the system is running, is MegaCli. Since it is
>> statically linked it works with any system, which is nice. But the
>> interface is painfully unclear, sadistically inconsistent, and badly
>> documented. If an emergency situation arose, I could explain mdadm and
>> dictate commands over the phone to a backup admin - there's not a
>> chance I could do that with MegaCli.
>>
> You can run the GUI on a different machine then the raid is on. The SW
> contains a "server" app and the X client side. The server app needs to
> run on the actual server, the GUI can run on any network connected machine.
>

I may not have read all the information correctly, but it seemed to me 
that I had the choice of installing the client only, or everything - 
meaning installing the gui client on the server as well.  Maybe I'll 
give it a try at some point.

However, realistically speaking I hope and expect that I won't need to 
do much with the raid anyway.  It's set up, and the chances are that it 
will continue to run for years without problems.  My aim with MegaCli 
was that I would be able to handle something like a disk replacement, or 
perhaps growing the raid, without taking the server offline during the 
whole process (I don't mind a reboot or two, as long as the rebuild 
itself can run while the server is online).  So MegaCli is good enough 
for that.

I guess it just struck me as strange that MegaCli is so awkward.  A 
quick google search shows I am not alone - others describe it as "the 
most cryptic command line utility in existence" (I wouldn't go that 
far), and that MegaRAID Storage Manager is a "java based monster that 
nobody really wants to have on a server".  The card itself is fine - and 
I've heard nothing but good opinions of it.  Surely a company that can 
make such good cards could make better software to let admins work with 
them in the way they want?