Write intent bitmaps

Write intent bitmaps

[Leslie Rhorer]

I have been reading a bit about write-intent bitmaps on RAID arrays, and I
am considering adding them to my arrays.  I have a few questions:

1. The write-intent bitmap seems to be rather similar to a file system
journal.  Are there any features of the bitmaps which distinguish them from
a journal, other than the fact they operate at the RAID layer, of course,
instead of the filesystem layer?

2. On a RAID5 or RAID6 array, how much of a performance hit might I expect?

3. The threads I have read all speak about the benefits during a power
failure.  Power failures are not the only source of dirty shutdowns,
however.  Are there any benefits to a bitmap for recovering a failed array
or a degraded array?  A resync can take more than a day, and the array is
vulnerable during that time.

4. How much space will be required for the bitmnap on an external drive?

5. What happens if the bitmap is lost or the external drive fills up?

6. Would it be best to allocate a small separate partition entirely for the
bitmap?  If so, would ext2 probably be the best choice?  How does one
guarantee the partition and filesystem will be available prior to the
assembly of  the RAID array during boot?

Re: Write intent bitmaps

[Carlos Carvalho]

Leslie Rhorer (lrhorer@satx.rr.com) wrote on 7 June 2009 21:10:
 >1. The write-intent bitmap seems to be rather similar to a file system
 >journal.  Are there any features of the bitmaps which distinguish them from
 >a journal, other than the fact they operate at the RAID layer, of course,
 >instead of the filesystem layer?

The bitmap doesn't have the info to be written, it's just a bit for
the whole region. The FS journal has the [journaled part of the] info,
which can be fully recovered later if necessary. Don't forget that
raid doesn't protect against unclean shutdowns; if the array is taken
down during a write, some disks may have the new version of the stripe
and others not. When it's resynced the parity will be recalculated
from what's on all disks, that is a mix of new and old versions of the
stripes. If later a disk fails before the blocks are re-written the
not-the-one-you-want parity will be used, and you'll have corruption.

 >2. On a RAID5 or RAID6 array, how much of a performance hit might I expect?

Depends on the chunk and where the bitmap is. With an internal one the
default chunk will cause a BIG hit. Fortunately it's very easy to try
different settings with the array live, so you can easily revert when
the world suddenly freezes around you... Our arrays are rather busy,
so performance is important and I gave up on it. If you can put it on
other disks I suppose it's possible to find a chunk size compatible
with performance.

 >3. The threads I have read all speak about the benefits during a power
 >failure.  Power failures are not the only source of dirty shutdowns,
 >however.  Are there any benefits to a bitmap for recovering a failed array
 >or a degraded array?  A resync can take more than a day, and the array is
 >vulnerable during that time.

That's the benefit and the purpose of the bitmap. Besides being
vulnerable during the resync, you also have a BIG performance hit if
your array is busy (or the resync takes forever), so it's worth trying.

 >4. How much space will be required for the bitmap on an external drive?

You decide it with the chunk size.

 >5. What happens if the bitmap is lost or the external drive fills up?

No idea.

 >6. Would it be best to allocate a small separate partition entirely for the
 >bitmap?

For performance what's important is to have the bitmap and array on
different disks. Having a filesystem exclusively for the bitmap might
avoid fragmentation but I suspect this is negligible in practice.
If the bitmap disks are being used for other things you'll have to
wait for the heads to come back, which may impact performance
noticeably.

 >If so, would ext2 probably be the best choice?

That's what the man page says. I find it strange since if it's a file
the filesystem shouldn't matter. Neil?

 >How does one guarantee the partition and filesystem will be
 >available prior to the assembly of the RAID array during boot?

Your boot process must guarantee it. If the bitmap is internal you get
it automatically from the kernel.

Re: Write intent bitmaps

[Goswin von Brederlow]

carlos@fisica.ufpr.br (Carlos Carvalho) writes:

> Leslie Rhorer (lrhorer@satx.rr.com) wrote on 7 June 2009 21:10:
>  >1. The write-intent bitmap seems to be rather similar to a file system
>  >journal.  Are there any features of the bitmaps which distinguish them from
>  >a journal, other than the fact they operate at the RAID layer, of course,
>  >instead of the filesystem layer?
>
> The bitmap doesn't have the info to be written, it's just a bit for
> the whole region. The FS journal has the [journaled part of the] info,
> which can be fully recovered later if necessary. Don't forget that
> raid doesn't protect against unclean shutdowns; if the array is taken
> down during a write, some disks may have the new version of the stripe
> and others not. When it's resynced the parity will be recalculated
> from what's on all disks, that is a mix of new and old versions of the
> stripes. If later a disk fails before the blocks are re-written the
> not-the-one-you-want parity will be used, and you'll have corruption.

Can that actualy happen? I would think the raid should refuse to
reassemble automatically if any one stripe does not have enough blocks
in sync. Just like when not enough disks are in sync without bitmap.

>  >2. On a RAID5 or RAID6 array, how much of a performance hit might I expect?
>
> Depends on the chunk and where the bitmap is. With an internal one the
> default chunk will cause a BIG hit. Fortunately it's very easy to try
> different settings with the array live, so you can easily revert when
> the world suddenly freezes around you... Our arrays are rather busy,
> so performance is important and I gave up on it. If you can put it on
> other disks I suppose it's possible to find a chunk size compatible
> with performance.

Worst case every write to the raid requires a write to the bitmap. So
your speed will be ~half. It is not (much) less than half though. You
could think that the seek to and from the bitmap must slow things down
even more but worst case is random access, which means there already
is a seek between each write. The bitmap just adds one write and one
seek for each write and seek.

The critical part seems to be the size one bit in the bitmap
covers. If you have 2 writes that are covered by the same bit then the
bit is only changed once. So the bigger the covered size the less
bitmap writes. On the other hand the benefits (specifically resyncs)
decrease with the covered size. Find a balance that works for you.

>  >3. The threads I have read all speak about the benefits during a power
>  >failure.  Power failures are not the only source of dirty shutdowns,
>  >however.  Are there any benefits to a bitmap for recovering a failed array
>  >or a degraded array?  A resync can take more than a day, and the array is
>  >vulnerable during that time.
>
> That's the benefit and the purpose of the bitmap. Besides being
> vulnerable during the resync, you also have a BIG performance hit if
> your array is busy (or the resync takes forever), so it's worth trying.

As long as the drives work again during reassembly it doesn't matter
what the cause of the failure was. It will only sync the parts
indicated by the bitmap. On the other hand if a drive fails and is
replaced there is no alternative to rewriting all of that drive.

One benefit of the bitmap during a full resync though is (afaik) that
the bitmap (better) indicates the amount done already. If the system
crashes and reboots the resync will resume instead of restart.

>  >4. How much space will be required for the bitmap on an external drive?
>
> You decide it with the chunk size.
>
>  >5. What happens if the bitmap is lost or the external drive fills up?
>
> No idea.

Afaik you can always start the raid without bitmap. If it is unclean
that means a full resync though. As for the external drive filling up
I don't think that has any effect, not with ext2. The bitmap uses a
fixed space on the disk and will never need more space. Even with a
completly full ext2 there should be no problem writing the bitmap.

>  >6. Would it be best to allocate a small separate partition entirely for the
>  >bitmap?
>
> For performance what's important is to have the bitmap and array on
> different disks. Having a filesystem exclusively for the bitmap might
> avoid fragmentation but I suspect this is negligible in practice.

From test I've done with some external enclosures there was absolutely
no difference between internal bitmap and external bitmap on a
partition on the same drive. But test it for your use case and
hardware.

Since the bitmap is only written once and then only rewritten just
make sure it is not fragmented to start with. Although it probably
doesn't matter even if it is.

> If the bitmap disks are being used for other things you'll have to
> wait for the heads to come back, which may impact performance
> noticeably.
>
>  >If so, would ext2 probably be the best choice?
>
> That's what the man page says. I find it strange since if it's a file
> the filesystem shouldn't matter. Neil?
>
>  >How does one guarantee the partition and filesystem will be
>  >available prior to the assembly of the RAID array during boot?
>
> Your boot process must guarantee it. If the bitmap is internal you get
> it automatically from the kernel.

MfG
        Goswin

Re: Write intent bitmaps

[Neil Brown]

On Thursday June 18, goswin-v-b@web.de wrote:
> carlos@fisica.ufpr.br (Carlos Carvalho) writes:
> 
> > Leslie Rhorer (lrhorer@satx.rr.com) wrote on 7 June 2009 21:10:
> >  >1. The write-intent bitmap seems to be rather similar to a file system
> >  >journal.  Are there any features of the bitmaps which distinguish them from
> >  >a journal, other than the fact they operate at the RAID layer, of course,
> >  >instead of the filesystem layer?
> >
> > The bitmap doesn't have the info to be written, it's just a bit for
> > the whole region. The FS journal has the [journaled part of the] info,
> > which can be fully recovered later if necessary. Don't forget that
> > raid doesn't protect against unclean shutdowns; if the array is taken
> > down during a write, some disks may have the new version of the stripe
> > and others not. When it's resynced the parity will be recalculated
> > from what's on all disks, that is a mix of new and old versions of the
> > stripes. If later a disk fails before the blocks are re-written the
> > not-the-one-you-want parity will be used, and you'll have corruption.
> 
> Can that actualy happen? I would think the raid should refuse to
> reassemble automatically if any one stripe does not have enough blocks
> in sync. Just like when not enough disks are in sync without bitmap.

You would think correctly.  If mdadm cannot be sure that the data is
still good, it will refuse to assemble the array. 
In particular, active degraded raid4/5/6 arrays will not normally be
assembled for you due to the above reason - the could be data
corruption.

If you want to get as whatever data there is there, you can assemble
with "--force".  But in that case be aware that there could be
corruption.
I have never actually heard of anyone doing that and discovering
corruption, but then the nature of the possible corruption is that it
would probably be quite hard to find.

> 
> >  >2. On a RAID5 or RAID6 array, how much of a performance hit might I expect?
> >
> > Depends on the chunk and where the bitmap is. With an internal one the
> > default chunk will cause a BIG hit. Fortunately it's very easy to try
> > different settings with the array live, so you can easily revert when
> > the world suddenly freezes around you... Our arrays are rather busy,
> > so performance is important and I gave up on it. If you can put it on
> > other disks I suppose it's possible to find a chunk size compatible
> > with performance.
> 
> Worst case every write to the raid requires a write to the bitmap. So
> your speed will be ~half. It is not (much) less than half though. You
> could think that the seek to and from the bitmap must slow things down
> even more but worst case is random access, which means there already
> is a seek between each write. The bitmap just adds one write and one
> seek for each write and seek.

I think half-speed would be very very unlikely.  md tries to gather
bitmap updates so that - where possible - it might update several bits
all at once.

I have measured a 10% performance drop.  However it is very dependant
on workload and, and you say, bitmap chunk size.

> 
> The critical part seems to be the size one bit in the bitmap
> covers. If you have 2 writes that are covered by the same bit then the
> bit is only changed once. So the bigger the covered size the less
> bitmap writes. On the other hand the benefits (specifically resyncs)
> decrease with the covered size. Find a balance that works for you.
> 
> >  >3. The threads I have read all speak about the benefits during a power
> >  >failure.  Power failures are not the only source of dirty shutdowns,
> >  >however.  Are there any benefits to a bitmap for recovering a failed array
> >  >or a degraded array?  A resync can take more than a day, and the array is
> >  >vulnerable during that time.
> >
> > That's the benefit and the purpose of the bitmap. Besides being
> > vulnerable during the resync, you also have a BIG performance hit if
> > your array is busy (or the resync takes forever), so it's worth trying.
> 
> As long as the drives work again during reassembly it doesn't matter
> what the cause of the failure was. It will only sync the parts
> indicated by the bitmap. On the other hand if a drive fails and is
> replaced there is no alternative to rewriting all of that drive.
> 
> One benefit of the bitmap during a full resync though is (afaik) that
> the bitmap (better) indicates the amount done already. If the system
> crashes and reboots the resync will resume instead of restart.

When you a rebuilding a drive that had failed, we call that "recovery"
not "resync".
With 0.90 metadata, a recovery will always restart at the beginning.
With 1.x metadata, we checkpoint the recovery so it won't duplicated
very much work.


NeilBrown

Re: Write intent bitmaps

[Goswin von Brederlow]

Neil Brown <neilb@suse.de> writes:

> On Thursday June 18, goswin-v-b@web.de wrote:
>> carlos@fisica.ufpr.br (Carlos Carvalho) writes:
>> >  >2. On a RAID5 or RAID6 array, how much of a performance hit might I expect?
>> >
>> > Depends on the chunk and where the bitmap is. With an internal one the
>> > default chunk will cause a BIG hit. Fortunately it's very easy to try
>> > different settings with the array live, so you can easily revert when
>> > the world suddenly freezes around you... Our arrays are rather busy,
>> > so performance is important and I gave up on it. If you can put it on
>> > other disks I suppose it's possible to find a chunk size compatible
>> > with performance.
>> 
>> Worst case every write to the raid requires a write to the bitmap. So
>> your speed will be ~half. It is not (much) less than half though. You
>> could think that the seek to and from the bitmap must slow things down
>> even more but worst case is random access, which means there already
>> is a seek between each write. The bitmap just adds one write and one
>> seek for each write and seek.
>
> I think half-speed would be very very unlikely.  md tries to gather
> bitmap updates so that - where possible - it might update several bits
> all at once.
>
> I have measured a 10% performance drop.  However it is very dependant
> on workload and, and you say, bitmap chunk size.

From my tests with internal bitmaps half is what you get with the
default size. At least that was what I got with a software raid over
external raid enclosures. Might be a side effect of bitmap writes not
covering a stripe on the external raid enclosures and them slowing
them down in the hope of getting more data for that stripe. But it was
quite unusable.

>> One benefit of the bitmap during a full resync though is (afaik) that
>> the bitmap (better) indicates the amount done already. If the system
>> crashes and reboots the resync will resume instead of restart.
>
> When you a rebuilding a drive that had failed, we call that "recovery"
> not "resync".
> With 0.90 metadata, a recovery will always restart at the beginning.
> With 1.x metadata, we checkpoint the recovery so it won't duplicated
> very much work.
>
>
> NeilBrown

One of these days I have to redo my home raids with newer metadata.

MfG
        Goswin

Re: Write intent bitmaps

[Neil Brown]

On Monday June 8, carlos@fisica.ufpr.br wrote:
> 
>  >5. What happens if the bitmap is lost or the external drive fills up?
> 
> No idea.

If the bitmap is lost, it is just as though you didn't have a bitmap 
(or as though all the bits in the bitmap were set to one).

The drive filling up is not relevant.  When the bitmap is in a
separate file, the file is preallocated to be the right size.

> 
>  >If so, would ext2 probably be the best choice?
> 
> That's what the man page says. I find it strange since if it's a file
> the filesystem shouldn't matter. Neil?

The way that md writes to the bitmap file is not entirely portable
across different filesystems.  As the man page say,  it is known to
work for ext2 and ext3.  Either is a fine choice.
A new filesystem interface is being introduced in 2.6.31 as part of
the swap-over-NFS work.  I might end up using that to write to bitmap
files, as it has the right characteristics.  But that is very much in
the future.

NeilBrown

Re: Write intent bitmaps

[Goswin von Brederlow]

Neil Brown <neilb@suse.de> writes:

> On Monday June 8, carlos@fisica.ufpr.br wrote:
>>  >If so, would ext2 probably be the best choice?
>> 
>> That's what the man page says. I find it strange since if it's a file
>> the filesystem shouldn't matter. Neil?
>
> The way that md writes to the bitmap file is not entirely portable
> across different filesystems.  As the man page say,  it is known to
> work for ext2 and ext3.  Either is a fine choice.
> A new filesystem interface is being introduced in 2.6.31 as part of
> the swap-over-NFS work.  I might end up using that to write to bitmap
> files, as it has the right characteristics.  But that is very much in
> the future.
>
> NeilBrown

How about support to write to a device instead of a file?

MfG
        Goswin

Re: Write intent bitmaps

[NeilBrown]

On Sat, June 20, 2009 1:01 am, Goswin von Brederlow wrote:
> Neil Brown <neilb@suse.de> writes:
>
>> On Monday June 8, carlos@fisica.ufpr.br wrote:
>>>  >If so, would ext2 probably be the best choice?
>>>
>>> That's what the man page says. I find it strange since if it's a file
>>> the filesystem shouldn't matter. Neil?
>>
>> The way that md writes to the bitmap file is not entirely portable
>> across different filesystems.  As the man page say,  it is known to
>> work for ext2 and ext3.  Either is a fine choice.
>> A new filesystem interface is being introduced in 2.6.31 as part of
>> the swap-over-NFS work.  I might end up using that to write to bitmap
>> files, as it has the right characteristics.  But that is very much in
>> the future.
>>
>> NeilBrown
>
> How about support to write to a device instead of a file?

One of my next little projects is to rationalise the bitmap support
and make it more generic - currently you cannot use bitmaps with
externally managed metadata because there is no way to say where
it should go.
I don't know yet how that will end up working, but it is entirely possible
that it will generalise to allows the bitmap to be written to an
arbitrary device.

However, as bitmaps are typically quite small (a few megabytes at
most, typically tens of kilobytes) it seems clumsy to create a
partition just for a bitmap..

NeilBrown

Re: Write intent bitmaps

[Goswin von Brederlow]

"NeilBrown" <neilb@suse.de> writes:

> On Sat, June 20, 2009 1:01 am, Goswin von Brederlow wrote:
>> Neil Brown <neilb@suse.de> writes:
>>
>>> On Monday June 8, carlos@fisica.ufpr.br wrote:
>>>>  >If so, would ext2 probably be the best choice?
>>>>
>>>> That's what the man page says. I find it strange since if it's a file
>>>> the filesystem shouldn't matter. Neil?
>>>
>>> The way that md writes to the bitmap file is not entirely portable
>>> across different filesystems.  As the man page say,  it is known to
>>> work for ext2 and ext3.  Either is a fine choice.
>>> A new filesystem interface is being introduced in 2.6.31 as part of
>>> the swap-over-NFS work.  I might end up using that to write to bitmap
>>> files, as it has the right characteristics.  But that is very much in
>>> the future.
>>>
>>> NeilBrown
>>
>> How about support to write to a device instead of a file?
>
> One of my next little projects is to rationalise the bitmap support
> and make it more generic - currently you cannot use bitmaps with
> externally managed metadata because there is no way to say where
> it should go.
> I don't know yet how that will end up working, but it is entirely possible
> that it will generalise to allows the bitmap to be written to an
> arbitrary device.
>
> However, as bitmaps are typically quite small (a few megabytes at
> most, typically tens of kilobytes) it seems clumsy to create a
> partition just for a bitmap..
>
> NeilBrown

lvcreate --extents 1 ...

Ideal would be if one could put multiple bitmaps on the same
device. But then again that is what an FS allows.

MfG
        Goswin

Re: Write intent bitmaps

[Bill Davidsen]

Neil Brown wrote:
> On Monday June 8, carlos@fisica.ufpr.br wrote:
>   
>>  >5. What happens if the bitmap is lost or the external drive fills up?
>>
>> No idea.
>>     
>
> If the bitmap is lost, it is just as though you didn't have a bitmap 
> (or as though all the bits in the bitmap were set to one).
>
> The drive filling up is not relevant.  When the bitmap is in a
> separate file, the file is preallocated to be the right size.
>
>   
>>  >If so, would ext2 probably be the best choice?
>>
>> That's what the man page says. I find it strange since if it's a file
>> the filesystem shouldn't matter. Neil?
>>     
>
> The way that md writes to the bitmap file is not entirely portable
> across different filesystems.  As the man page say,  it is known to
> work for ext2 and ext3.  Either is a fine choice.
> A new filesystem interface is being introduced in 2.6.31 as part of
> the swap-over-NFS work.  I might end up using that to write to bitmap
> files, as it has the right characteristics.  But that is very much in
> the future.
>   

I never understood why swap over anything wasn't supported... we had 
swap over NFS in SunOS, in 1992, and that was over 10Mbit thicknet to 
drives sized in MB (RAM in MB, too). I have been trying some things with 
SSD, putting swap on it, bitmaps both raid and f/s, and journals. At the 
moment I'm developing the scripts and tools to do valid measurement.

-- 
Bill Davidsen <davidsen@tmr.com>
  Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one error occurs during
wildcard (glob) expansion.

RE: Write intent bitmaps

[Leslie Rhorer]

> >  >5. What happens if the bitmap is lost or the external drive fills up?
> >
> > No idea.
> 
> If the bitmap is lost, it is just as though you didn't have a bitmap
> (or as though all the bits in the bitmap were set to one).

	OK, what about the thread in this list where (as I recall) the
fellow's bitmap was corrupted or lost and it evidently caused the entire
array to go splat?  That's worrisome.

> The drive filling up is not relevant.  When the bitmap is in a
> separate file, the file is preallocated to be the right size.

	Let me take a little different tack.  I don't have any ext2 or ext3
file systems on the server.  If I am going to set up an external bitmap, I
am going to have to steal some space from one of the other file systems -
perhaps from the swap area (5G), the boot file system (reiserfs - 109G), or
the Windows XP filesystem (NTFS - 40G).  I need to be able to calculate how
much space to steal to fit the bitmap, and allow for later expansion.

> >  >If so, would ext2 probably be the best choice?
> >
> > That's what the man page says. I find it strange since if it's a file
> > the filesystem shouldn't matter. Neil?
> 
> The way that md writes to the bitmap file is not entirely portable
> across different filesystems.  As the man page say,  it is known to
> work for ext2 and ext3.  Either is a fine choice.

	I'm fine with shrinking one of the existing file systems and
creating a tiny partition just for the bitmap.

Re: Write intent bitmaps

[Goswin von Brederlow]

"Leslie Rhorer" <lrhorer@satx.rr.com> writes:

>> >  >5. What happens if the bitmap is lost or the external drive fills up?
>> >
>> > No idea.
>> 
>> If the bitmap is lost, it is just as though you didn't have a bitmap
>> (or as though all the bits in the bitmap were set to one).
>
> 	OK, what about the thread in this list where (as I recall) the
> fellow's bitmap was corrupted or lost and it evidently caused the entire
> array to go splat?  That's worrisome.
>
>> The drive filling up is not relevant.  When the bitmap is in a
>> separate file, the file is preallocated to be the right size.
>
> 	Let me take a little different tack.  I don't have any ext2 or ext3
> file systems on the server.  If I am going to set up an external bitmap, I
> am going to have to steal some space from one of the other file systems -
> perhaps from the swap area (5G), the boot file system (reiserfs - 109G), or
> the Windows XP filesystem (NTFS - 40G).  I need to be able to calculate how
> much space to steal to fit the bitmap, and allow for later expansion.

That is pretty flexible and the size is miniscule. For example my big
raid5 with internal bitmap has

      bitmap: 0/233 pages [0KB], 2048KB chunk

That is slightly less than 1MB. With 512MB bitmap I could do 4KB
chunks but from a few tests a larger chunk size is better for me.

MfG
        Goswin

RAID 5 recovery to not degrade device on bad block

[Anshuman Aggarwal]

Here is a simple feature request which I assume would not be much
logic change for kernel devs familiar with the code.

Essentially, if I understand correctly, the kernel raid code will try
to let the drive fix a bad sector and otherwise fail the device and
degrade the array.
However, if an array is already degraded then this behvaviour can be
very limiting because typically you are in recovery mode and want to
get as much data out to your new disk as you can.

I would say that for an already degraded array, bad blocks should
*NOT* by default cause a single bad block to fail the whole
array...instead just log the bad blocks to the syslog and let the
admin take care of it.

Right now, the big benefit of RAID5 is being affected

Ideally, I'd like to see Neil's road map bad block device handler
implemented (have often thought of tinkering with the block device
code in the kernel to do just that)...but till then a simple check
that an array is  degraded before failing a device which would render
the whole array inoperable should suffice? This could throw big errors
in the syslog but at least the a 2 TB MD array won't be down because
of 1 512 byte sector?

Thanks,
Anshuman

Re: RAID 5 recovery to not degrade device on bad block

[Goswin von Brederlow]

Anshuman Aggarwal <anshuman.aggarwal@gmail.com> writes:

> Here is a simple feature request which I assume would not be much
> logic change for kernel devs familiar with the code.
>
> Essentially, if I understand correctly, the kernel raid code will try
> to let the drive fix a bad sector and otherwise fail the device and
> degrade the array.
> However, if an array is already degraded then this behvaviour can be
> very limiting because typically you are in recovery mode and want to
> get as much data out to your new disk as you can.
>
> I would say that for an already degraded array, bad blocks should
> *NOT* by default cause a single bad block to fail the whole
> array...instead just log the bad blocks to the syslog and let the
> admin take care of it.

Big problem there.

As long as the raid is degrade a bad block can be reported to the
system as I/O error.

But consider what happens when you resync the drive and don't stop on
a bad block. The block on the new drive coresponding to the bad block
can not be initialized corectly. But a read of the bad block would
trigger the block to be recomputed from the remaining disks. Instead
of an I/O error you would get invalid data.

What would be needed is the ability to mark blocks as bad. Even with
bitmap support the bit cover too large an area.

> Right now, the big benefit of RAID5 is being affected
>
> Ideally, I'd like to see Neil's road map bad block device handler
> implemented (have often thought of tinkering with the block device
> code in the kernel to do just that)...but till then a simple check
> that an array is  degraded before failing a device which would render
> the whole array inoperable should suffice? This could throw big errors
> in the syslog but at least the a 2 TB MD array won't be down because
> of 1 512 byte sector?
>
> Thanks,
> Anshuman

MfG
        Goswin

Write intent bitmaps.

[Simon Jackson]

I am trying to use write intent bitmaps on some RAID 1 volumes to reduce the rebuild times in the event of hard resets that cause the md driver to kick members out of my arrays.

I used the mdadm --grow /dev/md0 --bitmap=internal  and this appeared to succeed, but when I tried to examine the bitmap I get an error. 


:~$ sudo mdadm --grow /dev/md0 --bitmap=internal
:~$ sudo mdadm -X /dev/md0
        Filename : /dev/md0
           Magic : 00000000
mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
         Version : 0
mdadm: unknown bitmap version 0, either the bitmap file is corrupted or you need to upgrade your tools

cat /proc/mdstat
Personalities : [raid1] 
      
md0 : active raid1 sda5[0] sdb5[1]
      7823552 blocks [2/2] [UU]
      bitmap: 29/239 pages [116KB], 16KB chunk
      
unused devices: <none>
sjackson@mercuryst5:~$ sudo mdadm -X /dev/md0
        Filename : /dev/md0
           Magic : 00000000
mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
         Version : 0
mdadm: unknown bitmap version 0, either the bitmap file is corrupted or you need to upgrade your tools

Do I really have a usable bitmap on the device in this case?

Thanks for any input.  

Re: Write intent bitmaps.

[NeilBrown]

On Tue, August 25, 2009 12:39 am, Simon Jackson wrote:
>
>
> I am trying to use write intent bitmaps on some RAID 1 volumes to reduce
> the rebuild times in the event of hard resets that cause the md driver to
> kick members out of my arrays.
>
> I used the mdadm --grow /dev/md0 --bitmap=internal  and this appeared to
> succeed, but when I tried to examine the bitmap I get an error.
>
>
> :~$ sudo mdadm --grow /dev/md0 --bitmap=internal
> :~$ sudo mdadm -X /dev/md0
>         Filename : /dev/md0
>            Magic : 00000000
> mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
>          Version : 0
> mdadm: unknown bitmap version 0, either the bitmap file is corrupted or
> you need to upgrade your tools

Quoting from the man page:

       -X, --examine-bitmap
              Report  information about a bitmap file.  The argument is
either
              an external bitmap file or an array  component  in  case  of
 an
              internal  bitmap.   Note  that  running  this on an array
device
              (e.g.  /dev/md0) does not report the bitmap for that array.


Particularly read the last sentence.
Then try
   mdadm -X /dev/sda5

NeilBrown


>
> cat /proc/mdstat
> Personalities : [raid1]
>
> md0 : active raid1 sda5[0] sdb5[1]
>       7823552 blocks [2/2] [UU]
>       bitmap: 29/239 pages [116KB], 16KB chunk
>
> unused devices: <none>
> sjackson@mercuryst5:~$ sudo mdadm -X /dev/md0
>         Filename : /dev/md0
>            Magic : 00000000
> mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
>          Version : 0
> mdadm: unknown bitmap version 0, either the bitmap file is corrupted or
> you need to upgrade your tools
>
> Do I really have a usable bitmap on the device in this case?
>
> Thanks for any input.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Write intent bitmaps.

[Bill Davidsen]

NeilBrown wrote:
> On Tue, August 25, 2009 12:39 am, Simon Jackson wrote:
>   
>> I am trying to use write intent bitmaps on some RAID 1 volumes to reduce
>> the rebuild times in the event of hard resets that cause the md driver to
>> kick members out of my arrays.
>>
>> I used the mdadm --grow /dev/md0 --bitmap=internal  and this appeared to
>> succeed, but when I tried to examine the bitmap I get an error.
>>
>>
>> :~$ sudo mdadm --grow /dev/md0 --bitmap=internal
>> :~$ sudo mdadm -X /dev/md0
>>         Filename : /dev/md0
>>            Magic : 00000000
>> mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
>>          Version : 0
>> mdadm: unknown bitmap version 0, either the bitmap file is corrupted or
>> you need to upgrade your tools
>>     
>
> Quoting from the man page:
>
>        -X, --examine-bitmap
>               Report  information about a bitmap file.  The argument is
> either
>               an external bitmap file or an array  component  in  case  of
>  an
>               internal  bitmap.   Note  that  running  this on an array
> device
>               (e.g.  /dev/md0) does not report the bitmap for that array.
>
>
> Particularly read the last sentence.
> Then try
>    mdadm -X /dev/sda5
>   

Well that's nice and clear, but raises the question "why not?" This 
would seem to be one of the most common things someone would do, to look 
at the bitmap for an array.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc

"Now we have another quarterback besides Kurt Warner telling us during postgame
interviews that he owes every great thing that happens to him on a football
field to his faith in Jesus. I knew there had to be a reason why the Almighty
included a mute button on my remote."
			-- Arthur Troyer on Tim Tebow (Sports Illustrated)

Re: Write intent bitmaps.

[Paul Clements]

Bill Davidsen wrote:
> NeilBrown wrote:
>> On Tue, August 25, 2009 12:39 am, Simon Jackson wrote:
>>  
>>> I am trying to use write intent bitmaps on some RAID 1 volumes to reduce
>>> the rebuild times in the event of hard resets that cause the md 
>>> driver to
>>> kick members out of my arrays.
>>>
>>> I used the mdadm --grow /dev/md0 --bitmap=internal  and this appeared to
>>> succeed, but when I tried to examine the bitmap I get an error.
>>>
>>>
>>> :~$ sudo mdadm --grow /dev/md0 --bitmap=internal
>>> :~$ sudo mdadm -X /dev/md0
>>>         Filename : /dev/md0
>>>            Magic : 00000000
>>> mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
>>>          Version : 0
>>> mdadm: unknown bitmap version 0, either the bitmap file is corrupted or
>>> you need to upgrade your tools
>>>     
>>
>> Quoting from the man page:
>>
>>        -X, --examine-bitmap
>>               Report  information about a bitmap file.  The argument is
>> either
>>               an external bitmap file or an array  component  in  
>> case  of
>>  an
>>               internal  bitmap.   Note  that  running  this on an array
>> device
>>               (e.g.  /dev/md0) does not report the bitmap for that array.
>>
>>
>> Particularly read the last sentence.
>> Then try
>>    mdadm -X /dev/sda5
>>   
> 
> Well that's nice and clear, but raises the question "why not?" This 
> would seem to be one of the most common things someone would do, to look 
> at the bitmap for an array.

Two reasons why not:

The examine code simply takes the device or file you give it and looks 
for a bitmap in that file or device. You'd have to do some hand-waving 
to "read the bitmap for /dev/md0". There actually is no bitmap on 
/dev/md0; there is a bitmap stored either in a file or on each of the 
component devices. So which version of the bitmap do you read? From the 
first, second, third ... component disk?

Also, mdadm's behavior would be ambiguous if you implemented the above. 
What if /dev/md0 is itself a component of another md device? Then how is 
mdadm to know which bitmap you want? The one that actually physically 
exists on md0, or the ones that the components of md0 contain?

Perhaps better would be to simply throw an error in this case?

--
Paul

Re: Write intent bitmaps.

[Ryan Wagoner]

You could just provide more information in the mdadm bitmap error message

mdadm: unknown bitmap version 0, either the bitmap file is corrupted
or you need to upgrade your tools

suggested change

mdadm: unknown bitmap version 0, either the bitmap file is corrupted,
you are looking looking at the array and not an array component, or
you need to upgrade your tools

Ryan

On Wed, Sep 2, 2009 at 12:28 PM, Paul
Clements<paul.clements@steeleye.com> wrote:
> Bill Davidsen wrote:
>>
>> NeilBrown wrote:
>>>
>>> On Tue, August 25, 2009 12:39 am, Simon Jackson wrote:
>>>
>>>>
>>>> I am trying to use write intent bitmaps on some RAID 1 volumes to reduce
>>>> the rebuild times in the event of hard resets that cause the md driver
>>>> to
>>>> kick members out of my arrays.
>>>>
>>>> I used the mdadm --grow /dev/md0 --bitmap=internal  and this appeared to
>>>> succeed, but when I tried to examine the bitmap I get an error.
>>>>
>>>>
>>>> :~$ sudo mdadm --grow /dev/md0 --bitmap=internal
>>>> :~$ sudo mdadm -X /dev/md0
>>>>        Filename : /dev/md0
>>>>           Magic : 00000000
>>>> mdadm: invalid bitmap magic 0x0, the bitmap file appears to be corrupted
>>>>         Version : 0
>>>> mdadm: unknown bitmap version 0, either the bitmap file is corrupted or
>>>> you need to upgrade your tools
>>>>
>>>
>>> Quoting from the man page:
>>>
>>>       -X, --examine-bitmap
>>>              Report  information about a bitmap file.  The argument is
>>> either
>>>              an external bitmap file or an array  component  in  case  of
>>>  an
>>>              internal  bitmap.   Note  that  running  this on an array
>>> device
>>>              (e.g.  /dev/md0) does not report the bitmap for that array.
>>>
>>>
>>> Particularly read the last sentence.
>>> Then try
>>>   mdadm -X /dev/sda5
>>>
>>
>> Well that's nice and clear, but raises the question "why not?" This would
>> seem to be one of the most common things someone would do, to look at the
>> bitmap for an array.
>
> Two reasons why not:
>
> The examine code simply takes the device or file you give it and looks for a
> bitmap in that file or device. You'd have to do some hand-waving to "read
> the bitmap for /dev/md0". There actually is no bitmap on /dev/md0; there is
> a bitmap stored either in a file or on each of the component devices. So
> which version of the bitmap do you read? From the first, second, third ...
> component disk?
>
> Also, mdadm's behavior would be ambiguous if you implemented the above. What
> if /dev/md0 is itself a component of another md device? Then how is mdadm to
> know which bitmap you want? The one that actually physically exists on md0,
> or the ones that the components of md0 contain?
>
> Perhaps better would be to simply throw an error in this case?
>
> --
> Paul
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html