mdadm: /dev/md0 has been started with 1 drive (out of 2).

mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

Hi,

I am new to mdadm/software raid and I've built myself a raid1 array, which after a resume from sleep is assembled with 1 out of 2 devices only.

I queried the web, read some threads on the mailing list and learned that event count on these two devices differe a little, and that in a case like this, it would seem, mdadm --assemble --scan --force is a right action. Tried that, but the array is still assembled with only one device.

Where do I go from here?

--examine output:

> % mdadm --examine /dev/sdc1
> /dev/sdc1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>            Name : sega:0  (local to host sega)
>   Creation Time : Fri Nov  1 16:24:18 2013
>      Raid Level : raid1
>    Raid Devices : 2
> 
>  Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>      Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>   Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : 814f6fb3:a8a93019:c04ef011:cfa16124
> 
>     Update Time : Tue Nov  5 01:03:48 2013
>        Checksum : 32aca1de - correct
>          Events : 22
> 
> 
>    Device Role : Active device 0
>    Array State : AA ('A' == active, '.' == missing)
> 
> 
> 
> % mdadm --examine /dev/sdd1
> /dev/sdd1:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>            Name : sega:0  (local to host sega)
>   Creation Time : Fri Nov  1 16:24:18 2013
>      Raid Level : raid1
>    Raid Devices : 2
> 
>  Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>      Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>   Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : cea7f341:435cdefd:5f883265:a75c5080
> 
>     Update Time : Tue Nov  5 07:53:09 2013
>        Checksum : 55110136 - correct
>          Events : 30
> 
> 
>    Device Role : Active device 1
>    Array State : .A ('A' == active, '.' == missing)


dmesg output:

> [548246.716474] scsi_verify_blk_ioctl: 18 callbacks suppressed
> [548246.716484] mdadm: sending ioctl 800c0910 to a partition!
> [548246.716492] mdadm: sending ioctl 800c0910 to a partition!
> [548246.716512] mdadm: sending ioctl 1261 to a partition!
> [548246.716518] mdadm: sending ioctl 1261 to a partition!
> [548246.718155] mdadm: sending ioctl 800c0910 to a partition!
> [548246.718163] mdadm: sending ioctl 800c0910 to a partition!
> [548246.718174] mdadm: sending ioctl 1261 to a partition!
> [548246.718180] mdadm: sending ioctl 1261 to a partition!
> [548246.720524] mdadm: sending ioctl 800c0910 to a partition!
> [548246.720533] mdadm: sending ioctl 800c0910 to a partition!
> [548247.265498] md: md0 stopped.
> [548247.269420] md: bind<sdc1>
> [548247.271426] md: bind<sdd1>
> [548247.271471] md: kicking non-fresh sdc1 from array!
> [548247.271478] md: unbind<sdc1>
> [548247.274669] md: export_rdev(sdc1)
> [548247.332487] md/raid1:md0: active with 1 out of 2 mirrors
> [548247.332531] md0: detected capacity change from 0 to 2000155377664
> [548247.334969]  md0: unknown partition table
> [548272.306149] md0: detected capacity change from 2000155377664 to 0
> [548272.306163] md: md0 stopped.
> [548272.306175] md: unbind<sdd1>
> [548272.308646] md: export_rdev(sdd1)

Ivan

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

Personally, I'd probably do something like:
mdadm --assemble /dev/md0 /dev/sdd1
mdadm --manage /dev/md0 --run
mdadm --manage /dev/md0 --add /dev/sdc1

This will cause a full sync from sdd1 to sdc1, which will then ensure
both copies are identical/up to date.

Personally, I would also do:
mdadm --grow /dev/md0 --bitmap=internal
This means next time you have a similar issue, when you add the older
drive, it will only sync the small parts of the drive that are out of
date, instead of the entire drive.

Note: The above assumes that both drives are fully functional. If you
get a read error on sdd1 during the resync, then you will have
additional problems.

Regards,
Adam

On 05/11/13 19:41, Ivan Lezhnjov IV wrote:
> Hi,
>
> I am new to mdadm/software raid and I've built myself a raid1 array, which after a resume from sleep is assembled with 1 out of 2 devices only.
>
> I queried the web, read some threads on the mailing list and learned that event count on these two devices differe a little, and that in a case like this, it would seem, mdadm --assemble --scan --force is a right action. Tried that, but the array is still assembled with only one device.
>
> Where do I go from here?
>
> --examine output:
>
>> % mdadm --examine /dev/sdc1
>> /dev/sdc1:
>>           Magic : a92b4efc
>>         Version : 1.2
>>     Feature Map : 0x0
>>      Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>>            Name : sega:0  (local to host sega)
>>   Creation Time : Fri Nov  1 16:24:18 2013
>>      Raid Level : raid1
>>    Raid Devices : 2
>>
>>  Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>>      Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>>   Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>>     Data Offset : 262144 sectors
>>    Super Offset : 8 sectors
>>           State : clean
>>     Device UUID : 814f6fb3:a8a93019:c04ef011:cfa16124
>>
>>     Update Time : Tue Nov  5 01:03:48 2013
>>        Checksum : 32aca1de - correct
>>          Events : 22
>>
>>
>>    Device Role : Active device 0
>>    Array State : AA ('A' == active, '.' == missing)
>>
>>
>>
>> % mdadm --examine /dev/sdd1
>> /dev/sdd1:
>>           Magic : a92b4efc
>>         Version : 1.2
>>     Feature Map : 0x0
>>      Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>>            Name : sega:0  (local to host sega)
>>   Creation Time : Fri Nov  1 16:24:18 2013
>>      Raid Level : raid1
>>    Raid Devices : 2
>>
>>  Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>>      Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>>   Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>>     Data Offset : 262144 sectors
>>    Super Offset : 8 sectors
>>           State : clean
>>     Device UUID : cea7f341:435cdefd:5f883265:a75c5080
>>
>>     Update Time : Tue Nov  5 07:53:09 2013
>>        Checksum : 55110136 - correct
>>          Events : 30
>>
>>
>>    Device Role : Active device 1
>>    Array State : .A ('A' == active, '.' == missing)
>
> dmesg output:
>
>> [548246.716474] scsi_verify_blk_ioctl: 18 callbacks suppressed
>> [548246.716484] mdadm: sending ioctl 800c0910 to a partition!
>> [548246.716492] mdadm: sending ioctl 800c0910 to a partition!
>> [548246.716512] mdadm: sending ioctl 1261 to a partition!
>> [548246.716518] mdadm: sending ioctl 1261 to a partition!
>> [548246.718155] mdadm: sending ioctl 800c0910 to a partition!
>> [548246.718163] mdadm: sending ioctl 800c0910 to a partition!
>> [548246.718174] mdadm: sending ioctl 1261 to a partition!
>> [548246.718180] mdadm: sending ioctl 1261 to a partition!
>> [548246.720524] mdadm: sending ioctl 800c0910 to a partition!
>> [548246.720533] mdadm: sending ioctl 800c0910 to a partition!
>> [548247.265498] md: md0 stopped.
>> [548247.269420] md: bind<sdc1>
>> [548247.271426] md: bind<sdd1>
>> [548247.271471] md: kicking non-fresh sdc1 from array!
>> [548247.271478] md: unbind<sdc1>
>> [548247.274669] md: export_rdev(sdc1)
>> [548247.332487] md/raid1:md0: active with 1 out of 2 mirrors
>> [548247.332531] md0: detected capacity change from 0 to 2000155377664
>> [548247.334969]  md0: unknown partition table
>> [548272.306149] md0: detected capacity change from 2000155377664 to 0
>> [548272.306163] md: md0 stopped.
>> [548272.306175] md: unbind<sdd1>
>> [548272.308646] md: export_rdev(sdd1)


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

These are brand-new drives, that had fully synced some 20 hours before the array broke this morning, no problems with SMART stats or in OS logs. So, I'm fairly confident that the drives are OK.

Thanks for the detailed reply. I'm going to read a man page for all those arguments.  Meanwhile, I'm wondering if there's still a chance to assemble the array without a complete resync? It would take some 30 hrs with these drives and I'd rather avoid that, besides event count difference seems very small and I can see a lot of people say it is safe to add a non-fresh drive back in that case?

Could somebody please comment on this?

Ivan

On Nov 5, 2013, at 12:04 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:

> Personally, I'd probably do something like:
> mdadm --assemble /dev/md0 /dev/sdd1
> mdadm --manage /dev/md0 --run
> mdadm --manage /dev/md0 --add /dev/sdc1
> 
> This will cause a full sync from sdd1 to sdc1, which will then ensure
> both copies are identical/up to date.
> 
> Personally, I would also do:
> mdadm --grow /dev/md0 --bitmap=internal
> This means next time you have a similar issue, when you add the older
> drive, it will only sync the small parts of the drive that are out of
> date, instead of the entire drive.
> 
> Note: The above assumes that both drives are fully functional. If you
> get a read error on sdd1 during the resync, then you will have
> additional problems.
> 
> Regards,
> Adam

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

I checked out mdadm man page, read some pages on the web, and well, what bitmap chunk size should I go for considering these are WD Elements 2TB drives (what kind of technical information about these drives are relevant when considering a bitmap chunk size?)

Ivan

On Nov 5, 2013, at 12:04 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:

> Personally, I would also do:
> mdadm --grow /dev/md0 --bitmap=internal
> This means next time you have a similar issue, when you add the older
> drive, it will only sync the small parts of the drive that are out of
> date, instead of the entire drive.

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

On 05/11/13 21:37, Ivan Lezhnjov IV wrote:
> These are brand-new drives, that had fully synced some 20 hours before the array broke this morning, no problems with SMART stats or in OS logs. So, I'm fairly confident that the drives are OK.
> 
> Thanks for the detailed reply. I'm going to read a man page for all those arguments.  Meanwhile, I'm wondering if there's still a chance to assemble the array without a complete resync? It would take some 30 hrs with these drives and I'd rather avoid that, besides event count difference seems very small and I can see a lot of people say it is safe to add a non-fresh drive back in that case?
> 
> Could somebody please comment on this?

Personally, I'd prefer to know that the data is correct. It's not like
you actually need to work for those 30 hours, the computer will sync
them for you.

The problem is if you ignore the different contents, for those small
sections of disk (which are sections which were actually written to
recently with live data) you will get different content depending on
which disk you read, up until that section is re-written. The
alternative is to force the array, then run a check, and then a repair.
This will at least allow you to get consistent data regardless of which
disk you read from, however, you won't determine whether it is the
"newer" or "older" data (md will choose at random AFAIK).

Of course, a check will probably take close to 30 hours anyway, so why
not just do it properly, ensure you will get consistent data on both
disks, and that it is the most recent version/up to date.

Finally, while it might be safe to force the event count, it is usually
followed with:
run an fsck
get your data off the array as quickly as possible
rebuild the array

ie, most people are doing this with RAID5/6/10 arrays, and just want to
recover their data. I'm not sure it is suggested to continue to use the
array as normal.

Finally, once you add the bitmap, you will avoid the 30 hour resync in
future, you run all the same commands, but md will magically only sync
the needed sections which might only take a few minutes (yet still
ensure the entire disks are totally in sync, and use the freshest
version of data).

Regards,
Adam

> On Nov 5, 2013, at 12:04 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
> 
>> Personally, I'd probably do something like:
>> mdadm --assemble /dev/md0 /dev/sdd1
>> mdadm --manage /dev/md0 --run
>> mdadm --manage /dev/md0 --add /dev/sdc1
>>
>> This will cause a full sync from sdd1 to sdc1, which will then ensure
>> both copies are identical/up to date.
>>
>> Personally, I would also do:
>> mdadm --grow /dev/md0 --bitmap=internal
>> This means next time you have a similar issue, when you add the older
>> drive, it will only sync the small parts of the drive that are out of
>> date, instead of the entire drive.
>>
>> Note: The above assumes that both drives are fully functional. If you
>> get a read error on sdd1 during the resync, then you will have
>> additional problems.
>>
>> Regards,
>> Adam
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

On 05/11/13 22:32, Ivan Lezhnjov IV wrote:
> I checked out mdadm man page, read some pages on the web, and well, what bitmap chunk size should I go for considering these are WD Elements 2TB drives (what kind of technical information about these drives are relevant when considering a bitmap chunk size?)
> 

Other users might have a better suggestion, but I usually just use the
default.

The only time this will matter is if performance is really important to
you, then you can force a larger bitmap chunk size which will result in
more data being copied during a resync, but less bitmap updates,
therefore possibly better performance.

I mainly use it where I can't afford the 30 hours of resync time.

Regards,
Adam

> Ivan
> 
> On Nov 5, 2013, at 12:04 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
> 
>> Personally, I would also do:
>> mdadm --grow /dev/md0 --bitmap=internal
>> This means next time you have a similar issue, when you add the older
>> drive, it will only sync the small parts of the drive that are out of
>> date, instead of the entire drive.


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

On Nov 5, 2013, at 1:36 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:

> The problem is if you ignore the different contents, for those small
> sections of disk (which are sections which were actually written to
> recently with live data) you will get different content depending on
> which disk you read, up until that section is re-written.

Sounds like we are assuming /dev/sdd1 in my situation has the newest data simply by virtue of being recognized as "fresh" by mdadm? Does this assumption represent the actual state of data in terms of that it is indeed the most recent version possible prior to array failure? Or is it more like this is the only option there is and it's better than nothing?

> The
> alternative is to force the array, then run a check, and then a repair.
> This will at least allow you to get consistent data regardless of which
> disk you read from, however, you won't determine whether it is the
> "newer" or "older" data (md will choose at random AFAIK).

What are these check and repair commands that you refer to when talking about forcing an array? Are they echo check|repair > /sys/block/mdX/md/sync_action ?

When you say force the array, does it translate to a different set of commands than what you showed in the very first reply? What would be those? I'm just curious to see how these things are done when managing the arrays, and examples help a lot!

Ivan

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

On 05/11/13 23:02, Ivan Lezhnjov IV wrote:
> On Nov 5, 2013, at 1:36 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
> 
>> The problem is if you ignore the different contents, for those small
>> sections of disk (which are sections which were actually written to
>> recently with live data) you will get different content depending on
>> which disk you read, up until that section is re-written.
> 
> Sounds like we are assuming /dev/sdd1 in my situation has the newest data simply by virtue of being recognized as "fresh" by mdadm? Does this assumption represent the actual state of data in terms of that it is indeed the most recent version possible prior to array failure? Or is it more like this is the only option there is and it's better than nothing?

The Event count is higher for sdd1, which means md was able to
successfully write to the drive more recently. Therefore, it is
definitely "fresher" or more recent or newer.

>> The
>> alternative is to force the array, then run a check, and then a repair.
>> This will at least allow you to get consistent data regardless of which
>> disk you read from, however, you won't determine whether it is the
>> "newer" or "older" data (md will choose at random AFAIK).
> 
> What are these check and repair commands that you refer to when talking about forcing an array? Are they echo check|repair > /sys/block/mdX/md/sync_action ?

Yes, I didn't look it up, but that looks right.

> When you say force the array, does it translate to a different set of commands than what you showed in the very first reply? What would be those? I'm just curious to see how these things are done when managing the arrays, and examples help a lot!

Yes, there are other ways (which I've never used, just seen people on
this list talk about them) that will force the event count on the older
device up to the newer one, and then md will accept both drives as being
up to date etc. Effectively it just lies and pretends that the data is
correct, forcing the metadata (MD data) to match, but the actual user
data (complete content of the drives) may not match and is not checked.

I won't provide advise on this because I've never done it...

BTW, you haven't mentioned what data is actually on the array (if any),
what you are using it for, or how you got into this state.

Depending on your array usage, you may or may not want to use bitmaps,
and there might be other performance options to tune.

Depending on the current content of the array (with such low event
numbers, it looks like you may not have put data on yet, it might be
easier to just re-create the array (although that will do a resync
anyway, unless you force that to be skipped).

Finally, it would be a good idea to work out how you reached this point.
You really want to avoid having this problem in 6 months when you have
1.8TB of data on the drives....

Hope this helps.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

On Nov 5, 2013, at 2:31 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:

>> When you say force the array, does it translate to a different set of commands than what you showed in the very first reply? What would be those? I'm just curious to see how these things are done when managing the arrays, and examples help a lot!
> 
> Yes, there are other ways (which I've never used, just seen people on
> this list talk about them) that will force the event count on the older
> device up to the newer one, and then md will accept both drives as being
> up to date etc. Effectively it just lies and pretends that the data is
> correct, forcing the metadata (MD data) to match, but the actual user
> data (complete content of the drives) may not match and is not checked.
> 
> I won't provide advise on this because I've never done it…

Hm, makes one wonder what the advantage of this approach is then. It sounds like either of two options let one get access to data immediately, whether they choose to force even count and proceed with recovery or assemble an array and start a resync. I mean, what is it that makes this strategy worthwhile pursuing then? Even offloading data to a separate disk, in case of raid levels that offer data redundancy capability seems unnecessary, as an array disk mirror serve essentially the same purpose.

> 
> BTW, you haven't mentioned what data is actually on the array (if any),
> what you are using it for, or how you got into this state.

Just a personal file storage, some iso images, pictures, music, videos, system backups, virtual machine disk images as well, seeding some torrents and such. Multipurpose, yeah.
The usage pattern is of that occasional heavy writing when making backups (typically once a week) or copying iso images/video (when needed), and more frequent reads of average i/o intensity.  

> Depending on your array usage, you may or may not want to use bitmaps,
> and there might be other performance options to tune.

Mind you, this is a raid1 made out of two external USB 3.0 drives connected to USB 2.0 ports. So, the throughput is not terribly impressive, but I've been working with this configuration using a single disk for a while now and it proved sufficient and stable for my needs. The raid that I've put together some 4-5 days ago is a lazy approach to backups/countermeasure against disk failures. I've had a drive die in my hands shortly before I assembled the array, and I figured it was silly not to have a raid1 in place which clearly could have saved me some pain of extracting most important bits of the data from various places (just other disks I have.. it just happens I have a few around) that I used as extra backup storage locations.

> Depending on the current content of the array (with such low event
> numbers, it looks like you may not have put data on yet, it might be
> easier to just re-create the array (although that will do a resync
> anyway, unless you force that to be skipped).

Actually, prior to the array degradation I had been copying data to it for a several days straight (yeah, as I said the throughput is not very good, peaks at 21-27Mb/s for writes when 3 USB disks are involved in action simultaneously.. that is, copying from one disk to this two disk array, all three connected to the same computer… which I think is still a good number when you think about it!), so it has about 1TB of data that I wouldn't like to lose now :P

> 
> Finally, it would be a good idea to work out how you reached this point.
> You really want to avoid having this problem in 6 months when you have
> 1.8TB of data on the drives….

True. So, my setup is an old Linux laptop that used to be my main workstation and as I've said before the array is connected to it via USB interface. This computer being a hybrid server/workstation now, runs GNOME as desktop environment and a VNC server, and most importantly for our discussion I treat it as a workstation and never shut it down in the night, instead I switch it to sleep mode/hybernate. And that's how the array got out of sync, I resumed the laptop from sleep and the array was already degraded, event counts mismatch and all.

I will have to figure out how pm-utils treats raid devices when doing sleep/resume, maybe intervene and script --stop --scan via a pm user hooks. I think internal bitmaps will be of great help here, because it may take some trying to get it done right.]

Unfortunately, abandoning this configuration will most probably be very time consuming, because the system is so heavily customized by now it will be still easier and quicker to make sure pm plays with the raid array nicely than to say install Ubuntu Server (or workstation that I assume is capable of handling arrays on sleep/resume just nicely).

> 
> Hope this helps.

Tremendously. I appreciate your help very much!

Ivan--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

On 06/11/13 18:20, Ivan Lezhnjov IV wrote:
> 
> On Nov 5, 2013, at 2:31 PM, Adam Goryachev
> <mailinglists@websitemanagers.com.au> wrote:
> 
>>> When you say force the array, does it translate to a different
>>> set of commands than what you showed in the very first reply?
>>> What would be those? I'm just curious to see how these things are
>>> done when managing the arrays, and examples help a lot!
>> 
>> Yes, there are other ways (which I've never used, just seen people
>> on this list talk about them) that will force the event count on
>> the older device up to the newer one, and then md will accept both
>> drives as being up to date etc. Effectively it just lies and
>> pretends that the data is correct, forcing the metadata (MD data)
>> to match, but the actual user data (complete content of the drives)
>> may not match and is not checked.
>> 
>> I won't provide advise on this because I've never done it…
> 
> Hm, makes one wonder what the advantage of this approach is then. It
> sounds like either of two options let one get access to data
> immediately, whether they choose to force even count and proceed with
> recovery or assemble an array and start a resync. I mean, what is it
> that makes this strategy worthwhile pursuing then? Even offloading
> data to a separate disk, in case of raid levels that offer data
> redundancy capability seems unnecessary, as an array disk mirror
> serve essentially the same purpose.

The advantage is in RAID5/6. eg, RAID5 with one disk totally dead, and a
second disk partially dead which only dropped out of the array recently.
You might force the event count to current, then you can get most of
your data back.

You are right, there is no advantage for RAID1.

If you just wanted your data back, then you could have done this:
mdadm --assemble /dev/md1 /dev/sdd1
mdadm --manage /dev/md1 --run
(I think it was sdd1 and md1, adjust as appropriate)...
That simply forces the array to run even though all disks are not
present. It will allow you to mount the array and use as normal, right
up to the point there are not enough data disks left.

>> BTW, you haven't mentioned what data is actually on the array (if
>> any), what you are using it for, or how you got into this state.
> 
> Just a personal file storage, some iso images, pictures, music,
> videos, system backups, virtual machine disk images as well, seeding
> some torrents and such. Multipurpose, yeah. The usage pattern is of
> that occasional heavy writing when making backups (typically once a
> week) or copying iso images/video (when needed), and more frequent
> reads of average i/o intensity.

I don't think you would notice the overhead of the bitmap then... or at
least, it probably won't matter in your scenario...

>> Depending on your array usage, you may or may not want to use
>> bitmaps, and there might be other performance options to tune.
> 
> Mind you, this is a raid1 made out of two external USB 3.0 drives
> connected to USB 2.0 ports. So, the throughput is not terribly
> impressive, but I've been working with this configuration using a
> single disk for a while now and it proved sufficient and stable for
> my needs. The raid that I've put together some 4-5 days ago is a lazy
> approach to backups/countermeasure against disk failures. I've had a
> drive die in my hands shortly before I assembled the array, and I
> figured it was silly not to have a raid1 in place which clearly could
> have saved me some pain of extracting most important bits of the data
> from various places (just other disks I have.. it just happens I have
> a few around) that I used as extra backup storage locations.

I would definitely use a bitmap.... I've found USB drives can be flaky
from time to time. A bitmap will reduce the resync time to minutes. eg,
today I had a similar issue where one 2TB drive is SATA/internal and one
is USB3/external. After bootup I started the array on just the internal
drive, so adding back the USB drive required a resync. It completed in
less than one minute due to the bitmap functionality.

>> Depending on the current content of the array (with such low event 
>> numbers, it looks like you may not have put data on yet, it might
>> be easier to just re-create the array (although that will do a
>> resync anyway, unless you force that to be skipped).
> 
> Actually, prior to the array degradation I had been copying data to
> it for a several days straight (yeah, as I said the throughput is not
> very good, peaks at 21-27Mb/s for writes when 3 USB disks are
> involved in action simultaneously.. that is, copying from one disk to
> this two disk array, all three connected to the same computer… which
> I think is still a good number when you think about it!), so it has
> about 1TB of data that I wouldn't like to lose now :P

You might find you have more than one real USB bus on the computer,
potentially you might get better performance if you move the three
devices to different USB buses. Also, the two members of the RAID1 would
behave better on different buses if possible. You may need to read your
motherboard manual, or trial and error to find this out.

>> Finally, it would be a good idea to work out how you reached this
>> point. You really want to avoid having this problem in 6 months
>> when you have 1.8TB of data on the drives….
> 
> True. So, my setup is an old Linux laptop that used to be my main
> workstation and as I've said before the array is connected to it via
> USB interface. This computer being a hybrid server/workstation now,
> runs GNOME as desktop environment and a VNC server, and most
> importantly for our discussion I treat it as a workstation and never
> shut it down in the night, instead I switch it to sleep
> mode/hybernate. And that's how the array got out of sync, I resumed
> the laptop from sleep and the array was already degraded, event
> counts mismatch and all.
> 
> I will have to figure out how pm-utils treats raid devices when doing
> sleep/resume, maybe intervene and script --stop --scan via a pm user
> hooks. I think internal bitmaps will be of great help here, because
> it may take some trying to get it done right.]
> 
> Unfortunately, abandoning this configuration will most probably be
> very time consuming, because the system is so heavily customized by
> now it will be still easier and quicker to make sure pm plays with
> the raid array nicely than to say install Ubuntu Server (or
> workstation that I assume is capable of handling arrays on
> sleep/resume just nicely).

Have you considered just doing a shutdown instead of sleep/resume?
Presumably it will only sleep at night, and resume in the morning, so it
shouldn't take all that long to startup in the morning. You can probably
set the bios to automatically power on at a preset time in the morning a
few minutes before you would normally want to use it.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2621 bytes --]

On Tue Nov 05, 2013 at 02:02:03PM +0200, Ivan Lezhnjov IV wrote:

> On Nov 5, 2013, at 1:36 PM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:
> 
> > The
> > alternative is to force the array, then run a check, and then a repair.
> > This will at least allow you to get consistent data regardless of which
> > disk you read from, however, you won't determine whether it is the
> > "newer" or "older" data (md will choose at random AFAIK).
> 
> What are these check and repair commands that you refer to when
> talking about forcing an array? Are they echo check|repair >
> /sys/block/mdX/md/sync_action ?
> 
> When you say force the array, does it translate to a different set of
> commands than what you showed in the very first reply? What would be
> those? I'm just curious to see how these things are done when managing
> the arrays, and examples help a lot!
> 
There are 3 ways to get an array back up and running. The basic one is
just to do an assemble. This will only work if there's enough disks with
the same event count (and array state info) to assemble a working array.
It should always work with a RAID1 array as only a single disk is ever
needed.

The second option is to assemble with the --force flag. This will tell
md to ignore the event count when assembling the array (providing
they're close enough). This is only useful when a normal assemble fails
to get an array up and running. As you found, it won't add any extra
disks available, only the minimum needed to get the array back up. I
don't think this will ever be of use with a RAID1 array.

The final option (and it's definitely a last-hope scenario) is to
recreate the array using the exact same parameters as were originally
used and adding the --assume-clean flag. This is fraught with danger as
different mdadm versions have used different offsets (and only the very
latest allow specifying the data offset to use), and any error in
parameters or disk order can result in severe data loss. This should
only be used when all other options fail, and generally only when
advised to by one of the more knowledgeable people on this list (once
they're sure you've tried every other option and have pulled out the
necessary info from the array metadata to run the correct create
command). Again, I don't see this being useful for a RAID1 array.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

On Nov 6, 2013, at 9:57 AM, Adam Goryachev <mailinglists@websitemanagers.com.au> wrote:

> I would definitely use a bitmap.... I've found USB drives can be flaky
> from time to time. A bitmap will reduce the resync time to minutes. eg,
> today I had a similar issue where one 2TB drive is SATA/internal and one
> is USB3/external. After bootup I started the array on just the internal
> drive, so adding back the USB drive required a resync. It completed in
> less than one minute due to the bitmap functionality.

That sounds amazing.

> 
>>> Depending on the current content of the array (with such low event 
>>> numbers, it looks like you may not have put data on yet, it might
>>> be easier to just re-create the array (although that will do a
>>> resync anyway, unless you force that to be skipped).
>> 
>> Actually, prior to the array degradation I had been copying data to
>> it for a several days straight (yeah, as I said the throughput is not
>> very good, peaks at 21-27Mb/s for writes when 3 USB disks are
>> involved in action simultaneously.. that is, copying from one disk to
>> this two disk array, all three connected to the same computer… which
>> I think is still a good number when you think about it!), so it has
>> about 1TB of data that I wouldn't like to lose now :P
> 
> You might find you have more than one real USB bus on the computer,
> potentially you might get better performance if you move the three
> devices to different USB buses. Also, the two members of the RAID1 would
> behave better on different buses if possible. You may need to read your
> motherboard manual, or trial and error to find this out.

Actually, this is a great tip! I will look into this.

> 
>>> Finally, it would be a good idea to work out how you reached this
>>> point. You really want to avoid having this problem in 6 months
>>> when you have 1.8TB of data on the drives….
>> 
>> True. So, my setup is an old Linux laptop that used to be my main
>> workstation and as I've said before the array is connected to it via
>> USB interface. This computer being a hybrid server/workstation now,
>> runs GNOME as desktop environment and a VNC server, and most
>> importantly for our discussion I treat it as a workstation and never
>> shut it down in the night, instead I switch it to sleep
>> mode/hybernate. And that's how the array got out of sync, I resumed
>> the laptop from sleep and the array was already degraded, event
>> counts mismatch and all.
>> 
>> I will have to figure out how pm-utils treats raid devices when doing
>> sleep/resume, maybe intervene and script --stop --scan via a pm user
>> hooks. I think internal bitmaps will be of great help here, because
>> it may take some trying to get it done right.]
>> 
>> Unfortunately, abandoning this configuration will most probably be
>> very time consuming, because the system is so heavily customized by
>> now it will be still easier and quicker to make sure pm plays with
>> the raid array nicely than to say install Ubuntu Server (or
>> workstation that I assume is capable of handling arrays on
>> sleep/resume just nicely).
> 
> Have you considered just doing a shutdown instead of sleep/resume?
> Presumably it will only sleep at night, and resume in the morning, so it
> shouldn't take all that long to startup in the morning. You can probably
> set the bios to automatically power on at a preset time in the morning a
> few minutes before you would normally want to use it.

See, this particular laptop runs on max memory it's motherboard can handle, and for whatever reason the test memory happens very slowly, so it takes about a minute or a minute and a half to just get to boot loader. Besides, I'm such a fan of sleep/resume because it allows for really quick suspend in those rare cases when electricity goes out (happens sometimes where I live), or when I just need to temporarily power off all the noisy equipment to have a nice nap lol And resuming from a sleep and having the system in the state where I left it is just so convenient. I see a lot of value in this functionality. It is perhaps about living comfortably, you know?

Ivan

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

Hello,

so I've successfully rebuilt the array, added internal bitmap, haven't run any extensive i/o tests but I continued with copying of my data off the old disks and I haven't really noticed a serious impact. This is a first impression only, but so far so good.

Now that I have bitmap I deliberately repeated sleep/resume cycle exactly as it was done the last time that led to array degradation and sure enough the system started up with a degraded array. in fact, it is way more messy this time because both devices were dynamically assigned new /dev/sdx devices: before sleep they were /dev/sdc1 and /dev/sdd1, after resume they became /dev/sdd1 and /dev/sdb1.

So, I unmounted filesystem on the array, and stopped the array. Then reassembled it, and it looks to be in a good shape. However, I am wondering if this is exactly due to the internal bitmap. Basically what surprised me was that the array was assembled and shown as in sync instantly. Worth noting, I should say that before the laptop went to sleep there were no processes writing to the array disks -- I made sure -- so the data should be consistent on both drives, but as we know from my very first message event count may be still different when upon resume from sleep.

Anyway, here's how it looked in more specific terms

> % mount
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> dev on /dev type devtmpfs (rw,nosuid,relatime,size=1551228k,nr_inodes=216877,mode=755)
> run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
> /dev/sda3 on / type ext3 (rw,relatime,data=ordered)
> devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
> shm on /dev/shm type tmpfs (rw,nosuid,nodev,relatime)
> binfmt on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
> tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime)
> gvfs-fuse-daemon on /home/ilj/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
> /dev/md0 on /mnt/RAIDVault type ext4 (rw,relatime,stripe=32,data=ordered)
> % cat /proc/mdstat 
> Personalities : [raid1] 
> md0 : active raid1 sdc1[2]
>      1953276736 blocks super 1.2 [2/1] [U_]
>      bitmap: 0/15 pages [0KB], 65536KB chunk
> 
> unused devices: <none>
> % ls -lah /mnt/RAIDVault/
> ls: reading directory /mnt/RAIDVault/: Input/output error
> total 0
> % su
> Password: 
> % umount /mnt/RAIDVault/
> % mdadm --stop --scan
> mdadm: stopped /dev/md0
> …
> here I checked the logs, learned that the drives are now represented by different block device files
> …
> % ls /dev/sd
> sda   sda1  sda2  sda3  sdb   sdb1  sdd   sdd1  sde   
> % mdadm --assemble --scan
> mdadm: /dev/md0 has been started with 2 drives.
> % cat /proc/mdstat 
> Personalities : [raid1] 
> md0 : active raid1 sdd1[2] sdb1[1]
>      1953276736 blocks super 1.2 [2/2] [UU]
>      bitmap: 0/15 pages [0KB], 65536KB chunk
> 
> unused devices: <none>
> % mdadm --examine /dev/sdd1
> /dev/sdd1:
>          Magic : a92b4efc
>        Version : 1.2
>    Feature Map : 0x1
>     Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>           Name : sega:0  (local to host sega)
>  Creation Time : Fri Nov  1 16:24:18 2013
>     Raid Level : raid1
>   Raid Devices : 2
> 
> Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>     Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>  Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>    Data Offset : 262144 sectors
>   Super Offset : 8 sectors
>          State : clean
>    Device UUID : 827ed4c3:baf1ba90:d8f21e10:e524d383
> 
> Internal Bitmap : 8 sectors from superblock
>    Update Time : Fri Nov  8 00:10:07 2013
>       Checksum : 18402838 - correct
>         Events : 56
> 
> 
>   Device Role : Active device 0
>   Array State : AA ('A' == active, '.' == missing)
> % mdadm --examine /dev/sdb1
> /dev/sdb1:
>          Magic : a92b4efc
>        Version : 1.2
>    Feature Map : 0x1
>     Array UUID : c4cf4a52:6daa94c8:6d88a2fa:8f604199
>           Name : sega:0  (local to host sega)
>  Creation Time : Fri Nov  1 16:24:18 2013
>     Raid Level : raid1
>   Raid Devices : 2
> 
> Avail Dev Size : 3906553856 (1862.79 GiB 2000.16 GB)
>     Array Size : 1953276736 (1862.79 GiB 2000.16 GB)
>  Used Dev Size : 3906553472 (1862.79 GiB 2000.16 GB)
>    Data Offset : 262144 sectors
>   Super Offset : 8 sectors
>          State : clean
>    Device UUID : cea7f341:435cdefd:5f883265:a75c5080
> 
> Internal Bitmap : 8 sectors from superblock
>    Update Time : Fri Nov  8 00:10:07 2013
>       Checksum : 55138955 - correct
>         Events : 56
> 
> 
>   Device Role : Active device 1
>   Array State : AA ('A' == active, '.' == missing)

My question is basically if I'm enjoying the benefits of having internal bitmap or maybe I got lucky and this time event count was the same for both drives?

Ivan--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

Alright, apparently that time even count was in sync. That being said, I've seen internal bitmap at work, and it was blazing fast. I've also been using the array for a couple of days now, and as Adam predicted the performance impact is not even noticeable in my particular configuration. Which makes me really happy, of course.

Ivan

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Adam Goryachev]

On 08/11/13 17:32, Ivan Lezhnjov IV wrote:
> Hello,
>
> so I've successfully rebuilt the array, added internal bitmap, haven't run any extensive i/o tests but I continued with copying of my data off the old disks and I haven't really noticed a serious impact. This is a first impression only, but so far so good.
>
> Now that I have bitmap I deliberately repeated sleep/resume cycle exactly as it was done the last time that led to array degradation and sure enough the system started up with a degraded array. in fact, it is way more messy this time because both devices were dynamically assigned new /dev/sdx devices: before sleep they were /dev/sdc1 and /dev/sdd1, after resume they became /dev/sdd1 and /dev/sdb1.
I think this is a different issue, raid is not responsible for device 
discovery and mapping to device names. I think udev may provide a 
solution to this where it will ensure each device identified by some 
distinct hardware feature (eg, serial number), will be configured as a 
specific device name. I use this often for ethernet devices, but I 
assume something similar is applicable to disk drives.

> So, I unmounted filesystem on the array, and stopped the array. Then reassembled it, and it looks to be in a good shape. However, I am wondering if this is exactly due to the internal bitmap. Basically what surprised me was that the array was assembled and shown as in sync instantly. Worth noting, I should say that before the laptop went to sleep there were no processes writing to the array disks -- I made sure -- so the data should be consistent on both drives, but as we know from my very first message event count may be still different when upon resume from sleep.
Yes, given that there were no writes to the array, (or minimal writes, 
probably there is always something), then the re-sync would have been so 
quick you would not have noticed it. As mentioned, it can easily be 
completed in a second... for me, it is often a minute or two due to a 
lot of writes happening during the bootup process.
> My question is basically if I'm enjoying the benefits of having 
> internal bitmap or maybe I got lucky and this time event count was the 
> same for both drives? 
The only way to know for sure would be to examine the drives during the 
bootup process before the raid array is assembled....

You might see some information in the logs about the resync.

Regards,
Adam

-- 
Adam Goryachev
Website Managers
P: +61 2 8304 0000                    adam@websitemanagers.com.au
F: +61 2 8304 0001                     www.websitemanagers.com.au

Re: mdadm: /dev/md0 has been started with 1 drive (out of 2).

[Ivan Lezhnjov IV]

On Nov 11, 2013, at 1:01 AM, Adam Goryachev <adam@websitemanagers.com.au> wrote:

> On 08/11/13 17:32, Ivan Lezhnjov IV wrote:
>> Hello,
>> 
>> so I've successfully rebuilt the array, added internal bitmap, haven't run any extensive i/o tests but I continued with copying of my data off the old disks and I haven't really noticed a serious impact. This is a first impression only, but so far so good.
>> 
>> Now that I have bitmap I deliberately repeated sleep/resume cycle exactly as it was done the last time that led to array degradation and sure enough the system started up with a degraded array. in fact, it is way more messy this time because both devices were dynamically assigned new /dev/sdx devices: before sleep they were /dev/sdc1 and /dev/sdd1, after resume they became /dev/sdd1 and /dev/sdb1.
> I think this is a different issue, raid is not responsible for device discovery and mapping to device names. I think udev may provide a solution to this where it will ensure each device identified by some distinct hardware feature (eg, serial number), will be configured as a specific device name. I use this often for ethernet devices, but I assume something similar is applicable to disk drives.

That is correct, and my problem happens only when system fails to stop array before going to sleep. I've poked around a bit and learned that SAMBA server that uses this array as a share wouldn't allow the system to unmount it cleanly before going into the sleep state. On resume it would become a mess with drives having stale /dev/* devices, /dev/md0 still mounted sometimes but filesystem inaccessible (I/O errors is the only thing that I would see), etc. Long story short, I create pm-utils sleep hook that stops SAMBA, then umounts /dev/md0 and stop array cleanly, and then reassembles, mounts filesystem and starts SAMBA again and that resolves the above mentioned issue. I can now sleep/resume and have the array work as a kind of plug'n'play device ha :)

> 
>> So, I unmounted filesystem on the array, and stopped the array. Then reassembled it, and it looks to be in a good shape. However, I am wondering if this is exactly due to the internal bitmap. Basically what surprised me was that the array was assembled and shown as in sync instantly. Worth noting, I should say that before the laptop went to sleep there were no processes writing to the array disks -- I made sure -- so the data should be consistent on both drives, but as we know from my very first message event count may be still different when upon resume from sleep.
> Yes, given that there were no writes to the array, (or minimal writes, probably there is always something), then the re-sync would have been so quick you would not have noticed it. As mentioned, it can easily be completed in a second... for me, it is often a minute or two due to a lot of writes happening during the bootup process.

Indeed, it is lighting fast. And luckily for me, I don't really see performance decline with default internal bitmap type and its default chunk size.

>> My question is basically if I'm enjoying the benefits of having internal bitmap or maybe I got lucky and this time event count was the same for both drives? 
> The only way to know for sure would be to examine the drives during the bootup process before the raid array is assembled....
> 
> You might see some information in the logs about the resync.

Either I do not know how to read entries produced by mdadm but logs would typically contain little descriptive (yeah, depends on whether I can speak the mdam code language lol) information.

At any rate, I think I have a fairly good understanding of what's happening now. I've seen resync happen in under 5 seconds so it confirmed for me that internal bitmap is working as expected and even better than I expected.

Thanks for the answer, anyway!

Ivan