What is the solution for USB HDs

What is the solution for USB HDs

[juca]

Dear Linux-RAID Developers,

I always wanted to have a small home server with a RAID-array to store 
pesonal and work files.

So I decided to create a fileserver with a raspberry pi and a RAID-1 
Array. For this I bought two USB 1TB HDs.

After a while I noticed that Mdadm was marking my drives as failed or 
even worse as faulty. This was strange since both HDs are new.

So I tested them. I deleted the RAID partitions completely with WIPEFS. 
Then I created Ext4 partitions and run a benchmark of the partitions in 
ubuntu. Everything is fine.

I created then again my RAID array. To my surprise one drive was again 
faulty. I googled a little  and I found that my USB drives went to sleep 
mode and when they were woken up, one of them changed name /dev/sdb/ -> 
/dev/sdc. Mdadm was unable to find the "new" drive and marked it as 
faulty. Re-add didn't work. Adding it again went into an unnecessary and 
long sync of 1TB.

I googled the problem and learnt about UDEV. It turns out that Linux 
gracefully creates symlinks to the disks-partitions under 
/dev/disks/by-id/... I thought that I could use this symlinks to fix the 
problem with Mdadm, but mdadm just followed the links to find again the 
changing /dev/sd* names.

I continued my journey and found more about the UDEV rules. I created a 
couple of them, using the serial number of the disks. I turns out that 
changing the name is not anymore allowed. (NAME="sdb" is ignored) There 
is even a warning about this. See here: 
http://unix.stackexchange.com/questions/119593/is-there-a-way-to-change-device-names-in-dev-directory
"NAME="pendrak" ignored, kernel device nodes can not be renamed; please 
fix it in /etc/udev/rules.d/99-local.rules:1"

My last desperate idea was to create a cron job that reads directly from 
the disks, to keep them awake and avoid a name change. Something like 
this:

dd if=/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0 
bs=1024 count=$(($[1<<10]*10)) skip=$(($SKIP)) ibs=1024 of=/dev/null 
iflag=direct status=none 2>&1

But even with that workaround the name of a disk just changed and Mdadm 
marked it as faulty. --- I'm out of ideas.

So I'm stuck with this wonderful situation:

- MDADM will follow symlinks and ignore the /dev/disk/by-id/... 
identification.
- UDEV does not allow to change the name of a device.
- The names will apparently change no matter what I do.

I'm starting to regret my decision to create this array. I expected some 
challenges but it is turning to be ridiculously difficult. Almost 
unusable.

Maybe you have a solution for this.

In any case it would be great if you consider some long-term 
out-of-the-box solution for this new reality of USB HDs. My ideas would 
be:

-- Allow to create arrays with symlinks 
("/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0")

-- Include a keep-alive functionality that avoids names to change 
randomly.

-- Store HD ids to recognize disks.

Thank you.

Best regards,

Juan Carlos Carvajal Bermúdez
+43 650 477 0005
juca@juan-carlos.info
Beingasse 17/2/3
AT 1150, Wien
www.juan-carlos.info

Re: What is the solution for USB HDs

[Adam Goryachev]

On 12/12/16 04:14, juca@juan-carlos.info wrote:
> Dear Linux-RAID Developers,
>
> I always wanted to have a small home server with a RAID-array to store 
> pesonal and work files.
>
> So I decided to create a fileserver with a raspberry pi and a RAID-1 
> Array. For this I bought two USB 1TB HDs.
>
> After a while I noticed that Mdadm was marking my drives as failed or 
> even worse as faulty. This was strange since both HDs are new.
>
> So I tested them. I deleted the RAID partitions completely with 
> WIPEFS. Then I created Ext4 partitions and run a benchmark of the 
> partitions in ubuntu. Everything is fine.
>
> I created then again my RAID array. To my surprise one drive was again 
> faulty. I googled a little  and I found that my USB drives went to 
> sleep mode and when they were woken up, one of them changed name 
> /dev/sdb/ -> /dev/sdc. Mdadm was unable to find the "new" drive and 
> marked it as faulty. Re-add didn't work. Adding it again went into an 
> unnecessary and long sync of 1TB.
>
> I googled the problem and learnt about UDEV. It turns out that Linux 
> gracefully creates symlinks to the disks-partitions under 
> /dev/disks/by-id/... I thought that I could use this symlinks to fix 
> the problem with Mdadm, but mdadm just followed the links to find 
> again the changing /dev/sd* names.
>
> I continued my journey and found more about the UDEV rules. I created 
> a couple of them, using the serial number of the disks. I turns out 
> that changing the name is not anymore allowed. (NAME="sdb" is ignored) 
> There is even a warning about this. See here: 
> http://unix.stackexchange.com/questions/119593/is-there-a-way-to-change-device-names-in-dev-directory
> "NAME="pendrak" ignored, kernel device nodes can not be renamed; 
> please fix it in /etc/udev/rules.d/99-local.rules:1"
>
> My last desperate idea was to create a cron job that reads directly 
> from the disks, to keep them awake and avoid a name change. Something 
> like this:
>
> dd if=/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0 
> bs=1024 count=$(($[1<<10]*10)) skip=$(($SKIP)) ibs=1024 of=/dev/null 
> iflag=direct status=none 2>&1
>
> But even with that workaround the name of a disk just changed and 
> Mdadm marked it as faulty. --- I'm out of ideas.
>
> So I'm stuck with this wonderful situation:
>
> - MDADM will follow symlinks and ignore the /dev/disk/by-id/... 
> identification.
> - UDEV does not allow to change the name of a device.
> - The names will apparently change no matter what I do.
>
> I'm starting to regret my decision to create this array. I expected 
> some challenges but it is turning to be ridiculously difficult. Almost 
> unusable.
>
> Maybe you have a solution for this.
>
> In any case it would be great if you consider some long-term 
> out-of-the-box solution for this new reality of USB HDs. My ideas 
> would be:
>
> -- Allow to create arrays with symlinks 
> ("/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0")
>
> -- Include a keep-alive functionality that avoids names to change 
> randomly.
>
> -- Store HD ids to recognize disks.
>

Hi,
I think you are going down the wrong path. It isn't for Linux MD to 
magically "keep drives awake" or to detect which drive is the one you 
want to use.

Also, you haven't provided enough information/detail on *why* the drives 
are vanishing. For a drive to enter sleep mode, and then wake up again, 
it shouldn't change device name. The only time it will change device 
name (in my experience) is if you unplug it, and then re-plug it.
Given you are using a Raspberry Pi, and I've quite some experience with 
them, my initial question would be to confirm that you are providing 
sufficient power to the RPi, and to the 2 x USB HDD's, if there is not 
enough power, then one drive could easily "die" and then wake up again 
when the other drive reduces its power demand. One of the best ways to 
solve this is to:
a) Use desktop drives (ie, drives that have their own power supply, and 
don't just draw power from USB
b) Use a powered USB hub (ie, a USB hub that has it's own power supply)
c) Use USB flash drives, or SSD's, I suspect both will have a lower 
power demand

If you are still having problems after that, then please try to post 
more details on what happens when the drives vanish (eg, kernel logs, 
system logs, etc).

Also, you might consider an alternative SBC, some have SATA ports 
already available (RPi are the only SBC's I've used, but there are many 
other options/variations).

Regards,
Adam



-- 
Adam Goryachev Website Managers www.websitemanagers.com.au

Re: What is the solution for USB HDs

[Phil Turmel]

On 12/11/2016 05:14 PM, Adam Goryachev wrote:
> On 12/12/16 04:14, juca@juan-carlos.info wrote:
> If you are still having problems after that, then please try to post
> more details on what happens when the drives vanish (eg, kernel logs,
> system logs, etc).
> 
> Also, you might consider an alternative SBC, some have SATA ports
> already available (RPi are the only SBC's I've used, but there are many
> other options/variations).

There's also a kernel boot parameter that can cut off all USB
auto-suspend, which I suspect is part of the problem.  It has been known
for a long time that USB is not robust enough to be trusted in MD arrays.

{ looking around..... }

usbcore.autosuspend=-1

is what you need in your bootloader or builtin to the kernel.

Phil

Re: What is the solution for USB HDs

[Wols Lists]

On 12/12/16 03:17, Phil Turmel wrote:
> On 12/11/2016 05:14 PM, Adam Goryachev wrote:
>> > On 12/12/16 04:14, juca@juan-carlos.info wrote:
>> > If you are still having problems after that, then please try to post
>> > more details on what happens when the drives vanish (eg, kernel logs,
>> > system logs, etc).
>> > 
>> > Also, you might consider an alternative SBC, some have SATA ports
>> > already available (RPi are the only SBC's I've used, but there are many
>> > other options/variations).
> There's also a kernel boot parameter that can cut off all USB
> auto-suspend, which I suspect is part of the problem.  It has been known
> for a long time that USB is not robust enough to be trusted in MD arrays.

Okay, I've just parroted it on the wiki, but the wiki does recommend
against USB disks, precisely because every mention (very few) I've seen
says it's a bad idea - because of suspend.

Cheers,
Wol

Re: What is the solution for USB HDs

[Juan Carlos Carvajal B.]

Hi Adam,

thanks for the suggestions. Looking at the syslog I managed to see
what is happening. Apparently there is a bug in the Raspberry that
resets USB devices. See here:
https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=47239

Apparently is the kernel itself that disconnects the devices.

Dec 13 07:52:52 fileserver kernel: [121050.860021] usb 1-1.3.2: USB
disconnect, device number 20
Dec 13 07:52:52 fileserver kernel: [121050.861653] sd 15:0:0:0: [sdc]
Synchronizing SCSI cache
Dec 13 07:52:52 fileserver kernel: [121050.861828] sd 15:0:0:0: [sdc]
Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00
Dec 13 07:52:52 fileserver udisksd[1963]: Unable to resolve
/sys/devices/virtual/block/md0/md/dev-sdb/block symlink
Dec 13 07:52:52 fileserver kernel: [121051.195770] usb 1-1.3.2: new
high-speed USB device number 21 using dwc_otg

The full dump is here:
http://pastebin.com/F18GHLjV

I'll post this also in the raspberry forums to see what they say.
thanks!

Best regards!

Juan Carlos Carvajal Bermúdez
+43 650 477 0005
juca@juan-carlos.info
Beingasse 17/2/3
AT 1150, Wien
www.juan-carlos.info

On 11 December 2016 at 23:14, Adam Goryachev
<mailinglists@websitemanagers.com.au> wrote:
>
> On 12/12/16 04:14, juca@juan-carlos.info wrote:
>>
>> Dear Linux-RAID Developers,
>>
>> I always wanted to have a small home server with a RAID-array to store pesonal and work files.
>>
>> So I decided to create a fileserver with a raspberry pi and a RAID-1 Array. For this I bought two USB 1TB HDs.
>>
>> After a while I noticed that Mdadm was marking my drives as failed or even worse as faulty. This was strange since both HDs are new.
>>
>> So I tested them. I deleted the RAID partitions completely with WIPEFS. Then I created Ext4 partitions and run a benchmark of the partitions in ubuntu. Everything is fine.
>>
>> I created then again my RAID array. To my surprise one drive was again faulty. I googled a little  and I found that my USB drives went to sleep mode and when they were woken up, one of them changed name /dev/sdb/ -> /dev/sdc. Mdadm was unable to find the "new" drive and marked it as faulty. Re-add didn't work. Adding it again went into an unnecessary and long sync of 1TB.
>>
>> I googled the problem and learnt about UDEV. It turns out that Linux gracefully creates symlinks to the disks-partitions under /dev/disks/by-id/... I thought that I could use this symlinks to fix the problem with Mdadm, but mdadm just followed the links to find again the changing /dev/sd* names.
>>
>> I continued my journey and found more about the UDEV rules. I created a couple of them, using the serial number of the disks. I turns out that changing the name is not anymore allowed. (NAME="sdb" is ignored) There is even a warning about this. See here: http://unix.stackexchange.com/questions/119593/is-there-a-way-to-change-device-names-in-dev-directory
>> "NAME="pendrak" ignored, kernel device nodes can not be renamed; please fix it in /etc/udev/rules.d/99-local.rules:1"
>>
>> My last desperate idea was to create a cron job that reads directly from the disks, to keep them awake and avoid a name change. Something like this:
>>
>> dd if=/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0 bs=1024 count=$(($[1<<10]*10)) skip=$(($SKIP)) ibs=1024 of=/dev/null iflag=direct status=none 2>&1
>>
>> But even with that workaround the name of a disk just changed and Mdadm marked it as faulty. --- I'm out of ideas.
>>
>> So I'm stuck with this wonderful situation:
>>
>> - MDADM will follow symlinks and ignore the /dev/disk/by-id/... identification.
>> - UDEV does not allow to change the name of a device.
>> - The names will apparently change no matter what I do.
>>
>> I'm starting to regret my decision to create this array. I expected some challenges but it is turning to be ridiculously difficult. Almost unusable.
>>
>> Maybe you have a solution for this.
>>
>> In any case it would be great if you consider some long-term out-of-the-box solution for this new reality of USB HDs. My ideas would be:
>>
>> -- Allow to create arrays with symlinks ("/dev/disk/by-id/usb-TOSHIBA_External_USB_3.0_20151214504D3-0:0")
>>
>> -- Include a keep-alive functionality that avoids names to change randomly.
>>
>> -- Store HD ids to recognize disks.
>>
>
> Hi,
> I think you are going down the wrong path. It isn't for Linux MD to magically "keep drives awake" or to detect which drive is the one you want to use.
>
> Also, you haven't provided enough information/detail on *why* the drives are vanishing. For a drive to enter sleep mode, and then wake up again, it shouldn't change device name. The only time it will change device name (in my experience) is if you unplug it, and then re-plug it.
> Given you are using a Raspberry Pi, and I've quite some experience with them, my initial question would be to confirm that you are providing sufficient power to the RPi, and to the 2 x USB HDD's, if there is not enough power, then one drive could easily "die" and then wake up again when the other drive reduces its power demand. One of the best ways to solve this is to:
> a) Use desktop drives (ie, drives that have their own power supply, and don't just draw power from USB
> b) Use a powered USB hub (ie, a USB hub that has it's own power supply)
> c) Use USB flash drives, or SSD's, I suspect both will have a lower power demand
>
> If you are still having problems after that, then please try to post more details on what happens when the drives vanish (eg, kernel logs, system logs, etc).
>
> Also, you might consider an alternative SBC, some have SATA ports already available (RPi are the only SBC's I've used, but there are many other options/variations).
>
> Regards,
> Adam
>
>
>
> --
> Adam Goryachev Website Managers www.websitemanagers.com.au

Re: What is the solution for USB HDs

[Juan Carlos Carvajal B.]

Hi Phil,

usbcore.autosuspend=-1

This might be the solution! I'll test it and let you know.

And regarding power issues:

- this was the first thing I notices and I bought a powered usb-hub to
provide enough current to the HDs.

Using USB drives is kind of a home user scenario. I understand that
MDADM might be more focused on large data centers, but would be great
to have such a great tool also available for the average joe.

Best!
Juan Carlos Carvajal Bermúdez
+43 650 477 0005
juca@juan-carlos.info
Beingasse 17/2/3
AT 1150, Wien
www.juan-carlos.info


On 12 December 2016 at 04:17, Phil Turmel <philip@turmel.org> wrote:
> On 12/11/2016 05:14 PM, Adam Goryachev wrote:
>> On 12/12/16 04:14, juca@juan-carlos.info wrote:
>> If you are still having problems after that, then please try to post
>> more details on what happens when the drives vanish (eg, kernel logs,
>> system logs, etc).
>>
>> Also, you might consider an alternative SBC, some have SATA ports
>> already available (RPi are the only SBC's I've used, but there are many
>> other options/variations).
>
> There's also a kernel boot parameter that can cut off all USB
> auto-suspend, which I suspect is part of the problem.  It has been known
> for a long time that USB is not robust enough to be trusted in MD arrays.
>
> { looking around..... }
>
> usbcore.autosuspend=-1
>
> is what you need in your bootloader or builtin to the kernel.
>
> Phil

Re: What is the solution for USB HDs

[Thomas Fjellstrom]

On Wednesday, December 14, 2016 12:08:47 PM MST you wrote:
> Hi Phil,
> 
> usbcore.autosuspend=-1
> 
> This might be the solution! I'll test it and let you know.
> 
> And regarding power issues:
> 
> - this was the first thing I notices and I bought a powered usb-hub to
> provide enough current to the HDs.
> 
> Using USB drives is kind of a home user scenario. I understand that
> MDADM might be more focused on large data centers, but would be great
> to have such a great tool also available for the average joe.

I don't know if I'd call linux md-raid a "home user" senario. home user is 
more just not having any kind of backup or at most a single usb stick or usb 
hdd. 

Now, I use md-raid but I am not typical. I have a little self built NAS box, 
with 5x5 TB disks in raid 5. (I had thought I did raid 6, but I think I 
skipped it cause I have a full backup of the array on another box using a 
bunch of my older drives). Nice litle box using the Lian-Li PC Q25 case. 
Though I'd like to upgrade to the PC Q26, but its hard to find in canada for a 
decent price :(

Most of my friends use synology or drobo NAS boxes so they don't have to care 
about all the inner-workings most of the time.

I find USB disks too finicky for any real use. Also many are rather slow. I do 
however use them on occasion, especially usb3/3.1 ssds for quick backup or re-
image work when a usb stick won't cut it.

> Best!
> Juan Carlos Carvajal Bermúdez
> +43 650 477 0005
> juca@juan-carlos.info
> Beingasse 17/2/3
> AT 1150, Wien
> www.juan-carlos.info
> 
> On 12 December 2016 at 04:17, Phil Turmel <philip@turmel.org> wrote:
> > On 12/11/2016 05:14 PM, Adam Goryachev wrote:
> >> On 12/12/16 04:14, juca@juan-carlos.info wrote:
> >> If you are still having problems after that, then please try to post
> >> more details on what happens when the drives vanish (eg, kernel logs,
> >> system logs, etc).
> >> 
> >> Also, you might consider an alternative SBC, some have SATA ports
> >> already available (RPi are the only SBC's I've used, but there are many
> >> other options/variations).
> > 
> > There's also a kernel boot parameter that can cut off all USB
> > auto-suspend, which I suspect is part of the problem.  It has been known
> > for a long time that USB is not robust enough to be trusted in MD arrays.
> > 
> > { looking around..... }
> > 
> > usbcore.autosuspend=-1
> > 
> > is what you need in your bootloader or builtin to the kernel.
> > 
> > Phil
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Thomas Fjellstrom
thomas@fjellstrom.ca