btrfs-tools: missing device delete/remove cancel option on disk failure

btrfs-tools: missing device delete/remove cancel option on disk failure

[g6094199]

He guys,

i'm running in an rare error which isnt covered by an implemented usecase atm.i have a 4 hdd raid5 array. i like to replace the 4 disks with bigger ones to gain more usable space the NAS has only 4 drive bays so i need to use an usb bay.
since the replace code was unreliable at least before 4.4 afaik, the plan was to add anew disk to the array, balance and remove one of the small disks this should be safe.

BUT what if, if the new drive i'm adding is failing while doing the remove/delete operation? 
this is what i'm running into right now. a brand new disk which has an upcounting raw error rate. 
but there is no option to cancel an delete/remove operation. 
atm it isnt a problem for me. i hope to complete the operation successfully and replace the failingdrive afterwards.

please consider to add an option to exit/cancel an delete/remove operation like in replace/scrub/...

thx!

sash




---
Mail & Cloud Made in Germany mit 3 GB Speicher! https://email.freenet.de/mail/Uebersicht?epid=e9900000450

Re: btrfs-tools: missing device delete/remove cancel option on disk failure

[Martin]

On 07/05/16 10:39, g6094199@freenet.de wrote:
> a brand new disk which has an upcounting raw error rate

Note that is the "raw error rate".

For a brand new disk being run for the first time at maximum data
writes, the "raw error rate" may well be expected to increase. Hard
disks deliberately make use of error correction for normal operation.

More importantly, what do the other smart values show?

For myself, my concern would only be raised for sector failures.


And... A very good test for a new disk is to first run "badblocks" to
test the disk surface. Read the man page first. (Hint: Non-destructive
is slow, destructive write is fast...)

Good luck,
Martin

Re: btrfs-tools: missing device delete/remove cancel option on disk failure

[g6094199]

Am 08.05.2016 um 02:54 schrieb Martin:
> On 07/05/16 10:39, g6094199@freenet.de wrote:
>> a brand new disk which has an upcounting raw error rate
> Note that is the "raw error rate".
>
> For a brand new disk being run for the first time at maximum data
> writes, the "raw error rate" may well be expected to increase. Hard
> disks deliberately make use of error correction for normal operation.
>
> More importantly, what do the other smart values show?
>
> For myself, my concern would only be raised for sector failures.
>
>
> And... A very good test for a new disk is to first run "badblocks" to
> test the disk surface. Read the man page first. (Hint: Non-destructive
> is slow, destructive write is fast...)
>
> Good luck,
> Martin
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

i guess this log is out of diskussion:

[44388.089321] sd 8:0:0:0: [sdf] tag#0 FAILED Result:
hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
[44388.089334] sd 8:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 00 43 1c 48
00 00 08 00
[44388.089340] blk_update_request: I/O error, dev sdf, sector 35185216

...

May  7 06:39:31 NAS-Sash kernel: [35777.520490] sd 8:0:0:0: [sdf] tag#0
FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May  7 06:39:31 NAS-Sash kernel: [35777.520500] sd 8:0:0:0: [sdf] tag#0
Sense Key : Medium Error [current]
May  7 06:39:31 NAS-Sash kernel: [35777.520508] sd 8:0:0:0: [sdf] tag#0
Add. Sense: Unrecovered read error
May  7 06:39:31 NAS-Sash kernel: [35777.520516] sd 8:0:0:0: [sdf] tag#0
CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00
May  7 06:39:31 NAS-Sash kernel: [35777.520522] blk_update_request:
critical medium error, dev sdf, sector 472347008
May  7 06:39:35 NAS-Sash kernel: [35781.364117] sd 8:0:0:0: [sdf] tag#0
FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May  7 06:39:35 NAS-Sash kernel: [35781.364138] sd 8:0:0:0: [sdf] tag#0
Sense Key : Medium Error [current]
May  7 06:39:35 NAS-Sash kernel: [35781.364146] sd 8:0:0:0: [sdf] tag#0
Add. Sense: Unrecovered read error
May  7 06:39:35 NAS-Sash kernel: [35781.364154] sd 8:0:0:0: [sdf] tag#0
CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00

and different vendors use the raw error rate differently. some count up
constantly, some do only log real destructive errors.

but i had the luck that the system froze completely. not even an log
entry. now the file system is broken.....arg!

now i need some advice what to do next....best practice wise? try to
mount degraded and copy off all data? then i will net at least 9TB of
new storage... :-(


sash

Re: btrfs-tools: missing device delete/remove cancel option on disk failure

[Chris Murphy]

On Sun, May 8, 2016 at 5:53 AM,  <g6094199@freenet.de> wrote:

>
> i guess this log is out of diskussion:
>
> [44388.089321] sd 8:0:0:0: [sdf] tag#0 FAILED Result:
> hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK
> [44388.089334] sd 8:0:0:0: [sdf] tag#0 CDB: Read(10) 28 00 00 43 1c 48
> 00 00 08 00
> [44388.089340] blk_update_request: I/O error, dev sdf, sector 35185216
>
> ...
>
> May  7 06:39:31 NAS-Sash kernel: [35777.520490] sd 8:0:0:0: [sdf] tag#0
> FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> May  7 06:39:31 NAS-Sash kernel: [35777.520500] sd 8:0:0:0: [sdf] tag#0
> Sense Key : Medium Error [current]
> May  7 06:39:31 NAS-Sash kernel: [35777.520508] sd 8:0:0:0: [sdf] tag#0
> Add. Sense: Unrecovered read error
> May  7 06:39:31 NAS-Sash kernel: [35777.520516] sd 8:0:0:0: [sdf] tag#0
> CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00
> May  7 06:39:31 NAS-Sash kernel: [35777.520522] blk_update_request:
> critical medium error, dev sdf, sector 472347008
> May  7 06:39:35 NAS-Sash kernel: [35781.364117] sd 8:0:0:0: [sdf] tag#0
> FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> May  7 06:39:35 NAS-Sash kernel: [35781.364138] sd 8:0:0:0: [sdf] tag#0
> Sense Key : Medium Error [current]
> May  7 06:39:35 NAS-Sash kernel: [35781.364146] sd 8:0:0:0: [sdf] tag#0
> Add. Sense: Unrecovered read error
> May  7 06:39:35 NAS-Sash kernel: [35781.364154] sd 8:0:0:0: [sdf] tag#0
> CDB: Read(10) 28 00 03 84 ee 30 00 00 04 00

What drive is this? Is it one of the new drives? It's very unusual
that this new drive which should be getting mostly writes (what is
there to read?) during a 'dev delete' or 'replace' operation, but in
particular to have a read error this soon after data has been written.
I suspect maybe it's some other drive.

>
> and different vendors use the raw error rate differently. some count up
> constantly, some do only log real destructive errors.
>
> but i had the luck that the system froze completely. not even an log
> entry. now the file system is broken.....arg!




>
> now i need some advice what to do next....best practice wise? try to
> mount degraded and copy off all data? then i will net at least 9TB of
> new storage... :-(

If you don't have a backup already, you're kinda twice behind best
practices, aren't you? One you should have one anyway, two you should
have one before doing anything risky like growing the array or doing
device replacements.

The best practice at this point would be to mount with -o degraded,ro
I wouldn't even do rw mounts. The less you change things the better
off the file system is. And then copy the data to a new array before
making any changes to the original volume. And hopefully mounting
degraded,ro works, if not then it's btrfs restore (offline scrape)
time.



-- 
Chris Murphy

Re: btrfs-tools: missing device delete/remove cancel option on disk failure

[Duncan]

g6094199 posted on Sun, 08 May 2016 13:53:41 +0200 as excerpted:

> now i need some advice what to do next....best practice wise? try to
> mount degraded and copy off all data? then i will net at least 9TB of
> new storage... :-(

Well, even in general, regardless of the filesystem or what it's stored 
on, the sysadmin's rule of backups, in simple form, says if it's not 
backed up to at least one level, you are by your lack of backup defining 
the data as worth less than the time, trouble and resources necessary to 
do that backup.

And btrfs, being still stabilizing, not yet fully stable, reinforces that 
rule.  I know nobody around here that would recommend usage of btrfs in 
any mode without backups, unless the data on it is truly of trivial 
value, testing only and not worth backing up.

And btrfs parity-raid, raid56 mode, which you're using as you reported 
raid5 (I'm assuming btrfs raid5 here), is new and not yet as stable as 
btrfs is in general, with at least two known bugs related to recovery 
from loss of device, making it for purposes of recovery planning raid0.  
Nobody I know is even recommending that for production use even with 
backups at this point.  It's recommended only for testing, not for in-use 
deployment unless you really /are/ prepared to be scrapping it and 
starting over more or less constantly.

And of course the only way you couldn't know that before starting is if 
you didn't care enough about the stability of your data to do your 
research on the filesystem you were choosing in the first place, which 
would again point to the data simply not being worth enough to care about.


Which means if that data wasn't already backed up, you were already 
defining it as trivial testing data, and the only reason you may have for 
trying to recover it is to test the recovery procedures.  Otherwise, 
consider it a loss and blow it away with a new badblocks or mkfs, ready 
for a new test, as you were already declaring the data as testing data, 
not worth trying to recover, by storing it on btrfs parity raid, which is 
known to be recommended only for testing with already backed up or 
trivial value data you plan on losing, at this point.


So you don't need 9 TB of new storage, because either you already have it 
available and are using it for the backup you already have, or you had 
already demonstrated in multiple different ways that the data was really 
of only trivial value to you, so you can simply blow away the existing 
broken filesystem and simply start over with a new test.  Words can claim 
otherwise, but actions don't lie.  8^0

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman