Re: btrfs device delete missing - Input/output error

Re: btrfs device delete missing - Input/output error

[Tomasz Chmielewski]

> The first way to try to recover the current volume would be to
> overwrite LBA 2262535088 which you should only do with the filesystem
> unmounted. That's the sector causing the read error. If this is a 512
> byte drive:
> 
> dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088

It is a 512 byte drive.

Unfortunately overwriting that fails - which is quite weird, given that 
there are 0 reallocated sectors, according to SMART.

Is there a way to determine what's stored there (i.e. if it's a file?).


# dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088
dd: writing to `/dev/sdb': Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 2.88439 s, 0.0 kB/s

# dmesg -c
[ 8177.713212] ata4.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 
0x0
[ 8177.713285] ata4.00: irq_stat 0x40000008
[ 8177.713349] ata4.00: failed command: READ FPDMA QUEUED
[ 8177.713419] ata4.00: cmd 60/08:50:b0:8b:db/00:00:86:00:00/40 tag 10 
ncq 4096 in
[ 8177.713419]          res 41/40:08:b0:8b:db/00:00:86:00:00/00 Emask 
0x409 (media error) <F>
[ 8177.713662] ata4.00: status: { DRDY ERR }
[ 8177.713725] ata4.00: error: { UNC }
[ 8177.755099] ata4.00: configured for UDMA/133
[ 8177.755175] sd 3:0:0:0: [sdb]
[ 8177.755252] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 8177.755318] sd 3:0:0:0: [sdb]
[ 8177.755380] Sense Key : Medium Error [current] [descriptor]
[ 8177.755449] Descriptor sense data with sense descriptors (in hex):
[ 8177.755516]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[ 8177.755600]         86 db 8b b0
[ 8177.755667] sd 3:0:0:0: [sdb]
[ 8177.755729] Add. Sense: Unrecovered read error - auto reallocate 
failed
[ 8177.757316] sd 3:0:0:0: [sdb] CDB:
[ 8177.757377] Read(16): 88 00 00 00 00 00 86 db 8b b0 00 00 00 08 00 00
[ 8177.757463] end_request: I/O error, dev sdb, sector 2262535088
[ 8177.757542] ata4: EH complete

-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: btrfs device delete missing - Input/output error

[Chris Murphy]

On Fri, Dec 5, 2014 at 3:03 AM, Tomasz Chmielewski <tch@virtall.com> wrote:
>> The first way to try to recover the current volume would be to
>> overwrite LBA 2262535088 which you should only do with the filesystem
>> unmounted. That's the sector causing the read error. If this is a 512
>> byte drive:
>>
>> dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088
>
>
> It is a 512 byte drive.
>
> Unfortunately overwriting that fails - which is quite weird, given that
> there are 0 reallocated sectors, according to SMART.
>
> Is there a way to determine what's stored there (i.e. if it's a file?).
>
>
> # dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088
> dd: writing to `/dev/sdb': Input/output error
> 1+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 2.88439 s, 0.0 kB/s
>
> # dmesg -c
> [ 8177.713212] ata4.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 0x0
> [ 8177.713285] ata4.00: irq_stat 0x40000008
> [ 8177.713349] ata4.00: failed command: READ FPDMA QUEUED



You're getting a read error when writing. This is expected when writing 512
bytes to a 4k sector.

What do you get for
parted /dev/sdb u s p



-- 
Chris Murphy

Re: btrfs device delete missing - Input/output error

[Tomasz Chmielewski]

On 2014-12-05 17:41, Chris Murphy wrote:

> You're getting a read error when writing. This is expected when
> writing 512 bytes to a 4k sector.
> 
> What do you get for
>  parted /dev/sdb u s p

Right - so we're getting these:

# parted /dev/sdb u s p
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 5860533168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start        End          Size         File system  Name  Flags
  5      2048s        4095s        2048s                           
bios_grub
  1      4096s        67112959s    67108864s                       raid
  2      67112960s    68161535s    1048576s                        raid
  3      68161536s    2215645183s  2147483648s                     raid
  4      2215645184s  5860533134s  3644887951s  btrfs              raid


-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: btrfs device delete missing - Input/output error

[Chris Murphy]

On Fri, Dec 5, 2014 at 10:28 AM, Tomasz Chmielewski <tch@virtall.com> wrote:
> On 2014-12-05 17:41, Chris Murphy wrote:
>
>> You're getting a read error when writing. This is expected when
>> writing 512 bytes to a 4k sector.
>>
>> What do you get for
>>  parted /dev/sdb u s p
>
>
> Right - so we're getting these:
>
> # parted /dev/sdb u s p
> Model: ATA ST3000DM001-9YN1 (scsi)
> Disk /dev/sdb: 5860533168s
> Sector size (logical/physical): 512B/4096B

OK it's a 4096 byte physical sector drive so you have to use the
bs=4096 command with the proper seek value (which is based on the bs
value).



-- 
Chris Murphy

Re: btrfs device delete missing - Input/output error

[Tomasz Chmielewski]

On 2014-12-06 05:57, Chris Murphy wrote:

>> Right - so we're getting these:
>> 
>> # parted /dev/sdb u s p
>> Model: ATA ST3000DM001-9YN1 (scsi)
>> Disk /dev/sdb: 5860533168s
>> Sector size (logical/physical): 512B/4096B
> 
> OK it's a 4096 byte physical sector drive so you have to use the
> bs=4096 command with the proper seek value (which is based on the bs
> value).

Yep.
So here is what I did:

# echo 2262535088 / 8|bc    # (512 * 8 = 4096)
282816886


* verify if it's this place

# dd if=/dev/sdb of=/dev/null bs=4096 count=1 skip=282816886
dd: reading `/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 2.83002 s, 0.0 kB/s


* overwrite it:

dd if=/dev/zero of=/dev/sdb bs=4096 count=1 seek=282816886
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 3.0277e-05 s, 135 MB/s


* try to read it:

# dd if=/dev/sdb of=/dev/null bs=4096 count=1 skip=282816886
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 2.6204e-05 s, 156 MB/s


* try to read with the old skip value (repeat for 2262535088 - 
2262535095, or use a different count value - this is where we were 
getting the errors, it's also 8 * 512):

dd if=/dev/sdb of=/dev/null count=1 skip=2262535088
...
dd if=/dev/sdb of=/dev/null count=1 skip=2262535095


* Unfortunately this is still an error for btrfs, because the checksum 
does not match:

# time btrfs device delete missing /home
ERROR: error removing the device 'missing' - Input/output error

# dmesg -c
[84200.109774] BTRFS info (device sda4): relocating block group 
1375492636672 flags 17
[84203.635559] BTRFS info (device sda4): csum failed ino 261 off 
384262144 csum 2566472073 expected csum 4193010590
[84203.650980] BTRFS info (device sda4): csum failed ino 261 off 
384262144 csum 2566472073 expected csum 4193010590
[84203.651444] BTRFS info (device sda4): csum failed ino 261 off 
384262144 csum 2566472073 expected csum 4193010590


* remounting with nodatacow (nodatasum) does not help (since it's for 
new data)

* let's find the inode printing the error - turned out to be a directory 
- so created a new one, moved the data from the "corrupt one", removed 
the directory with that inode:

# find /home -mount -inum 261
/home/backup/ma-int/weekly


* repeat for any other found:

# dmesg -c
[85197.300494] BTRFS info (device sda4): relocating block group 
1375492636672 flags 17
[85200.448713] BTRFS info (device sda4): csum failed ino 267 off 
384262144 csum 2566472073 expected csum 4193010590
[85200.472581] BTRFS info (device sda4): csum failed ino 267 off 
384262144 csum 2566472073 expected csum 4193010590
[85200.473551] BTRFS info (device sda4): csum failed ino 267 off 
384262144 csum 2566472073 expected csum 4193010590


* unfortunately it never ends - let's say we have 
/home/backup/ma-int/weekly, which we "fixed" with:

mkdir /home/backup/ma-int/weekly.tmp
mv /home/backup/ma-int/weekly/* /home/backup/ma-int/weekly.tmp
rmdir /home/backup/ma-int/weekly

After we run again btrfs device delete missing /home, the newly created 
directory eventually (/home/backup/ma-int/weekly.tmp) is being detected 
as "csum failed ino ...".


-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: btrfs device delete missing - Input/output error

[Chris Murphy]

On Sat, Dec 6, 2014 at 2:17 AM, Tomasz Chmielewski <tch@virtall.com> wrote:

> After we run again btrfs device delete missing /home, the newly created
> directory eventually (/home/backup/ma-int/weekly.tmp) is being detected as
> "csum failed ino ...".

Unmount the volume and run btrfs check, without repair:

1.
If that comes up clean then chances are this is file corruption and
you can mount and do:

btrfs scrub start -r <MP>

This will show you filenames for corrupt files you'll need to locate
backups to restore from eventually. Then unmount it again and do:

btrfs check --init-csum-tree <dev>

This will remove the current checksum tree and calculate new checksums
(only very recent btrfs-progs does this, I tested with 3.17.3), so any
files that are corrupt will no longer be flagged as corrupt, and you
should be able to remove the missing device and resume the balance
needed to restore normal operation for the mirrored volume. If you
don't have recent btrfs-progs, the initialization of the csum tree
will strip the csums entirely, and you'll need to recreate the file
system (or live without checksums).

2.
I'm going to bet btrfs check doesn't come up clean, based on the
additional problems you reported. In this case it's not file
corruption, but filesystem corruption. Even though the mirror isn't
intact, there's a possibility this small hole in the file system can
be repaired with very recent fsck patches from Josef. You'd have to
check whether the patches he's submitted in the last month or so to
the list have been incorporated into btrfs-progs integration branch;
and build a btrfs-progs that has them. Then use --repair and cross
fingers.


Certainly before 2. and probably even before 1, you ought to go ahead
and get anything off the volume you care about, seeing as right now
it's at least mounting. It'd be good to know either way if it works.
And if it doesn't work, take a btrfs-image of the fs, Josef might be
interested in testing against it seeing as only a 4KB hole was punched
into the file system.



-- 
Chris Murphy

Re: btrfs device delete missing - Input/output error

[Tomasz Chmielewski]

On 2014-12-07 06:26, Chris Murphy wrote:
> On Sat, Dec 6, 2014 at 2:17 AM, Tomasz Chmielewski <tch@virtall.com> 
> wrote:
> 
>> After we run again btrfs device delete missing /home, the newly 
>> created
>> directory eventually (/home/backup/ma-int/weekly.tmp) is being 
>> detected as
>> "csum failed ino ...".

> If that comes up clean then chances are this is file corruption and
> you can mount and do:
> 
> btrfs scrub start -r <MP>

Actually, this seemed to be enough.

I didn't realize that scrub would print the names of corrupted files.

After removing them, "btrfs device delete missing" is finally making 
some progress (previously, it would break at 300+GB; still running now):

# btrfs fi show /home
Label: none  uuid: 92e93437-cc9f-475d-a739-085f3270896b
         Total devices 3 FS bytes used 1.40TiB
         devid    2 size 1.70TiB used 1.42TiB path /dev/sdb4
         devid    3 size 1.70TiB used 807.00GiB path /dev/sda4
         *** Some devices missing

Btrfs v3.14.2


Thanks for your hints!

-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: btrfs device delete missing - Input/output error

[Duncan]

Chris Murphy posted on Thu, 04 Dec 2014 15:25:08 -0700 as excerpted:

> If this is a 512 byte drive:

LOL!  And I used to think a (so-called) 1.44 MB floppy was small!  Now 
you're worried about a half a KiB drive!  =:^)

(Yes, I know what you /meant/.  But the image this created in my head was 
too amusing to pass up! =:^)


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs device delete missing - Input/output error

[Chris Murphy]

On Thu, Dec 4, 2014 at 1:55 PM, Tomasz Chmielewski <tch@virtall.com> wrote:
> btrfs RAID-1, one of the drive failed and was replaced.
>
> Unfortunately the second drive is also showing errors and "btrfs device
> delete missing" exits with error:
>
> # btrfs device delete missing /home
> ERROR: error removing the device 'missing' - Input/output error
>
>
> Is there a way for btrfs to ignore these errors? I'd rather loose a few
> files here and there than the whole filesystem.

If this is file system metadata that it can't read due to the UNC
error, and if that metadata isn't already replicated on the mirror,
then it's not ignorable. It might not be fatal if it's file specific
metadata or data, if you can determine which file it is, and then
either do an rsync excluding that file to a new file system, or faster
and a bit more risky is to delete the file and then do a btrfs
send/receive to a new file system.

What I'd probably do is reboot, mount the filesystem read only, and
rsync what I wanted from this troubled volume onto a new volume. This
is the most conservative approach, not trusting either the hardware or
the file system. Only once I've got the data I want safe, would I
entertain trying to fix the existing troubled volume; or if the rsync
persistently fails to do a complete recovery.

The first way to try to recover the current volume would be to
overwrite LBA 2262535088 which you should only do with the filesystem
unmounted. That's the sector causing the read error. If this is a 512
byte drive:

dd if=/dev/zero of=/dev/sdb count=1 seek=2262535088

If it's 4096 bytes then you need to do:
dd if=/dev/zero of=/dev/sdb bs=4096 count=1 seek=282816886

Next do btrfs check without repair and see if it finds problems and
report that. If it doesn't find problems, then chances are it's either
file specific metadata or data in which case a scrub will report the
affected file but otherwise continue on. And now you can just delete
that file.

If you do a weekly or even a monthly scrub, the chances bad sectors
crop up that aren't automatically repairable with raid1 is quite low.

-- 
Chris Murphy

btrfs device delete missing - Input/output error

[Tomasz Chmielewski]

btrfs RAID-1, one of the drive failed and was replaced.

Unfortunately the second drive is also showing errors and "btrfs device 
delete missing" exits with error:

# btrfs device delete missing /home
ERROR: error removing the device 'missing' - Input/output error


Is there a way for btrfs to ignore these errors? I'd rather loose a few 
files here and there than the whole filesystem.


# btrfs fi show /home
Label: none  uuid: 92e93437-cc9f-475d-a739-085f3270896b
         Total devices 3 FS bytes used 1.40TiB
         devid    2 size 1.70TiB used 1.46TiB path /dev/sdb4
         devid    3 size 1.70TiB used 322.00GiB path /dev/sda4
         *** Some devices missing


# smartctl -a /dev/sdb | grep -i Reallocated
   5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always 
       -       0


# dmesg -c
[26677.752448] BTRFS info (device sdb4): relocating block group 
1375492636672 flags 17
[26684.803713] ata4.00: exception Emask 0x0 SAct 0x7c000003 SErr 0x0 
action 0x0
[26684.803786] ata4.00: irq_stat 0x40000008
[26684.803846] ata4.00: failed command: READ FPDMA QUEUED
[26684.803899] ata4.00: cmd 60/00:d0:00:89:db/04:00:86:00:00/40 tag 26 
ncq 524288 in
[26684.803899]          res 41/40:00:b0:8b:db/00:04:86:00:00/00 Emask 
0x409 (media error) <F>
[26684.805329] ata4.00: status: { DRDY ERR }
[26684.805388] ata4.00: error: { UNC }
[26684.876401] ata4.00: configured for UDMA/133
[26684.876499] sd 3:0:0:0: [sdb]
[26684.876564] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[26684.876628] sd 3:0:0:0: [sdb]
[26684.876687] Sense Key : Medium Error [current] [descriptor]
[26684.876749] Descriptor sense data with sense descriptors (in hex):
[26684.876810]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[26684.876871]         86 db 8b b0
[26684.876921] sd 3:0:0:0: [sdb]
[26684.876970] Add. Sense: Unrecovered read error - auto reallocate 
failed
[26684.877032] sd 3:0:0:0: [sdb] CDB:
[26684.877081] Read(16): 88 00 00 00 00 00 86 db 89 00 00 00 04 00 00 00
[26684.877143] end_request: I/O error, dev sdb, sector 2262535088
[26684.877196] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 15, flush 0, corrupt 
0, gen 0
[26684.877294] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 16, flush 0, corrupt 
0, gen 0
[26684.877393] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 17, flush 0, corrupt 
0, gen 0
[26684.877511] ata4: EH complete
[26687.978787] ata4.00: exception Emask 0x0 SAct 0x400 SErr 0x0 action 
0x0
[26687.978872] ata4.00: irq_stat 0x40000008
[26687.978937] ata4.00: failed command: READ FPDMA QUEUED
[26687.979006] ata4.00: cmd 60/80:50:80:8b:db/01:00:86:00:00/40 tag 10 
ncq 196608 in
[26687.979006]          res 41/40:80:b0:8b:db/00:01:86:00:00/00 Emask 
0x409 (media error) <F>
[26687.979250] ata4.00: status: { DRDY ERR }
[26687.979312] ata4.00: error: { UNC }
[26688.059820] ata4.00: configured for UDMA/133
[26688.059909] sd 3:0:0:0: [sdb]
[26688.059970] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[26688.060041] sd 3:0:0:0: [sdb]
[26688.060115] Sense Key : Medium Error [current] [descriptor]
[26688.060184] Descriptor sense data with sense descriptors (in hex):
[26688.060251]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[26688.060334]         86 db 8b b0
[26688.060399] sd 3:0:0:0: [sdb]
[26688.060462] Add. Sense: Unrecovered read error - auto reallocate 
failed
[26688.060530] sd 3:0:0:0: [sdb] CDB:
[26688.060590] Read(16): 88 00 00 00 00 00 86 db 8b 80 00 00 01 80 00 00
[26688.060677] end_request: I/O error, dev sdb, sector 2262535088
[26688.060744] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 18, flush 0, corrupt 
0, gen 0
[26688.060875] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 19, flush 0, corrupt 
0, gen 0
[26688.061011] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 20, flush 0, corrupt 
0, gen 0
[26688.061151] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 21, flush 0, corrupt 
0, gen 0
[26688.061275] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 22, flush 0, corrupt 
0, gen 0
[26688.061446] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 23, flush 0, corrupt 
0, gen 0
[26688.061602] ata4: EH complete
[26690.921047] ata4.00: exception Emask 0x0 SAct 0x800 SErr 0x0 action 
0x0
[26690.921119] ata4.00: irq_stat 0x40000008
[26690.921179] ata4.00: failed command: READ FPDMA QUEUED
[26690.921232] ata4.00: cmd 60/08:58:b0:8b:db/00:00:86:00:00/40 tag 11 
ncq 4096 in
[26690.921232]          res 41/40:08:b0:8b:db/00:00:86:00:00/00 Emask 
0x409 (media error) <F>
[26690.921426] ata4.00: status: { DRDY ERR }
[26690.921475] ata4.00: error: { UNC }
[26691.002497] ata4.00: configured for UDMA/133
[26691.002576] sd 3:0:0:0: [sdb]
[26691.002634] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[26691.002686] sd 3:0:0:0: [sdb]
[26691.002734] Sense Key : Medium Error [current] [descriptor]
[26691.002787] Descriptor sense data with sense descriptors (in hex):
[26691.002839]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[26691.002900]         86 db 8b b0
[26691.002951] sd 3:0:0:0: [sdb]
[26691.002999] Add. Sense: Unrecovered read error - auto reallocate 
failed
[26691.003053] sd 3:0:0:0: [sdb] CDB:
[26691.003101] Read(16): 88 00 00 00 00 00 86 db 8b b0 00 00 00 08 00 00
[26691.003163] end_request: I/O error, dev sdb, sector 2262535088
[26691.003215] btrfs_dev_stat_print_on_error: 36 callbacks suppressed
[26691.003268] BTRFS: bdev /dev/sdb4 errs: wr 0, rd 60, flush 0, corrupt 
0, gen 0
[26691.003383] ata4: EH complete



-- 
Tomasz Chmielewski
http://www.sslrack.com