* suspected BTRFS errors resulting in file system becoming unrecovable
@ 2016-01-25 14:58 WillIam Thorne
2016-01-25 16:36 ` Patrik Lundquist
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: WillIam Thorne @ 2016-01-25 14:58 UTC (permalink / raw)
To: linux-btrfs
Hi
I have a WD 3TB external HD attached over USB to an arm based micro PC (rasp pi). I was experimenting with btrfs for storing email archives but recently encountered some problems which resulted in the filesystem becoming apparently unrecoverable. I’m not an expert and it was quicker to switch back to ext4 and restored from backup so no support needed. Here what appears to be the relevant part of the syslog including the stack trace in case it is useful:
Best
W
pi@mail /var/log $ btrfs --version
Btrfs Btrfs v0.19
pi@mail /var/log $ uname -a
Linux mail 4.1.7-v7+ #817 SMP PREEMPT Sat Sep 19 15:32:00 BST 2015 armv7l GNU/Linux
Jan 20 09:42:08 mail kernel: [2762753.507576] usb 1-1.5: reset high-speed USB device number 4 using dwc_otg
Jan 20 09:43:18 mail kernel: [2762823.972777] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.972806] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.972819] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.972837] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2c 20 00 00 f0 00
Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
Jan 20 09:43:18 mail kernel: [2762823.972867] BTRFS: bdev /dev/sda1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.976998] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.977015] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.977028] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.977042] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2d 10 00 00 f0 00
Jan 20 09:43:18 mail kernel: [2762823.977052] blk_update_request: I/O error, dev sda, sector 16198928
Jan 20 09:43:18 mail kernel: [2762823.977066] BTRFS: bdev /dev/sda1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.978226] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.978273] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.978286] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.978301] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2e 00 00 00 20 00
Jan 20 09:43:18 mail kernel: [2762823.978318] blk_update_request: I/O error, dev sda, sector 16199168
Jan 20 09:43:18 mail kernel: [2762823.978339] BTRFS: bdev /dev/sda1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.983241] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.983264] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.983277] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.983292] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c3 f8 00 00 f0 00
Jan 20 09:43:18 mail kernel: [2762823.983304] blk_update_request: I/O error, dev sda, sector 16172024
Jan 20 09:43:18 mail kernel: [2762823.983319] BTRFS: bdev /dev/sda1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.987799] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.987818] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.987830] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.987844] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c4 e8 00 00 f0 00
Jan 20 09:43:18 mail kernel: [2762823.987855] blk_update_request: I/O error, dev sda, sector 16172264
Jan 20 09:43:18 mail kernel: [2762823.987869] BTRFS: bdev /dev/sda1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.989136] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.989155] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.989168] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.989182] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c5 d8 00 00 20 00
Jan 20 09:43:18 mail kernel: [2762823.989193] blk_update_request: I/O error, dev sda, sector 16172504
Jan 20 09:43:18 mail kernel: [2762823.989207] BTRFS: bdev /dev/sda1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.993965] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.993988] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.994000] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.994015] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 17 53 68 00 00 c0 00
Jan 20 09:43:18 mail kernel: [2762823.994028] blk_update_request: I/O error, dev sda, sector 1528680
Jan 20 09:43:18 mail kernel: [2762823.994043] BTRFS: bdev /dev/sda1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.994058] BTRFS: bdev /dev/sda1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.994070] BTRFS: bdev /dev/sda1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.997419] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Jan 20 09:43:18 mail kernel: [2762823.997435] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
Jan 20 09:43:18 mail kernel: [2762823.997448] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
Jan 20 09:43:18 mail kernel: [2762823.997461] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 37 53 68 00 00 c0 00
Jan 20 09:43:18 mail kernel: [2762823.997471] blk_update_request: I/O error, dev sda, sector 3625832
Jan 20 09:43:18 mail kernel: [2762823.997484] BTRFS: bdev /dev/sda1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Jan 20 09:43:18 mail kernel: [2762823.997601] BTRFS: error (device sda1) in btrfs_commit_transaction:2068: errno=-5 IO failure (Error while writing out transaction)
Jan 20 09:43:18 mail kernel: [2762824.011517] BTRFS info (device sda1): forced readonly
Jan 20 09:43:18 mail kernel: [2762824.011537] BTRFS warning (device sda1): Skipping commit of aborted transaction.
Jan 20 09:43:18 mail kernel: [2762824.011576] ------------[ cut here ]------------
Jan 20 09:43:18 mail kernel: [2762824.011682] WARNING: CPU: 0 PID: 1318 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0xd8/0x128 [btrfs]()
Jan 20 09:43:18 mail kernel: [2762824.011709] BTRFS: Transaction aborted (error -5)
Jan 20 09:43:18 mail kernel: [2762824.011717] Modules linked in: cfg80211 rfkill snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd btrfs xor xor_neon raid6_pq zlib_deflate sg bcm2835_gpiomem uio_pdrv_genirq uio
Jan 20 09:43:18 mail kernel: [2762824.011790] CPU: 0 PID: 1318 Comm: btrfs-transacti Not tainted 4.1.7-v7+ #817
Jan 20 09:43:18 mail kernel: [2762824.011797] Hardware name: BCM2709
Jan 20 09:43:18 mail kernel: [2762824.011832] [<80018440>] (unwind_backtrace) from [<80013e0c>] (show_stack+0x20/0x24)
Jan 20 09:43:18 mail kernel: [2762824.011852] [<80013e0c>] (show_stack) from [<80558548>] (dump_stack+0x98/0xe0)
Jan 20 09:43:18 mail kernel: [2762824.011872] [<80558548>] (dump_stack) from [<80026a4c>] (warn_slowpath_common+0x8c/0xc8)
Jan 20 09:43:18 mail kernel: [2762824.011892] [<80026a4c>] (warn_slowpath_common) from [<80026ac8>] (warn_slowpath_fmt+0x40/0x48)
Jan 20 09:43:18 mail kernel: [2762824.011971] [<80026ac8>] (warn_slowpath_fmt) from [<7f051790>] (__btrfs_abort_transaction+0xd8/0x128 [btrfs])
Jan 20 09:43:18 mail kernel: [2762824.012153] [<7f051790>] (__btrfs_abort_transaction [btrfs]) from [<7f082a84>] (btrfs_commit_transaction+0x330/0xd40 [btrfs])
Jan 20 09:43:18 mail kernel: [2762824.012353] [<7f082a84>] (btrfs_commit_transaction [btrfs]) from [<7f07e95c>] (transaction_kthread+0x174/0x1ec [btrfs])
Jan 20 09:43:18 mail kernel: [2762824.012463] [<7f07e95c>] (transaction_kthread [btrfs]) from [<80042498>] (kthread+0xe8/0x104)
Jan 20 09:43:18 mail kernel: [2762824.012481] [<80042498>] (kthread) from [<8000fa58>] (ret_from_fork+0x14/0x3c)
Jan 20 09:43:18 mail kernel: [2762824.012492] ---[ end trace 1c48a450ca505104 ]---
Jan 20 09:43:18 mail kernel: [2762824.012505] BTRFS: error (device sda1) in cleanup_transaction:1692: errno=-5 IO failure
Jan 20 09:43:18 mail kernel: [2762824.022734] BTRFS info (device sda1): delayed_refs has NO entry
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 14:58 suspected BTRFS errors resulting in file system becoming unrecovable WillIam Thorne
@ 2016-01-25 16:36 ` Patrik Lundquist
2016-01-25 16:43 ` Austin S. Hemmelgarn
2016-01-25 20:48 ` Chris Murphy
2 siblings, 0 replies; 11+ messages in thread
From: Patrik Lundquist @ 2016-01-25 16:36 UTC (permalink / raw)
To: WillIam Thorne; +Cc: linux-btrfs
On 25 January 2016 at 15:58, WillIam Thorne <will.thorne@me.com> wrote:
>
> Hi
>
> I have a WD 3TB external HD attached over USB to an arm based micro PC (rasp pi). I was experimenting with btrfs for storing email archives but recently encountered some problems which resulted in the filesystem becoming apparently unrecoverable. I’m not an expert and it was quicker to switch back to ext4 and restored from backup so no support needed. Here what appears to be the relevant part of the syslog including the stack trace in case it is useful:
>
> Best
> W
>
> pi@mail /var/log $ btrfs --version
> Btrfs Btrfs v0.19
You're using an ancient btrfs-progs.
> pi@mail /var/log $ uname -a
> Linux mail 4.1.7-v7+ #817 SMP PREEMPT Sat Sep 19 15:32:00 BST 2015 armv7l GNU/Linux
>
> Jan 20 09:42:08 mail kernel: [2762753.507576] usb 1-1.5: reset high-speed USB device number 4 using dwc_otg
> Jan 20 09:43:18 mail kernel: [2762823.972777] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.972806] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.972819] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.972837] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2c 20 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
> Jan 20 09:43:18 mail kernel: [2762823.972867] BTRFS: bdev /dev/sda1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Your USB controller resets device #4 and then you get write errors on
sda. Not a Btrfs problem.
> Jan 20 09:43:18 mail kernel: [2762823.976998] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.977015] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.977028] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.977042] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2d 10 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.977052] blk_update_request: I/O error, dev sda, sector 16198928
> Jan 20 09:43:18 mail kernel: [2762823.977066] BTRFS: bdev /dev/sda1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.978226] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.978273] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.978286] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.978301] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2e 00 00 00 20 00
> Jan 20 09:43:18 mail kernel: [2762823.978318] blk_update_request: I/O error, dev sda, sector 16199168
> Jan 20 09:43:18 mail kernel: [2762823.978339] BTRFS: bdev /dev/sda1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.983241] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.983264] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.983277] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.983292] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c3 f8 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.983304] blk_update_request: I/O error, dev sda, sector 16172024
> Jan 20 09:43:18 mail kernel: [2762823.983319] BTRFS: bdev /dev/sda1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.987799] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.987818] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.987830] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.987844] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c4 e8 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.987855] blk_update_request: I/O error, dev sda, sector 16172264
> Jan 20 09:43:18 mail kernel: [2762823.987869] BTRFS: bdev /dev/sda1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.989136] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.989155] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.989168] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.989182] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c5 d8 00 00 20 00
> Jan 20 09:43:18 mail kernel: [2762823.989193] blk_update_request: I/O error, dev sda, sector 16172504
> Jan 20 09:43:18 mail kernel: [2762823.989207] BTRFS: bdev /dev/sda1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.993965] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.993988] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.994000] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.994015] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 17 53 68 00 00 c0 00
> Jan 20 09:43:18 mail kernel: [2762823.994028] blk_update_request: I/O error, dev sda, sector 1528680
> Jan 20 09:43:18 mail kernel: [2762823.994043] BTRFS: bdev /dev/sda1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.994058] BTRFS: bdev /dev/sda1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.994070] BTRFS: bdev /dev/sda1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.997419] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.997435] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.997448] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.997461] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 37 53 68 00 00 c0 00
> Jan 20 09:43:18 mail kernel: [2762823.997471] blk_update_request: I/O error, dev sda, sector 3625832
> Jan 20 09:43:18 mail kernel: [2762823.997484] BTRFS: bdev /dev/sda1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.997601] BTRFS: error (device sda1) in btrfs_commit_transaction:2068: errno=-5 IO failure (Error while writing out transaction)
> Jan 20 09:43:18 mail kernel: [2762824.011517] BTRFS info (device sda1): forced readonly
> Jan 20 09:43:18 mail kernel: [2762824.011537] BTRFS warning (device sda1): Skipping commit of aborted transaction.
> Jan 20 09:43:18 mail kernel: [2762824.011576] ------------[ cut here ]------------
> Jan 20 09:43:18 mail kernel: [2762824.011682] WARNING: CPU: 0 PID: 1318 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0xd8/0x128 [btrfs]()
> Jan 20 09:43:18 mail kernel: [2762824.011709] BTRFS: Transaction aborted (error -5)
> Jan 20 09:43:18 mail kernel: [2762824.011717] Modules linked in: cfg80211 rfkill snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd btrfs xor xor_neon raid6_pq zlib_deflate sg bcm2835_gpiomem uio_pdrv_genirq uio
> Jan 20 09:43:18 mail kernel: [2762824.011790] CPU: 0 PID: 1318 Comm: btrfs-transacti Not tainted 4.1.7-v7+ #817
> Jan 20 09:43:18 mail kernel: [2762824.011797] Hardware name: BCM2709
> Jan 20 09:43:18 mail kernel: [2762824.011832] [<80018440>] (unwind_backtrace) from [<80013e0c>] (show_stack+0x20/0x24)
> Jan 20 09:43:18 mail kernel: [2762824.011852] [<80013e0c>] (show_stack) from [<80558548>] (dump_stack+0x98/0xe0)
> Jan 20 09:43:18 mail kernel: [2762824.011872] [<80558548>] (dump_stack) from [<80026a4c>] (warn_slowpath_common+0x8c/0xc8)
> Jan 20 09:43:18 mail kernel: [2762824.011892] [<80026a4c>] (warn_slowpath_common) from [<80026ac8>] (warn_slowpath_fmt+0x40/0x48)
> Jan 20 09:43:18 mail kernel: [2762824.011971] [<80026ac8>] (warn_slowpath_fmt) from [<7f051790>] (__btrfs_abort_transaction+0xd8/0x128 [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012153] [<7f051790>] (__btrfs_abort_transaction [btrfs]) from [<7f082a84>] (btrfs_commit_transaction+0x330/0xd40 [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012353] [<7f082a84>] (btrfs_commit_transaction [btrfs]) from [<7f07e95c>] (transaction_kthread+0x174/0x1ec [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012463] [<7f07e95c>] (transaction_kthread [btrfs]) from [<80042498>] (kthread+0xe8/0x104)
> Jan 20 09:43:18 mail kernel: [2762824.012481] [<80042498>] (kthread) from [<8000fa58>] (ret_from_fork+0x14/0x3c)
> Jan 20 09:43:18 mail kernel: [2762824.012492] ---[ end trace 1c48a450ca505104 ]---
> Jan 20 09:43:18 mail kernel: [2762824.012505] BTRFS: error (device sda1) in cleanup_transaction:1692: errno=-5 IO failure
> Jan 20 09:43:18 mail kernel: [2762824.022734] BTRFS info (device sda1): delayed_refs has NO entry
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 14:58 suspected BTRFS errors resulting in file system becoming unrecovable WillIam Thorne
2016-01-25 16:36 ` Patrik Lundquist
@ 2016-01-25 16:43 ` Austin S. Hemmelgarn
2016-01-25 21:12 ` Chris Murphy
[not found] ` <6F3946F5-07EC-43EB-A86F-41D4E3491EDD@me.com>
2016-01-25 20:48 ` Chris Murphy
2 siblings, 2 replies; 11+ messages in thread
From: Austin S. Hemmelgarn @ 2016-01-25 16:43 UTC (permalink / raw)
To: WillIam Thorne, linux-btrfs
On 2016-01-25 09:58, WillIam Thorne wrote:
> Hi
>
> I have a WD 3TB external HD attached over USB to an arm based micro PC (rasp pi). I was experimenting with btrfs for storing email archives but recently encountered some problems which resulted in the filesystem becoming apparently unrecoverable. I’m not an expert and it was quicker to switch back to ext4 and restored from backup so no support needed. Here what appears to be the relevant part of the syslog including the stack trace in case it is useful:
>
> Best
> W
>
> pi@mail /var/log $ btrfs --version
> Btrfs Btrfs v0.19
In general, if you plan to use BTRFS on Debian (or Raspbian), you should
be building the tools yourself locally, Debian is almost as bad about
staying up to date as most enterprise distros.
>
> pi@mail /var/log $ uname -a
> Linux mail 4.1.7-v7+ #817 SMP PREEMPT Sat Sep 19 15:32:00 BST 2015 armv7l GNU/Linux
>
> Jan 20 09:42:08 mail kernel: [2762753.507576] usb 1-1.5: reset high-speed USB device number 4 using dwc_otg
> Jan 20 09:43:18 mail kernel: [2762823.972777] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.972806] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.972819] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.972837] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2c 20 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
This line right here ^^^ indicates that it was triggered by an issue
with the USB device. I don't personally know enough about USB-MSC and
SCSI to know for certain what is happening, but you should probably scan
your logs and make sure you're not still getting stuff like this,
because if you are, you're likely to get data corruption on any
filesystem on the device. Based on this, the BTRFS trace you got is
probably a result of problems with the USB device.
> Jan 20 09:43:18 mail kernel: [2762823.997601] BTRFS: error (device sda1) in btrfs_commit_transaction:2068: errno=-5 IO failure (Error while writing out transaction)
> Jan 20 09:43:18 mail kernel: [2762824.011517] BTRFS info (device sda1): forced readonly
> Jan 20 09:43:18 mail kernel: [2762824.011537] BTRFS warning (device sda1): Skipping commit of aborted transaction.
> Jan 20 09:43:18 mail kernel: [2762824.011576] ------------[ cut here ]------------
> Jan 20 09:43:18 mail kernel: [2762824.011682] WARNING: CPU: 0 PID: 1318 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0xd8/0x128 [btrfs]()
> Jan 20 09:43:18 mail kernel: [2762824.011709] BTRFS: Transaction aborted (error -5)
> Jan 20 09:43:18 mail kernel: [2762824.011717] Modules linked in: cfg80211 rfkill snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd btrfs xor xor_neon raid6_pq zlib_deflate sg bcm2835_gpiomem uio_pdrv_genirq uio
> Jan 20 09:43:18 mail kernel: [2762824.011790] CPU: 0 PID: 1318 Comm: btrfs-transacti Not tainted 4.1.7-v7+ #817
> Jan 20 09:43:18 mail kernel: [2762824.011797] Hardware name: BCM2709
> Jan 20 09:43:18 mail kernel: [2762824.011832] [<80018440>] (unwind_backtrace) from [<80013e0c>] (show_stack+0x20/0x24)
> Jan 20 09:43:18 mail kernel: [2762824.011852] [<80013e0c>] (show_stack) from [<80558548>] (dump_stack+0x98/0xe0)
> Jan 20 09:43:18 mail kernel: [2762824.011872] [<80558548>] (dump_stack) from [<80026a4c>] (warn_slowpath_common+0x8c/0xc8)
> Jan 20 09:43:18 mail kernel: [2762824.011892] [<80026a4c>] (warn_slowpath_common) from [<80026ac8>] (warn_slowpath_fmt+0x40/0x48)
> Jan 20 09:43:18 mail kernel: [2762824.011971] [<80026ac8>] (warn_slowpath_fmt) from [<7f051790>] (__btrfs_abort_transaction+0xd8/0x128 [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012153] [<7f051790>] (__btrfs_abort_transaction [btrfs]) from [<7f082a84>] (btrfs_commit_transaction+0x330/0xd40 [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012353] [<7f082a84>] (btrfs_commit_transaction [btrfs]) from [<7f07e95c>] (transaction_kthread+0x174/0x1ec [btrfs])
> Jan 20 09:43:18 mail kernel: [2762824.012463] [<7f07e95c>] (transaction_kthread [btrfs]) from [<80042498>] (kthread+0xe8/0x104)
> Jan 20 09:43:18 mail kernel: [2762824.012481] [<80042498>] (kthread) from [<8000fa58>] (ret_from_fork+0x14/0x3c)
> Jan 20 09:43:18 mail kernel: [2762824.012492] ---[ end trace 1c48a450ca505104 ]---
> Jan 20 09:43:18 mail kernel: [2762824.012505] BTRFS: error (device sda1) in cleanup_transaction:1692: errno=-5 IO failure
> Jan 20 09:43:18 mail kernel: [2762824.022734] BTRFS info (device sda1): delayed_refs has NO entry
The bit about 'transaction aborted' is almost always indicative of an
error with the storage path (in your case, the USB controller, the USB
cable, or the USB device), not BTRFS. That said, something like this
shouldn't usually cause the FS to be irreparably damaged, although it
will make the FS unusable until you remount (or possibly until you
reboot, I'm not certain about the error handling here because I've never
dealt with it myself).
Now, just a general caution: Avoid using USB storage for persistent
online storage, there's just to many things that can go wrong, and quite
a few USB storage controllers are absolute crap. I understand that this
can be somewhat tricky with something like a Raspberry Pi, but with
BTRFS especially, there's not sufficient error recovery in Linux to
safely use most USB storage devices for anything other than file
transfers or possibly off-line backups. That said, there are some
brands that work well provided they get enough power (I've personally
had really good results using a SanDisk Cruzer Fit flash drive (the USB
3.0 version, I've had only intermittent success with the USB 2.0 ones)
with a Raspberry Pi via a powered hub).
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 16:43 ` Austin S. Hemmelgarn
@ 2016-01-25 21:12 ` Chris Murphy
2016-01-26 12:22 ` Austin S. Hemmelgarn
[not found] ` <6F3946F5-07EC-43EB-A86F-41D4E3491EDD@me.com>
1 sibling, 1 reply; 11+ messages in thread
From: Chris Murphy @ 2016-01-25 21:12 UTC (permalink / raw)
To: Austin S. Hemmelgarn; +Cc: WillIam Thorne, Btrfs BTRFS
On Mon, Jan 25, 2016 at 9:43 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> Now, just a general caution: Avoid using USB storage for persistent online
> storage, there's just to many things that can go wrong, and quite a few USB
> storage controllers are absolute crap.
Yes. Lately, USB 3 stuff is better in that the speeds are approaching
rocket science, so if the manufacturer doesn't get it right, things
rapidly implode. But as far as getting useful error messages and such,
it's almost like the state of networking 20 years ago. You just had to
follow the rules and swap stuff out if it wasn't working. Error codes
might mean something to a developer but they're useless for mortal
users.
I really think USB hubs help fix a lot of USB related problems, even
when it's not a power related problem. Currently I'm using internal
SATA in a NUC for the primary storage, but use send/receive to two
separate raid1 volumes that are USB drives. I can balance/scrub, and
read/write to any and all drives with no problems since getting a
dyconn "industrial" (probably a design term, it's aluminum not
plastic) hub. Before that, one drive or another would just
intermittently reset, usually to no ill effect, but once it wigged
out, vanished, and reappeared as a completely different /dev/sdx
device. The enclosures are crap, the manufacturer (ASMedia Technology
Inc.) didn't bother to fully populate all of the USB descriptors and
it doesn't pass through physical sector size properly, or report max
power correctly, etc.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 21:12 ` Chris Murphy
@ 2016-01-26 12:22 ` Austin S. Hemmelgarn
2016-02-08 2:02 ` Chris Murphy
0 siblings, 1 reply; 11+ messages in thread
From: Austin S. Hemmelgarn @ 2016-01-26 12:22 UTC (permalink / raw)
To: Chris Murphy; +Cc: WillIam Thorne, Btrfs BTRFS
On 2016-01-25 16:12, Chris Murphy wrote:
> On Mon, Jan 25, 2016 at 9:43 AM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> Now, just a general caution: Avoid using USB storage for persistent online
>> storage, there's just to many things that can go wrong, and quite a few USB
>> storage controllers are absolute crap.
>
> Yes. Lately, USB 3 stuff is better in that the speeds are approaching
> rocket science, so if the manufacturer doesn't get it right, things
> rapidly implode. But as far as getting useful error messages and such,
> it's almost like the state of networking 20 years ago. You just had to
> follow the rules and swap stuff out if it wasn't working. Error codes
> might mean something to a developer but they're useless for mortal
> users.
Yeah, with storage devices at least it's a bit easier, they just speak a
form of SCSI over the USB interface (even the older mass storage ones),
so if you have some understanding of SCSI, it's not too hard to
translate. Part of the issue though is that USB was designed originally
for input devices, and that has had a significant impact on how it got
designed. Add to that that most USB devices other than input devices
aren't standards compliant in some way (usually it's incomplete
descriptors or not properly negotiating power usage), and it starts to
become rather impressive that things actually work to the degree they do.
>
> I really think USB hubs help fix a lot of USB related problems, even
> when it's not a power related problem. Currently I'm using internal
> SATA in a NUC for the primary storage, but use send/receive to two
> separate raid1 volumes that are USB drives. I can balance/scrub, and
> read/write to any and all drives with no problems since getting a
> dyconn "industrial" (probably a design term, it's aluminum not
> plastic) hub. Before that, one drive or another would just
> intermittently reset, usually to no ill effect, but once it wigged
> out, vanished, and reappeared as a completely different /dev/sdx
> device. The enclosures are crap, the manufacturer (ASMedia Technology
> Inc.) didn't bother to fully populate all of the USB descriptors and
> it doesn't pass through physical sector size properly, or report max
> power correctly, etc.
That's interesting, I've usually had really good results with ASMedia's
USB devices (for reference, they're the semiconductor division of the
same company that ASUS and ASRock are owned by); as far as not passing
through physical sector size properly though, that's normal on almost
all external IDE/SATA adapters (both enclosures and regular adapters),
as is not properly handling commands other than the basic ones needed
for disk access (such as SCT stuff and SMART related commands). That
said, the USB issues on the NUC don't surprise me too much, Intel's USB
controller chips have been known to have hardware bugs (especially their
original XHCI implementation), and I've seen stuff similar to what
you're talking about before with other Intel systems. Now, for other
systems, you have to keep in mind that most of them have hubs internal
to the system, and the external USB ports don't go directly to the USB
controllers, but quite often these hubs are not particularly high
quality, so they often make things worse.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-26 12:22 ` Austin S. Hemmelgarn
@ 2016-02-08 2:02 ` Chris Murphy
0 siblings, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2016-02-08 2:02 UTC (permalink / raw)
To: Austin S. Hemmelgarn; +Cc: Chris Murphy, WillIam Thorne, Btrfs BTRFS
On Tue, Jan 26, 2016 at 5:22 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-01-25 16:12, Chris Murphy wrote:
>> I really think USB hubs help fix a lot of USB related problems, even
>> when it's not a power related problem. Currently I'm using internal
>> SATA in a NUC for the primary storage, but use send/receive to two
>> separate raid1 volumes that are USB drives. I can balance/scrub, and
>> read/write to any and all drives with no problems since getting a
>> dyconn "industrial" (probably a design term, it's aluminum not
>> plastic) hub. Before that, one drive or another would just
>> intermittently reset, usually to no ill effect, but once it wigged
>> out, vanished, and reappeared as a completely different /dev/sdx
>> device. The enclosures are crap, the manufacturer (ASMedia Technology
>> Inc.) didn't bother to fully populate all of the USB descriptors and
>> it doesn't pass through physical sector size properly, or report max
>> power correctly, etc.
>
> That's interesting, I've usually had really good results with ASMedia's USB
> devices (for reference, they're the semiconductor division of the same
> company that ASUS and ASRock are owned by); as far as not passing through
> physical sector size properly though, that's normal on almost all external
> IDE/SATA adapters (both enclosures and regular adapters), as is not properly
> handling commands other than the basic ones needed for disk access (such as
> SCT stuff and SMART related commands). That said, the USB issues on the NUC
> don't surprise me too much, Intel's USB controller chips have been known to
> have hardware bugs (especially their original XHCI implementation), and I've
> seen stuff similar to what you're talking about before with other Intel
> systems. Now, for other systems, you have to keep in mind that most of them
> have hubs internal to the system, and the external USB ports don't go
> directly to the USB controllers, but quite often these hubs are not
> particularly high quality, so they often make things worse.
>
Fair enough, it may just be some weirdness between Intel and ASMedia.
But the external hub in between them has solved the intermittant
"reset" messages I'd get, and so far I haven't had any drive vanish,
only to come back as a different /dev/sdX letter, as I had a couple
times before the hub was present.
Curiously, the NUC has a high powered port. No enclosure complains
when on that high powered port. All drives are rated in the 1A range,
but that's just for spin up, they aren't pulling that current once
spun up, even with heavy read/write. So a regular 900mA port should be
plenty adequate. But those are the ports I have the problem on...
UNLESS there's a powered external hub being used on that same 900mA
port. And then I can have 4 drives (on the hub) doing simultaneous
balance or scrubs and not a peep of an error. So I don't actually
think this is a power problem. I think it's something to do with the
internal hub you're referring to.
Also, the normal 900mA ports provide no power when the NUC is powered
off. The orange 1.5A high powered port does provide power, despite the
NUC being off.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <6F3946F5-07EC-43EB-A86F-41D4E3491EDD@me.com>]
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
[not found] ` <6F3946F5-07EC-43EB-A86F-41D4E3491EDD@me.com>
@ 2016-02-08 16:42 ` Austin S. Hemmelgarn
0 siblings, 0 replies; 11+ messages in thread
From: Austin S. Hemmelgarn @ 2016-02-08 16:42 UTC (permalink / raw)
To: WillIam Thorne; +Cc: linux-btrfs, raspbian-devel
On 2016-02-08 11:23, WillIam Thorne wrote:
> Thanks all for the help. Here’s a bit more info below. Seeing as its
> possibly related to the USB implementation on the pi, I have cc’d their
> mailing list.
Glad we could be of assistance.
>
>> On 25 Jan 2016, at 16:43, Austin S. Hemmelgarn <ahferroin7@gmail.com
>> <mailto:ahferroin7@gmail.com>> wrote:
>>
>> On 2016-01-25 09:58, WillIam Thorne wrote:
>>> Hi
>>>
>>> I have a WD 3TB external HD attached over USB to an arm based micro
>>> PC (rasp pi). I was experimenting with btrfs for storing email
>>> archives but recently encountered some problems which resulted in the
>>> filesystem becoming apparently unrecoverable. I’m not an expert and
>>> it was quicker to switch back to ext4 and restored from backup so no
>>> support needed. Here what appears to be the relevant part of the
>>> syslog including the stack trace in case it is useful:
>>>
>>> Best
>>> W
>>>
>>> pi@mail /var/log $ btrfs --version
>>> Btrfs Btrfs v0.19
>> In general, if you plan to use BTRFS on Debian (or Raspbian), you
>> should be building the tools yourself locally, Debian is almost as bad
>> about staying up to date as most enterprise distros.
>>>
>>> pi@mail /var/log $ uname -a
>>> Linux mail 4.1.7-v7+ #817 SMP PREEMPT Sat Sep 19 15:32:00 BST 2015
>>> armv7l GNU/Linux
>>>
>>> Jan 20 09:42:08 mail kernel: [2762753.507576] usb 1-1.5: reset
>>> high-speed USB device number 4 using dwc_otg
> The device reset always seemed to happen directly after my tarsnap
> <http://www.tarsnap.com/> backup ran, although this had been running
> fine for a month or so before hand. I noticed the problems when I came
> back from holiday over christmas. Maybe it’s load related, the usb
> driver / controller on the pi used to be a little buggy, maybe they
> didn’t catch everything.
If it was working correctly that long before this happened, that says
one of two things to me:
1. It's a non-periodic intermittent error due to a design flaw or
manufacturing defect in part of the hardware.
2. Some part of the hardware is failing.
Based on what you say below, I think the first one is the case. Either
way though, I would suggest you make sure you have working backups of
any data you care about on this device, as either case is likely to
cause data loss.
>
>>> Jan 20 09:43:18 mail kernel: [2762823.972777] sd 0:0:0:0: [sda]
>>> UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
>>> Jan 20 09:43:18 mail kernel: [2762823.972806] sd 0:0:0:0: [sda] Sense
>>> Key : 0x2 [current]
>>> Jan 20 09:43:18 mail kernel: [2762823.972819] sd 0:0:0:0: [sda]
>>> ASC=0x3a ASCQ=0x0
>>> Jan 20 09:43:18 mail kernel: [2762823.972837] sd 0:0:0:0: [sda] CDB:
>>> opcode=0x2a 2a 00 00 f7 2c 20 00 00 f0 00
>>> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O
>>> error, dev sda, sector 16198688
>> This line right here ^^^ indicates that it was triggered by an issue
>> with the USB device. I don't personally know enough about USB-MSC and
>> SCSI to know for certain what is happening, but you should probably
>> scan your logs and make sure you're not still getting stuff like this,
>> because if you are, you're likely to get data corruption on any
>> filesystem on the device. Based on this, the BTRFS trace you got is
>> probably a result of problems with the USB device.
> I reformatted the disk to ext4 on the 22nd of Jan and restored the
> backed up data in full to the disk. Since then I have grepped for
> ‘error’ and ‘dwc_otg’ in my syslog every week, but have not seen the
> errors again. I will ping an email to the list in a month or two if I am
> still not seeing these.
It may have been some design flaw in the USB device that caused it to
not handle BTRFS write patterns well. I've seen similar behavior with
some really cheap SATA controllers before as well. I'd be interested to
see if similar issues occur with the same disk hooked up to a regular
x86 system instead of a single-board computer like the Pi.
>>> Jan 20 09:43:18 mail kernel: [2762823.997601] BTRFS: error (device
>>> sda1) in btrfs_commit_transaction:2068: errno=-5 IO failure (Error
>>> while writing out transaction)
>>> Jan 20 09:43:18 mail kernel: [2762824.011517] BTRFS info (device
>>> sda1): forced readonly
>>> Jan 20 09:43:18 mail kernel: [2762824.011537] BTRFS warning (device
>>> sda1): Skipping commit of aborted transaction.
>>> Jan 20 09:43:18 mail kernel: [2762824.011576] ------------[ cut here
>>> ]------------
>>> Jan 20 09:43:18 mail kernel: [2762824.011682] WARNING: CPU: 0 PID:
>>> 1318 at fs/btrfs/super.c:260 __btrfs_abort_transaction+0xd8/0x128
>>> [btrfs]()
>>> Jan 20 09:43:18 mail kernel: [2762824.011709] BTRFS: Transaction
>>> aborted (error -5)
>>> Jan 20 09:43:18 mail kernel: [2762824.011717] Modules linked in:
>>> cfg80211 rfkill snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer
>>> snd btrfs xor xor_neon raid6_pq zlib_deflate sg bcm2835_gpiomem
>>> uio_pdrv_genirq uio
>>> Jan 20 09:43:18 mail kernel: [2762824.011790] CPU: 0 PID: 1318 Comm:
>>> btrfs-transacti Not tainted 4.1.7-v7+ #817
>>> Jan 20 09:43:18 mail kernel: [2762824.011797] Hardware name: BCM2709
>>> Jan 20 09:43:18 mail kernel: [2762824.011832] [<80018440>]
>>> (unwind_backtrace) from [<80013e0c>] (show_stack+0x20/0x24)
>>> Jan 20 09:43:18 mail kernel: [2762824.011852] [<80013e0c>]
>>> (show_stack) from [<80558548>] (dump_stack+0x98/0xe0)
>>> Jan 20 09:43:18 mail kernel: [2762824.011872] [<80558548>]
>>> (dump_stack) from [<80026a4c>] (warn_slowpath_common+0x8c/0xc8)
>>> Jan 20 09:43:18 mail kernel: [2762824.011892] [<80026a4c>]
>>> (warn_slowpath_common) from [<80026ac8>] (warn_slowpath_fmt+0x40/0x48)
>>> Jan 20 09:43:18 mail kernel: [2762824.011971] [<80026ac8>]
>>> (warn_slowpath_fmt) from [<7f051790>]
>>> (__btrfs_abort_transaction+0xd8/0x128 [btrfs])
>>> Jan 20 09:43:18 mail kernel: [2762824.012153] [<7f051790>]
>>> (__btrfs_abort_transaction [btrfs]) from [<7f082a84>]
>>> (btrfs_commit_transaction+0x330/0xd40 [btrfs])
>>> Jan 20 09:43:18 mail kernel: [2762824.012353] [<7f082a84>]
>>> (btrfs_commit_transaction [btrfs]) from [<7f07e95c>]
>>> (transaction_kthread+0x174/0x1ec [btrfs])
>>> Jan 20 09:43:18 mail kernel: [2762824.012463] [<7f07e95c>]
>>> (transaction_kthread [btrfs]) from [<80042498>] (kthread+0xe8/0x104)
>>> Jan 20 09:43:18 mail kernel: [2762824.012481] [<80042498>] (kthread)
>>> from [<8000fa58>] (ret_from_fork+0x14/0x3c)
>>> Jan 20 09:43:18 mail kernel: [2762824.012492] ---[ end trace
>>> 1c48a450ca505104 ]---
>>> Jan 20 09:43:18 mail kernel: [2762824.012505] BTRFS: error (device
>>> sda1) in cleanup_transaction:1692: errno=-5 IO failure
>>> Jan 20 09:43:18 mail kernel: [2762824.022734] BTRFS info (device
>>> sda1): delayed_refs has NO entry
>> The bit about 'transaction aborted' is almost always indicative of an
>> error with the storage path (in your case, the USB controller, the USB
>> cable, or the USB device), not BTRFS. That said, something like this
>> shouldn't usually cause the FS to be irreparably damaged, although it
>> will make the FS unusable until you remount (or possibly until you
>> reboot, I'm not certain about the error handling here because I've
>> never dealt with it myself).
>>
>> Now, just a general caution: Avoid using USB storage for persistent
>> online storage, there's just to many things that can go wrong, and
>> quite a few USB storage controllers are absolute crap. I understand
>> that this can be somewhat tricky with something like a Raspberry Pi,
>> but with BTRFS especially, there's not sufficient error recovery in
>> Linux to safely use most USB storage devices for anything other than
>> file transfers or possibly off-line backups. That said, there are
>> some brands that work well provided they get enough power (I've
>> personally had really good results using a SanDisk Cruzer Fit flash
>> drive (the USB 3.0 version, I've had only intermittent success with
>> the USB 2.0 ones) with a Raspberry Pi via a powered hub).
> The disk is an externally powered HD so hopefully this rules out power
> related ‘brownouts’ which I have heard can be a problem on the Pi. I’m
> just using it as a box to sling old emails on as a holding area so that
> they are out of my main email account and backed up to the cloud while
> also available to be accessed reasonably quickly if needs be.
You might consider putting a hub between the Pi and the disk itself.
That's resolved most USB issues I've seen that weren't power related.
If you do go this way, look for one of the ones that you could power the
Pi itself from, those tend to be high quality.
That said, just because the device is externally powered doesn't mean it
isn't drawing any power from the USB port, a lot of external disk
enclosures use the external power for the disk itself, and still power
the USB chip off the bus (this actually simplifies the hardware design
somewhat). I doubt that this is what was causing the issues, but it's
still something to consider.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 14:58 suspected BTRFS errors resulting in file system becoming unrecovable WillIam Thorne
2016-01-25 16:36 ` Patrik Lundquist
2016-01-25 16:43 ` Austin S. Hemmelgarn
@ 2016-01-25 20:48 ` Chris Murphy
2 siblings, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2016-01-25 20:48 UTC (permalink / raw)
To: WillIam Thorne; +Cc: Btrfs BTRFS
On Mon, Jan 25, 2016 at 7:58 AM, WillIam Thorne <will.thorne@me.com> wrote:
> Hi
>
> I have a WD 3TB external HD attached over USB to an arm based micro PC (rasp pi). I was experimenting with btrfs for storing email archives but recently encountered some problems which resulted in the filesystem becoming apparently unrecoverable. I’m not an expert and it was quicker to switch back to ext4 and restored from backup so no support needed. Here what appears to be the relevant part of the syslog including the stack trace in case it is useful:
>
> Best
> W
>
> pi@mail /var/log $ btrfs --version
> Btrfs Btrfs v0.19
>
> pi@mail /var/log $ uname -a
> Linux mail 4.1.7-v7+ #817 SMP PREEMPT Sat Sep 19 15:32:00 BST 2015 armv7l GNU/Linux
>
> Jan 20 09:42:08 mail kernel: [2762753.507576] usb 1-1.5: reset high-speed USB device number 4 using dwc_otg
> Jan 20 09:43:18 mail kernel: [2762823.972777] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.972806] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.972819] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.972837] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2c 20 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
These are not Btrfs errors, they're libata errors, it's having a
problem writing to the drive. That this is a USB connected drive it
could be related to insufficient power or some incompatibility with
the bridge chipset in the enclosure and the USB controller in the
computer. I actually had a similar problem until I put a hub in
between the drive and computer (a NUC).
> Jan 20 09:43:18 mail kernel: [2762823.972867] BTRFS: bdev /dev/sda1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.976998] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.977015] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.977028] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.977042] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2d 10 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.977052] blk_update_request: I/O error, dev sda, sector 16198928
> Jan 20 09:43:18 mail kernel: [2762823.977066] BTRFS: bdev /dev/sda1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.978226] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.978273] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.978286] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.978301] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f7 2e 00 00 00 20 00
> Jan 20 09:43:18 mail kernel: [2762823.978318] blk_update_request: I/O error, dev sda, sector 16199168
> Jan 20 09:43:18 mail kernel: [2762823.978339] BTRFS: bdev /dev/sda1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.983241] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.983264] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.983277] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.983292] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c3 f8 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.983304] blk_update_request: I/O error, dev sda, sector 16172024
> Jan 20 09:43:18 mail kernel: [2762823.983319] BTRFS: bdev /dev/sda1 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.987799] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.987818] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.987830] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.987844] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c4 e8 00 00 f0 00
> Jan 20 09:43:18 mail kernel: [2762823.987855] blk_update_request: I/O error, dev sda, sector 16172264
> Jan 20 09:43:18 mail kernel: [2762823.987869] BTRFS: bdev /dev/sda1 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.989136] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.989155] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.989168] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.989182] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 f6 c5 d8 00 00 20 00
> Jan 20 09:43:18 mail kernel: [2762823.989193] blk_update_request: I/O error, dev sda, sector 16172504
> Jan 20 09:43:18 mail kernel: [2762823.989207] BTRFS: bdev /dev/sda1 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.993965] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.993988] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.994000] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.994015] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 17 53 68 00 00 c0 00
> Jan 20 09:43:18 mail kernel: [2762823.994028] blk_update_request: I/O error, dev sda, sector 1528680
> Jan 20 09:43:18 mail kernel: [2762823.994043] BTRFS: bdev /dev/sda1 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.994058] BTRFS: bdev /dev/sda1 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.994070] BTRFS: bdev /dev/sda1 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.997419] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
> Jan 20 09:43:18 mail kernel: [2762823.997435] sd 0:0:0:0: [sda] Sense Key : 0x2 [current]
> Jan 20 09:43:18 mail kernel: [2762823.997448] sd 0:0:0:0: [sda] ASC=0x3a ASCQ=0x0
> Jan 20 09:43:18 mail kernel: [2762823.997461] sd 0:0:0:0: [sda] CDB: opcode=0x2a 2a 00 00 37 53 68 00 00 c0 00
> Jan 20 09:43:18 mail kernel: [2762823.997471] blk_update_request: I/O error, dev sda, sector 3625832
> Jan 20 09:43:18 mail kernel: [2762823.997484] BTRFS: bdev /dev/sda1 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
> Jan 20 09:43:18 mail kernel: [2762823.997601] BTRFS: error (device sda1) in btrfs_commit_transaction:2068: errno=-5 IO failure (Error while writing out transaction)
> Jan 20 09:43:18 mail kernel: [2762824.011517] BTRFS info (device sda1): forced readonly
> Jan 20 09:43:18 mail kernel: [2762824.011537] BTRFS warning (device sda1): Skipping commit of aborted transaction.
Well this is nifty, I like that it's eventually giving up when it can't write!
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
@ 2016-01-25 21:27 Piotr Szymaniak
0 siblings, 0 replies; 11+ messages in thread
From: Piotr Szymaniak @ 2016-01-25 21:27 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]
Hi,
Sorry, forgot to ansewer on-list, but I'm forwarding my question to
William and his answer in next message.
----- Forwarded message from Piotr Szymaniak <szarpaj@grubelek.pl> -----
Date: Mon, 25 Jan 2016 17:48:49 +0100
From: Piotr Szymaniak <szarpaj@grubelek.pl>
To: WillIam Thorne <will.thorne@me.com>
Subject: Re: suspected BTRFS errors resulting in file system becoming unrecovable
User-Agent: Mutt/1.5.23 (2014-03-12)
On Mon, Jan 25, 2016 at 02:58:31PM +0000, WillIam Thorne wrote:
> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
> Jan 20 09:43:18 mail kernel: [2762823.977052] blk_update_request: I/O error, dev sda, sector 16198928
> Jan 20 09:43:18 mail kernel: [2762823.978318] blk_update_request: I/O error, dev sda, sector 16199168
> Jan 20 09:43:18 mail kernel: [2762823.983304] blk_update_request: I/O error, dev sda, sector 16172024
> Jan 20 09:43:18 mail kernel: [2762823.987855] blk_update_request: I/O error, dev sda, sector 16172264
> Jan 20 09:43:18 mail kernel: [2762823.989193] blk_update_request: I/O error, dev sda, sector 16172504
Hi,
This looks like bad sectors, not btrfs fault (on the other hand, few bad
sectors should not make fs unrecoverable).
Can you paste smartctl -x /dev/[yourdevice] output?
Best regards,
Piotr Szymaniak.
----- End forwarded message -----
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: suspected BTRFS errors resulting in file system becoming unrecovable
@ 2016-01-25 21:29 Piotr Szymaniak
2016-01-25 22:30 ` Chris Murphy
0 siblings, 1 reply; 11+ messages in thread
From: Piotr Szymaniak @ 2016-01-25 21:29 UTC (permalink / raw)
To: linux-btrfs
And an answer with smartctl output.
----- Forwarded message from WillIam Thorne <will.thorne@me.com> -----
Date: Mon, 25 Jan 2016 16:52:44 +0000
From: WillIam Thorne <will.thorne@me.com>
To: Piotr Szymaniak <szarpaj@grubelek.pl>
Subject: Re: suspected BTRFS errors resulting in file system becoming unrecovable
X-Mailer: Apple Mail (2.3112)
Hi Piotr,
Thanks for your help, here’s the output from smartctl. By the way, should I cc answers to linux-btrfs@vger.kernel.org?
pi@mail /var/log $ sudo smartctl -d sat /dev/sda -x
smartctl 5.41 2011-06-09 r3365 [armv7l-linux-4.1.7-v7+] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Device Model: WDC WD20EZRX-00D8PB0
Serial Number: WD-WMC4N0J17JYP
LU WWN Device Id: 5 0014ee 605b50871
Firmware Version: 80.00A80
User Capacity: 2,000,365,380,096 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 9
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Mon Jan 25 16:51:57 2016 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (32340) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x7035) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-K 195 195 051 - 78
3 Spin_Up_Time POS--K 199 172 021 - 5050
4 Start_Stop_Count -O--CK 100 100 000 - 382
5 Reallocated_Sector_Ct PO--CK 200 200 140 - 0
7 Seek_Error_Rate -OSR-K 200 200 000 - 0
9 Power_On_Hours -O--CK 100 100 000 - 269
10 Spin_Retry_Count -O--CK 100 100 000 - 0
11 Calibration_Retry_Count -O--CK 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 316
192 Power-Off_Retract_Count -O--CK 200 200 000 - 241
193 Load_Cycle_Count -O--CK 199 199 000 - 4765
194 Temperature_Celsius -O---K 113 108 000 - 37
196 Reallocated_Event_Count -O--CK 200 200 000 - 0
197 Current_Pending_Sector -O--CK 200 200 000 - 0
198 Offline_Uncorrectable ----CK 200 200 000 - 0
199 UDMA_CRC_Error_Count -O--CK 200 200 000 - 0
200 Multi_Zone_Error_Rate ---R-- 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
GP/S Log at address 0x00 has 1 sectors [Log Directory]
SMART Log at address 0x01 has 1 sectors [Summary SMART error log]
SMART Log at address 0x02 has 5 sectors [Comprehensive SMART error log]
GP Log at address 0x03 has 6 sectors [Ext. Comprehensive SMART error log]
SMART Log at address 0x06 has 1 sectors [SMART self-test log]
GP Log at address 0x07 has 1 sectors [Extended self-test log]
SMART Log at address 0x09 has 1 sectors [Selective self-test log]
GP Log at address 0x10 has 1 sectors [NCQ Command Error]
GP Log at address 0x11 has 1 sectors [SATA Phy Event Counters]
GP/S Log at address 0x80 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x81 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x82 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x83 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x84 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x85 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x86 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x87 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x88 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x89 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x8f has 16 sectors [Host vendor specific log]
GP/S Log at address 0x90 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x91 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x92 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x93 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x94 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x95 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x96 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x97 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x98 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x99 has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9a has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9b has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9c has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9d has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9e has 16 sectors [Host vendor specific log]
GP/S Log at address 0x9f has 16 sectors [Host vendor specific log]
GP/S Log at address 0xa0 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa1 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa2 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa3 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa4 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa5 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa6 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa7 has 16 sectors [Device vendor specific log]
GP/S Log at address 0xa8 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xa9 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaa has 1 sectors [Device vendor specific log]
GP/S Log at address 0xab has 1 sectors [Device vendor specific log]
GP/S Log at address 0xac has 1 sectors [Device vendor specific log]
GP/S Log at address 0xad has 1 sectors [Device vendor specific log]
GP/S Log at address 0xae has 1 sectors [Device vendor specific log]
GP/S Log at address 0xaf has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb0 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb1 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb2 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb3 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb4 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb5 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb6 has 1 sectors [Device vendor specific log]
GP/S Log at address 0xb7 has 39 sectors [Device vendor specific log]
GP/S Log at address 0xbd has 1 sectors [Device vendor specific log]
GP/S Log at address 0xc0 has 1 sectors [Device vendor specific log]
GP Log at address 0xc1 has 93 sectors [Device vendor specific log]
GP Log at address 0xcf has 65535 sectors [Device vendor specific log]
GP/S Log at address 0xe0 has 1 sectors [SCT Command/Status]
GP/S Log at address 0xe1 has 1 sectors [SCT Data Transfer]
SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 64 (device log contains only the most recent 24 errors)
CR = Command Register
FEATR = Features Register
COUNT = Count (was: Sector Count) Register
LBA_48 = Upper bytes of LBA High/Mid/Low Registers ] ATA-8
LH = LBA High (was: Cylinder High) Register ] LBA
LM = LBA Mid (was: Cylinder Low) Register ] Register
LL = LBA Low (was: Sector Number) Register ]
DV = Device (was: Device/Head) Register
DC = Device Control Register
ER = Error register
ST = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 64 [15] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:50.583 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:47.209 READ DMA EXT
25 00 d0 00 08 00 00 20 00 08 00 40 00 00:30:47.207 READ DMA EXT
25 00 d0 00 08 00 00 00 02 08 00 40 00 00:30:47.207 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:43.833 READ DMA EXT
Error 63 [14] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:47.209 READ DMA EXT
25 00 d0 00 08 00 00 20 00 08 00 40 00 00:30:47.207 READ DMA EXT
25 00 d0 00 08 00 00 00 02 08 00 40 00 00:30:47.207 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:43.833 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:39.856 READ DMA EXT
Error 62 [13] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:43.833 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:39.856 READ DMA EXT
25 00 d0 00 08 00 00 20 00 00 00 40 00 00:30:36.678 READ DMA EXT
25 00 d0 00 08 00 00 00 02 00 00 40 00 00:30:36.678 READ DMA EXT
25 00 d0 00 08 00 00 00 00 00 80 40 00 00:30:36.677 READ DMA EXT
Error 61 [12] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:39.856 READ DMA EXT
25 00 d0 00 08 00 00 20 00 00 00 40 00 00:30:36.678 READ DMA EXT
25 00 d0 00 08 00 00 00 02 00 00 40 00 00:30:36.678 READ DMA EXT
25 00 d0 00 08 00 00 00 00 00 80 40 00 00:30:36.677 READ DMA EXT
25 00 d0 00 08 00 00 20 00 00 00 40 00 00:30:36.675 READ DMA EXT
Error 60 [11] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:27.477 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:24.070 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 20 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:17.324 READ DMA EXT
Error 59 [10] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:24.070 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 20 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:17.324 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b f8 40 00 00:22:17.324 READ DMA EXT
Error 58 [9] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 20 40 00 00:22:20.695 READ DMA EXT
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:17.324 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b f8 40 00 00:22:17.324 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b f0 40 00 00:22:17.323 READ DMA EXT
Error 57 [8] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER -- ST COUNT LBA_48 LH LM LL DV DC
-- -- -- == -- == == == -- -- -- -- --
40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
Commands leading to the command that caused the error were:
CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
-- == -- == -- == == == -- -- -- -- -- --------------- --------------------
25 00 d0 00 08 00 00 00 00 08 80 40 00 00:22:17.324 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b f8 40 00 00:22:17.324 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b f0 40 00 00:22:17.323 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b e8 40 00 00:22:17.323 READ DMA EXT
25 00 d0 00 08 00 00 00 00 0b e0 40 00 00:22:17.322 READ DMA EXT
SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 258 (0x0102)
SCT Support Level: 1
Device State: Active (0)
Current Temperature: 37 Celsius
Power Cycle Min/Max Temperature: 29/37 Celsius
Lifetime Min/Max Temperature: 2/42 Celsius
Under/Over Temperature Limit Count: 0/0
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -41/85 Celsius
Temperature History Size (Index): 478 (458)
Index Estimated Time Temperature Celsius
459 2016-01-25 08:54 35 ****************
460 2016-01-25 08:55 36 *****************
... ..( 11 skipped). .. *****************
472 2016-01-25 09:07 36 *****************
473 2016-01-25 09:08 ? -
474 2016-01-25 09:09 19 -
475 2016-01-25 09:10 ? -
476 2016-01-25 09:11 22 ***
477 2016-01-25 09:12 ? -
0 2016-01-25 09:13 35 ****************
... ..( 9 skipped). .. ****************
10 2016-01-25 09:23 35 ****************
11 2016-01-25 09:24 ? -
12 2016-01-25 09:25 37 ******************
... ..( 4 skipped). .. ******************
17 2016-01-25 09:30 37 ******************
18 2016-01-25 09:31 ? -
19 2016-01-25 09:32 25 ******
20 2016-01-25 09:33 25 ******
21 2016-01-25 09:34 26 *******
22 2016-01-25 09:35 26 *******
23 2016-01-25 09:36 27 ********
24 2016-01-25 09:37 27 ********
25 2016-01-25 09:38 ? -
26 2016-01-25 09:39 28 *********
... ..( 2 skipped). .. *********
29 2016-01-25 09:42 28 *********
30 2016-01-25 09:43 29 **********
31 2016-01-25 09:44 29 **********
32 2016-01-25 09:45 30 ***********
33 2016-01-25 09:46 30 ***********
34 2016-01-25 09:47 30 ***********
35 2016-01-25 09:48 ? -
36 2016-01-25 09:49 31 ************
... ..( 2 skipped). .. ************
39 2016-01-25 09:52 31 ************
40 2016-01-25 09:53 32 *************
41 2016-01-25 09:54 31 ************
42 2016-01-25 09:55 32 *************
... ..( 11 skipped). .. *************
54 2016-01-25 10:07 32 *************
55 2016-01-25 10:08 33 **************
56 2016-01-25 10:09 33 **************
57 2016-01-25 10:10 33 **************
58 2016-01-25 10:11 34 ***************
59 2016-01-25 10:12 33 **************
... ..( 3 skipped). .. **************
63 2016-01-25 10:16 33 **************
64 2016-01-25 10:17 34 ***************
... ..( 3 skipped). .. ***************
68 2016-01-25 10:21 34 ***************
69 2016-01-25 10:22 35 ****************
70 2016-01-25 10:23 35 ****************
71 2016-01-25 10:24 ? -
72 2016-01-25 10:25 41 **********************
73 2016-01-25 10:26 ? -
74 2016-01-25 10:27 33 **************
... ..( 28 skipped). .. **************
103 2016-01-25 10:56 33 **************
104 2016-01-25 10:57 34 ***************
105 2016-01-25 10:58 ? -
106 2016-01-25 10:59 29 **********
... ..( 6 skipped). .. **********
113 2016-01-25 11:06 29 **********
114 2016-01-25 11:07 30 ***********
115 2016-01-25 11:08 30 ***********
116 2016-01-25 11:09 30 ***********
117 2016-01-25 11:10 31 ************
... ..( 11 skipped). .. ************
129 2016-01-25 11:22 31 ************
130 2016-01-25 11:23 32 *************
... ..( 9 skipped). .. *************
140 2016-01-25 11:33 32 *************
141 2016-01-25 11:34 33 **************
... ..( 6 skipped). .. **************
148 2016-01-25 11:41 33 **************
149 2016-01-25 11:42 34 ***************
... ..( 6 skipped). .. ***************
156 2016-01-25 11:49 34 ***************
157 2016-01-25 11:50 35 ****************
... ..( 9 skipped). .. ****************
167 2016-01-25 12:00 35 ****************
168 2016-01-25 12:01 36 *****************
... ..( 18 skipped). .. *****************
187 2016-01-25 12:20 36 *****************
188 2016-01-25 12:21 37 ******************
... ..(269 skipped). .. ******************
458 2016-01-25 16:51 37 ******************
Warning: device does not support SCT Error Recovery Control command
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 0 Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 1 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000f 2 0 R_ERR response for host-to-device data FIS, CRC
0x0012 2 0 R_ERR response for host-to-device non-data FIS, CRC
0x8000 4 21174 Vendor specific
> On 25 Jan 2016, at 16:48, Piotr Szymaniak <szarpaj@grubelek.pl> wrote:
>
> On Mon, Jan 25, 2016 at 02:58:31PM +0000, WillIam Thorne wrote:
>> Jan 20 09:43:18 mail kernel: [2762823.972851] blk_update_request: I/O error, dev sda, sector 16198688
>> Jan 20 09:43:18 mail kernel: [2762823.977052] blk_update_request: I/O error, dev sda, sector 16198928
>> Jan 20 09:43:18 mail kernel: [2762823.978318] blk_update_request: I/O error, dev sda, sector 16199168
>> Jan 20 09:43:18 mail kernel: [2762823.983304] blk_update_request: I/O error, dev sda, sector 16172024
>> Jan 20 09:43:18 mail kernel: [2762823.987855] blk_update_request: I/O error, dev sda, sector 16172264
>> Jan 20 09:43:18 mail kernel: [2762823.989193] blk_update_request: I/O error, dev sda, sector 16172504
>
> Hi,
>
> This looks like bad sectors, not btrfs fault (on the other hand, few bad
> sectors should not make fs unrecoverable).
>
> Can you paste smartctl -x /dev/[yourdevice] output?
>
>
> Best regards,
> Piotr Szymaniak.
----- End forwarded message -----
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: suspected BTRFS errors resulting in file system becoming unrecovable
2016-01-25 21:29 Piotr Szymaniak
@ 2016-01-25 22:30 ` Chris Murphy
0 siblings, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2016-01-25 22:30 UTC (permalink / raw)
To: Piotr Szymaniak; +Cc: Btrfs BTRFS
>
> Error 64 [15] occurred at disk power-on lifetime: 262 hours (10 days + 22 hours)
> When the command that caused the error occurred, the device was active or idle.
>
> After command completion occurred, registers were:
> ER -- ST COUNT LBA_48 LH LM LL DV DC
> -- -- -- == -- == == == -- -- -- -- --
> 40 -- 51 00 08 00 00 00 00 08 80 40 00 Error: UNC 8 sectors at LBA = 0x00000880 = 2176
This is an unrecoverable read or write error. I'm not having much luck
figuring out if 40 or 51 refers to a read or a write. A write failure
is fatal, the drive needs to be replaced under warranty. A read
failure is more of a soft fail, it needs to be written over for the
problem to get fixed by the drive firmware, but there's no indication
there are pending (bad) sectors.
>
> Commands leading to the command that caused the error were:
> CR FEATR COUNT LBA_48 LH LM LL DV DC Powered_Up_Time Command/Feature_Name
> -- == -- == -- == == == -- -- -- -- -- --------------- --------------------
> 25 00 d0 00 08 00 00 00 00 08 80 40 00 00:30:50.583 READ DMA EXT
Huh and this is a read error.
So I wonder if what's really going on is a bad sector is just taking a
long time to recover, and we're getting the USB equivalent of libata's
link reset due to SCSI command timer (the topic of the month
apparently)? And that's cause writes to "fail" possibly because they
were lost in the command queue in the drive when the reset happened.
Does this drive contain vital information or can it be destructively
written to? If you can destructively write to it you could set it to
do
badblocks -b 4096 -c 256 -svw /dev/sdX
Use whole block device *OR* if it's a partition, make sure the start
LBA is divisible by 8 (it needs to be 4096 byte aligned on any recent
drive, which will be a 512e AF drive).
Again this is destructive, data will be lost on this drive. And it
shouldn't be mounted at the time you do this (badblocks should fail).
If you can't do a destructive write to this drive just yet, this is
non-destructive:
smartctl -t long /dev/sdX
That starts an extended offline (readonly) test for bad sectors which
will be aborted at the first bad sector it finds. You can check with
smartctl -a, and look in the section "SMART Self-test log" for the
first line.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2016-02-08 16:43 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-01-25 14:58 suspected BTRFS errors resulting in file system becoming unrecovable WillIam Thorne
2016-01-25 16:36 ` Patrik Lundquist
2016-01-25 16:43 ` Austin S. Hemmelgarn
2016-01-25 21:12 ` Chris Murphy
2016-01-26 12:22 ` Austin S. Hemmelgarn
2016-02-08 2:02 ` Chris Murphy
[not found] ` <6F3946F5-07EC-43EB-A86F-41D4E3491EDD@me.com>
2016-02-08 16:42 ` Austin S. Hemmelgarn
2016-01-25 20:48 ` Chris Murphy
2016-01-25 21:27 Piotr Szymaniak
2016-01-25 21:29 Piotr Szymaniak
2016-01-25 22:30 ` Chris Murphy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.