experiences running btrfs on external USB disks?

experiences running btrfs on external USB disks?

[Tomasz Chmielewski]

I'm trying to use btrfs on an external USB drive, without much success.

When the drive is connected for 2-3+ days, the filesystem gets remounted 
readonly, with BTRFS saying "IO failure":

[77760.444607] BTRFS error (device sdb1): bad tree block start, want 
378372096 have 0
[77760.550933] BTRFS error (device sdb1): bad tree block start, want 
378372096 have 0
[77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: 
errno=-5 IO failure
[77760.550979] BTRFS info (device sdb1): forced readonly
[77760.551003] BTRFS: error (device sdb1) in 
btrfs_run_delayed_refs:2935: errno=-5 IO failure
[77760.553223] BTRFS error (device sdb1): pending csums is 4096


Note that there are no other kernel messages (i.e. that would indicate a 
problem with disk, cable disconnection etc.).

The load on the drive itself can be quite heavy at times (i.e. 100% IO 
for 1-2 h and more) - can it contribute to the problem (i.e. btrfs 
thinks there is some timeout somewhere)?

Running 4.19.6 right now, but was experiencing the issue also with 4.18 
kernels.



# btrfs device stats /data
[/dev/sda1].write_io_errs    0
[/dev/sda1].read_io_errs     0
[/dev/sda1].flush_io_errs    0
[/dev/sda1].corruption_errs  0
[/dev/sda1].generation_errs  0



Tomasz Chmielewski

Re: experiences running btrfs on external USB disks?

[Chris Murphy]

On Mon, Dec 3, 2018 at 10:44 PM Tomasz Chmielewski <mangoo@wpkg.org> wrote:
>
> I'm trying to use btrfs on an external USB drive, without much success.
>
> When the drive is connected for 2-3+ days, the filesystem gets remounted
> readonly, with BTRFS saying "IO failure":
>
> [77760.444607] BTRFS error (device sdb1): bad tree block start, want
> 378372096 have 0
> [77760.550933] BTRFS error (device sdb1): bad tree block start, want
> 378372096 have 0
> [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804:
> errno=-5 IO failure
> [77760.550979] BTRFS info (device sdb1): forced readonly
> [77760.551003] BTRFS: error (device sdb1) in
> btrfs_run_delayed_refs:2935: errno=-5 IO failure
> [77760.553223] BTRFS error (device sdb1): pending csums is 4096
>
>
> Note that there are no other kernel messages (i.e. that would indicate a
> problem with disk, cable disconnection etc.).
>
> The load on the drive itself can be quite heavy at times (i.e. 100% IO
> for 1-2 h and more) - can it contribute to the problem (i.e. btrfs
> thinks there is some timeout somewhere)?
>
> Running 4.19.6 right now, but was experiencing the issue also with 4.18
> kernels.
>
>
>
> # btrfs device stats /data
> [/dev/sda1].write_io_errs    0
> [/dev/sda1].read_io_errs     0
> [/dev/sda1].flush_io_errs    0
> [/dev/sda1].corruption_errs  0
> [/dev/sda1].generation_errs  0


Hard to say without a complete dmesg; but errno=-5 IO failure is
pretty much some kind of hardware problem in my experience. I haven't
seen it be a bug.

-- 
Chris Murphy

Re: experiences running btrfs on external USB disks?

[Tomasz Chmielewski]

On 2018-12-04 14:59, Chris Murphy wrote:

>> Running 4.19.6 right now, but was experiencing the issue also with 
>> 4.18
>> kernels.
>> 
>> 
>> 
>> # btrfs device stats /data
>> [/dev/sda1].write_io_errs    0
>> [/dev/sda1].read_io_errs     0
>> [/dev/sda1].flush_io_errs    0
>> [/dev/sda1].corruption_errs  0
>> [/dev/sda1].generation_errs  0
> 
> 
> Hard to say without a complete dmesg; but errno=-5 IO failure is
> pretty much some kind of hardware problem in my experience. I haven't
> seen it be a bug.

It is a complete dmesg - in sense:

# grep -i btrfs -A5 -B5 /var/log/syslog
Dec  4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed 
remove event for unknown device 
"/sys/skbuff_head_cache(1481:anacron.service)"
Dec  4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed 
remove event for unknown device "/sys/buffer_head(1481:anacron.service)"
Dec  4 05:06:56 step snapd[747]: udevmon.go:184: udev monitor observed 
remove event for unknown device 
"/sys/ext4_inode_cache(1481:anacron.service)"
Dec  4 05:15:01 step CRON[9352]: (root) CMD (command -v debian-sa1 > 
/dev/null && debian-sa1 1 1)
Dec  4 05:17:01 step CRON[9358]: (root) CMD (   cd / && run-parts 
--report /etc/cron.hourly)
Dec  4 05:23:13 step kernel: [77760.444607] BTRFS error (device sdb1): 
bad tree block start, want 378372096 have 0
Dec  4 05:23:13 step kernel: [77760.550933] BTRFS error (device sdb1): 
bad tree block start, want 378372096 have 0
Dec  4 05:23:13 step kernel: [77760.550972] BTRFS: error (device sdb1) 
in __btrfs_free_extent:6804: errno=-5 IO failure
Dec  4 05:23:13 step kernel: [77760.550979] BTRFS info (device sdb1): 
forced readonly
Dec  4 05:23:13 step kernel: [77760.551003] BTRFS: error (device sdb1) 
in btrfs_run_delayed_refs:2935: errno=-5 IO failure
Dec  4 05:23:13 step kernel: [77760.553223] BTRFS error (device sdb1): 
pending csums is 4096
Dec  4 05:23:14 step postfix/pickup[8993]: 13BBE460F86: uid=0 
from=<root@step>
Dec  4 05:23:14 step postfix/cleanup[9398]: 13BBE460F86: 
message-id=<20181204052314.13BBE460F86@step>
Dec  4 05:23:14 step postfix/qmgr[2745]: 13BBE460F86: from=<root@step>, 
size=404, nrcpt=1 (queue active)
Dec  4 05:23:14 step postfix/pickup[8993]: 40A964603EC: uid=0 
from=<root@step>

[...some emails follow, usual CRON messages etc., but noting at all 
generated by the kernel, no hardware issue reported...]



Tomasz Chmielewski

Re: experiences running btrfs on external USB disks?

[Austin S. Hemmelgarn]

On 2018-12-04 00:37, Tomasz Chmielewski wrote:
> I'm trying to use btrfs on an external USB drive, without much success.
> 
> When the drive is connected for 2-3+ days, the filesystem gets remounted 
> readonly, with BTRFS saying "IO failure":
> 
> [77760.444607] BTRFS error (device sdb1): bad tree block start, want 
> 378372096 have 0
> [77760.550933] BTRFS error (device sdb1): bad tree block start, want 
> 378372096 have 0
> [77760.550972] BTRFS: error (device sdb1) in __btrfs_free_extent:6804: 
> errno=-5 IO failure
> [77760.550979] BTRFS info (device sdb1): forced readonly
> [77760.551003] BTRFS: error (device sdb1) in 
> btrfs_run_delayed_refs:2935: errno=-5 IO failure
> [77760.553223] BTRFS error (device sdb1): pending csums is 4096
> 
> 
> Note that there are no other kernel messages (i.e. that would indicate a 
> problem with disk, cable disconnection etc.).
> 
> The load on the drive itself can be quite heavy at times (i.e. 100% IO 
> for 1-2 h and more) - can it contribute to the problem (i.e. btrfs 
> thinks there is some timeout somewhere)?
> 
> Running 4.19.6 right now, but was experiencing the issue also with 4.18 
> kernels.
> 
> 
> 
> # btrfs device stats /data
> [/dev/sda1].write_io_errs    0
> [/dev/sda1].read_io_errs     0
> [/dev/sda1].flush_io_errs    0
> [/dev/sda1].corruption_errs  0
> [/dev/sda1].generation_errs  0
It looks to me like the typical USB issues that are present with almost 
all filesystems but only seem to be noticed by BTRFS because it does 
more rigorous checking of data.

In short, USB is _crap_ for fixed storage, don't use it like that, even 
if you are using filesystems which don't appear to complain.

Re: experiences running btrfs on external USB disks?

[Graham Cobb]

On 04/12/2018 12:38, Austin S. Hemmelgarn wrote:
> In short, USB is _crap_ for fixed storage, don't use it like that, even
> if you are using filesystems which don't appear to complain.

That's useful advice, thanks.

Do you (or anyone else) have any experience of using btrfs over iSCSI? I
was thinking about this for three different use cases:

1) Giving my workstation a data disk that is actually a partition on a
server -- keeping all the data on the big disks on the server and
reducing power consumption (just a small boot SSD in the workstation).

2) Splitting a btrfs RAID1 between a local disk and a remote iSCSI
mirror to provide  redundancy without putting more disks in the local
system. Of course, this would mean that one of the RAID1 copies would
have higher latency than the other.

3) Like case 1 but actually exposing an LVM logical volume from the
server using iSCSI, rather than a simple disk partition. I would then
put both encryption and RAID running on the server below that logical
volume.

NBD could also be an alternative to iSCSI in these cases as well.

Any thoughts?

Graham

Re: experiences running btrfs on external USB disks?

[Austin S. Hemmelgarn]

On 2018-12-04 08:37, Graham Cobb wrote:
> On 04/12/2018 12:38, Austin S. Hemmelgarn wrote:
>> In short, USB is _crap_ for fixed storage, don't use it like that, even
>> if you are using filesystems which don't appear to complain.
> 
> That's useful advice, thanks.
> 
> Do you (or anyone else) have any experience of using btrfs over iSCSI? I
> was thinking about this for three different use cases:
> 
> 1) Giving my workstation a data disk that is actually a partition on a
> server -- keeping all the data on the big disks on the server and
> reducing power consumption (just a small boot SSD in the workstation).
> 
> 2) Splitting a btrfs RAID1 between a local disk and a remote iSCSI
> mirror to provide  redundancy without putting more disks in the local
> system. Of course, this would mean that one of the RAID1 copies would
> have higher latency than the other.
> 
> 3) Like case 1 but actually exposing an LVM logical volume from the
> server using iSCSI, rather than a simple disk partition. I would then
> put both encryption and RAID running on the server below that logical
> volume.
> 
> NBD could also be an alternative to iSCSI in these cases as well.
> 
> Any thoughts?
I've not run it over iSCSI (I tend to avoid that overly-complicated 
mess), but I have done it over NBD and ATAoE, as well as some more 
exotic arrangements, and it's really not too bad.  The important part is 
making sure your block layer and all the stuff under it are reliable, 
and USB is not.