Input/output error, nothing appended in dmesg

Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

I'm getting occasional (every few weeks) input/output errors on a btrfs 
filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2 
kernel:

# cat 2016-08-02/serverX/syslog.log
cat: 2016-08-02/serverX/syslog.log: Input/output error


Strangely, nothing gets appended in dmesg:

# dmesg -c
#


The filesystem stores mostly remote syslog files (so, all text files, 
appended to).

Expected?



# btrfs fi show /var/log/remote/
Label: none  uuid: 5cec93a8-7894-41f6-94a4-9d9b58216dd4
         Total devices 1 FS bytes used 146.55GiB
         devid    1 size 200.00GiB used 153.01GiB path /dev/xvdb


# btrfs fi df /var/log/remote/
Data, single: total=149.00GiB, used=144.50GiB
System, single: total=4.00MiB, used=48.00KiB
Metadata, single: total=4.01GiB, used=2.05GiB
GlobalReserve, single: total=512.00MiB, used=0.00B



Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Chris Mason]

On 08/05/2016 07:42 AM, Tomasz Chmielewski wrote:
> I'm getting occasional (every few weeks) input/output errors on a btrfs
> filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2
> kernel:
>
> # cat 2016-08-02/serverX/syslog.log
> cat: 2016-08-02/serverX/syslog.log: Input/output error

How big is the file?  We had one bug with inline files that might have 
caused this.

-chris

Re: Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

On 2016-08-05 23:26, Chris Mason wrote:
> On 08/05/2016 07:42 AM, Tomasz Chmielewski wrote:
>> I'm getting occasional (every few weeks) input/output errors on a 
>> btrfs
>> filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2
>> kernel:
>> 
>> # cat 2016-08-02/serverX/syslog.log
>> cat: 2016-08-02/serverX/syslog.log: Input/output error
> 
> How big is the file?  We had one bug with inline files that might have
> caused this.

This one's tiny, 158137 bytes.



Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Chris Mason]

On 08/05/2016 10:44 AM, Tomasz Chmielewski wrote:
> On 2016-08-05 23:26, Chris Mason wrote:
>> On 08/05/2016 07:42 AM, Tomasz Chmielewski wrote:
>>> I'm getting occasional (every few weeks) input/output errors on a btrfs
>>> filesystem with compress-force=zlib, running on Amazon EC2, with 4.5.2
>>> kernel:
>>>
>>> # cat 2016-08-02/serverX/syslog.log
>>> cat: 2016-08-02/serverX/syslog.log: Input/output error
>>
>> How big is the file?  We had one bug with inline files that might have
>> caused this.
>
> This one's tiny, 158137 bytes.

Too big for the known problem though.  Still, can you btrfs-debug-tree 
and just make sure it doesn't have inline items?

-chris

Re: Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

On 2016-08-06 00:15, Chris Mason wrote:

>>>> # cat 2016-08-02/serverX/syslog.log
>>>> cat: 2016-08-02/serverX/syslog.log: Input/output error
>>> 
>>> How big is the file?  We had one bug with inline files that might 
>>> have
>>> caused this.
>> 
>> This one's tiny, 158137 bytes.
> 
> Too big for the known problem though.  Still, can you btrfs-debug-tree
> and just make sure it doesn't have inline items?

Hmmm....

# btrfs-debug-tree /dev/xvdb > /root/debug.tree
parent transid verify failed on 355229302784 wanted 49943295 found 
49943301
parent transid verify failed on 355229302784 wanted 49943295 found 
49943301
Ignoring transid failure
parent transid verify failed on 355233251328 wanted 49943299 found 
49943303
parent transid verify failed on 355233251328 wanted 49943299 found 
49943303
Ignoring transid failure
print-tree.c:1105: btrfs_print_tree: Assertion failed.
btrfs-debug-tree[0x418d99]
btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
btrfs-debug-tree(main+0x9a5)[0x432589]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
btrfs-debug-tree[0x4070e9]



Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Chris Mason]

On 08/05/2016 11:38 AM, Tomasz Chmielewski wrote:
> On 2016-08-06 00:15, Chris Mason wrote:
>
>>>>> # cat 2016-08-02/serverX/syslog.log
>>>>> cat: 2016-08-02/serverX/syslog.log: Input/output error
>>>>
>>>> How big is the file?  We had one bug with inline files that might have
>>>> caused this.
>>>
>>> This one's tiny, 158137 bytes.
>>
>> Too big for the known problem though.  Still, can you btrfs-debug-tree
>> and just make sure it doesn't have inline items?
>
> Hmmm....
>
> # btrfs-debug-tree /dev/xvdb > /root/debug.tree
> parent transid verify failed on 355229302784 wanted 49943295 found 49943301
> parent transid verify failed on 355229302784 wanted 49943295 found 49943301
> Ignoring transid failure
> parent transid verify failed on 355233251328 wanted 49943299 found 49943303
> parent transid verify failed on 355233251328 wanted 49943299 found 49943303
> Ignoring transid failure
> print-tree.c:1105: btrfs_print_tree: Assertion failed.
> btrfs-debug-tree[0x418d99]
> btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
> btrfs-debug-tree(main+0x9a5)[0x432589]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
> btrfs-debug-tree[0x4070e9]

Looks like the FS is mounted?

-chris

Re: Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

On 2016-08-06 00:38, Tomasz Chmielewski wrote:

>> Too big for the known problem though.  Still, can you btrfs-debug-tree
>> and just make sure it doesn't have inline items?
> 
> Hmmm....
> 
> # btrfs-debug-tree /dev/xvdb > /root/debug.tree
> parent transid verify failed on 355229302784 wanted 49943295 found 
> 49943301
> parent transid verify failed on 355229302784 wanted 49943295 found 
> 49943301
> Ignoring transid failure
> parent transid verify failed on 355233251328 wanted 49943299 found 
> 49943303
> parent transid verify failed on 355233251328 wanted 49943299 found 
> 49943303
> Ignoring transid failure
> print-tree.c:1105: btrfs_print_tree: Assertion failed.
> btrfs-debug-tree[0x418d99]
> btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
> btrfs-debug-tree(main+0x9a5)[0x432589]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
> btrfs-debug-tree[0x4070e9]

And, miracle cure O_o

# file ./2016-08-02/serverX/syslog.log
ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output 
error)

# echo 3 > /proc/sys/vm/drop_caches

# file 2016-08-02/serverX/syslog.log
2016-08-02/serverX/syslog.log: ASCII text, with very long lines

# cat 2016-08-02/serverX/syslog.log
(...)


Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

On 2016-08-06 00:40, Chris Mason wrote:

>>> Too big for the known problem though.  Still, can you 
>>> btrfs-debug-tree
>>> and just make sure it doesn't have inline items?
>> 
>> Hmmm....
>> 
>> # btrfs-debug-tree /dev/xvdb > /root/debug.tree
>> parent transid verify failed on 355229302784 wanted 49943295 found 
>> 49943301
>> parent transid verify failed on 355229302784 wanted 49943295 found 
>> 49943301
>> Ignoring transid failure
>> parent transid verify failed on 355233251328 wanted 49943299 found 
>> 49943303
>> parent transid verify failed on 355233251328 wanted 49943299 found 
>> 49943303
>> Ignoring transid failure
>> print-tree.c:1105: btrfs_print_tree: Assertion failed.
>> btrfs-debug-tree[0x418d99]
>> btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
>> btrfs-debug-tree(main+0x9a5)[0x432589]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
>> btrfs-debug-tree[0x4070e9]
> 
> Looks like the FS is mounted?

It is mounted, yes. Does btrfs-debug-tree need an unmounted FS?

I'm not able to unmount it unfortunately (in sense, the system has to 
work).


Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Chris Mason]

On 08/05/2016 11:45 AM, Tomasz Chmielewski wrote:
> On 2016-08-06 00:38, Tomasz Chmielewski wrote:
> 
>>> Too big for the known problem though.  Still, can you btrfs-debug-tree
>>> and just make sure it doesn't have inline items?
>>
>> Hmmm....
>>
>> # btrfs-debug-tree /dev/xvdb > /root/debug.tree
>> parent transid verify failed on 355229302784 wanted 49943295 found
>> 49943301
>> parent transid verify failed on 355229302784 wanted 49943295 found
>> 49943301
>> Ignoring transid failure
>> parent transid verify failed on 355233251328 wanted 49943299 found
>> 49943303
>> parent transid verify failed on 355233251328 wanted 49943299 found
>> 49943303
>> Ignoring transid failure
>> print-tree.c:1105: btrfs_print_tree: Assertion failed.
>> btrfs-debug-tree[0x418d99]
>> btrfs-debug-tree(btrfs_print_tree+0x26a)[0x41acf6]
>> btrfs-debug-tree(main+0x9a5)[0x432589]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f2369de0f45]
>> btrfs-debug-tree[0x4070e9]
> 
> And, miracle cure O_o
> 
> # file ./2016-08-02/serverX/syslog.log
> ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output error)
> 
> # echo 3 > /proc/sys/vm/drop_caches
> 
> # file 2016-08-02/serverX/syslog.log
> 2016-08-02/serverX/syslog.log: ASCII text, with very long lines
> 
> # cat 2016-08-02/serverX/syslog.log
> (...)
> 

If you don't already have this commit, please give it a try.  Should fix things up.

commit 8dff9c85341032767d7b519217a79ea04cd676b0
Author: Chris Mason <clm@fb.com>
Date:   Sat Sep 19 11:28:25 2015 -0700

    Btrfs: deal with duplciates during extent_map insertion in btrfs_get_extent

Re: Input/output error, nothing appended in dmesg

[Tomasz Chmielewski]

On 2016-08-06 00:45, Tomasz Chmielewski wrote:

> And, miracle cure O_o
> 
> # file ./2016-08-02/serverX/syslog.log
> ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output 
> error)
> 
> # echo 3 > /proc/sys/vm/drop_caches
> 
> # file 2016-08-02/serverX/syslog.log
> 2016-08-02/serverX/syslog.log: ASCII text, with very long lines
> 
> # cat 2016-08-02/serverX/syslog.log
> (...)

A few mins after the previous "echo 3 > /proc/sys/vm/drop_caches" (this 
file is around 1.5 MB and wasn't touched since 2016-06-21):

# file ./2016-06-21/serverY/nginx-dashboard-error.log
./2016-06-21/serverY/nginx-dashboard-error.log: ERROR: cannot read 
`./2016-06-21/serverY/nginx-dashboard-error.log' (Input/output error)

# echo 3 > /proc/sys/vm/drop_caches

# file ./2016-06-21/serverY/nginx-dashboard-error.log
./2016-06-21/serverY/nginx-dashboard-error.log: ASCII text, with very 
long lines

# cat ./2016-06-21/serverY/nginx-dashboard-error.log
(...works OK, no corruption...)


Tomasz Chmielewski
https://lxadm.com

Re: Input/output error, nothing appended in dmesg

[Diego Calleja]

El sábado, 6 de agosto de 2016 0:45:13 (CEST) Tomasz Chmielewski escribió:
> And, miracle cure O_o
> 
> # file ./2016-08-02/serverX/syslog.log
> ERROR: cannot read `./2016-08-02/serverX/syslog.log' (Input/output
> error)
> 
> # echo 3 > /proc/sys/vm/drop_caches
> 
> # file 2016-08-02/serverX/syslog.log
> 2016-08-02/serverX/syslog.log: ASCII text, with very long lines

FWIW, bugs similar to this one were reported in the past:

http://www.spinics.net/lists/linux-btrfs/msg54962.html
http://www.spinics.net/lists/linux-btrfs/msg52371.html