btrfs-transacti causing IO problem to btrfs

btrfs-transacti causing IO problem to btrfs

[Ian Gordon]

Hello,

I have a btrfs filesystem which has been working ok for about 90days,
but on Monday it become very slow (takes about 6hours to rsync backup a
3GB Ubuntu server - despite minimal changes from previous backup). I
noticed, that even with no processes reading or writing to the
filesystem, that btrfs-transaci was writing to the disk (averaging at
about 5MB/s) for a few hours before stopping until I wrote to the
filesystem again and then the process would repeat.

The btrfs filesystem uses skinny-metadata and is mounted with relatime
It has 744 subvolumes (of which about 700 are readonly snapshots)

Any ideas? Is there some sort of cleanup getting automatically run in
the background?

 * Linux hutton-cis.cc.strath.ac.uk 3.16.0-30-generic #40~14.04.1-Ubuntu
SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
 * Btrfs v3.12
 * Label: none  uuid: a3e49694-a66d-4c0d-b5bd-d2b7ced68535
	Total devices 1 FS bytes used 4.59TiB
	devid    1 size 8.76TiB used 4.77TiB path /dev/dm-0
 * Data, single: total=4.41TiB, used=4.41TiB
   System, DUP: total=8.00MiB, used=544.00KiB
   System, single: total=4.00MiB, used=0.00
   Metadata, DUP: total=186.00GiB, used=184.17GiB
   Metadata, single: total=8.00MiB, used=0.00
   unknown, single: total=512.00MiB, used=16.30MiB
 * There is nothing logged in dmesg except for the standard boot
   messages

Thanks for any help anyone can be,

-- 
Ian Gordon,
Department of Computer and Information Sciences,
University of Strathclyde.
Tel: 0141 548 3592    Fax: 0141 548 4523        Room: LT1307
The University of Strathclyde is a charitable body, registered in
Scotland, with registration number SC015263.

Re: btrfs-transacti causing IO problem to btrfs (skinny-metadata?)

[Martin]

On 25/03/15 08:29, Ian Gordon wrote:
> 
> Hello,
> 
> I have a btrfs filesystem which has been working ok for about 90days,
> but on Monday it become very slow (takes about 6hours to rsync backup a
> 3GB Ubuntu server - despite minimal changes from previous backup). I
> noticed, that even with no processes reading or writing to the
> filesystem, that btrfs-transaci was writing to the disk (averaging at
> about 5MB/s) for a few hours before stopping until I wrote to the
> filesystem again and then the process would repeat.
> 
> The btrfs filesystem uses skinny-metadata and is mounted with relatime
> It has 744 subvolumes (of which about 700 are readonly snapshots)
> 
> Any ideas? Is there some sort of cleanup getting automatically run in
> the background?

Is that btrfs system formatted with a previous kernel (pre-
skinny-metadata) and is now being used with a kernel that newly has
skinny-metadata enabled by default?...


I've stumbled across a bug/change/patch listed for a mixed pre/post
skinny-metadata whereby you get to see lots of csum errors in the logs...

For one 16TB btrfs raid1 system that brought things down to read-only...


I'm now on kernel 3.18.9, and Btrfs v3.18.2. Presently copying the
latest data onto a backup before trying a scrub! :-)

LOTs of:

kernel: parent transid verify failed on 5992676900864 wanted 70743 found
70709
kernel: parent transid verify failed on 5992676900864 wanted 70743 found
70709
kernel: BTRFS info (device sdj): no csum found for inode 50675726 start
16384
kernel: BTRFS info (device sdj): csum failed ino 50675726 off 0 csum
42383870 expected csum 0
kernel: BTRFS info (device sdj): csum failed ino 50675726 off 4096 csum
815939273 expected csum 0

and variations seen.

(Or advice for that one welcomed! ;-) )


Cheers,
Martin

Re: btrfs-transacti causing IO problem to btrfs (skinny-metadata?)

[Martin]

For anyone stumbling across this thread:

For my example, all was cleaned up by using the btrfs scrub:

btrfs scrub start /mountpoint


No data lost.

Good luck,
Martin


On 25/03/15 09:38, Martin wrote:

> 
> Is that btrfs system formatted with a previous kernel (pre-
> skinny-metadata) and is now being used with a kernel that newly has
> skinny-metadata enabled by default?...
> 
> 
> I've stumbled across a bug/change/patch listed for a mixed pre/post
> skinny-metadata whereby you get to see lots of csum errors in the logs...
> 
> For one 16TB btrfs raid1 system that brought things down to read-only...
> 
> 
> I'm now on kernel 3.18.9, and Btrfs v3.18.2. Presently copying the
> latest data onto a backup before trying a scrub! :-)
> 
> LOTs of:
> 
> kernel: parent transid verify failed on 5992676900864 wanted 70743 found
> 70709
> kernel: parent transid verify failed on 5992676900864 wanted 70743 found
> 70709
> kernel: BTRFS info (device sdj): no csum found for inode 50675726 start
> 16384
> kernel: BTRFS info (device sdj): csum failed ino 50675726 off 0 csum
> 42383870 expected csum 0
> kernel: BTRFS info (device sdj): csum failed ino 50675726 off 4096 csum
> 815939273 expected csum 0
> 
> and variations seen.
> 
> (Or advice for that one welcomed! ;-) )