* checksum error...
@ 2019-04-08 15:48 Scott E. Blomquist
2019-04-08 16:29 ` Hugo Mills
2019-04-15 13:14 ` Patrik Lundquist
0 siblings, 2 replies; 4+ messages in thread
From: Scott E. Blomquist @ 2019-04-08 15:48 UTC (permalink / raw)
To: linux-btrfs
Hi All,
The weekend btrfs scrub/balance came back with this following...
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 62378, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270894, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271453, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271449, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271435, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271432, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271421, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271411, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271401, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271391, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271345, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271253, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271163, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271074, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270983, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr 7 06:57:10 2019] BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Sun Apr 7 06:57:10 2019] BTRFS error (device sdb1): unable to fixup (regular) error at logical 274820497408 on dev /dev/sda1
Here is what I have...
root@cbmm-fsb:~# uname -a
Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
root@cbmm-fsb:~# btrfs --version
btrfs-progs v4.15.1
root@cbmm-fsb:~# btrfs fi show
Label: none uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
Total devices 2 FS bytes used 80.09TiB
devid 1 size 76.40TiB used 62.49TiB path /dev/sda1
devid 2 size 32.74TiB used 18.83TiB path /dev/sdb1
root@cbmm-fsb:~# btrfs fi df /home/cbcl
Data, single: total=79.80TiB, used=79.80TiB
System, RAID1: total=32.00MiB, used=9.09MiB
Metadata, RAID1: total=757.00GiB, used=281.34GiB
Metadata, DUP: total=22.50GiB, used=19.27GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
sda and sdb are megaraid raid6 with BBU and both are optimal.
Any tips? Thanks.
sb. Scott Blomquist
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: checksum error...
2019-04-08 15:48 checksum error Scott E. Blomquist
@ 2019-04-08 16:29 ` Hugo Mills
2019-04-08 18:40 ` Scott E. Blomquist
2019-04-15 13:14 ` Patrik Lundquist
1 sibling, 1 reply; 4+ messages in thread
From: Hugo Mills @ 2019-04-08 16:29 UTC (permalink / raw)
To: Scott E. Blomquist; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]
On Mon, Apr 08, 2019 at 11:48:03AM -0400, Scott E. Blomquist wrote:
>
> Hi All,
>
> The weekend btrfs scrub/balance came back with this following...
>
> [Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[snip]
Since there doesn't seem to be anything else wrong (no messages
without a filename, which would imply metadata corruption), this is
most likely a simple case of on-device corruption.
Delete yyyy/yyyyy and restore it from backups. At least, do so in
the working copy; The snapshots of it can safely remain until they get
rotated out normally.
Check your SMART statistics and see if anything looks wrong there
on the hardware side. Also check dmesg and earlier kernel logs for
signs of the hardware showing an error on read -- it may have tried
several times to read that location before giving up and/or returning
bad data.
Hugo.
>
> Here is what I have...
>
> root@cbmm-fsb:~# uname -a
> Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> root@cbmm-fsb:~# btrfs --version
> btrfs-progs v4.15.1
>
> root@cbmm-fsb:~# btrfs fi show
> Label: none uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
> Total devices 2 FS bytes used 80.09TiB
> devid 1 size 76.40TiB used 62.49TiB path /dev/sda1
> devid 2 size 32.74TiB used 18.83TiB path /dev/sdb1
>
> root@cbmm-fsb:~# btrfs fi df /home/cbcl
> Data, single: total=79.80TiB, used=79.80TiB
> System, RAID1: total=32.00MiB, used=9.09MiB
> Metadata, RAID1: total=757.00GiB, used=281.34GiB
> Metadata, DUP: total=22.50GiB, used=19.27GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> sda and sdb are megaraid raid6 with BBU and both are optimal.
>
> Any tips? Thanks.
>
> sb. Scott Blomquist
>
--
Hugo Mills | If it ain't broke, hit it again.
hugo@... carfax.org.uk |
http://carfax.org.uk/ |
PGP: E2AB1DE4 | Foon
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: checksum error...
2019-04-08 16:29 ` Hugo Mills
@ 2019-04-08 18:40 ` Scott E. Blomquist
0 siblings, 0 replies; 4+ messages in thread
From: Scott E. Blomquist @ 2019-04-08 18:40 UTC (permalink / raw)
To: Hugo Mills; +Cc: Scott E. Blomquist, linux-btrfs
Hugo Mills writes:
> On Mon, Apr 08, 2019 at 11:48:03AM -0400, Scott E. Blomquist wrote:
> >
> > Hi All,
> >
> > The weekend btrfs scrub/balance came back with this following...
> >
> > [Sun Apr 7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
> [snip]
>
> Since there doesn't seem to be anything else wrong (no messages
> without a filename, which would imply metadata corruption), this is
> most likely a simple case of on-device corruption.
>
> Delete yyyy/yyyyy and restore it from backups. At least, do so in
> the working copy; The snapshots of it can safely remain until they get
> rotated out normally.
>
> Check your SMART statistics and see if anything looks wrong there
> on the hardware side. Also check dmesg and earlier kernel logs for
> signs of the hardware showing an error on read -- it may have tried
> several times to read that location before giving up and/or returning
> bad data.
>
> Hugo.
Thanks, Hugo. Very helpful.
Turns out event log from MegaCli is showing some unexpected sense in
the eventlog.
Cheers,
sb. Scott Blomquist
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: checksum error...
2019-04-08 15:48 checksum error Scott E. Blomquist
2019-04-08 16:29 ` Hugo Mills
@ 2019-04-15 13:14 ` Patrik Lundquist
1 sibling, 0 replies; 4+ messages in thread
From: Patrik Lundquist @ 2019-04-15 13:14 UTC (permalink / raw)
To: Scott E. Blomquist; +Cc: linux-btrfs
On Mon, 8 Apr 2019 at 18:27, Scott E. Blomquist <sb@techsquare.com> wrote:
>
> root@cbmm-fsb:~# btrfs fi df /home/cbcl
> Data, single: total=79.80TiB, used=79.80TiB
> System, RAID1: total=32.00MiB, used=9.09MiB
> Metadata, RAID1: total=757.00GiB, used=281.34GiB
> Metadata, DUP: total=22.50GiB, used=19.27GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> sda and sdb are megaraid raid6 with BBU and both are optimal.
You should convert the remaining "Metadata, DUP" to RAID1.
btrfs balance start -v -mconvert=raid1,soft /home/cbcl
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-04-15 13:15 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-08 15:48 checksum error Scott E. Blomquist
2019-04-08 16:29 ` Hugo Mills
2019-04-08 18:40 ` Scott E. Blomquist
2019-04-15 13:14 ` Patrik Lundquist
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.