* BTRFS scrub reports an error but check doesn't find any errors.
@ 2021-07-25 17:39 Dave T
2021-07-25 23:49 ` Qu Wenruo
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Dave T @ 2021-07-25 17:39 UTC (permalink / raw)
To: Btrfs BTRFS
What does the list recommend I do in this case?
starting btrfs scrub ...
scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
Scrub started: Sun Jul 25 00:40:43 2021
Status: finished
Duration: 2:52:45
Total to scrub: 1.26TiB
Rate: 113.72MiB/s
Error summary: read=1
Corrected: 0
Uncorrectable: 1
Unverified: 0
ERROR: there are uncorrectable errors
dmesg | grep "checksum error at" | tail -n 20
(no output)
# dmesg | grep -i checksum
[ +0.001698] xor: automatically using best checksumming function avx
(not related to BTRFS, right?)
# btrfs fi us /path/to/xyz
Overall:
Device size: 2.73TiB
Device allocated: 1.26TiB
Device unallocated: 1.47TiB
Device missing: 0.00B
Used: 1.12TiB
Free (estimated): 1.60TiB (min: 888.70GiB)
Free (statfs, df): 1.60TiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:1.25TiB, Used:1.11TiB (89.38%)
/dev/mapper/userluks 1.25TiB
Metadata,DUP: Size:6.00GiB, Used:5.26GiB (87.67%)
/dev/mapper/userluks 12.00GiB
System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
/dev/mapper/userluks 64.00MiB
Unallocated:
/dev/mapper/userluks 1.47TiB
# btrfs check /dev/mapper/xyz
Opening filesystem to check...
Checking filesystem on /dev/mapper/xyz
UUID: 56cea9cf-5374-4a43-b19d-6b0b143dc635
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 1230187327496 bytes used, no error found
total csum bytes: 1195610680
total tree bytes: 5648285696
total fs tree bytes: 4011016192
total extent tree bytes: 379256832
btree space waste bytes: 827370015
file data blocks allocated: 5497457123328
referenced 5523039584256
If more info is needed, please let me know. Recommendations and advice
are appreciated.
Thank you.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-25 17:39 BTRFS scrub reports an error but check doesn't find any errors Dave T
@ 2021-07-25 23:49 ` Qu Wenruo
2021-07-26 18:38 ` Chris Murphy
2021-07-27 21:41 ` Zygo Blaxell
2 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2021-07-25 23:49 UTC (permalink / raw)
To: Dave T, Btrfs BTRFS
On 2021/7/26 上午1:39, Dave T wrote:
> What does the list recommend I do in this case?
>
> starting btrfs scrub ...
> scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
> Scrub started: Sun Jul 25 00:40:43 2021
> Status: finished
> Duration: 2:52:45
> Total to scrub: 1.26TiB
> Rate: 113.72MiB/s
> Error summary: read=1
Scrub checks the csum for data and metadata, while btrfs-check only
checks the metadata, it doesn't check the csum of data, unless with
--check-data-csum option.
> Corrected: 0
> Uncorrectable: 1
> Unverified: 0
> ERROR: there are uncorrectable errors
>
> dmesg | grep "checksum error at" | tail -n 20
> (no output)
I'm more interested in this problem, scrub finds one csum error but no
output is pretty weird already.
>
> # dmesg | grep -i checksum
> [ +0.001698] xor: automatically using best checksumming function avx
> (not related to BTRFS, right?)
>
> # btrfs fi us /path/to/xyz
> Overall:
> Device size: 2.73TiB
> Device allocated: 1.26TiB
> Device unallocated: 1.47TiB
> Device missing: 0.00B
> Used: 1.12TiB
> Free (estimated): 1.60TiB (min: 888.70GiB)
> Free (statfs, df): 1.60TiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:1.25TiB, Used:1.11TiB (89.38%)
> /dev/mapper/userluks 1.25TiB
>
> Metadata,DUP: Size:6.00GiB, Used:5.26GiB (87.67%)
> /dev/mapper/userluks 12.00GiB
>
> System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
> /dev/mapper/userluks 64.00MiB
>
> Unallocated:
> /dev/mapper/userluks 1.47TiB
>
> # btrfs check /dev/mapper/xyz
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/xyz
> UUID: 56cea9cf-5374-4a43-b19d-6b0b143dc635
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 1230187327496 bytes used, no error found
Since btrfs-check reports no error, it means if there is some real
error, it's in data, not metadata.
And since no error message, the only way to catch the problem is through
"btrfs device stats" command to see which device gets is error
accounting increased.
And since the values are accumulated after the creation of the fs, it
may not be that obvious.
So you may want to record the output, run scrub again, then compare the
output to determine which device is affected.
Or, you can use "btrfs check --check-data-csum" to do a "scrub" in user
space.
Thanks,
Qu
> total csum bytes: 1195610680
> total tree bytes: 5648285696
> total fs tree bytes: 4011016192
> total extent tree bytes: 379256832
> btree space waste bytes: 827370015
> file data blocks allocated: 5497457123328
> referenced 5523039584256
>
> If more info is needed, please let me know. Recommendations and advice
> are appreciated.
> Thank you.
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-25 17:39 BTRFS scrub reports an error but check doesn't find any errors Dave T
2021-07-25 23:49 ` Qu Wenruo
@ 2021-07-26 18:38 ` Chris Murphy
2021-07-27 21:41 ` Zygo Blaxell
2 siblings, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2021-07-26 18:38 UTC (permalink / raw)
To: Dave T; +Cc: Btrfs BTRFS
On Sun, Jul 25, 2021 at 11:40 AM Dave T <davestechshop@gmail.com> wrote:
> dmesg | grep "checksum error at" | tail -n 20
> (no output)
>
> # dmesg | grep -i checksum
> [ +0.001698] xor: automatically using best checksumming function avx
> (not related to BTRFS, right?)
Search for "csum" or use -i btrfs instead.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-25 17:39 BTRFS scrub reports an error but check doesn't find any errors Dave T
2021-07-25 23:49 ` Qu Wenruo
2021-07-26 18:38 ` Chris Murphy
@ 2021-07-27 21:41 ` Zygo Blaxell
2021-07-28 15:15 ` Dave T
2 siblings, 1 reply; 11+ messages in thread
From: Zygo Blaxell @ 2021-07-27 21:41 UTC (permalink / raw)
To: Dave T; +Cc: Btrfs BTRFS
On Sun, Jul 25, 2021 at 01:39:55PM -0400, Dave T wrote:
> What does the list recommend I do in this case?
>
> starting btrfs scrub ...
> scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
> Scrub started: Sun Jul 25 00:40:43 2021
> Status: finished
> Duration: 2:52:45
> Total to scrub: 1.26TiB
> Rate: 113.72MiB/s
> Error summary: read=1
> Corrected: 0
> Uncorrectable: 1
> Unverified: 0
> ERROR: there are uncorrectable errors
This is a read failure (data not available from device), not a csum error
(data available but not correct).
> dmesg | grep "checksum error at" | tail -n 20
> (no output)
You should be looking for a IO failure on the underlying device (the
one below /dev/mapper/userluks). Look for log messages that appear just
before btrfs errors, or errors mentioning the device itself:
dmesg | grep -B99 -i btrfs
dmesg | grep -C9 sda
> # dmesg | grep -i checksum
> [ +0.001698] xor: automatically using best checksumming function avx
> (not related to BTRFS, right?)
>
> # btrfs fi us /path/to/xyz
> Overall:
> Device size: 2.73TiB
> Device allocated: 1.26TiB
> Device unallocated: 1.47TiB
> Device missing: 0.00B
> Used: 1.12TiB
> Free (estimated): 1.60TiB (min: 888.70GiB)
> Free (statfs, df): 1.60TiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:1.25TiB, Used:1.11TiB (89.38%)
> /dev/mapper/userluks 1.25TiB
>
> Metadata,DUP: Size:6.00GiB, Used:5.26GiB (87.67%)
> /dev/mapper/userluks 12.00GiB
>
> System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
> /dev/mapper/userluks 64.00MiB
Since the error was not corrected, it likely occurred in the data blocks.
A metadata error would be correctable, so check wouldn't report it because
the scrub will have already corrected it (assuming the underlying drive
is still healthy enough to remap bad sectors).
> Unallocated:
> /dev/mapper/userluks 1.47TiB
>
> # btrfs check /dev/mapper/xyz
That command won't read any data blocks, so it won't see any errors there.
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/xyz
> UUID: 56cea9cf-5374-4a43-b19d-6b0b143dc635
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 1230187327496 bytes used, no error found
> total csum bytes: 1195610680
> total tree bytes: 5648285696
> total fs tree bytes: 4011016192
> total extent tree bytes: 379256832
> btree space waste bytes: 827370015
> file data blocks allocated: 5497457123328
> referenced 5523039584256
>
> If more info is needed, please let me know. Recommendations and advice
> are appreciated.
> Thank you.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-27 21:41 ` Zygo Blaxell
@ 2021-07-28 15:15 ` Dave T
2021-07-28 16:11 ` Andrei Borzenkov
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Dave T @ 2021-07-28 15:15 UTC (permalink / raw)
To: Zygo Blaxell; +Cc: Btrfs BTRFS
On Tue, Jul 27, 2021 at 5:44 PM Zygo Blaxell
<ce3g8jdj@umail.furryterror.org> wrote:
>
> On Sun, Jul 25, 2021 at 01:39:55PM -0400, Dave T wrote:
> > What does the list recommend I do in this case?
> >
> > starting btrfs scrub ...
> > scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
> > Scrub started: Sun Jul 25 00:40:43 2021
> > Status: finished
> > Duration: 2:52:45
> > Total to scrub: 1.26TiB
> > Rate: 113.72MiB/s
> > Error summary: read=1
> > Corrected: 0
> > Uncorrectable: 1
> > Unverified: 0
> > ERROR: there are uncorrectable errors
>
> This is a read failure (data not available from device), not a csum error
> (data available but not correct).
Thank you.
>
> > dmesg | grep "checksum error at" | tail -n 20
> > (no output)
>
> You should be looking for a IO failure on the underlying device (the
> one below /dev/mapper/userluks).
That would be /dev/sde in my case.
> Look for log messages that appear just
> before btrfs errors, or errors mentioning the device itself:
>
> dmesg | grep -B99 -i btrfs
>
> dmesg | grep -C9 sda
>
Here is the complete output from a new scrub, including the output of
the commands above.
# fpath="/home/userdata"
# check_space "$fpath"
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/userluks 2.8T 1.2T 1.7T 42% /home/userdata
Data, single: total=1.25TiB, used=1.11TiB
System, DUP: total=32.00MiB, used=160.00KiB
Metadata, DUP: total=6.00GiB, used=5.32GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
# balance "$fpath"
starting btrfs balance for /home/userdata
Done, had to relocate 10 out of 1284 chunks
real 101m35.135s
user 0m0.000s
sys 0m32.991s
-----------------------
# scrub "$fpath"
starting btrfs scrub ...
scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
Scrub started: Tue Jul 27 19:46:25 2021
Status: finished
Duration: 2:09:17
Total to scrub: 1.25TiB
Rate: 151.98MiB/s
Error summary: read=1
Corrected: 0
Uncorrectable: 1
Unverified: 0
ERROR: there are uncorrectable errors
# btrfs fi us /home/userdata/
Overall:
Device size: 2.73TiB
Device allocated: 1.25TiB
Device unallocated: 1.48TiB
Device missing: 0.00B
Used: 1.12TiB
Free (estimated): 1.60TiB (min: 883.62GiB)
Free (statfs, df): 1.60TiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:1.24TiB, Used:1.11TiB (90.10%)
/dev/mapper/userluks 1.24TiB
Metadata,DUP: Size:6.00GiB, Used:5.35GiB (89.17%)
/dev/mapper/userluks 12.00GiB
System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
/dev/mapper/userluks 64.00MiB
Unallocated:
/dev/mapper/userluks 1.48TiB
# lsblk
NAME LABEL UUID PARTUUID
MODEL SIZE SERIAL
MOUNTPOINT
sde
ST3000DM001-1CH166 2.7T XXXXXXX
└─sde1 56cea9cf-3566-49f3-8abf-e59246f88a43
5db1ed2f-572e-4388-920b-6e4bfabf9e72 2.7T
└─userluks USERCOMMON 56cea9cf-5374-4a43-b19d-6b0b143dc635
2.7T
/mnt/temp/user
dmesg | grep -B99 -i btrfs
The only relevant output is:
[ +0.025578] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[ +0.000983] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[ +0.026646] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[ +0.001383] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[Jul28 00:03] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[ +0.001119] BTRFS warning (device dm-4): qgroup rescan init failed,
qgroup is not enabled
[ +1.828343] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +0.001132] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +0.029263] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +0.000969] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +0.068749] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +0.005558] BTRFS warning (device dm-0): qgroup rescan init failed,
qgroup is not enabled
[ +2.872178] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
[ +0.024708] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
[ +0.041130] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
[ +4.115641] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
[ +0.032633] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
[ +0.059885] BTRFS warning (device dm-2): qgroup rescan init failed,
qgroup is not enabled
# dmesg | grep -C9 sde
(no output)
The journal shows:
Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
0xffffffff SErr 0x0 action 0x0
Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
Jul 27 21:54:39 server kernel: ata10.00: cmd
60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
res
41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
Medium Error [current]
Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
Unrecovered read error - auto reallocate failed
Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
Jul 27 21:54:39 server kernel: ata10: EH complete
Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
0x4000000 SErr 0x0 action 0x0
Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
Jul 27 21:54:45 server kernel: ata10.00: cmd
60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
res
41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
Medium Error [current]
Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
Unrecovered read error - auto reallocate failed
Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
Jul 27 21:54:45 server kernel: ata10: EH complete
Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
at logical 1567691653120 on dev /dev/mapper/userluks, physical
1414087852032, root 19911, inode 624993, offset 5717954560, length
4096, links 1 (path: path/to/file/filename.ext)
Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
at logical 1567691653120 on dev /dev/mapper/userluks, physical
1414087852032, root 19989, inode 624993, offset 5717954560, length
4096, links 1 (path: path/to/file/filename.ext)
Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
at logical 1567691653120 on dev /dev/mapper/userluks, physical
1414087852032, root 20199, inode 624993, offset 5717954560, length
4096, links 1 (path: path/to/file/filename.ext)
Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
/dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
fixup (regular) error at logical 1567691653120 on dev
/dev/mapper/userluks
Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
finished on devid 1 with status: 0
btrfs su li /home/userdata/
...
ID 19911 gen 152144 top level 257 path @usertop/18053/snapshot
ID 19989 gen 152144 top level 257 path @usertop/18131/snapshot
ID 20199 gen 152144 top level 257 path @usertop/18313/snapshot
...
snapper -c userdata ls
...
18053 | single | | Fri 01 Jan 2021 12:00:18 AM EST | root |
timeline | timeline |
18131 | single | | Mon 04 Jan 2021 12:00:15 AM EST | root |
timeline | timeline |
18313 | single | | Mon 11 Jan 2021 12:00:20 AM EST | root |
timeline | timeline |
...
There are snapshots after that date without any errors. The live (r/w)
file system does not show any errors.
The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
SATA HDD).
Is there a way to mark sectors on the disk as bad? If so, is it
advisable to keep using this physical disk?
Thanks for sharing your knowledge. This is very helpful.
> > # dmesg | grep -i checksum
> > [ +0.001698] xor: automatically using best checksumming function avx
> > (not related to BTRFS, right?)
> >
> > # btrfs fi us /path/to/xyz
> > Overall:
> > Device size: 2.73TiB
> > Device allocated: 1.26TiB
> > Device unallocated: 1.47TiB
> > Device missing: 0.00B
> > Used: 1.12TiB
> > Free (estimated): 1.60TiB (min: 888.70GiB)
> > Free (statfs, df): 1.60TiB
> > Data ratio: 1.00
> > Metadata ratio: 2.00
> > Global reserve: 512.00MiB (used: 0.00B)
> > Multiple profiles: no
> >
> > Data,single: Size:1.25TiB, Used:1.11TiB (89.38%)
> > /dev/mapper/userluks 1.25TiB
> >
> > Metadata,DUP: Size:6.00GiB, Used:5.26GiB (87.67%)
> > /dev/mapper/userluks 12.00GiB
> >
> > System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
> > /dev/mapper/userluks 64.00MiB
>
> Since the error was not corrected, it likely occurred in the data blocks.
Yes, it appears so from the info above.
>
> A metadata error would be correctable, so check wouldn't report it because
> the scrub will have already corrected it (assuming the underlying drive
> is still healthy enough to remap bad sectors).
>
> > Unallocated:
> > /dev/mapper/userluks 1.47TiB
> >
> > # btrfs check /dev/mapper/xyz
>
> That command won't read any data blocks, so it won't see any errors there.
>
> > Opening filesystem to check...
> > Checking filesystem on /dev/mapper/xyz
> > UUID: 56cea9cf-5374-4a43-b19d-6b0b143dc635
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space cache
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 1230187327496 bytes used, no error found
> > total csum bytes: 1195610680
> > total tree bytes: 5648285696
> > total fs tree bytes: 4011016192
> > total extent tree bytes: 379256832
> > btree space waste bytes: 827370015
> > file data blocks allocated: 5497457123328
> > referenced 5523039584256
> >
> > If more info is needed, please let me know. Recommendations and advice
> > are appreciated.
> > Thank you.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 15:15 ` Dave T
@ 2021-07-28 16:11 ` Andrei Borzenkov
2021-07-28 16:21 ` Dave T
2021-07-28 19:18 ` Zygo Blaxell
2021-07-29 3:12 ` Chris Murphy
2 siblings, 1 reply; 11+ messages in thread
From: Andrei Borzenkov @ 2021-07-28 16:11 UTC (permalink / raw)
To: Dave T, Zygo Blaxell; +Cc: Btrfs BTRFS
On 28.07.2021 18:15, Dave T wrote:
...
>
> Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
> 0xffffffff SErr 0x0 action 0x0
> Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
> Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> Jul 27 21:54:39 server kernel: ata10.00: cmd
> 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
> res
> 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
> Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
> Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
> Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
> Medium Error [current]
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
> Unrecovered read error - auto reallocate failed
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
> 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
> Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
> sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
> Jul 27 21:54:39 server kernel: ata10: EH complete
> Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
> 0x4000000 SErr 0x0 action 0x0
> Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
> Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> Jul 27 21:54:45 server kernel: ata10.00: cmd
> 60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
> res
> 41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
> Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
> Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
> Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
> Medium Error [current]
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
> Unrecovered read error - auto reallocate failed
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
> 88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
> Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
> sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> Jul 27 21:54:45 server kernel: ata10: EH complete
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 19911, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 19989, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 20199, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
> /dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
> fixup (regular) error at logical 1567691653120 on dev
> /dev/mapper/userluks
> Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
> finished on devid 1 with status: 0
>
...>
> The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
> SATA HDD).
>
> Is there a way to mark sectors on the disk as bad? If so, is it
Directly overwriting sector may "fix" it (of course, data is still lost)
or trigger sector replacement. hdparm has --write-sector command
although I do not have any experience with it. Or simple dd may suffice.
Difference is that hdparm will bypass any kernel block layer recovery.
If you had redundant data profile, btrfs scrub would likely have fixed
it for you.
> advisable to keep using this physical disk?
>
Well, this happens, if this is just one sector so far I would say yes.
You probably need to keep an eye on it though.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 16:11 ` Andrei Borzenkov
@ 2021-07-28 16:21 ` Dave T
2021-07-28 18:17 ` Andrei Borzenkov
2021-07-28 19:19 ` Zygo Blaxell
0 siblings, 2 replies; 11+ messages in thread
From: Dave T @ 2021-07-28 16:21 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: ce3g8jdj, Btrfs BTRFS
On Wed, Jul 28, 2021 at 12:11 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
> On 28.07.2021 18:15, Dave T wrote:
> ...
> >
> > Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
> > 0xffffffff SErr 0x0 action 0x0
> > Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
> > Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> > Jul 27 21:54:39 server kernel: ata10.00: cmd
> > 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
> > res
> > 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
> > Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
> > Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
> > Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
> > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
> > hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
> > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
> > Medium Error [current]
> > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
> > Unrecovered read error - auto reallocate failed
> > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
> > 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
> > Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
> > sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
> > Jul 27 21:54:39 server kernel: ata10: EH complete
> > Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
> > 0x4000000 SErr 0x0 action 0x0
> > Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
> > Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> > Jul 27 21:54:45 server kernel: ata10.00: cmd
> > 60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
> > res
> > 41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
> > Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
> > Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
> > Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
> > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
> > hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
> > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
> > Medium Error [current]
> > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
> > Unrecovered read error - auto reallocate failed
> > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
> > 88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
> > Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
> > sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> > Jul 27 21:54:45 server kernel: ata10: EH complete
> > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > 1414087852032, root 19911, inode 624993, offset 5717954560, length
> > 4096, links 1 (path: path/to/file/filename.ext)
> > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > 1414087852032, root 19989, inode 624993, offset 5717954560, length
> > 4096, links 1 (path: path/to/file/filename.ext)
> > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > 1414087852032, root 20199, inode 624993, offset 5717954560, length
> > 4096, links 1 (path: path/to/file/filename.ext)
> > Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
> > /dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> > Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
> > fixup (regular) error at logical 1567691653120 on dev
> > /dev/mapper/userluks
> > Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
> > finished on devid 1 with status: 0
> >
> ...>
> > The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
> > SATA HDD).
> >
> > Is there a way to mark sectors on the disk as bad? If so, is it
>
> Directly overwriting sector may "fix" it (of course, data is still lost)
> or trigger sector replacement. hdparm has --write-sector command
> although I do not have any experience with it. Or simple dd may suffice.
> Difference is that hdparm will bypass any kernel block layer recovery.
> If you had redundant data profile, btrfs scrub would likely have fixed
> it for you.
I never knew BTRFS could duplicate data without RAID. This looks like
a great feature for my situation. I think I may upgrade this disk to a
larger one and enable DUP for data.
Is this a good tutorial to follow?
https://zejn.net/b/2017/04/30/single-device-data-redundancy-with-btrfs/
Should I expect my data to take twice as much space after enabling DUP?
>
> > advisable to keep using this physical disk?
> >
>
> Well, this happens, if this is just one sector so far I would say yes.
> You probably need to keep an eye on it though.
Thanks. My guess is that this is an isolated issue. It doesn't seem to
be growing, but I will watch it.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 16:21 ` Dave T
@ 2021-07-28 18:17 ` Andrei Borzenkov
2021-07-28 19:19 ` Zygo Blaxell
1 sibling, 0 replies; 11+ messages in thread
From: Andrei Borzenkov @ 2021-07-28 18:17 UTC (permalink / raw)
To: Dave T; +Cc: ce3g8jdj, Btrfs BTRFS
On 28.07.2021 19:21, Dave T wrote:
> On Wed, Jul 28, 2021 at 12:11 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>>
>> On 28.07.2021 18:15, Dave T wrote:
>> ...
>>>
>>> Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
>>> 0xffffffff SErr 0x0 action 0x0
>>> Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
>>> Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
>>> Jul 27 21:54:39 server kernel: ata10.00: cmd
>>> 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
>>> res
>>> 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
>>> Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
>>> Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
>>> Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
>>> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
>>> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
>>> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
>>> Medium Error [current]
>>> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
>>> Unrecovered read error - auto reallocate failed
>>> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
>>> 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
>>> Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
>>> sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
>>> Jul 27 21:54:39 server kernel: ata10: EH complete
>>> Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
>>> 0x4000000 SErr 0x0 action 0x0
>>> Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
>>> Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
>>> Jul 27 21:54:45 server kernel: ata10.00: cmd
>>> 60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
>>> res
>>> 41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
>>> Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
>>> Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
>>> Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
>>> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
>>> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
>>> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
>>> Medium Error [current]
>>> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
>>> Unrecovered read error - auto reallocate failed
>>> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
>>> 88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
>>> Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
>>> sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
>>> Jul 27 21:54:45 server kernel: ata10: EH complete
>>> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
>>> at logical 1567691653120 on dev /dev/mapper/userluks, physical
>>> 1414087852032, root 19911, inode 624993, offset 5717954560, length
>>> 4096, links 1 (path: path/to/file/filename.ext)
>>> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
>>> at logical 1567691653120 on dev /dev/mapper/userluks, physical
>>> 1414087852032, root 19989, inode 624993, offset 5717954560, length
>>> 4096, links 1 (path: path/to/file/filename.ext)
>>> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
>>> at logical 1567691653120 on dev /dev/mapper/userluks, physical
>>> 1414087852032, root 20199, inode 624993, offset 5717954560, length
>>> 4096, links 1 (path: path/to/file/filename.ext)
>>> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
>>> /dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
>>> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
>>> fixup (regular) error at logical 1567691653120 on dev
>>> /dev/mapper/userluks
>>> Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
>>> finished on devid 1 with status: 0
>>>
>> ...>
>>> The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
>>> SATA HDD).
>>>
>>> Is there a way to mark sectors on the disk as bad? If so, is it
>>
>> Directly overwriting sector may "fix" it (of course, data is still lost)
>> or trigger sector replacement. hdparm has --write-sector command
>> although I do not have any experience with it. Or simple dd may suffice.
>> Difference is that hdparm will bypass any kernel block layer recovery.
>> If you had redundant data profile, btrfs scrub would likely have fixed
>> it for you.
>
> I never knew BTRFS could duplicate data without RAID. This looks like
> a great feature for my situation. I think I may upgrade this disk to a
> larger one and enable DUP for data.
>
> Is this a good tutorial to follow?
> https://zejn.net/b/2017/04/30/single-device-data-redundancy-with-btrfs/
>
> Should I expect my data to take twice as much space after enabling DUP?
>
Yes, of course. Your data will be stored twice.
>>
>>> advisable to keep using this physical disk?
>>>
>>
>> Well, this happens, if this is just one sector so far I would say yes.
>> You probably need to keep an eye on it though.
>
> Thanks. My guess is that this is an isolated issue. It doesn't seem to
> be growing, but I will watch it.
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 15:15 ` Dave T
2021-07-28 16:11 ` Andrei Borzenkov
@ 2021-07-28 19:18 ` Zygo Blaxell
2021-07-29 3:12 ` Chris Murphy
2 siblings, 0 replies; 11+ messages in thread
From: Zygo Blaxell @ 2021-07-28 19:18 UTC (permalink / raw)
To: Dave T; +Cc: Btrfs BTRFS
On Wed, Jul 28, 2021 at 11:15:31AM -0400, Dave T wrote:
> On Tue, Jul 27, 2021 at 5:44 PM Zygo Blaxell
> <ce3g8jdj@umail.furryterror.org> wrote:
> >
> > On Sun, Jul 25, 2021 at 01:39:55PM -0400, Dave T wrote:
> > > What does the list recommend I do in this case?
> > >
> > > starting btrfs scrub ...
> > > scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
> > > Scrub started: Sun Jul 25 00:40:43 2021
> > > Status: finished
> > > Duration: 2:52:45
> > > Total to scrub: 1.26TiB
> > > Rate: 113.72MiB/s
> > > Error summary: read=1
> > > Corrected: 0
> > > Uncorrectable: 1
> > > Unverified: 0
> > > ERROR: there are uncorrectable errors
> >
> > This is a read failure (data not available from device), not a csum error
> > (data available but not correct).
>
> Thank you.
>
> >
> > > dmesg | grep "checksum error at" | tail -n 20
> > > (no output)
> >
> > You should be looking for a IO failure on the underlying device (the
> > one below /dev/mapper/userluks).
>
> That would be /dev/sde in my case.
>
> > Look for log messages that appear just
> > before btrfs errors, or errors mentioning the device itself:
> >
> > dmesg | grep -B99 -i btrfs
> >
> > dmesg | grep -C9 sda
> >
>
> Here is the complete output from a new scrub, including the output of
> the commands above.
>
> # fpath="/home/userdata"
>
> # check_space "$fpath"
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/userluks 2.8T 1.2T 1.7T 42% /home/userdata
> Data, single: total=1.25TiB, used=1.11TiB
> System, DUP: total=32.00MiB, used=160.00KiB
> Metadata, DUP: total=6.00GiB, used=5.32GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> # balance "$fpath"
> starting btrfs balance for /home/userdata
> Done, had to relocate 10 out of 1284 chunks
>
> real 101m35.135s
> user 0m0.000s
> sys 0m32.991s
> -----------------------
>
> # scrub "$fpath"
> starting btrfs scrub ...
> scrub done for 56cea9cf-5374-4a43-b19d-6b0b143dc635
> Scrub started: Tue Jul 27 19:46:25 2021
> Status: finished
> Duration: 2:09:17
> Total to scrub: 1.25TiB
> Rate: 151.98MiB/s
> Error summary: read=1
> Corrected: 0
> Uncorrectable: 1
> Unverified: 0
> ERROR: there are uncorrectable errors
>
>
> # btrfs fi us /home/userdata/
> Overall:
> Device size: 2.73TiB
> Device allocated: 1.25TiB
> Device unallocated: 1.48TiB
> Device missing: 0.00B
> Used: 1.12TiB
> Free (estimated): 1.60TiB (min: 883.62GiB)
> Free (statfs, df): 1.60TiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:1.24TiB, Used:1.11TiB (90.10%)
> /dev/mapper/userluks 1.24TiB
>
> Metadata,DUP: Size:6.00GiB, Used:5.35GiB (89.17%)
> /dev/mapper/userluks 12.00GiB
>
> System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
> /dev/mapper/userluks 64.00MiB
>
> Unallocated:
> /dev/mapper/userluks 1.48TiB
>
> # lsblk
> NAME LABEL UUID PARTUUID
> MODEL SIZE SERIAL
> MOUNTPOINT
>
> sde
> ST3000DM001-1CH166 2.7T XXXXXXX
> └─sde1 56cea9cf-3566-49f3-8abf-e59246f88a43
> 5db1ed2f-572e-4388-920b-6e4bfabf9e72 2.7T
> └─userluks USERCOMMON 56cea9cf-5374-4a43-b19d-6b0b143dc635
> 2.7T
> /mnt/temp/user
>
>
> dmesg | grep -B99 -i btrfs
> The only relevant output is:
>
> [ +0.025578] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.000983] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.026646] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.001383] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [Jul28 00:03] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.001119] BTRFS warning (device dm-4): qgroup rescan init failed,
> qgroup is not enabled
> [ +1.828343] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.001132] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.029263] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.000969] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.068749] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.005558] BTRFS warning (device dm-0): qgroup rescan init failed,
> qgroup is not enabled
> [ +2.872178] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.024708] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.041130] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
> [ +4.115641] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.032633] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
> [ +0.059885] BTRFS warning (device dm-2): qgroup rescan init failed,
> qgroup is not enabled
>
> # dmesg | grep -C9 sde
> (no output)
>
> The journal shows:
>
> Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
> 0xffffffff SErr 0x0 action 0x0
> Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
> Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> Jul 27 21:54:39 server kernel: ata10.00: cmd
> 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
> res
> 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
> Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
> Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
> Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
> Medium Error [current]
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
> Unrecovered read error - auto reallocate failed
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
> 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
> Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
> sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
> Jul 27 21:54:39 server kernel: ata10: EH complete
> Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
> 0x4000000 SErr 0x0 action 0x0
> Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
> Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> Jul 27 21:54:45 server kernel: ata10.00: cmd
> 60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
> res
> 41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
> Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
> Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
> Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
> Medium Error [current]
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
> Unrecovered read error - auto reallocate failed
> Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
> 88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
> Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
> sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> Jul 27 21:54:45 server kernel: ata10: EH complete
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 19911, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 19989, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> at logical 1567691653120 on dev /dev/mapper/userluks, physical
> 1414087852032, root 20199, inode 624993, offset 5717954560, length
> 4096, links 1 (path: path/to/file/filename.ext)
> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
> /dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
> fixup (regular) error at logical 1567691653120 on dev
> /dev/mapper/userluks
> Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
> finished on devid 1 with status: 0
>
>
> btrfs su li /home/userdata/
>
> ...
> ID 19911 gen 152144 top level 257 path @usertop/18053/snapshot
> ID 19989 gen 152144 top level 257 path @usertop/18131/snapshot
> ID 20199 gen 152144 top level 257 path @usertop/18313/snapshot
> ...
>
> snapper -c userdata ls
>
> ...
> 18053 | single | | Fri 01 Jan 2021 12:00:18 AM EST | root |
> timeline | timeline |
> 18131 | single | | Mon 04 Jan 2021 12:00:15 AM EST | root |
> timeline | timeline |
> 18313 | single | | Mon 11 Jan 2021 12:00:20 AM EST | root |
> timeline | timeline |
> ...
>
> There are snapshots after that date without any errors. The live (r/w)
> file system does not show any errors.
>
> The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
> SATA HDD).
>
> Is there a way to mark sectors on the disk as bad? If so, is it
> advisable to keep using this physical disk?
Delete or replace the file so the blocks are released to free space.
The next write over that sector will clobber the bad sector with fresh
data, and the drive should remap the bad sector at that time.
You can also try to expedite that process with dd, but that's error
prone (you have to calculate the offsets and not typo them into dd
commands...) and not really necessary (I never bother with that).
You should monitor the drive SMART data and run SMART self-tests
periodically (in addition to btrfs scrubs, they check different things).
If the number of UNC sectors increases over time, it's a good indicator
something is wrong with the drive and it may need replacement soon.
Seagate Barracudas often follow a pattern where over a period of a few
days, there's one UNC sector, then, two, then ten, then a thousand,
then the drive doesn't spin up any more and all data is lost. On those
drives I'd run a SMART long self-test daily if possible, or as close
to daily as your system load allows.
> Thanks for sharing your knowledge. This is very helpful.
>
> > > # dmesg | grep -i checksum
> > > [ +0.001698] xor: automatically using best checksumming function avx
> > > (not related to BTRFS, right?)
> > >
> > > # btrfs fi us /path/to/xyz
> > > Overall:
> > > Device size: 2.73TiB
> > > Device allocated: 1.26TiB
> > > Device unallocated: 1.47TiB
> > > Device missing: 0.00B
> > > Used: 1.12TiB
> > > Free (estimated): 1.60TiB (min: 888.70GiB)
> > > Free (statfs, df): 1.60TiB
> > > Data ratio: 1.00
> > > Metadata ratio: 2.00
> > > Global reserve: 512.00MiB (used: 0.00B)
> > > Multiple profiles: no
> > >
> > > Data,single: Size:1.25TiB, Used:1.11TiB (89.38%)
> > > /dev/mapper/userluks 1.25TiB
> > >
> > > Metadata,DUP: Size:6.00GiB, Used:5.26GiB (87.67%)
> > > /dev/mapper/userluks 12.00GiB
> > >
> > > System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
> > > /dev/mapper/userluks 64.00MiB
> >
> > Since the error was not corrected, it likely occurred in the data blocks.
>
> Yes, it appears so from the info above.
>
> >
> > A metadata error would be correctable, so check wouldn't report it because
> > the scrub will have already corrected it (assuming the underlying drive
> > is still healthy enough to remap bad sectors).
> >
> > > Unallocated:
> > > /dev/mapper/userluks 1.47TiB
> > >
> > > # btrfs check /dev/mapper/xyz
> >
> > That command won't read any data blocks, so it won't see any errors there.
> >
> > > Opening filesystem to check...
> > > Checking filesystem on /dev/mapper/xyz
> > > UUID: 56cea9cf-5374-4a43-b19d-6b0b143dc635
> > > [1/7] checking root items
> > > [2/7] checking extents
> > > [3/7] checking free space cache
> > > [4/7] checking fs roots
> > > [5/7] checking only csums items (without verifying data)
> > > [6/7] checking root refs
> > > [7/7] checking quota groups skipped (not enabled on this FS)
> > > found 1230187327496 bytes used, no error found
> > > total csum bytes: 1195610680
> > > total tree bytes: 5648285696
> > > total fs tree bytes: 4011016192
> > > total extent tree bytes: 379256832
> > > btree space waste bytes: 827370015
> > > file data blocks allocated: 5497457123328
> > > referenced 5523039584256
> > >
> > > If more info is needed, please let me know. Recommendations and advice
> > > are appreciated.
> > > Thank you.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 16:21 ` Dave T
2021-07-28 18:17 ` Andrei Borzenkov
@ 2021-07-28 19:19 ` Zygo Blaxell
1 sibling, 0 replies; 11+ messages in thread
From: Zygo Blaxell @ 2021-07-28 19:19 UTC (permalink / raw)
To: Dave T; +Cc: Andrei Borzenkov, Btrfs BTRFS
On Wed, Jul 28, 2021 at 12:21:43PM -0400, Dave T wrote:
> On Wed, Jul 28, 2021 at 12:11 PM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
> >
> > On 28.07.2021 18:15, Dave T wrote:
> > ...
> > >
> > > Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
> > > 0xffffffff SErr 0x0 action 0x0
> > > Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
> > > Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> > > Jul 27 21:54:39 server kernel: ata10.00: cmd
> > > 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
> > > res
> > > 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
> > > Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
> > > Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
> > > Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
> > > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
> > > hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
> > > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
> > > Medium Error [current]
> > > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
> > > Unrecovered read error - auto reallocate failed
> > > Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
> > > 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
> > > Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
> > > sector 2761896480 op 0x0:(READ) flags 0x0 phys_seg 15 prio class 0
> > > Jul 27 21:54:39 server kernel: ata10: EH complete
> > > Jul 27 21:54:45 server kernel: ata10.00: exception Emask 0x0 SAct
> > > 0x4000000 SErr 0x0 action 0x0
> > > Jul 27 21:54:45 server kernel: ata10.00: irq_stat 0x40000008
> > > Jul 27 21:54:45 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> > > Jul 27 21:54:45 server kernel: ata10.00: cmd
> > > 60/08:d0:20:32:9f/00:00:a4:00:00/40 tag 26 ncq dma 4096 in
> > > res
> > > 41/40:08:20:32:9f/00:00:a4:00:00/00 Emask 0x409 (media error) <F>
> > > Jul 27 21:54:45 server kernel: ata10.00: status: { DRDY ERR }
> > > Jul 27 21:54:45 server kernel: ata10.00: error: { UNC }
> > > Jul 27 21:54:45 server kernel: ata10.00: configured for UDMA/133
> > > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 FAILED Result:
> > > hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=4s
> > > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Sense Key :
> > > Medium Error [current]
> > > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 Add. Sense:
> > > Unrecovered read error - auto reallocate failed
> > > Jul 27 21:54:45 server kernel: sd 9:0:0:0: [sde] tag#26 CDB: Read(16)
> > > 88 00 00 00 00 00 a4 9f 32 20 00 00 00 08 00 00
> > > Jul 27 21:54:45 server kernel: blk_update_request: I/O error, dev sde,
> > > sector 2761896480 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> > > Jul 27 21:54:45 server kernel: ata10: EH complete
> > > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > > 1414087852032, root 19911, inode 624993, offset 5717954560, length
> > > 4096, links 1 (path: path/to/file/filename.ext)
> > > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > > 1414087852032, root 19989, inode 624993, offset 5717954560, length
> > > 4096, links 1 (path: path/to/file/filename.ext)
> > > Jul 27 21:54:45 server kernel: BTRFS warning (device dm-2): i/o error
> > > at logical 1567691653120 on dev /dev/mapper/userluks, physical
> > > 1414087852032, root 20199, inode 624993, offset 5717954560, length
> > > 4096, links 1 (path: path/to/file/filename.ext)
> > > Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): bdev
> > > /dev/mapper/userluks errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
> > > Jul 27 21:54:45 server kernel: BTRFS error (device dm-2): unable to
> > > fixup (regular) error at logical 1567691653120 on dev
> > > /dev/mapper/userluks
> > > Jul 27 21:55:42 server kernel: BTRFS info (device dm-2): scrub:
> > > finished on devid 1 with status: 0
> > >
> > ...>
> > > The volume is a 3TB disk, model ST3000DM001-1CH166 (Seagate Barracuda
> > > SATA HDD).
> > >
> > > Is there a way to mark sectors on the disk as bad? If so, is it
> >
> > Directly overwriting sector may "fix" it (of course, data is still lost)
> > or trigger sector replacement. hdparm has --write-sector command
> > although I do not have any experience with it. Or simple dd may suffice.
> > Difference is that hdparm will bypass any kernel block layer recovery.
> > If you had redundant data profile, btrfs scrub would likely have fixed
> > it for you.
>
> I never knew BTRFS could duplicate data without RAID. This looks like
> a great feature for my situation. I think I may upgrade this disk to a
> larger one and enable DUP for data.
>
> Is this a good tutorial to follow?
> https://zejn.net/b/2017/04/30/single-device-data-redundancy-with-btrfs/
>
> Should I expect my data to take twice as much space after enabling DUP?
Twice as much space, and also there is a seeking cost because the data is
written in two locations some distance apart on the media. It doesn't
help if the entire device fails, so at best it typically only delays
the inevitable total failure...but maybe that gives you time to finish
a backup before the drive dies.
> > > advisable to keep using this physical disk?
> > >
> >
> > Well, this happens, if this is just one sector so far I would say yes.
> > You probably need to keep an eye on it though.
>
> Thanks. My guess is that this is an isolated issue. It doesn't seem to
> be growing, but I will watch it.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: BTRFS scrub reports an error but check doesn't find any errors.
2021-07-28 15:15 ` Dave T
2021-07-28 16:11 ` Andrei Borzenkov
2021-07-28 19:18 ` Zygo Blaxell
@ 2021-07-29 3:12 ` Chris Murphy
2 siblings, 0 replies; 11+ messages in thread
From: Chris Murphy @ 2021-07-29 3:12 UTC (permalink / raw)
To: Dave T; +Cc: Zygo Blaxell, Btrfs BTRFS
On Wed, Jul 28, 2021 at 9:15 AM Dave T <davestechshop@gmail.com> wrote:
>
> The journal shows:
>
> Jul 27 21:54:39 server kernel: ata10.00: exception Emask 0x0 SAct
> 0xffffffff SErr 0x0 action 0x0
> Jul 27 21:54:39 server kernel: ata10.00: irq_stat 0x40000008
> Jul 27 21:54:39 server kernel: ata10.00: failed command: READ FPDMA QUEUED
> Jul 27 21:54:39 server kernel: ata10.00: cmd
> 60/00:90:98:2f:9f/03:00:a4:00:00/40 tag 18 ncq dma 393216 in
> res
> 41/40:00:20:32:9f/00:03:a4:00:00/00 Emask 0x409 (media error) <F>
> Jul 27 21:54:39 server kernel: ata10.00: status: { DRDY ERR }
> Jul 27 21:54:39 server kernel: ata10.00: error: { UNC }
> Jul 27 21:54:39 server kernel: ata10.00: configured for UDMA/133
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Sense Key :
> Medium Error [current]
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 Add. Sense:
> Unrecovered read error - auto reallocate failed
> Jul 27 21:54:39 server kernel: sd 9:0:0:0: [sde] tag#18 CDB: Read(16)
> 88 00 00 00 00 00 a4 9f 2f 98 00 00 03 00 00 00
> Jul 27 21:54:39 server kernel: blk_update_request: I/O error, dev sde,
Bad sector. It'll need to be overwritten to be remapped by the drive
firmware. I can't tell if it's 512n or 512e but the write needs to be
the size of the physical sector or else the firmware turns the write
into read-modify-write and you'll just get the same UNC error on the
read.
DUP profile will recover from this automatically. The file system is
informed of the physical sector, it does a reverse lookup to find the
logical block, reads good data from the good copy and overwrites the
bad copy. And either that'll stick or fail with an internal write
failure in which case the firmware remaps to a reserve sector. Once
reserve sectors are gone, you get UNC write errors - and that pretty
much means the drive is toast.
--
Chris Murphy
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-07-29 3:12 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-25 17:39 BTRFS scrub reports an error but check doesn't find any errors Dave T
2021-07-25 23:49 ` Qu Wenruo
2021-07-26 18:38 ` Chris Murphy
2021-07-27 21:41 ` Zygo Blaxell
2021-07-28 15:15 ` Dave T
2021-07-28 16:11 ` Andrei Borzenkov
2021-07-28 16:21 ` Dave T
2021-07-28 18:17 ` Andrei Borzenkov
2021-07-28 19:19 ` Zygo Blaxell
2021-07-28 19:18 ` Zygo Blaxell
2021-07-29 3:12 ` Chris Murphy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.