* many csum warning/errors on qemu guests using btrfs
@ 2020-04-27 15:23 Michal Soltys
2020-04-29 15:45 ` Michal Soltys
2020-04-29 19:10 ` Input/output errors (unreadables files) Ferry Toth
0 siblings, 2 replies; 10+ messages in thread
From: Michal Soltys @ 2020-04-27 15:23 UTC (permalink / raw)
To: linux-btrfs
Hi,
I have a guest machine (more about host it's running on later) with current debian buster running on debian's 5.4.0-0.bpo.4-amd64 kernel
(so from what I can see, 5.4.19 + whatever stuff debian did with it).
This machine is essentially nexus3 using btrfs for its storage needs, using snapshotting to ease backup tasks - so basically we prepare nexus to have its data in coherent state, make a ro snapshot of the subvolume with its data, then make backup from that to another subvolume.
The guest is running in qemu, using lvm thin pool as its storage - and journaled mdraid (raid5 physical disks, raid1 journal on ssds) as physical volumes. There are quite a few other virtual machines using this thin pool for - so far - over a year, with no apparent corruption or other issues.
More detailed stack description:
1) guest
debian buster, 5.4.0-0.bpo.4-amd64
☠ btrfs subvolume list /storage
ID 258 gen 62951 top level 5 path nexus-backup
ID 2624 gen 62969 top level 5 path nexus-data
ID 7418 gen 61532 top level 5 path nexus-data-snapshot
2) host
debian buster, 5.2.0-0.bpo.3-amd64
qemu 3.1.0 (3.1+dfsg-8+deb10u2)
the guest in question is using: cache=none, io=native, discards passed down, virtio-scsi with 1 iothread
3) storage
lvm thin-pool using mirrored ssds for metadata, raid5 on md for data (with journal on md raid1 consisting of two ssds)
the disks are behind:
24 disk backplane, connected via:
mpt2sas_cm0: Current Controller Queue Depth(8056),Max Controller Queue Depth(8192)
mpt2sas_cm0: Scatter Gather Elements per IO(128)
mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.02.00)
The lsi controller is in "IT" mode.
I can of course provide whatever more details are needed (mdadm, lvs, etc.)
Anyway we noticed rather worrysome amount of csum warnings. First a few files from actual production data (we have 13 files pinpointed by btrfs scrub):
Apr 24 16:12:04 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1116939354112 on dev /dev/sdb1, physical 528537223168, root 2624, inode 2019088, offset 4788224, length 4096, links 1 (path: blobs/touk_hosted/content/vol-04/chap-02/b52ea3b8-d072-4088-9bbc-6bbc279a06dc.bytes)
Apr 24 16:17:12 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1206004457472 on dev /dev/sdb1, physical 624044777472, root 2624, inode 2053325, offset 38678528, length 4096, links 1 (path: blobs/touk_hosted/content/vol-11/chap-09/9b6e7c7d-96f1-4b77-81a8-1d3a1b8d5053.bytes)
Apr 24 16:38:45 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1653594550272 on dev /dev/sdb1, physical 1072708612096, root 2624, inode 4764888, offset 1732608, length 4096, links 1 (path: blobs/touk_hosted/content/vol-02/chap-33/a64e7204-922a-46d5-b907-ba7cadfdcc92.bytes)
Apr 24 16:42:23 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1720708112384 on dev /dev/sdb1, physical 1141969657856, root 2624, inode 8460204, offset 77910016, length 4096, links 1 (path: blobs/touk_hosted/content/vol-07/chap-12/b01daa03-8345-440a-8830-07aebc6b03fc.bytes)
In this case, each of the files is either zip or jar. In case of zip files, unzip verifies everything correctly until the 4kb block.
But when we were doing fresh backup recently (as outlined above - from snapshotted ro subvolume to another subvolume):
Apr 27 13:31:45 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1731807096832 on dev /dev/sdb1, physical 1154142384128, root 258, inode 285, offset 3560243200, length 4096, links 1 (path: nexus3_blob_store_backup-touk_hosted-full-2020-04-25_01-00.tar)
Apr 27 13:32:00 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1735887626240 on dev /dev/sdb1, physical 1158222913536, root 258, inode 285, offset 7288856576, length 4096, links 1 (path: nexus3_blob_store_backup-touk_hosted-full-2020-04-25_01-00.tar)
Apr 27 13:32:08 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1738657308672 on dev /dev/sdb1, physical 1160992595968, root 258, inode 285, offset 9750835200, length 4096, links 1 (path: nexus3_blob_store_backup-touk_hosted-full-2020-04-25_01-00.tar)
Apr 27 13:33:44 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1761327661056 on dev /dev/sdb1, physical 1184736690176, root 258, inode 285, offset 29991952384, length 4096, links 1 (path: nexus3_blob_store_backup-touk_hosted-full-2020-04-25_01-00.tar)
Apr 27 13:34:29 nexus3 kernel: BTRFS warning (device sdb1): checksum error at logical 1774246481920 on dev /dev/sdb1, physical 1197655511040, root 258, inode 285, offset 41648664576, length 4096, links 1 (path: nexus3_blob_store_backup-touk_hosted-full-2020-04-25_01-00.tar)
(and many more - over 180 total)
At this point we're somewhat perplexed as to what is going on. This level of errors would nearly for sure cause visible issues with other virtual machines using this lvm pool. While it's possible we have corruptions creeping under without even realizing it, but after all this time we would have likely noticed something going on on other guests ... but:
On this host, there is one other machine with btrfs as its main filesystem (this one on arch, with kernel 5.3.8-arch1-1), I did quick test just a moment ago there and scrubbed it afterwards:
dd if=/dev/zero of=/atest bs=262144 count=$((16*4096))
[3802380.689492] BTRFS info (device sda3): scrub: started on devid 1
[3802448.842611] BTRFS warning (device sda3): checksum error at logical 20659965952 on dev /dev/sda3, physical 21708541952, root 5, inode 6665778, offset 9469747200, length 4096, links 1 (path: atest)
[3802448.843220] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[3802448.844983] BTRFS error (device sda3): unable to fixup (regular) error at logical 20659965952 on dev /dev/sda3
....
Afterwards I did repeat the above *directly on the host machine*, creating 16gib files (with dcfldd, so not 0s anymore) for half an hour, on same thin-pool using freshly made btrfs - no issues observed after btrfs scrubs. This might imply that the friction is somewhere around qemu (qemu version ? machine options ? virtio-scsi ? kernel ?). I did check ECC state of memory, but as far as edac-util is concered, no issues here either.
Were there any bugs around those kernel versions (5.2+ ?) related to btrfs / checksums or btrfs living in qemu (or perhaps that particular qemu version) ? Any suggestion how to pinpoint the issue ?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-27 15:23 many csum warning/errors on qemu guests using btrfs Michal Soltys
@ 2020-04-29 15:45 ` Michal Soltys
2020-04-29 19:21 ` Chris Murphy
2020-04-29 19:10 ` Input/output errors (unreadables files) Ferry Toth
1 sibling, 1 reply; 10+ messages in thread
From: Michal Soltys @ 2020-04-29 15:45 UTC (permalink / raw)
To: linux-btrfs
Short update:
1) turned out to not be btrfs fault in any way or form, as we recreated
the same issue with ext4 while manually checksumming the files; so if
anything, btrfs told us we have actual issues somewhere =)
2) qemu/vm scenario is also not to be blamed, as we recreated the issue
directly on the host as well
So as far as I can see, both of the above narrows the potential culprits
to either faulty/buggy hardware/firmware somewhere - or - some subtle
lvm/md/kernel issues. Though so far pinpointing the issue is proving
rather frustrating.
Anyway, sorry for the noise.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Input/output errors (unreadables files)
2020-04-27 15:23 many csum warning/errors on qemu guests using btrfs Michal Soltys
2020-04-29 15:45 ` Michal Soltys
@ 2020-04-29 19:10 ` Ferry Toth
2020-04-29 23:45 ` Qu Wenruo
1 sibling, 1 reply; 10+ messages in thread
From: Ferry Toth @ 2020-04-29 19:10 UTC (permalink / raw)
To: linux-btrfs
While upgrading Ubuntu the upgrade fails due to input/output errors on 4
files. Journal shows the same block is causing the problem for all 4 files.
I would like to delete the files, but that appears impossible.
I fear to reboot before finishing the upgrade, so this is with the old
kernel:
ferry@ferry-quad:~$ uname -a
Linux ferry-quad 5.3.0-51-generic #44-Ubuntu SMP Wed Apr 22 21:09:44 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
ferry@ferry-quad:~$ journalctl -b -e
...
apr 29 20:51:39 ferry-quad kernel: BTRFS critical (device sda2): corrupt
leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode generat>
apr 29 20:51:39 ferry-quad kernel: BTRFS error (device sda2):
block=1027628027904 read time tree block corruption detected
apr 29 20:51:55 ferry-quad kernel: BTRFS critical (device sda2): corrupt
leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode generat>
apr 29 20:51:55 ferry-quad kernel: BTRFS error (device sda2):
block=1027628027904 read time tree block corruption detected
ferry@ferry-quad:~$ sudo btrfs ins dump-tree -b 1027628027904 /dev/sda2
btrfs-progs v5.3-rc1
leaf 1027628027904 items 36 free space 966 generation 4933124 owner 294
leaf 1027628027904 flags 0x1(WRITTEN) backref revision 1
fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
location key (915987 INODE_ITEM 0) type FILE
transid 7782235549259005952 data_len 0 name_len 8
name: smb.conf
item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
generation 1 transid 18446744073709551492 size 12464
nbytes 16384
block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1350489744.0 (2012-10-17 18:02:24)
ctime 1353328654.0 (2012-11-19 13:37:34)
mtime 1350489744.0 (2012-10-17 18:02:24)
otime 0.0 (1970-01-01 01:00:00)
item 2 key (915987 INODE_REF 915986) itemoff 3779 itemsize 18
index 2 namelen 8 name: smb.conf
item 3 key (915987 EXTENT_DATA 0) itemoff 3726 itemsize 53
generation 18 type 1 (regular)
extent data disk byte 1110664871936 nr 16384
extent data offset 0 nr 16384 ram 16384
extent compression 0 (none)
item 4 key (915989 INODE_ITEM 0) itemoff 3566 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 144 flags 0x0(none)
atime 1571487761.716511096 (2019-10-19 14:22:41)
ctime 1571414492.451090452 (2019-10-18 18:01:32)
mtime 1571414492.451090452 (2019-10-18 18:01:32)
otime 0.0 (1970-01-01 01:00:00)
item 5 key (915989 INODE_REF 659081) itemoff 3546 itemsize 20
index 1101 namelen 10 name: libassuan0
item 6 key (915989 DIR_ITEM 653215628) itemoff 3497 itemsize 49
location key (61471579 INODE_ITEM 0) type FILE
transid 4900555 data_len 0 name_len 19
name: changelog.Debian.gz
item 7 key (915989 DIR_ITEM 1600214284) itemoff 3458 itemsize 39
location key (61471580 INODE_ITEM 0) type FILE
transid 4900555 data_len 0 name_len 9
name: copyright
item 8 key (915989 DIR_INDEX 56) itemoff 3409 itemsize 49
location key (61471579 INODE_ITEM 0) type FILE
transid 4900555 data_len 0 name_len 19
name: changelog.Debian.gz
item 9 key (915989 DIR_INDEX 57) itemoff 3370 itemsize 39
location key (61471580 INODE_ITEM 0) type FILE
transid 4900555 data_len 0 name_len 9
name: copyright
item 10 key (915990 INODE_ITEM 0) itemoff 3210 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 96 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1477470910.249847328 (2016-10-26 10:35:10)
mtime 1477470910.249847328 (2016-10-26 10:35:10)
otime 0.0 (1970-01-01 01:00:00)
item 11 key (915990 INODE_REF 659081) itemoff 3189 itemsize 21
index 1343 namelen 11 name: libasyncns0
item 12 key (915990 DIR_ITEM 653215628) itemoff 3140 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 13 key (915990 DIR_ITEM 1600214284) itemoff 3101 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 14 key (915990 DIR_INDEX 38) itemoff 3052 itemsize 49
location key (25762857 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 19
name: changelog.Debian.gz
item 15 key (915990 DIR_INDEX 39) itemoff 3013 itemsize 39
location key (25762858 INODE_ITEM 0) type FILE
transid 2068663 data_len 0 name_len 9
name: copyright
item 16 key (915991 INODE_ITEM 0) itemoff 2853 itemsize 160
generation 1 transid 4933124 size 68 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 48 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1540813807.330187853 (2018-10-29 12:50:07)
mtime 1540813807.330187853 (2018-10-29 12:50:07)
otime 0.0 (1970-01-01 01:00:00)
item 17 key (915991 INODE_REF 659081) itemoff 2831 itemsize 22
index 1545 namelen 12 name: libatasmart4
item 18 key (915991 DIR_ITEM 653215628) itemoff 2782 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 19 key (915991 DIR_ITEM 1600214284) itemoff 2743 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 20 key (915991 DIR_ITEM 3650993379) itemoff 2707 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 21 key (915991 DIR_INDEX 20) itemoff 2671 itemsize 36
location key (52273680 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 6
name: README
item 22 key (915991 DIR_INDEX 21) itemoff 2622 itemsize 49
location key (52273681 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 19
name: changelog.Debian.gz
item 23 key (915991 DIR_INDEX 22) itemoff 2583 itemsize 39
location key (52273682 INODE_ITEM 0) type FILE
transid 4036682 data_len 0 name_len 9
name: copyright
item 24 key (915992 INODE_ITEM 0) itemoff 2423 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 595 flags 0x0(none)
atime 1571487761.728511097 (2019-10-19 14:22:41)
ctime 1571414569.458782758 (2019-10-18 18:02:49)
mtime 1571414569.458782758 (2019-10-18 18:02:49)
otime 0.0 (1970-01-01 01:00:00)
item 25 key (915992 INODE_REF 659081) itemoff 2402 itemsize 21
index 2141 namelen 11 name: libatk1.0-0
item 26 key (915992 DIR_ITEM 653215628) itemoff 2353 itemsize 49
location key (61511250 INODE_ITEM 0) type FILE
transid 4902445 data_len 0 name_len 19
name: changelog.Debian.gz
item 27 key (915992 DIR_ITEM 1600214284) itemoff 2314 itemsize 39
location key (61511251 INODE_ITEM 0) type FILE
transid 4902445 data_len 0 name_len 9
name: copyright
item 28 key (915992 DIR_INDEX 227) itemoff 2265 itemsize 49
location key (61511250 INODE_ITEM 0) type FILE
transid 4902445 data_len 0 name_len 19
name: changelog.Debian.gz
item 29 key (915992 DIR_INDEX 228) itemoff 2226 itemsize 39
location key (61511251 INODE_ITEM 0) type FILE
transid 4902445 data_len 0 name_len 9
name: copyright
item 30 key (915993 INODE_ITEM 0) itemoff 2066 itemsize 160
generation 1 transid 4933124 size 56 nbytes 0
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 299 flags 0x0(none)
atime 1571487761.720511096 (2019-10-19 14:22:41)
ctime 1571414569.158783954 (2019-10-18 18:02:49)
mtime 1571414569.158783954 (2019-10-18 18:02:49)
otime 0.0 (1970-01-01 01:00:00)
item 31 key (915993 INODE_REF 659081) itemoff 2042 itemsize 24
index 1639 namelen 14 name: libatk1.0-data
item 32 key (915993 DIR_ITEM 653215628) itemoff 1993 itemsize 49
location key (61511215 INODE_ITEM 0) type FILE
transid 4902428 data_len 0 name_len 19
name: changelog.Debian.gz
item 33 key (915993 DIR_ITEM 1600214284) itemoff 1954 itemsize 39
location key (61511216 INODE_ITEM 0) type FILE
transid 4902428 data_len 0 name_len 9
name: copyright
item 34 key (915993 DIR_INDEX 116) itemoff 1905 itemsize 49
location key (61511215 INODE_ITEM 0) type FILE
transid 4902428 data_len 0 name_len 19
name: changelog.Debian.gz
item 35 key (915993 DIR_INDEX 117) itemoff 1866 itemsize 39
location key (61511216 INODE_ITEM 0) type FILE
transid 4902428 data_len 0 name_len 9
name: copyright
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-29 15:45 ` Michal Soltys
@ 2020-04-29 19:21 ` Chris Murphy
2020-04-30 1:46 ` Qu Wenruo
0 siblings, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2020-04-29 19:21 UTC (permalink / raw)
To: Michal Soltys; +Cc: Btrfs BTRFS
On Wed, Apr 29, 2020 at 9:45 AM Michal Soltys <msoltyspl@yandex.pl> wrote:
>
> Short update:
>
> 1) turned out to not be btrfs fault in any way or form, as we recreated
> the same issue with ext4 while manually checksumming the files; so if
> anything, btrfs told us we have actual issues somewhere =)
>
> 2) qemu/vm scenario is also not to be blamed, as we recreated the issue
> directly on the host as well
>
> So as far as I can see, both of the above narrows the potential culprits
> to either faulty/buggy hardware/firmware somewhere - or - some subtle
> lvm/md/kernel issues. Though so far pinpointing the issue is proving
> rather frustrating.
>
>
> Anyway, sorry for the noise.
It's not noise. I think it's useful to see how Btrfs can help isolate
such cases.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Input/output errors (unreadables files)
2020-04-29 19:10 ` Input/output errors (unreadables files) Ferry Toth
@ 2020-04-29 23:45 ` Qu Wenruo
2020-04-30 8:59 ` Ferry Toth
0 siblings, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2020-04-29 23:45 UTC (permalink / raw)
To: Ferry Toth, linux-btrfs
[-- Attachment #1.1: Type: text/plain, Size: 13441 bytes --]
On 2020/4/30 上午3:10, Ferry Toth wrote:
> While upgrading Ubuntu the upgrade fails due to input/output errors on 4
> files. Journal shows the same block is causing the problem for all 4 files.
>
> I would like to delete the files, but that appears impossible.
>
> I fear to reboot before finishing the upgrade, so this is with the old
> kernel:
>
> ferry@ferry-quad:~$ uname -a
> Linux ferry-quad 5.3.0-51-generic #44-Ubuntu SMP Wed Apr 22 21:09:44 UTC
> 2020 x86_64 x86_64 x86_64 GNU/Linux
>
> ferry@ferry-quad:~$ journalctl -b -e
> ...
> apr 29 20:51:39 ferry-quad kernel: BTRFS critical (device sda2): corrupt
> leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode
> generat>
This line explains the problem.
Older kernel tends to create inode with invalid generation.
Newer kernel is pretty picky about anything suspicious, and reject the
whole tree block.
Latest btrfs-progs should be able to repair it.
Or, revert to older kernel and delete these files, and call it a day.
Thanks,
Qu
> apr 29 20:51:39 ferry-quad kernel: BTRFS error (device sda2):
> block=1027628027904 read time tree block corruption detected
> apr 29 20:51:55 ferry-quad kernel: BTRFS critical (device sda2): corrupt
> leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode
> generat>
> apr 29 20:51:55 ferry-quad kernel: BTRFS error (device sda2):
> block=1027628027904 read time tree block corruption detected
>
> ferry@ferry-quad:~$ sudo btrfs ins dump-tree -b 1027628027904 /dev/sda2
> btrfs-progs v5.3-rc1
> leaf 1027628027904 items 36 free space 966 generation 4933124 owner 294
> leaf 1027628027904 flags 0x1(WRITTEN) backref revision 1
> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
> location key (915987 INODE_ITEM 0) type FILE
> transid 7782235549259005952 data_len 0 name_len 8
> name: smb.conf
> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
> generation 1 transid 18446744073709551492 size 12464
> nbytes 16384
> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
> sequence 0 flags 0x0(none)
> atime 1350489744.0 (2012-10-17 18:02:24)
> ctime 1353328654.0 (2012-11-19 13:37:34)
> mtime 1350489744.0 (2012-10-17 18:02:24)
> otime 0.0 (1970-01-01 01:00:00)
> item 2 key (915987 INODE_REF 915986) itemoff 3779 itemsize 18
> index 2 namelen 8 name: smb.conf
> item 3 key (915987 EXTENT_DATA 0) itemoff 3726 itemsize 53
> generation 18 type 1 (regular)
> extent data disk byte 1110664871936 nr 16384
> extent data offset 0 nr 16384 ram 16384
> extent compression 0 (none)
> item 4 key (915989 INODE_ITEM 0) itemoff 3566 itemsize 160
> generation 1 transid 4933124 size 56 nbytes 0
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 144 flags 0x0(none)
> atime 1571487761.716511096 (2019-10-19 14:22:41)
> ctime 1571414492.451090452 (2019-10-18 18:01:32)
> mtime 1571414492.451090452 (2019-10-18 18:01:32)
> otime 0.0 (1970-01-01 01:00:00)
> item 5 key (915989 INODE_REF 659081) itemoff 3546 itemsize 20
> index 1101 namelen 10 name: libassuan0
> item 6 key (915989 DIR_ITEM 653215628) itemoff 3497 itemsize 49
> location key (61471579 INODE_ITEM 0) type FILE
> transid 4900555 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 7 key (915989 DIR_ITEM 1600214284) itemoff 3458 itemsize 39
> location key (61471580 INODE_ITEM 0) type FILE
> transid 4900555 data_len 0 name_len 9
> name: copyright
> item 8 key (915989 DIR_INDEX 56) itemoff 3409 itemsize 49
> location key (61471579 INODE_ITEM 0) type FILE
> transid 4900555 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 9 key (915989 DIR_INDEX 57) itemoff 3370 itemsize 39
> location key (61471580 INODE_ITEM 0) type FILE
> transid 4900555 data_len 0 name_len 9
> name: copyright
> item 10 key (915990 INODE_ITEM 0) itemoff 3210 itemsize 160
> generation 1 transid 4933124 size 56 nbytes 0
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 96 flags 0x0(none)
> atime 1571487761.720511096 (2019-10-19 14:22:41)
> ctime 1477470910.249847328 (2016-10-26 10:35:10)
> mtime 1477470910.249847328 (2016-10-26 10:35:10)
> otime 0.0 (1970-01-01 01:00:00)
> item 11 key (915990 INODE_REF 659081) itemoff 3189 itemsize 21
> index 1343 namelen 11 name: libasyncns0
> item 12 key (915990 DIR_ITEM 653215628) itemoff 3140 itemsize 49
> location key (25762857 INODE_ITEM 0) type FILE
> transid 2068663 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 13 key (915990 DIR_ITEM 1600214284) itemoff 3101 itemsize 39
> location key (25762858 INODE_ITEM 0) type FILE
> transid 2068663 data_len 0 name_len 9
> name: copyright
> item 14 key (915990 DIR_INDEX 38) itemoff 3052 itemsize 49
> location key (25762857 INODE_ITEM 0) type FILE
> transid 2068663 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 15 key (915990 DIR_INDEX 39) itemoff 3013 itemsize 39
> location key (25762858 INODE_ITEM 0) type FILE
> transid 2068663 data_len 0 name_len 9
> name: copyright
> item 16 key (915991 INODE_ITEM 0) itemoff 2853 itemsize 160
> generation 1 transid 4933124 size 68 nbytes 0
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 48 flags 0x0(none)
> atime 1571487761.720511096 (2019-10-19 14:22:41)
> ctime 1540813807.330187853 (2018-10-29 12:50:07)
> mtime 1540813807.330187853 (2018-10-29 12:50:07)
> otime 0.0 (1970-01-01 01:00:00)
> item 17 key (915991 INODE_REF 659081) itemoff 2831 itemsize 22
> index 1545 namelen 12 name: libatasmart4
> item 18 key (915991 DIR_ITEM 653215628) itemoff 2782 itemsize 49
> location key (52273681 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 19 key (915991 DIR_ITEM 1600214284) itemoff 2743 itemsize 39
> location key (52273682 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 9
> name: copyright
> item 20 key (915991 DIR_ITEM 3650993379) itemoff 2707 itemsize 36
> location key (52273680 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 6
> name: README
> item 21 key (915991 DIR_INDEX 20) itemoff 2671 itemsize 36
> location key (52273680 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 6
> name: README
> item 22 key (915991 DIR_INDEX 21) itemoff 2622 itemsize 49
> location key (52273681 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 23 key (915991 DIR_INDEX 22) itemoff 2583 itemsize 39
> location key (52273682 INODE_ITEM 0) type FILE
> transid 4036682 data_len 0 name_len 9
> name: copyright
> item 24 key (915992 INODE_ITEM 0) itemoff 2423 itemsize 160
> generation 1 transid 4933124 size 56 nbytes 0
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 595 flags 0x0(none)
> atime 1571487761.728511097 (2019-10-19 14:22:41)
> ctime 1571414569.458782758 (2019-10-18 18:02:49)
> mtime 1571414569.458782758 (2019-10-18 18:02:49)
> otime 0.0 (1970-01-01 01:00:00)
> item 25 key (915992 INODE_REF 659081) itemoff 2402 itemsize 21
> index 2141 namelen 11 name: libatk1.0-0
> item 26 key (915992 DIR_ITEM 653215628) itemoff 2353 itemsize 49
> location key (61511250 INODE_ITEM 0) type FILE
> transid 4902445 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 27 key (915992 DIR_ITEM 1600214284) itemoff 2314 itemsize 39
> location key (61511251 INODE_ITEM 0) type FILE
> transid 4902445 data_len 0 name_len 9
> name: copyright
> item 28 key (915992 DIR_INDEX 227) itemoff 2265 itemsize 49
> location key (61511250 INODE_ITEM 0) type FILE
> transid 4902445 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 29 key (915992 DIR_INDEX 228) itemoff 2226 itemsize 39
> location key (61511251 INODE_ITEM 0) type FILE
> transid 4902445 data_len 0 name_len 9
> name: copyright
> item 30 key (915993 INODE_ITEM 0) itemoff 2066 itemsize 160
> generation 1 transid 4933124 size 56 nbytes 0
> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
> sequence 299 flags 0x0(none)
> atime 1571487761.720511096 (2019-10-19 14:22:41)
> ctime 1571414569.158783954 (2019-10-18 18:02:49)
> mtime 1571414569.158783954 (2019-10-18 18:02:49)
> otime 0.0 (1970-01-01 01:00:00)
> item 31 key (915993 INODE_REF 659081) itemoff 2042 itemsize 24
> index 1639 namelen 14 name: libatk1.0-data
> item 32 key (915993 DIR_ITEM 653215628) itemoff 1993 itemsize 49
> location key (61511215 INODE_ITEM 0) type FILE
> transid 4902428 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 33 key (915993 DIR_ITEM 1600214284) itemoff 1954 itemsize 39
> location key (61511216 INODE_ITEM 0) type FILE
> transid 4902428 data_len 0 name_len 9
> name: copyright
> item 34 key (915993 DIR_INDEX 116) itemoff 1905 itemsize 49
> location key (61511215 INODE_ITEM 0) type FILE
> transid 4902428 data_len 0 name_len 19
> name: changelog.Debian.gz
> item 35 key (915993 DIR_INDEX 117) itemoff 1866 itemsize 39
> location key (61511216 INODE_ITEM 0) type FILE
> transid 4902428 data_len 0 name_len 9
> name: copyright
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-29 19:21 ` Chris Murphy
@ 2020-04-30 1:46 ` Qu Wenruo
2020-04-30 5:01 ` Chris Murphy
2020-04-30 13:12 ` Michal Soltys
0 siblings, 2 replies; 10+ messages in thread
From: Qu Wenruo @ 2020-04-30 1:46 UTC (permalink / raw)
To: Chris Murphy, Michal Soltys; +Cc: Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1003 bytes --]
On 2020/4/30 上午3:21, Chris Murphy wrote:
> On Wed, Apr 29, 2020 at 9:45 AM Michal Soltys <msoltyspl@yandex.pl> wrote:
>>
>> Short update:
>>
>> 1) turned out to not be btrfs fault in any way or form, as we recreated
>> the same issue with ext4 while manually checksumming the files; so if
>> anything, btrfs told us we have actual issues somewhere =)
Is that related to mixing buffered write with DIO write?
If so, maybe changing the qemu cache mode may help?
Thanks,
Qu
>>
>> 2) qemu/vm scenario is also not to be blamed, as we recreated the issue
>> directly on the host as well
>>
>> So as far as I can see, both of the above narrows the potential culprits
>> to either faulty/buggy hardware/firmware somewhere - or - some subtle
>> lvm/md/kernel issues. Though so far pinpointing the issue is proving
>> rather frustrating.
>>
>>
>> Anyway, sorry for the noise.
>
> It's not noise. I think it's useful to see how Btrfs can help isolate
> such cases.
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-30 1:46 ` Qu Wenruo
@ 2020-04-30 5:01 ` Chris Murphy
2020-04-30 7:01 ` Qu Wenruo
2020-04-30 13:12 ` Michal Soltys
1 sibling, 1 reply; 10+ messages in thread
From: Chris Murphy @ 2020-04-30 5:01 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Chris Murphy, Michal Soltys, Btrfs BTRFS
On Wed, Apr 29, 2020 at 7:46 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2020/4/30 上午3:21, Chris Murphy wrote:
> > On Wed, Apr 29, 2020 at 9:45 AM Michal Soltys <msoltyspl@yandex.pl> wrote:
> >>
> >> Short update:
> >>
> >> 1) turned out to not be btrfs fault in any way or form, as we recreated
> >> the same issue with ext4 while manually checksumming the files; so if
> >> anything, btrfs told us we have actual issues somewhere =)
>
> Is that related to mixing buffered write with DIO write?
>
> If so, maybe changing the qemu cache mode may help?
I thought this would only happen if the host is Btrfs? Maybe it's a
bit crazy but these days I only use Btrfs on Btrfs with cache=unsafe.
I do lots of VM force quits, never see any problems. I haven't tested
it, but I think unsafe is quite unsafe if the host crashes/power fails
while the guest is active. Performance is much better though.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-30 5:01 ` Chris Murphy
@ 2020-04-30 7:01 ` Qu Wenruo
0 siblings, 0 replies; 10+ messages in thread
From: Qu Wenruo @ 2020-04-30 7:01 UTC (permalink / raw)
To: Chris Murphy; +Cc: Michal Soltys, Btrfs BTRFS
[-- Attachment #1.1: Type: text/plain, Size: 1578 bytes --]
On 2020/4/30 下午1:01, Chris Murphy wrote:
> On Wed, Apr 29, 2020 at 7:46 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>>
>> On 2020/4/30 上午3:21, Chris Murphy wrote:
>>> On Wed, Apr 29, 2020 at 9:45 AM Michal Soltys <msoltyspl@yandex.pl> wrote:
>>>>
>>>> Short update:
>>>>
>>>> 1) turned out to not be btrfs fault in any way or form, as we recreated
>>>> the same issue with ext4 while manually checksumming the files; so if
>>>> anything, btrfs told us we have actual issues somewhere =)
>>
>> Is that related to mixing buffered write with DIO write?
>>
>> If so, maybe changing the qemu cache mode may help?
>
> I thought this would only happen if the host is Btrfs?
Mixed buffered write with direct IO is known to cause problem, not only
for btrfs, but almost all fses.
> Maybe it's a
> bit crazy but these days I only use Btrfs on Btrfs with cache=unsafe.
Unlike the name, if you're using cache=unsafe, all writes are buffered,
thus you won't hit any csum mismatch problem caused by this.
But you can hit other problems though, e.g. if memory pressure is
forcing some image data to be written, it can break the COW requirement
of the VM (but the file is still completely sane).
> I do lots of VM force quits, never see any problems. I haven't tested
> it, but I think unsafe is quite unsafe if the host crashes/power fails
> while the guest is active. Performance is much better though.
>
Yeah, host power loss is another problem.
But at least, cache=unsafe actually avoids the csum problem.
Thanks,
Qu
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Input/output errors (unreadables files)
2020-04-29 23:45 ` Qu Wenruo
@ 2020-04-30 8:59 ` Ferry Toth
0 siblings, 0 replies; 10+ messages in thread
From: Ferry Toth @ 2020-04-30 8:59 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
Op 30-04-2020 om 01:45 schreef Qu Wenruo:
>
> On 2020/4/30 上午3:10, Ferry Toth wrote:
>> While upgrading Ubuntu the upgrade fails due to input/output errors on 4
>> files. Journal shows the same block is causing the problem for all 4 files.
>>
>> I would like to delete the files, but that appears impossible.
>>
>> I fear to reboot before finishing the upgrade, so this is with the old
>> kernel:
>>
>> ferry@ferry-quad:~$ uname -a
>> Linux ferry-quad 5.3.0-51-generic #44-Ubuntu SMP Wed Apr 22 21:09:44 UTC
>> 2020 x86_64 x86_64 x86_64 GNU/Linux
>>
>> ferry@ferry-quad:~$ journalctl -b -e
>> ...
>> apr 29 20:51:39 ferry-quad kernel: BTRFS critical (device sda2): corrupt
>> leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode
>> generat>
> This line explains the problem.
> Older kernel tends to create inode with invalid generation.
>
> Newer kernel is pretty picky about anything suspicious, and reject the
> whole tree block.
>
> Latest btrfs-progs should be able to repair it.
Yes, but is root file system and mounted.
> Or, revert to older kernel and delete these files, and call it a day.
I installed 4.15 kernel from Ubuntu kernel ppa and gambled that I would
be able to reboot.
And indeed, lucky for once. With 4.15 Synaptic (dpkg) is able to
complete package updates.
Thanks for the good advice!
>
> Thanks,
> Qu
>> apr 29 20:51:39 ferry-quad kernel: BTRFS error (device sda2):
>> block=1027628027904 read time tree block corruption detected
>> apr 29 20:51:55 ferry-quad kernel: BTRFS critical (device sda2): corrupt
>> leaf: root=294 block=1027628027904 slot=1 ino=915987, invalid inode
>> generat>
>> apr 29 20:51:55 ferry-quad kernel: BTRFS error (device sda2):
>> block=1027628027904 read time tree block corruption detected
>>
>> ferry@ferry-quad:~$ sudo btrfs ins dump-tree -b 1027628027904 /dev/sda2
>> btrfs-progs v5.3-rc1
>> leaf 1027628027904 items 36 free space 966 generation 4933124 owner 294
>> leaf 1027628027904 flags 0x1(WRITTEN) backref revision 1
>> fs uuid 27155120-9ef8-47fb-b248-eaac2b7c8375
>> chunk uuid 5704f1ba-08fd-4f6b-9117-0e080b4e9ef0
>> item 0 key (915986 DIR_INDEX 2) itemoff 3957 itemsize 38
>> location key (915987 INODE_ITEM 0) type FILE
>> transid 7782235549259005952 data_len 0 name_len 8
>> name: smb.conf
>> item 1 key (915987 INODE_ITEM 0) itemoff 3797 itemsize 160
>> generation 1 transid 18446744073709551492 size 12464
>> nbytes 16384
>> block group 0 mode 100644 links 1 uid 0 gid 0 rdev 0
>> sequence 0 flags 0x0(none)
>> atime 1350489744.0 (2012-10-17 18:02:24)
>> ctime 1353328654.0 (2012-11-19 13:37:34)
>> mtime 1350489744.0 (2012-10-17 18:02:24)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 2 key (915987 INODE_REF 915986) itemoff 3779 itemsize 18
>> index 2 namelen 8 name: smb.conf
>> item 3 key (915987 EXTENT_DATA 0) itemoff 3726 itemsize 53
>> generation 18 type 1 (regular)
>> extent data disk byte 1110664871936 nr 16384
>> extent data offset 0 nr 16384 ram 16384
>> extent compression 0 (none)
>> item 4 key (915989 INODE_ITEM 0) itemoff 3566 itemsize 160
>> generation 1 transid 4933124 size 56 nbytes 0
>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>> sequence 144 flags 0x0(none)
>> atime 1571487761.716511096 (2019-10-19 14:22:41)
>> ctime 1571414492.451090452 (2019-10-18 18:01:32)
>> mtime 1571414492.451090452 (2019-10-18 18:01:32)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 5 key (915989 INODE_REF 659081) itemoff 3546 itemsize 20
>> index 1101 namelen 10 name: libassuan0
>> item 6 key (915989 DIR_ITEM 653215628) itemoff 3497 itemsize 49
>> location key (61471579 INODE_ITEM 0) type FILE
>> transid 4900555 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 7 key (915989 DIR_ITEM 1600214284) itemoff 3458 itemsize 39
>> location key (61471580 INODE_ITEM 0) type FILE
>> transid 4900555 data_len 0 name_len 9
>> name: copyright
>> item 8 key (915989 DIR_INDEX 56) itemoff 3409 itemsize 49
>> location key (61471579 INODE_ITEM 0) type FILE
>> transid 4900555 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 9 key (915989 DIR_INDEX 57) itemoff 3370 itemsize 39
>> location key (61471580 INODE_ITEM 0) type FILE
>> transid 4900555 data_len 0 name_len 9
>> name: copyright
>> item 10 key (915990 INODE_ITEM 0) itemoff 3210 itemsize 160
>> generation 1 transid 4933124 size 56 nbytes 0
>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>> sequence 96 flags 0x0(none)
>> atime 1571487761.720511096 (2019-10-19 14:22:41)
>> ctime 1477470910.249847328 (2016-10-26 10:35:10)
>> mtime 1477470910.249847328 (2016-10-26 10:35:10)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 11 key (915990 INODE_REF 659081) itemoff 3189 itemsize 21
>> index 1343 namelen 11 name: libasyncns0
>> item 12 key (915990 DIR_ITEM 653215628) itemoff 3140 itemsize 49
>> location key (25762857 INODE_ITEM 0) type FILE
>> transid 2068663 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 13 key (915990 DIR_ITEM 1600214284) itemoff 3101 itemsize 39
>> location key (25762858 INODE_ITEM 0) type FILE
>> transid 2068663 data_len 0 name_len 9
>> name: copyright
>> item 14 key (915990 DIR_INDEX 38) itemoff 3052 itemsize 49
>> location key (25762857 INODE_ITEM 0) type FILE
>> transid 2068663 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 15 key (915990 DIR_INDEX 39) itemoff 3013 itemsize 39
>> location key (25762858 INODE_ITEM 0) type FILE
>> transid 2068663 data_len 0 name_len 9
>> name: copyright
>> item 16 key (915991 INODE_ITEM 0) itemoff 2853 itemsize 160
>> generation 1 transid 4933124 size 68 nbytes 0
>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>> sequence 48 flags 0x0(none)
>> atime 1571487761.720511096 (2019-10-19 14:22:41)
>> ctime 1540813807.330187853 (2018-10-29 12:50:07)
>> mtime 1540813807.330187853 (2018-10-29 12:50:07)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 17 key (915991 INODE_REF 659081) itemoff 2831 itemsize 22
>> index 1545 namelen 12 name: libatasmart4
>> item 18 key (915991 DIR_ITEM 653215628) itemoff 2782 itemsize 49
>> location key (52273681 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 19 key (915991 DIR_ITEM 1600214284) itemoff 2743 itemsize 39
>> location key (52273682 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 9
>> name: copyright
>> item 20 key (915991 DIR_ITEM 3650993379) itemoff 2707 itemsize 36
>> location key (52273680 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 6
>> name: README
>> item 21 key (915991 DIR_INDEX 20) itemoff 2671 itemsize 36
>> location key (52273680 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 6
>> name: README
>> item 22 key (915991 DIR_INDEX 21) itemoff 2622 itemsize 49
>> location key (52273681 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 23 key (915991 DIR_INDEX 22) itemoff 2583 itemsize 39
>> location key (52273682 INODE_ITEM 0) type FILE
>> transid 4036682 data_len 0 name_len 9
>> name: copyright
>> item 24 key (915992 INODE_ITEM 0) itemoff 2423 itemsize 160
>> generation 1 transid 4933124 size 56 nbytes 0
>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>> sequence 595 flags 0x0(none)
>> atime 1571487761.728511097 (2019-10-19 14:22:41)
>> ctime 1571414569.458782758 (2019-10-18 18:02:49)
>> mtime 1571414569.458782758 (2019-10-18 18:02:49)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 25 key (915992 INODE_REF 659081) itemoff 2402 itemsize 21
>> index 2141 namelen 11 name: libatk1.0-0
>> item 26 key (915992 DIR_ITEM 653215628) itemoff 2353 itemsize 49
>> location key (61511250 INODE_ITEM 0) type FILE
>> transid 4902445 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 27 key (915992 DIR_ITEM 1600214284) itemoff 2314 itemsize 39
>> location key (61511251 INODE_ITEM 0) type FILE
>> transid 4902445 data_len 0 name_len 9
>> name: copyright
>> item 28 key (915992 DIR_INDEX 227) itemoff 2265 itemsize 49
>> location key (61511250 INODE_ITEM 0) type FILE
>> transid 4902445 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 29 key (915992 DIR_INDEX 228) itemoff 2226 itemsize 39
>> location key (61511251 INODE_ITEM 0) type FILE
>> transid 4902445 data_len 0 name_len 9
>> name: copyright
>> item 30 key (915993 INODE_ITEM 0) itemoff 2066 itemsize 160
>> generation 1 transid 4933124 size 56 nbytes 0
>> block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
>> sequence 299 flags 0x0(none)
>> atime 1571487761.720511096 (2019-10-19 14:22:41)
>> ctime 1571414569.158783954 (2019-10-18 18:02:49)
>> mtime 1571414569.158783954 (2019-10-18 18:02:49)
>> otime 0.0 (1970-01-01 01:00:00)
>> item 31 key (915993 INODE_REF 659081) itemoff 2042 itemsize 24
>> index 1639 namelen 14 name: libatk1.0-data
>> item 32 key (915993 DIR_ITEM 653215628) itemoff 1993 itemsize 49
>> location key (61511215 INODE_ITEM 0) type FILE
>> transid 4902428 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 33 key (915993 DIR_ITEM 1600214284) itemoff 1954 itemsize 39
>> location key (61511216 INODE_ITEM 0) type FILE
>> transid 4902428 data_len 0 name_len 9
>> name: copyright
>> item 34 key (915993 DIR_INDEX 116) itemoff 1905 itemsize 49
>> location key (61511215 INODE_ITEM 0) type FILE
>> transid 4902428 data_len 0 name_len 19
>> name: changelog.Debian.gz
>> item 35 key (915993 DIR_INDEX 117) itemoff 1866 itemsize 39
>> location key (61511216 INODE_ITEM 0) type FILE
>> transid 4902428 data_len 0 name_len 9
>> name: copyright
>>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: many csum warning/errors on qemu guests using btrfs
2020-04-30 1:46 ` Qu Wenruo
2020-04-30 5:01 ` Chris Murphy
@ 2020-04-30 13:12 ` Michal Soltys
1 sibling, 0 replies; 10+ messages in thread
From: Michal Soltys @ 2020-04-30 13:12 UTC (permalink / raw)
To: Qu Wenruo, Chris Murphy; +Cc: Btrfs BTRFS
On 4/30/20 3:46 AM, Qu Wenruo wrote:
>
>
> On 2020/4/30 上午3:21, Chris Murphy wrote:
>> On Wed, Apr 29, 2020 at 9:45 AM Michal Soltys <msoltyspl@yandex.pl> wrote:
>>>
>>> Short update:
>>>
>>> 1) turned out to not be btrfs fault in any way or form, as we recreated
>>> the same issue with ext4 while manually checksumming the files; so if
>>> anything, btrfs told us we have actual issues somewhere =)
>
> Is that related to mixing buffered write with DIO write?
>
> If so, maybe changing the qemu cache mode may help?
>
> Thanks,
> Qu
>
Well, we initially thought the issue was with VMs only - but we also
managed to hit the problem with host machine directly. As for VMs - they
are all on separate lvm volumes (raw, not as images via filesystem - if
that's what you meant in context of mixing writing modes).
The without-qemu stack looks like this:
- on the bottom 24 disk backplane connected to lsi 2308 controller (v20
firwmare - for the record I found some tidbits, that this particular
firmware versions proved problematic for some people)
- md raid5 - 4 mechanical disks using write-back journal (the journal
device is md raid1 (2 ssds in the same backplane))
- the above raid device is added to lvm vg as a pv
- this pv is used for thin pool's data and 2 other ssds (mirrored on lvm
level, physically not in backplane) are used for thin pool's metadata
Then it's a matter of simple mkfs.ext4 or mkfs.btrfs on a lv created in
the above pool. Then a 16gb file created with e.g.:
dcfldd textpattern=$(hexdump -v -n 8192 -e '1/4 "%08X"' /dev/urandom)
hash=md5 hashlog=./test.bin bs=262144 count=$((16*4096)) of=test.md5
totalhashformat="#hash#"
Will usually (though not always) produce image that will read back
(after dropping caches) with different checksum (ext4 case) or will have
btrfs scrub complaining (btrfs case).
The culprits in the file will be one - few 4kb pieces with junk in them.
This is also unlike any other sizes used across the stack (md raid -
default 512kb chunk (1.5m stripe), lvm extents: 120m, thin-pool chunks:
1.5m).
While trying to get the issue replicated, what didn't work:
- I put 4 other disks in the backlane and created another raid5 in the
same way - using the same ssds as above for its journal - no issues
- used the new md raid as lvm linear volume - no issues either
- used the new md raid for lvm thin pool (using same ssds as earlier) -
no issues either
- used the old (!?!) md raid (the one giving issues) but creating a
linear volume on it - no issues
By "no issue" I mean the above dcfldd running in a loop for 3-6 hours,
interleaved with sync/fstrim/drop_caches as appropriate.
By "issue" I mean 1-3 runs are enough to create file with silent
corruptions.
What's worse, the "working" and "non-working" cases weirdly overlap with
each other, making it hard to even reasonably pinpoint the reason (as
for example, "it always happens if I use X").
While I realize it turned out not to be exactly btrfs mailing list
material - I'll appreciate any suggestions. For now I'm planning to drop
the firmware from 20 to 19 and update kernel - and see if that happens
to help.
>>>
>>> 2) qemu/vm scenario is also not to be blamed, as we recreated the issue
>>> directly on the host as well
>>>
>>> So as far as I can see, both of the above narrows the potential culprits
>>> to either faulty/buggy hardware/firmware somewhere - or - some subtle
>>> lvm/md/kernel issues. Though so far pinpointing the issue is proving
>>> rather frustrating.
>>>
>>>
>>> Anyway, sorry for the noise.
>>
>> It's not noise. I think it's useful to see how Btrfs can help isolate
>> such cases.
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-04-30 13:18 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-27 15:23 many csum warning/errors on qemu guests using btrfs Michal Soltys
2020-04-29 15:45 ` Michal Soltys
2020-04-29 19:21 ` Chris Murphy
2020-04-30 1:46 ` Qu Wenruo
2020-04-30 5:01 ` Chris Murphy
2020-04-30 7:01 ` Qu Wenruo
2020-04-30 13:12 ` Michal Soltys
2020-04-29 19:10 ` Input/output errors (unreadables files) Ferry Toth
2020-04-29 23:45 ` Qu Wenruo
2020-04-30 8:59 ` Ferry Toth
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.