* Leaf corruption due to csum range
@ 2021-05-10 20:50 Philipp Fent
2021-05-11 8:18 ` Wang Yugui
2021-05-11 8:56 ` Filipe Manana
0 siblings, 2 replies; 8+ messages in thread
From: Philipp Fent @ 2021-05-10 20:50 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]
I encountered a btrfs error on my system. I run Microsoft SQL Server in
a docker container on a btrfs filesystem on an SSD. When bulk-loading
some benchmark data, my system reproducibly enters in the following
failing state:
[ 366.665714] BTRFS critical (device sda): corrupt leaf:
root=18446744073709551610 block=507544305664 slot=0, csum end range
(308900515840) goes beyond the start range (308900384768) of the next
csum item
[ 366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292
total ptrs 4 free space 3 owner 18446744073709551610
[ 366.665725] item 0 key (18446744073709551606 128 308891275264)
itemoff 7259 itemsize 9024
[ 366.665727] item 1 key (18446744073709551606 128 308900384768)
itemoff 7067 itemsize 192
[ 366.665728] item 2 key (18446744073709551606 128 309036716032)
itemoff 2587 itemsize 4480
[ 366.665730] item 3 key (18446744073709551606 128 309041303552)
itemoff 103 itemsize 2484
[ 366.665731] BTRFS error (device sda): block=507544305664 write time
tree block corruption detected
[ 366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136:
errno=-5 IO failure
[ 366.665824] BTRFS info (device sda): forced readonly
Please note the erroring ranges:
csum end: 308900515840
Start next: 308900384768
which is a difference of (1 << 17) == 0b100000000000000000 == 128KB
To me, this looks suspiciously like an off-by-one error, but I'm not too
versed in debugging btrfs.
I reproduced this several times on my machine using the attached
scripts. The only obvious similarity between the crashes is this 128KB
csum end / start next. Sometimes a get one corrupt leaf, sometimes many.
I tried to reproduce it on another machine with an HDD, but didn't
encounter this error there.
Can you help me to debug this further?
# uname -a
Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000
x86_64 GNU/Linux
# btrfs --version
btrfs-progs v5.11.1
# btrfs fi show
Label: none uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720
Total devices 1 FS bytes used 187.11GiB
devid 1 size 931.51GiB used 208.03GiB path /dev/sda
# btrfs fi df /ssdSpace
Data, single: total=207.00GiB, used=186.67GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=1.00GiB, used=450.08MiB
GlobalReserve, single: total=215.41MiB, used=0.00B
[-- Attachment #2: btrfsbug.tar.gz --]
[-- Type: application/gzip, Size: 1911 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Leaf corruption due to csum range 2021-05-10 20:50 Leaf corruption due to csum range Philipp Fent @ 2021-05-11 8:18 ` Wang Yugui 2021-05-11 8:44 ` Qu Wenruo 2021-05-11 8:56 ` Filipe Manana 1 sibling, 1 reply; 8+ messages in thread From: Wang Yugui @ 2021-05-11 8:18 UTC (permalink / raw) To: Philipp Fent; +Cc: linux-btrfs hi, the last 'write time tree block corruption detected' is marked as memory ECC error. From: chil L1n <devchill1n@gmail.com> To: linux-btrfs@vger.kernel.org Date: Sat, 6 Mar 2021 10:10:11 +0100 Subject: btrfs error: write time tree block corruption detected Is this a server with ECC memory? Best Regards Wang Yugui (wangyugui@e16-tech.com) 2021/05/11 > I encountered a btrfs error on my system. I run Microsoft SQL Server in > a docker container on a btrfs filesystem on an SSD. When bulk-loading > some benchmark data, my system reproducibly enters in the following > failing state: > > [ 366.665714] BTRFS critical (device sda): corrupt leaf: > root=18446744073709551610 block=507544305664 slot=0, csum end range > (308900515840) goes beyond the start range (308900384768) of the next > csum item > [ 366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292 > total ptrs 4 free space 3 owner 18446744073709551610 > [ 366.665725] item 0 key (18446744073709551606 128 308891275264) > itemoff 7259 itemsize 9024 > [ 366.665727] item 1 key (18446744073709551606 128 308900384768) > itemoff 7067 itemsize 192 > [ 366.665728] item 2 key (18446744073709551606 128 309036716032) > itemoff 2587 itemsize 4480 > [ 366.665730] item 3 key (18446744073709551606 128 309041303552) > itemoff 103 itemsize 2484 > [ 366.665731] BTRFS error (device sda): block=507544305664 write time > tree block corruption detected > [ 366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136: > errno=-5 IO failure > [ 366.665824] BTRFS info (device sda): forced readonly > > Please note the erroring ranges: > csum end: 308900515840 > Start next: 308900384768 > which is a difference of (1 << 17) == 0b100000000000000000 == 128KB > To me, this looks suspiciously like an off-by-one error, but I'm not too > versed in debugging btrfs. > > I reproduced this several times on my machine using the attached > scripts. The only obvious similarity between the crashes is this 128KB > csum end / start next. Sometimes a get one corrupt leaf, sometimes many. > I tried to reproduce it on another machine with an HDD, but didn't > encounter this error there. > Can you help me to debug this further? > > # uname -a > Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000 > x86_64 GNU/Linux > # btrfs --version > btrfs-progs v5.11.1 > # btrfs fi show > Label: none uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720 > Total devices 1 FS bytes used 187.11GiB > devid 1 size 931.51GiB used 208.03GiB path /dev/sda > # btrfs fi df /ssdSpace > Data, single: total=207.00GiB, used=186.67GiB > System, single: total=32.00MiB, used=48.00KiB > Metadata, single: total=1.00GiB, used=450.08MiB > GlobalReserve, single: total=215.41MiB, used=0.00B ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Leaf corruption due to csum range 2021-05-11 8:18 ` Wang Yugui @ 2021-05-11 8:44 ` Qu Wenruo 0 siblings, 0 replies; 8+ messages in thread From: Qu Wenruo @ 2021-05-11 8:44 UTC (permalink / raw) To: Wang Yugui, Philipp Fent; +Cc: linux-btrfs On 2021/5/11 下午4:18, Wang Yugui wrote: > hi, > > the last 'write time tree block corruption detected' is marked as > memory ECC error. So ECC can failed to recovery the bitflip? Now I can't even rely on ECC memories nowadays? (At least tree-check rocks again) Thanks, QU > > From: chil L1n <devchill1n@gmail.com> > To: linux-btrfs@vger.kernel.org > Date: Sat, 6 Mar 2021 10:10:11 +0100 > Subject: btrfs error: write time tree block corruption detected > > Is this a server with ECC memory? > > Best Regards > Wang Yugui (wangyugui@e16-tech.com) > 2021/05/11 > >> I encountered a btrfs error on my system. I run Microsoft SQL Server in >> a docker container on a btrfs filesystem on an SSD. When bulk-loading >> some benchmark data, my system reproducibly enters in the following >> failing state: >> >> [ 366.665714] BTRFS critical (device sda): corrupt leaf: >> root=18446744073709551610 block=507544305664 slot=0, csum end range >> (308900515840) goes beyond the start range (308900384768) of the next >> csum item >> [ 366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292 >> total ptrs 4 free space 3 owner 18446744073709551610 >> [ 366.665725] item 0 key (18446744073709551606 128 308891275264) >> itemoff 7259 itemsize 9024 >> [ 366.665727] item 1 key (18446744073709551606 128 308900384768) >> itemoff 7067 itemsize 192 >> [ 366.665728] item 2 key (18446744073709551606 128 309036716032) >> itemoff 2587 itemsize 4480 >> [ 366.665730] item 3 key (18446744073709551606 128 309041303552) >> itemoff 103 itemsize 2484 >> [ 366.665731] BTRFS error (device sda): block=507544305664 write time >> tree block corruption detected >> [ 366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136: >> errno=-5 IO failure >> [ 366.665824] BTRFS info (device sda): forced readonly >> >> Please note the erroring ranges: >> csum end: 308900515840 >> Start next: 308900384768 >> which is a difference of (1 << 17) == 0b100000000000000000 == 128KB >> To me, this looks suspiciously like an off-by-one error, but I'm not too >> versed in debugging btrfs. >> >> I reproduced this several times on my machine using the attached >> scripts. The only obvious similarity between the crashes is this 128KB >> csum end / start next. Sometimes a get one corrupt leaf, sometimes many. >> I tried to reproduce it on another machine with an HDD, but didn't >> encounter this error there. >> Can you help me to debug this further? >> >> # uname -a >> Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000 >> x86_64 GNU/Linux >> # btrfs --version >> btrfs-progs v5.11.1 >> # btrfs fi show >> Label: none uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720 >> Total devices 1 FS bytes used 187.11GiB >> devid 1 size 931.51GiB used 208.03GiB path /dev/sda >> # btrfs fi df /ssdSpace >> Data, single: total=207.00GiB, used=186.67GiB >> System, single: total=32.00MiB, used=48.00KiB >> Metadata, single: total=1.00GiB, used=450.08MiB >> GlobalReserve, single: total=215.41MiB, used=0.00B > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Leaf corruption due to csum range 2021-05-10 20:50 Leaf corruption due to csum range Philipp Fent 2021-05-11 8:18 ` Wang Yugui @ 2021-05-11 8:56 ` Filipe Manana [not found] ` <ad414944-2418-3728-ac1a-5d4d37e37ac1@in.tum.de> 1 sibling, 1 reply; 8+ messages in thread From: Filipe Manana @ 2021-05-11 8:56 UTC (permalink / raw) To: Philipp Fent; +Cc: linux-btrfs On Mon, May 10, 2021 at 10:01 PM Philipp Fent <fent@in.tum.de> wrote: > > I encountered a btrfs error on my system. I run Microsoft SQL Server in > a docker container on a btrfs filesystem on an SSD. When bulk-loading > some benchmark data, my system reproducibly enters in the following > failing state: > > [ 366.665714] BTRFS critical (device sda): corrupt leaf: > root=18446744073709551610 block=507544305664 slot=0, csum end range > (308900515840) goes beyond the start range (308900384768) of the next > csum item > [ 366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292 > total ptrs 4 free space 3 owner 18446744073709551610 > [ 366.665725] item 0 key (18446744073709551606 128 308891275264) > itemoff 7259 itemsize 9024 > [ 366.665727] item 1 key (18446744073709551606 128 308900384768) > itemoff 7067 itemsize 192 > [ 366.665728] item 2 key (18446744073709551606 128 309036716032) > itemoff 2587 itemsize 4480 > [ 366.665730] item 3 key (18446744073709551606 128 309041303552) > itemoff 103 itemsize 2484 > [ 366.665731] BTRFS error (device sda): block=507544305664 write time > tree block corruption detected > [ 366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136: > errno=-5 IO failure > [ 366.665824] BTRFS info (device sda): forced readonly > > Please note the erroring ranges: > csum end: 308900515840 > Start next: 308900384768 > which is a difference of (1 << 17) == 0b100000000000000000 == 128KB > To me, this looks suspiciously like an off-by-one error, but I'm not too > versed in debugging btrfs. Most likely it's a race when adding checksums. In this case for the log tree (fsync). This has happened in the past and the most recent fix was: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e289f03ea79bbc6574b78ac25682555423a91cbb There were cases too that affected the csum tree and not the log tree, but those are many years old now. > > I reproduced this several times on my machine using the attached > scripts. The only obvious similarity between the crashes is this 128KB > csum end / start next. Sometimes a get one corrupt leaf, sometimes many. > I tried to reproduce it on another machine with an HDD, but didn't > encounter this error there. > Can you help me to debug this further? Try to see if there are reflink operations (clone and dedupe) done by sql server (or maybe docker), in case there aren't, that excludes shared extents being the cause of the problem. I'll have to look at the code and think what might go wrong to lead to that, so I can't say that I have exact steps on how to debug that. Thanks. > > # uname -a > Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000 > x86_64 GNU/Linux > # btrfs --version > btrfs-progs v5.11.1 > # btrfs fi show > Label: none uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720 > Total devices 1 FS bytes used 187.11GiB > devid 1 size 931.51GiB used 208.03GiB path /dev/sda > # btrfs fi df /ssdSpace > Data, single: total=207.00GiB, used=186.67GiB > System, single: total=32.00MiB, used=48.00KiB > Metadata, single: total=1.00GiB, used=450.08MiB > GlobalReserve, single: total=215.41MiB, used=0.00B -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <ad414944-2418-3728-ac1a-5d4d37e37ac1@in.tum.de>]
* Re: Leaf corruption due to csum range [not found] ` <ad414944-2418-3728-ac1a-5d4d37e37ac1@in.tum.de> @ 2021-05-11 12:35 ` Filipe Manana [not found] ` <ef9ea56e-fb47-f719-137b-ffb545a09db7@in.tum.de> 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2021-05-11 12:35 UTC (permalink / raw) To: Philipp Fent; +Cc: Wang Yugui, linux-btrfs On Tue, May 11, 2021 at 12:35 PM Philipp Fent <fent@in.tum.de> wrote: > > On 11.05.21 10:18, Wang Yugui wrote: > > Is this a server with ECC memory? > > My machine does not have ECC memory. So I guess a random bitflip is a > possibility, but I would rule this out. Memtester didn't report any > errors in an extended test, and I run quite a lot of write heavy > databases like sql server. This sql server workload is the only > application where I ever got csum errors. And I really get them *every > time* I run it. I think is more likely an issue in sql server's write > pattern. > > > On 11.05.21 10:56, Filipe Manana wrote: > > Most likely it's a race when adding checksums. In this case for the > > log tree (fsync). > > Try to see if there are reflink operations (clone and dedupe) done by > > sql server (or maybe docker), in case there aren't, that excludes > > shared extents being the cause of the problem. > > I now also ran the sql server docker container under sysdig, which gave > me the following breakdown of executed system calls of the container: > CALLS/S TOT TIME AVG TIME SYSCALL > 13.05 702.41s 420ms futex > 12.11 193us 124ns gettid > 7.55 1.05ms 1.09us mprotect > 6.55 70ms 84us nanosleep > 5.70 128us 175ns getcpu > 5.44 1.04ms 1.49us read > 5.41 1.18ms 1.71us mmap > 5.25 1.99ms 2.96us openat > 4.44 122us 215ns close > 4.23 308us 568ns fstat > 1.50 298us 1.55us access > 1.11 21us 153ns getpid > 1.08 38us 279ns rt_sigprocmask > 0.94 41us 343ns lseek > 0.91 542us 4.67us munmap > 0.77 25us 260ns sigaltstack > 0.77 22us 226ns set_robust_list > 0.70 616us 6.92us clone > 0.55 653us 9.33us sendto > 0.53 88.96s 1.30s recvfrom > 0.44 867us 15us madvise > 0.44 11us 198ns rt_sigaction > > Filipe's suspicion of a race condition might be a good lead. There are > several clone operations in the trace and the majority of runtime seems Not familiar with sysdig myself, but those clone calls are likely to be the system call for creating process [1], and not the ioctl for cloning [2] or dedupe [3]. strace would be clear to me, which I'm more familiar with (or even better, bpftrace). Also, seeing that mmap is in the log, I just remembered that 5.13-rc1 includes a fix for races between mmap writes and fsync that could fix that: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=885f46d87f29a94eafe3cc707d5c4dea2be248f3 The changelog identifies logging file extent items with overlapping ranges, as the result of the race, but thinking out loud now, I think it could also result in logging checksum items with overlapping ranges because of that. If you want to test it, either try 5.13-rc1 or pick that patch and all its dependencies into 5.12.x (the dependencies are listed at the end of the changelog, all of them landed in 5.13-rc1). I'll see if later today or tomorrow I can get the reproducer to run here. [1] https://man7.org/linux/man-pages/man2/clone.2.html [2] https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html [3] https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html > to be spent contending locks, so I assume there are multiple threads at > work. > I've attached the full sysdig log (~1MB), not sure if this helps. > > Is there a way to increase the btrfs log output, so I could try to > observe which leafs are written? CONFIG_BTRFS_DEBUG looks like something > I could enable. Is this the right approach to narrow down a race? There's no way to log which leaves are written and dump their contents on the fly. That would produce tremendous amount of text to log. Most likely this is a race condition that the sql server workload is good for triggering (at least on your box). Thanks > > Thanks. -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <ef9ea56e-fb47-f719-137b-ffb545a09db7@in.tum.de>]
* Re: Leaf corruption due to csum range [not found] ` <ef9ea56e-fb47-f719-137b-ffb545a09db7@in.tum.de> @ 2021-05-13 9:57 ` Filipe Manana 2021-05-13 10:50 ` Filipe Manana 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2021-05-13 9:57 UTC (permalink / raw) To: Philipp Fent; +Cc: linux-btrfs On Tue, May 11, 2021 at 7:19 PM Philipp Fent <fent@in.tum.de> wrote: > > Thanks for the explanation! I wasn't aware of these ioctls. > > > strace would be clear to me, which I'm more familiar with (or even > better, bpftrace). > > I've attached an strace output that decompresses to about 200MB > logfiles. I can't make heads or tails of it, but I hope it helps. > I have never used bpftrace, do you have any pointers where I could start? There's some documentation and examples on their github. > > > I just remembered that 5.13-rc1 includes a fix for races between mmap > writes and fsync that could fix that > > I tried 5.13-rc1, but I'm running into the same csum range issue: > > > > Linux version 5.13.0-rc1-1-mainline (linux-mainline@archlinux) (gcc > (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Tue, 11 May > 2021 15:34:19 +0000 > ... > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610 > block=507430633472 slot=5, csum end range (293918547968) goes beyond the > start range (293918416896) of the next csum item > BTRFS info (device sda): leaf 507430633472 gen 18451 total ptrs 11 free > space 5016 owner 18446744073709551610 > item 0 key (18446744073709551606 128 293837238272) itemoff 15923 > itemsize 360 > item 1 key (18446744073709551606 128 293838544896) itemoff 15863 > itemsize 60 > item 2 key (18446744073709551606 128 293838675968) itemoff 15563 > itemsize 300 > item 3 key (18446744073709551606 128 293839527936) itemoff 15503 > itemsize 60 > item 4 key (18446744073709551606 128 293872295936) itemoff 15263 > itemsize 240 > item 5 key (18446744073709551606 128 293913763840) itemoff 10591 > itemsize 4672 > item 6 key (18446744073709551606 128 293918416896) itemoff 8351 > itemsize 2240 > item 7 key (18446744073709551606 128 293947658240) itemoff 8347 > itemsize 4 > item 8 key (18446744073709551606 128 293965193216) itemoff 8287 > itemsize 60 > item 9 key (18446744073709551606 128 293965848576) itemoff 8227 > itemsize 60 > item 10 key (18446744073709551606 128 293966176256) itemoff 5291 > itemsize 2936 > BTRFS error (device sda): block=507430633472 write time tree block > corruption detected > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610 > block=507447197696 slot=0, csum end range (320352133120) goes beyond the > start range (320352116736) of the next csum item > BTRFS info (device sda): leaf 507447197696 gen 18451 total ptrs 3 free > space 116 owner 18446744073709551610 > item 0 key (18446744073709551606 128 320336326656) itemoff 847 > itemsize 15436 > item 1 key (18446744073709551606 128 320352116736) itemoff 831 > itemsize 16 > item 2 key (18446744073709551606 128 320352247808) itemoff 191 > itemsize 640 > BTRFS error (device sda): block=507447197696 write time tree block > corruption detected > BTRFS: error (device sda) in btrfs_sync_log:3136: errno=-5 IO failure > BTRFS info (device sda): forced readonly Ok, then it's something else. When I run you reproducer I get an error: $ ./runMssql.sh Starting MSSQL docker container... 9943b714ed210a2937d5fce27ec110981b471e6e9f2c619629cb66501621ebb5 Loading TPC-H schema... Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed for user 'sa'.. dbgen.sh ran successfully before. Any idea? > > > > Curiously, the second leaf range overshot by only 16KB.... > Let me know, if I can try anything else. -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Leaf corruption due to csum range 2021-05-13 9:57 ` Filipe Manana @ 2021-05-13 10:50 ` Filipe Manana 2021-05-13 11:11 ` Philipp Fent 0 siblings, 1 reply; 8+ messages in thread From: Filipe Manana @ 2021-05-13 10:50 UTC (permalink / raw) To: Philipp Fent; +Cc: linux-btrfs On Thu, May 13, 2021 at 10:57 AM Filipe Manana <fdmanana@gmail.com> wrote: > > On Tue, May 11, 2021 at 7:19 PM Philipp Fent <fent@in.tum.de> wrote: > > > > Thanks for the explanation! I wasn't aware of these ioctls. > > > > > strace would be clear to me, which I'm more familiar with (or even > > better, bpftrace). > > > > I've attached an strace output that decompresses to about 200MB > > logfiles. I can't make heads or tails of it, but I hope it helps. > > I have never used bpftrace, do you have any pointers where I could start? > > There's some documentation and examples on their github. > > > > > > I just remembered that 5.13-rc1 includes a fix for races between mmap > > writes and fsync that could fix that > > > > I tried 5.13-rc1, but I'm running into the same csum range issue: > > > > > > > > Linux version 5.13.0-rc1-1-mainline (linux-mainline@archlinux) (gcc > > (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Tue, 11 May > > 2021 15:34:19 +0000 > > ... > > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610 > > block=507430633472 slot=5, csum end range (293918547968) goes beyond the > > start range (293918416896) of the next csum item > > BTRFS info (device sda): leaf 507430633472 gen 18451 total ptrs 11 free > > space 5016 owner 18446744073709551610 > > item 0 key (18446744073709551606 128 293837238272) itemoff 15923 > > itemsize 360 > > item 1 key (18446744073709551606 128 293838544896) itemoff 15863 > > itemsize 60 > > item 2 key (18446744073709551606 128 293838675968) itemoff 15563 > > itemsize 300 > > item 3 key (18446744073709551606 128 293839527936) itemoff 15503 > > itemsize 60 > > item 4 key (18446744073709551606 128 293872295936) itemoff 15263 > > itemsize 240 > > item 5 key (18446744073709551606 128 293913763840) itemoff 10591 > > itemsize 4672 > > item 6 key (18446744073709551606 128 293918416896) itemoff 8351 > > itemsize 2240 > > item 7 key (18446744073709551606 128 293947658240) itemoff 8347 > > itemsize 4 > > item 8 key (18446744073709551606 128 293965193216) itemoff 8287 > > itemsize 60 > > item 9 key (18446744073709551606 128 293965848576) itemoff 8227 > > itemsize 60 > > item 10 key (18446744073709551606 128 293966176256) itemoff 5291 > > itemsize 2936 > > BTRFS error (device sda): block=507430633472 write time tree block > > corruption detected > > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610 > > block=507447197696 slot=0, csum end range (320352133120) goes beyond the > > start range (320352116736) of the next csum item > > BTRFS info (device sda): leaf 507447197696 gen 18451 total ptrs 3 free > > space 116 owner 18446744073709551610 > > item 0 key (18446744073709551606 128 320336326656) itemoff 847 > > itemsize 15436 > > item 1 key (18446744073709551606 128 320352116736) itemoff 831 > > itemsize 16 > > item 2 key (18446744073709551606 128 320352247808) itemoff 191 > > itemsize 640 > > BTRFS error (device sda): block=507447197696 write time tree block > > corruption detected > > BTRFS: error (device sda) in btrfs_sync_log:3136: errno=-5 IO failure > > BTRFS info (device sda): forced readonly > > Ok, then it's something else. > > When I run you reproducer I get an error: > > $ ./runMssql.sh > Starting MSSQL docker container... > 9943b714ed210a2937d5fce27ec110981b471e6e9f2c619629cb66501621ebb5 > Loading TPC-H schema... > Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed > for user 'sa'.. Ok, never mind. I changed the 'sleep 5' to 'sleep 15' in the script and it works now. Seems like 5 seconds is too little on this vm for starting the server. I can also trigger the bug. I'll see what causes the bug. Thanks for providing the reliable reproducer, it's really helpful. > > dbgen.sh ran successfully before. > > Any idea? > > > > > > > > > Curiously, the second leaf range overshot by only 16KB.... > > Let me know, if I can try anything else. > > > > -- > Filipe David Manana, > > “Whether you think you can, or you think you can't — you're right.” -- Filipe David Manana, “Whether you think you can, or you think you can't — you're right.” ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Leaf corruption due to csum range 2021-05-13 10:50 ` Filipe Manana @ 2021-05-13 11:11 ` Philipp Fent 0 siblings, 0 replies; 8+ messages in thread From: Philipp Fent @ 2021-05-13 11:11 UTC (permalink / raw) To: fdmanana; +Cc: linux-btrfs On 13.05.21 12:50, Filipe Manana wrote: > Ok, never mind. > I changed the 'sleep 5' to 'sleep 15' in the script and it works now. > Seems like 5 seconds is too little on this vm for starting the server. Oh, that timing is unfortunate and sleeping is admittedly a not a good solution. You can try to add '-l 60' to the first sqlcmd to increase the login timeout, which should eliminate any unnecessary waiting time. > I can also trigger the bug. I'll see what causes the bug. > > Thanks for providing the reliable reproducer, it's really helpful. I'm glad this worked! Thank you for taking the time. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-05-13 11:11 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-10 20:50 Leaf corruption due to csum range Philipp Fent
2021-05-11 8:18 ` Wang Yugui
2021-05-11 8:44 ` Qu Wenruo
2021-05-11 8:56 ` Filipe Manana
[not found] ` <ad414944-2418-3728-ac1a-5d4d37e37ac1@in.tum.de>
2021-05-11 12:35 ` Filipe Manana
[not found] ` <ef9ea56e-fb47-f719-137b-ffb545a09db7@in.tum.de>
2021-05-13 9:57 ` Filipe Manana
2021-05-13 10:50 ` Filipe Manana
2021-05-13 11:11 ` Philipp Fent
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).