Leaf corruption due to csum range

Leaf corruption due to csum range

[Philipp Fent]

[-- Attachment #1: Type: text/plain, Size: 2304 bytes --]

I encountered a btrfs error on my system. I run Microsoft SQL Server in
a docker container on a btrfs filesystem on an SSD. When bulk-loading
some benchmark data, my system reproducibly enters in the following
failing state:

[  366.665714] BTRFS critical (device sda): corrupt leaf:
root=18446744073709551610 block=507544305664 slot=0, csum end range
(308900515840) goes beyond the start range (308900384768) of the next
csum item
[  366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292
total ptrs 4 free space 3 owner 18446744073709551610
[  366.665725]  item 0 key (18446744073709551606 128 308891275264)
itemoff 7259 itemsize 9024
[  366.665727]  item 1 key (18446744073709551606 128 308900384768)
itemoff 7067 itemsize 192
[  366.665728]  item 2 key (18446744073709551606 128 309036716032)
itemoff 2587 itemsize 4480
[  366.665730]  item 3 key (18446744073709551606 128 309041303552)
itemoff 103 itemsize 2484
[  366.665731] BTRFS error (device sda): block=507544305664 write time
tree block corruption detected
[  366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136:
errno=-5 IO failure
[  366.665824] BTRFS info (device sda): forced readonly

Please note the erroring ranges:
csum end:   308900515840
Start next: 308900384768
which is a difference of (1 << 17) == 0b100000000000000000 == 128KB
To me, this looks suspiciously like an off-by-one error, but I'm not too
versed in debugging btrfs.

I reproduced this several times on my machine using the attached
scripts. The only obvious similarity between the crashes is this 128KB
csum end / start next. Sometimes a get one corrupt leaf, sometimes many.
I tried to reproduce it on another machine with an HDD, but didn't
encounter this error there.
Can you help me to debug this further?

# uname -a
Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000
x86_64 GNU/Linux
# btrfs --version
btrfs-progs v5.11.1
# btrfs fi show
Label: none  uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720
        Total devices 1 FS bytes used 187.11GiB
        devid    1 size 931.51GiB used 208.03GiB path /dev/sda
# btrfs fi df /ssdSpace
Data, single: total=207.00GiB, used=186.67GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=1.00GiB, used=450.08MiB
GlobalReserve, single: total=215.41MiB, used=0.00B

[-- Attachment #2: btrfsbug.tar.gz --]
[-- Type: application/gzip, Size: 1911 bytes --]

Re: Leaf corruption due to csum range

[Wang Yugui]

hi,

the last 'write time tree block corruption detected' is marked as
memory ECC error.

 From:    chil L1n <devchill1n@gmail.com>
 To:      linux-btrfs@vger.kernel.org
 Date:    Sat, 6 Mar 2021 10:10:11 +0100
 Subject: btrfs error: write time tree block corruption detected

Is this a server with ECC memory?

Best Regards
Wang Yugui (wangyugui@e16-tech.com)
2021/05/11

> I encountered a btrfs error on my system. I run Microsoft SQL Server in
> a docker container on a btrfs filesystem on an SSD. When bulk-loading
> some benchmark data, my system reproducibly enters in the following
> failing state:
> 
> [  366.665714] BTRFS critical (device sda): corrupt leaf:
> root=18446744073709551610 block=507544305664 slot=0, csum end range
> (308900515840) goes beyond the start range (308900384768) of the next
> csum item
> [  366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292
> total ptrs 4 free space 3 owner 18446744073709551610
> [  366.665725]  item 0 key (18446744073709551606 128 308891275264)
> itemoff 7259 itemsize 9024
> [  366.665727]  item 1 key (18446744073709551606 128 308900384768)
> itemoff 7067 itemsize 192
> [  366.665728]  item 2 key (18446744073709551606 128 309036716032)
> itemoff 2587 itemsize 4480
> [  366.665730]  item 3 key (18446744073709551606 128 309041303552)
> itemoff 103 itemsize 2484
> [  366.665731] BTRFS error (device sda): block=507544305664 write time
> tree block corruption detected
> [  366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136:
> errno=-5 IO failure
> [  366.665824] BTRFS info (device sda): forced readonly
> 
> Please note the erroring ranges:
> csum end:   308900515840
> Start next: 308900384768
> which is a difference of (1 << 17) == 0b100000000000000000 == 128KB
> To me, this looks suspiciously like an off-by-one error, but I'm not too
> versed in debugging btrfs.
> 
> I reproduced this several times on my machine using the attached
> scripts. The only obvious similarity between the crashes is this 128KB
> csum end / start next. Sometimes a get one corrupt leaf, sometimes many.
> I tried to reproduce it on another machine with an HDD, but didn't
> encounter this error there.
> Can you help me to debug this further?
> 
> # uname -a
> Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000
> x86_64 GNU/Linux
> # btrfs --version
> btrfs-progs v5.11.1
> # btrfs fi show
> Label: none  uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720
>         Total devices 1 FS bytes used 187.11GiB
>         devid    1 size 931.51GiB used 208.03GiB path /dev/sda
> # btrfs fi df /ssdSpace
> Data, single: total=207.00GiB, used=186.67GiB
> System, single: total=32.00MiB, used=48.00KiB
> Metadata, single: total=1.00GiB, used=450.08MiB
> GlobalReserve, single: total=215.41MiB, used=0.00B

Re: Leaf corruption due to csum range

[Qu Wenruo]

On 2021/5/11 下午4:18, Wang Yugui wrote:
> hi,
>
> the last 'write time tree block corruption detected' is marked as
> memory ECC error.

So ECC can failed to recovery the bitflip?

Now I can't even rely on ECC memories nowadays?
(At least tree-check rocks again)

Thanks,
QU
>
>   From:    chil L1n <devchill1n@gmail.com>
>   To:      linux-btrfs@vger.kernel.org
>   Date:    Sat, 6 Mar 2021 10:10:11 +0100
>   Subject: btrfs error: write time tree block corruption detected
>
> Is this a server with ECC memory?
>
> Best Regards
> Wang Yugui (wangyugui@e16-tech.com)
> 2021/05/11
>
>> I encountered a btrfs error on my system. I run Microsoft SQL Server in
>> a docker container on a btrfs filesystem on an SSD. When bulk-loading
>> some benchmark data, my system reproducibly enters in the following
>> failing state:
>>
>> [  366.665714] BTRFS critical (device sda): corrupt leaf:
>> root=18446744073709551610 block=507544305664 slot=0, csum end range
>> (308900515840) goes beyond the start range (308900384768) of the next
>> csum item
>> [  366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292
>> total ptrs 4 free space 3 owner 18446744073709551610
>> [  366.665725]  item 0 key (18446744073709551606 128 308891275264)
>> itemoff 7259 itemsize 9024
>> [  366.665727]  item 1 key (18446744073709551606 128 308900384768)
>> itemoff 7067 itemsize 192
>> [  366.665728]  item 2 key (18446744073709551606 128 309036716032)
>> itemoff 2587 itemsize 4480
>> [  366.665730]  item 3 key (18446744073709551606 128 309041303552)
>> itemoff 103 itemsize 2484
>> [  366.665731] BTRFS error (device sda): block=507544305664 write time
>> tree block corruption detected
>> [  366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136:
>> errno=-5 IO failure
>> [  366.665824] BTRFS info (device sda): forced readonly
>>
>> Please note the erroring ranges:
>> csum end:   308900515840
>> Start next: 308900384768
>> which is a difference of (1 << 17) == 0b100000000000000000 == 128KB
>> To me, this looks suspiciously like an off-by-one error, but I'm not too
>> versed in debugging btrfs.
>>
>> I reproduced this several times on my machine using the attached
>> scripts. The only obvious similarity between the crashes is this 128KB
>> csum end / start next. Sometimes a get one corrupt leaf, sometimes many.
>> I tried to reproduce it on another machine with an HDD, but didn't
>> encounter this error there.
>> Can you help me to debug this further?
>>
>> # uname -a
>> Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000
>> x86_64 GNU/Linux
>> # btrfs --version
>> btrfs-progs v5.11.1
>> # btrfs fi show
>> Label: none  uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720
>>          Total devices 1 FS bytes used 187.11GiB
>>          devid    1 size 931.51GiB used 208.03GiB path /dev/sda
>> # btrfs fi df /ssdSpace
>> Data, single: total=207.00GiB, used=186.67GiB
>> System, single: total=32.00MiB, used=48.00KiB
>> Metadata, single: total=1.00GiB, used=450.08MiB
>> GlobalReserve, single: total=215.41MiB, used=0.00B
>
>

Re: Leaf corruption due to csum range

[Filipe Manana]

On Mon, May 10, 2021 at 10:01 PM Philipp Fent <fent@in.tum.de> wrote:
>
> I encountered a btrfs error on my system. I run Microsoft SQL Server in
> a docker container on a btrfs filesystem on an SSD. When bulk-loading
> some benchmark data, my system reproducibly enters in the following
> failing state:
>
> [  366.665714] BTRFS critical (device sda): corrupt leaf:
> root=18446744073709551610 block=507544305664 slot=0, csum end range
> (308900515840) goes beyond the start range (308900384768) of the next
> csum item
> [  366.665723] BTRFS info (device sda): leaf 507544305664 gen 18292
> total ptrs 4 free space 3 owner 18446744073709551610
> [  366.665725]  item 0 key (18446744073709551606 128 308891275264)
> itemoff 7259 itemsize 9024
> [  366.665727]  item 1 key (18446744073709551606 128 308900384768)
> itemoff 7067 itemsize 192
> [  366.665728]  item 2 key (18446744073709551606 128 309036716032)
> itemoff 2587 itemsize 4480
> [  366.665730]  item 3 key (18446744073709551606 128 309041303552)
> itemoff 103 itemsize 2484
> [  366.665731] BTRFS error (device sda): block=507544305664 write time
> tree block corruption detected
> [  366.665821] BTRFS: error (device sda) in btrfs_sync_log:3136:
> errno=-5 IO failure
> [  366.665824] BTRFS info (device sda): forced readonly
>
> Please note the erroring ranges:
> csum end:   308900515840
> Start next: 308900384768
> which is a difference of (1 << 17) == 0b100000000000000000 == 128KB
> To me, this looks suspiciously like an off-by-one error, but I'm not too
> versed in debugging btrfs.

Most likely it's a race when adding checksums. In this case for the
log tree (fsync).
This has happened in the past and the most recent fix was:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e289f03ea79bbc6574b78ac25682555423a91cbb

There were cases too that affected the csum tree and not the log tree,
but those are many years old now.

>
> I reproduced this several times on my machine using the attached
> scripts. The only obvious similarity between the crashes is this 128KB
> csum end / start next. Sometimes a get one corrupt leaf, sometimes many.
> I tried to reproduce it on another machine with an HDD, but didn't
> encounter this error there.
> Can you help me to debug this further?

Try to see if there are reflink operations (clone and dedupe) done by
sql server (or maybe docker), in case there aren't, that excludes
shared extents being the cause of the problem.

I'll have to look at the code and think what might go wrong to lead to
that, so I can't say that I have exact steps on how to debug that.

Thanks.

>
> # uname -a
> Linux desk 5.12.2-arch1-1 #1 SMP PREEMPT Fri, 07 May 2021 15:36:06 +0000
> x86_64 GNU/Linux
> # btrfs --version
> btrfs-progs v5.11.1
> # btrfs fi show
> Label: none  uuid: 6733acf5-be40-4fe2-9d6f-819d39e49720
>         Total devices 1 FS bytes used 187.11GiB
>         devid    1 size 931.51GiB used 208.03GiB path /dev/sda
> # btrfs fi df /ssdSpace
> Data, single: total=207.00GiB, used=186.67GiB
> System, single: total=32.00MiB, used=48.00KiB
> Metadata, single: total=1.00GiB, used=450.08MiB
> GlobalReserve, single: total=215.41MiB, used=0.00B



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Leaf corruption due to csum range

[Filipe Manana]

On Tue, May 11, 2021 at 12:35 PM Philipp Fent <fent@in.tum.de> wrote:
>
> On 11.05.21 10:18, Wang Yugui wrote:
> > Is this a server with ECC memory?
>
> My machine does not have ECC memory. So I guess a random bitflip is a
> possibility, but I would rule this out. Memtester didn't report any
> errors in an extended test, and I run quite a lot of write heavy
> databases like sql server. This sql server workload is the only
> application where I ever got csum errors. And I really get them *every
> time* I run it. I think is more likely an issue in sql server's write
> pattern.
>
>
> On 11.05.21 10:56, Filipe Manana wrote:
> > Most likely it's a race when adding checksums. In this case for the
> > log tree (fsync).
> > Try to see if there are reflink operations (clone and dedupe) done by
> > sql server (or maybe docker), in case there aren't, that excludes
> > shared extents being the cause of the problem.
>
> I now also ran the sql server docker container under sysdig, which gave
> me the following breakdown of executed system calls of the container:
> CALLS/S  TOT TIME  AVG TIME SYSCALL
>   13.05   702.41s     420ms futex
>   12.11     193us     124ns gettid
>    7.55    1.05ms    1.09us mprotect
>    6.55      70ms      84us nanosleep
>    5.70     128us     175ns getcpu
>    5.44    1.04ms    1.49us read
>    5.41    1.18ms    1.71us mmap
>    5.25    1.99ms    2.96us openat
>    4.44     122us     215ns close
>    4.23     308us     568ns fstat
>    1.50     298us    1.55us access
>    1.11      21us     153ns getpid
>    1.08      38us     279ns rt_sigprocmask
>    0.94      41us     343ns lseek
>    0.91     542us    4.67us munmap
>    0.77      25us     260ns sigaltstack
>    0.77      22us     226ns set_robust_list
>    0.70     616us    6.92us clone
>    0.55     653us    9.33us sendto
>    0.53    88.96s     1.30s recvfrom
>    0.44     867us      15us madvise
>    0.44      11us     198ns rt_sigaction
>
> Filipe's suspicion of a race condition might be a good lead. There are
> several clone operations in the trace and the majority of runtime seems

Not familiar with sysdig myself, but those clone calls are likely to
be the system call for creating process [1], and not the ioctl for
cloning [2] or dedupe [3].

strace would be clear to me, which I'm more familiar with (or even
better, bpftrace).

Also, seeing that mmap is in the log, I just remembered that 5.13-rc1
includes a fix for races between mmap writes and fsync that could fix
that:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=885f46d87f29a94eafe3cc707d5c4dea2be248f3

The changelog identifies logging file extent items with overlapping
ranges, as the result of the race, but thinking out loud now, I think
it could also result in logging checksum items with overlapping ranges
because of that.
If you want to test it, either try 5.13-rc1 or pick that patch and all
its dependencies into 5.12.x (the dependencies are listed at the end
of the changelog, all of them landed in 5.13-rc1).

I'll see if later today or tomorrow I can get the reproducer to run here.

[1] https://man7.org/linux/man-pages/man2/clone.2.html
[2] https://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html
[3] https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html

> to be spent contending locks, so I assume there are multiple threads at
> work.
> I've attached the full sysdig log (~1MB), not sure if this helps.
>
> Is there a way to increase the btrfs log output, so I could try to
> observe which leafs are written? CONFIG_BTRFS_DEBUG looks like something
> I could enable. Is this the right approach to narrow down a race?

There's no way to log which leaves are written and dump their contents
on the fly.
That would produce tremendous amount of text to log.

Most likely this is a race condition that the sql server workload is
good for triggering (at least on your box).

Thanks

>
> Thanks.



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Leaf corruption due to csum range

[Filipe Manana]

On Tue, May 11, 2021 at 7:19 PM Philipp Fent <fent@in.tum.de> wrote:
>
> Thanks for the explanation! I wasn't aware of these ioctls.
>
> > strace would be clear to me, which I'm more familiar with (or even
> better, bpftrace).
>
> I've attached an strace output that decompresses to about 200MB
> logfiles. I can't make heads or tails of it, but I hope it helps.
> I have never used bpftrace, do you have any pointers where I could start?

There's some documentation and examples on their github.

>
> > I just remembered that 5.13-rc1 includes a fix for races between mmap
> writes and fsync that could fix that
>
> I tried 5.13-rc1, but I'm running into the same csum range issue:
>
>
>
> Linux version 5.13.0-rc1-1-mainline (linux-mainline@archlinux) (gcc
> (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Tue, 11 May
> 2021 15:34:19 +0000
> ...
> BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> block=507430633472 slot=5, csum end range (293918547968) goes beyond the
> start range (293918416896) of the next csum item
> BTRFS info (device sda): leaf 507430633472 gen 18451 total ptrs 11 free
> space 5016 owner 18446744073709551610
>         item 0 key (18446744073709551606 128 293837238272) itemoff 15923
> itemsize 360
>         item 1 key (18446744073709551606 128 293838544896) itemoff 15863
> itemsize 60
>         item 2 key (18446744073709551606 128 293838675968) itemoff 15563
> itemsize 300
>         item 3 key (18446744073709551606 128 293839527936) itemoff 15503
> itemsize 60
>         item 4 key (18446744073709551606 128 293872295936) itemoff 15263
> itemsize 240
>         item 5 key (18446744073709551606 128 293913763840) itemoff 10591
> itemsize 4672
>         item 6 key (18446744073709551606 128 293918416896) itemoff 8351
> itemsize 2240
>         item 7 key (18446744073709551606 128 293947658240) itemoff 8347
> itemsize 4
>         item 8 key (18446744073709551606 128 293965193216) itemoff 8287
> itemsize 60
>         item 9 key (18446744073709551606 128 293965848576) itemoff 8227
> itemsize 60
>         item 10 key (18446744073709551606 128 293966176256) itemoff 5291
> itemsize 2936
> BTRFS error (device sda): block=507430633472 write time tree block
> corruption detected
> BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> block=507447197696 slot=0, csum end range (320352133120) goes beyond the
> start range (320352116736) of the next csum item
> BTRFS info (device sda): leaf 507447197696 gen 18451 total ptrs 3 free
> space 116 owner 18446744073709551610
>         item 0 key (18446744073709551606 128 320336326656) itemoff 847
> itemsize 15436
>         item 1 key (18446744073709551606 128 320352116736) itemoff 831
> itemsize 16
>         item 2 key (18446744073709551606 128 320352247808) itemoff 191
> itemsize 640
> BTRFS error (device sda): block=507447197696 write time tree block
> corruption detected
> BTRFS: error (device sda) in btrfs_sync_log:3136: errno=-5 IO failure
> BTRFS info (device sda): forced readonly

Ok, then it's something else.

When I run you reproducer I get an error:

$ ./runMssql.sh
Starting MSSQL docker container...
9943b714ed210a2937d5fce27ec110981b471e6e9f2c619629cb66501621ebb5
Loading TPC-H schema...
Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed
for user 'sa'..

dbgen.sh ran successfully before.

Any idea?

>
>
>
> Curiously, the second leaf range overshot by only 16KB....
> Let me know, if I can try anything else.



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Leaf corruption due to csum range

[Filipe Manana]

On Thu, May 13, 2021 at 10:57 AM Filipe Manana <fdmanana@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 7:19 PM Philipp Fent <fent@in.tum.de> wrote:
> >
> > Thanks for the explanation! I wasn't aware of these ioctls.
> >
> > > strace would be clear to me, which I'm more familiar with (or even
> > better, bpftrace).
> >
> > I've attached an strace output that decompresses to about 200MB
> > logfiles. I can't make heads or tails of it, but I hope it helps.
> > I have never used bpftrace, do you have any pointers where I could start?
>
> There's some documentation and examples on their github.
>
> >
> > > I just remembered that 5.13-rc1 includes a fix for races between mmap
> > writes and fsync that could fix that
> >
> > I tried 5.13-rc1, but I'm running into the same csum range issue:
> >
> >
> >
> > Linux version 5.13.0-rc1-1-mainline (linux-mainline@archlinux) (gcc
> > (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Tue, 11 May
> > 2021 15:34:19 +0000
> > ...
> > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> > block=507430633472 slot=5, csum end range (293918547968) goes beyond the
> > start range (293918416896) of the next csum item
> > BTRFS info (device sda): leaf 507430633472 gen 18451 total ptrs 11 free
> > space 5016 owner 18446744073709551610
> >         item 0 key (18446744073709551606 128 293837238272) itemoff 15923
> > itemsize 360
> >         item 1 key (18446744073709551606 128 293838544896) itemoff 15863
> > itemsize 60
> >         item 2 key (18446744073709551606 128 293838675968) itemoff 15563
> > itemsize 300
> >         item 3 key (18446744073709551606 128 293839527936) itemoff 15503
> > itemsize 60
> >         item 4 key (18446744073709551606 128 293872295936) itemoff 15263
> > itemsize 240
> >         item 5 key (18446744073709551606 128 293913763840) itemoff 10591
> > itemsize 4672
> >         item 6 key (18446744073709551606 128 293918416896) itemoff 8351
> > itemsize 2240
> >         item 7 key (18446744073709551606 128 293947658240) itemoff 8347
> > itemsize 4
> >         item 8 key (18446744073709551606 128 293965193216) itemoff 8287
> > itemsize 60
> >         item 9 key (18446744073709551606 128 293965848576) itemoff 8227
> > itemsize 60
> >         item 10 key (18446744073709551606 128 293966176256) itemoff 5291
> > itemsize 2936
> > BTRFS error (device sda): block=507430633472 write time tree block
> > corruption detected
> > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> > block=507447197696 slot=0, csum end range (320352133120) goes beyond the
> > start range (320352116736) of the next csum item
> > BTRFS info (device sda): leaf 507447197696 gen 18451 total ptrs 3 free
> > space 116 owner 18446744073709551610
> >         item 0 key (18446744073709551606 128 320336326656) itemoff 847
> > itemsize 15436
> >         item 1 key (18446744073709551606 128 320352116736) itemoff 831
> > itemsize 16
> >         item 2 key (18446744073709551606 128 320352247808) itemoff 191
> > itemsize 640
> > BTRFS error (device sda): block=507447197696 write time tree block
> > corruption detected
> > BTRFS: error (device sda) in btrfs_sync_log:3136: errno=-5 IO failure
> > BTRFS info (device sda): forced readonly
>
> Ok, then it's something else.
>
> When I run you reproducer I get an error:
>
> $ ./runMssql.sh
> Starting MSSQL docker container...
> 9943b714ed210a2937d5fce27ec110981b471e6e9f2c619629cb66501621ebb5
> Loading TPC-H schema...
> Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed
> for user 'sa'..

Ok, never mind.
I changed the 'sleep 5' to 'sleep 15' in the script and it works now.
Seems like 5 seconds is too little on this vm for starting the server.

I can also trigger the bug. I'll see what causes the bug.

Thanks for providing the reliable reproducer, it's really helpful.


>
> dbgen.sh ran successfully before.
>
> Any idea?
>
> >
> >
> >
> > Curiously, the second leaf range overshot by only 16KB....
> > Let me know, if I can try anything else.
>
>
>
> --
> Filipe David Manana,
>
> “Whether you think you can, or you think you can't — you're right.”



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: Leaf corruption due to csum range

[Philipp Fent]

On 13.05.21 12:50, Filipe Manana wrote:
> Ok, never mind.
> I changed the 'sleep 5' to 'sleep 15' in the script and it works now.
> Seems like 5 seconds is too little on this vm for starting the server.

Oh, that timing is unfortunate and sleeping is admittedly a not a good
solution. You can try to add '-l 60' to the first sqlcmd to increase the
login timeout, which should eliminate any unnecessary waiting time.

> I can also trigger the bug. I'll see what causes the bug.
>
> Thanks for providing the reliable reproducer, it's really helpful.

I'm glad this worked! Thank you for taking the time.