Re: Leaf corruption due to csum range

[Filipe Manana <fdmanana@gmail.com>]

On Thu, May 13, 2021 at 10:57 AM Filipe Manana <fdmanana@gmail.com> wrote:
>
> On Tue, May 11, 2021 at 7:19 PM Philipp Fent <fent@in.tum.de> wrote:
> >
> > Thanks for the explanation! I wasn't aware of these ioctls.
> >
> > > strace would be clear to me, which I'm more familiar with (or even
> > better, bpftrace).
> >
> > I've attached an strace output that decompresses to about 200MB
> > logfiles. I can't make heads or tails of it, but I hope it helps.
> > I have never used bpftrace, do you have any pointers where I could start?
>
> There's some documentation and examples on their github.
>
> >
> > > I just remembered that 5.13-rc1 includes a fix for races between mmap
> > writes and fsync that could fix that
> >
> > I tried 5.13-rc1, but I'm running into the same csum range issue:
> >
> >
> >
> > Linux version 5.13.0-rc1-1-mainline (linux-mainline@archlinux) (gcc
> > (GCC) 10.2.0, GNU ld (GNU Binutils) 2.36.1) #1 SMP PREEMPT Tue, 11 May
> > 2021 15:34:19 +0000
> > ...
> > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> > block=507430633472 slot=5, csum end range (293918547968) goes beyond the
> > start range (293918416896) of the next csum item
> > BTRFS info (device sda): leaf 507430633472 gen 18451 total ptrs 11 free
> > space 5016 owner 18446744073709551610
> >         item 0 key (18446744073709551606 128 293837238272) itemoff 15923
> > itemsize 360
> >         item 1 key (18446744073709551606 128 293838544896) itemoff 15863
> > itemsize 60
> >         item 2 key (18446744073709551606 128 293838675968) itemoff 15563
> > itemsize 300
> >         item 3 key (18446744073709551606 128 293839527936) itemoff 15503
> > itemsize 60
> >         item 4 key (18446744073709551606 128 293872295936) itemoff 15263
> > itemsize 240
> >         item 5 key (18446744073709551606 128 293913763840) itemoff 10591
> > itemsize 4672
> >         item 6 key (18446744073709551606 128 293918416896) itemoff 8351
> > itemsize 2240
> >         item 7 key (18446744073709551606 128 293947658240) itemoff 8347
> > itemsize 4
> >         item 8 key (18446744073709551606 128 293965193216) itemoff 8287
> > itemsize 60
> >         item 9 key (18446744073709551606 128 293965848576) itemoff 8227
> > itemsize 60
> >         item 10 key (18446744073709551606 128 293966176256) itemoff 5291
> > itemsize 2936
> > BTRFS error (device sda): block=507430633472 write time tree block
> > corruption detected
> > BTRFS critical (device sda): corrupt leaf: root=18446744073709551610
> > block=507447197696 slot=0, csum end range (320352133120) goes beyond the
> > start range (320352116736) of the next csum item
> > BTRFS info (device sda): leaf 507447197696 gen 18451 total ptrs 3 free
> > space 116 owner 18446744073709551610
> >         item 0 key (18446744073709551606 128 320336326656) itemoff 847
> > itemsize 15436
> >         item 1 key (18446744073709551606 128 320352116736) itemoff 831
> > itemsize 16
> >         item 2 key (18446744073709551606 128 320352247808) itemoff 191
> > itemsize 640
> > BTRFS error (device sda): block=507447197696 write time tree block
> > corruption detected
> > BTRFS: error (device sda) in btrfs_sync_log:3136: errno=-5 IO failure
> > BTRFS info (device sda): forced readonly
>
> Ok, then it's something else.
>
> When I run you reproducer I get an error:
>
> $ ./runMssql.sh
> Starting MSSQL docker container...
> 9943b714ed210a2937d5fce27ec110981b471e6e9f2c619629cb66501621ebb5
> Loading TPC-H schema...
> Sqlcmd: Error: Microsoft ODBC Driver 17 for SQL Server : Login failed
> for user 'sa'..

Ok, never mind.
I changed the 'sleep 5' to 'sleep 15' in the script and it works now.
Seems like 5 seconds is too little on this vm for starting the server.

I can also trigger the bug. I'll see what causes the bug.

Thanks for providing the reliable reproducer, it's really helpful.


>
> dbgen.sh ran successfully before.
>
> Any idea?
>
> >
> >
> >
> > Curiously, the second leaf range overshot by only 16KB....
> > Let me know, if I can try anything else.
>
>
>
> --
> Filipe David Manana,
>
> “Whether you think you can, or you think you can't — you're right.”



-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”