regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)

regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)

[Ganapathi Kamath]

Hello,

A) Included below is I think a sure-way to reproduce the bug. (I wasn't sure if you had done reproduction yet, and were looking only at provided kernel traces) 
B) Is there a specific kernel-bug ## I can cite and watch for progress/fix ? 

I ran into the same while doing similar rsync tests related to qemu bug 
https://gitlab.com/qemu-project/qemu/-/issues/727 

1) 
create empty qcow2 file
[root@sirius gana]# qemu-img create -f qcow2  /mnt/a16/gkpics01.qcow2 99723771904

2) 
create another qcow2/vhdx containing sgdata, Ensure that data is good/not be corrupt. (I prepare my vhdx on windows) 
That issue filing mentioned earlier has scriptlets for "synthetic generated data"  (sgdata)
https://gitlab.com/qemu-project/qemu/-/issues/727#note_739930694 

3) download a fedora rawhide iso
https://archive.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/

4)  
[root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20211230.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga-gl" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15"

Inside VM, open a terminal tab
1) use gdisk make type-0700 partition /dev/sda1
2) format /dev/sda1 as  ntfs
mkfs -t ntfs -Q -L fs_gkpics01 /dev/sda1
3) mount /dev/sda and /dev/sdb 
mount /dev/sda1 /mnt/a
mount /dev/sdb2 /mnt/b 
4) ( sdate=`date`; cd /mnt/b ; rsync -avH ./photos001 /mnt/a | tee /tmp/rsynclog.txt ; echo $sdate ; date ) 

monitor dmesg -w in a second tab

Basically, in the above test, I am trying to copy about 85GiB of data from /dev/sdb2 to /dev/sda1

The kernel trace happens within less than a couple of Gb of transfer, after which rsync bails out. 

-Gana

Re: regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)

[Ganapathi Kamath]

I filed the bug in kernel.org bugzilla as
https://bugzilla.kernel.org/show_bug.cgi?id=215460
summary: fs/ntfs3: page_cache_ra_unbounded on rsync from ntfs3 to ext4 

I hope someone is looking into this. 

In the below, email, I had to mention that to produce the trace as mentioned in the other emails
one should load manjaro or arch live-cds
* archlinux-2022.01.01-x86_64.iso (kernel-5.15.12)
* manjaro-gnome-21.2.0-211220-linux515.iso (kernel-5.15.7) 

The bug happens if the src partition is mounted as ntfs3 on vhdx.
So i believe it is triggered from the "read" side filesystem of rsync. 
The dst partition/filesystem can be ext4 and is on a fixed qcow2 or raw disk. 

Bug does not happen if src partition is mounted using ntfs-fuseblk driver.

-Gana

From: Ganapathi Kamath
Sent: Friday, December 31, 2021 11:22 AM
To: Matthew Wilcox <willy@infradead.org>; ntfs3@lists.linux.dev <ntfs3@lists.linux.dev>
Subject: regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux) 
 
Hello,

A) Included below is I think a sure-way to reproduce the bug. (I wasn't sure if you had done reproduction yet, and were looking only at provided kernel traces) 
B) Is there a specific kernel-bug ## I can cite and watch for progress/fix ? 

I ran into the same while doing similar rsync tests related to qemu bug 
https://gitlab.com/qemu-project/qemu/-/issues/727 

1) 
create empty qcow2 file
[root@sirius gana]# qemu-img create -f qcow2  /mnt/a16/gkpics01.qcow2 99723771904

2) 
create another qcow2/vhdx containing sgdata, Ensure that data is good/not be corrupt. (I prepare my vhdx on windows) 
That issue filing mentioned earlier has scriptlets for "synthetic generated data"  (sgdata)
https://gitlab.com/qemu-project/qemu/-/issues/727#note_739930694 

3) download a fedora rawhide iso
https://archive.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/

4)  
[root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20211230.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga-gl" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15"

Inside VM, open a terminal tab
1) use gdisk make type-0700 partition /dev/sda1
2) format /dev/sda1 as  ntfs
mkfs -t ntfs -Q -L fs_gkpics01 /dev/sda1
3) mount /dev/sda and /dev/sdb 
mount /dev/sda1 /mnt/a
mount /dev/sdb2 /mnt/b 
4) ( sdate=`date`; cd /mnt/b ; rsync -avH ./photos001 /mnt/a | tee /tmp/rsynclog.txt ; echo $sdate ; date ) 

monitor dmesg -w in a second tab

Basically, in the above test, I am trying to copy about 85GiB of data from /dev/sdb2 to /dev/sda1

The kernel trace happens within less than a couple of Gb of transfer, after which rsync bails out. 

-Gana

Re: regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)

[Matthew Wilcox]

On Sun, Jan 09, 2022 at 12:06:07PM +0000, Ganapathi Kamath wrote:
> 
> I filed the bug in kernel.org bugzilla as
> https://bugzilla.kernel.org/show_bug.cgi?id=215460
> summary: fs/ntfs3: page_cache_ra_unbounded on rsync from ntfs3 to ext4 
> 
> I hope someone is looking into this. 

You should probably cc the maintainer.  I was just trying to help.

> In the below, email, I had to mention that to produce the trace as mentioned in the other emails
> one should load manjaro or arch live-cds
> * archlinux-2022.01.01-x86_64.iso (kernel-5.15.12)
> * manjaro-gnome-21.2.0-211220-linux515.iso (kernel-5.15.7) 
> 
> The bug happens if the src partition is mounted as ntfs3 on vhdx.
> So i believe it is triggered from the "read" side filesystem of rsync. 
> The dst partition/filesystem can be ext4 and is on a fixed qcow2 or raw disk. 
> 
> Bug does not happen if src partition is mounted using ntfs-fuseblk driver.
> 
> -Gana
> 
> From: Ganapathi Kamath
> Sent: Friday, December 31, 2021 11:22 AM
> To: Matthew Wilcox <willy@infradead.org>; ntfs3@lists.linux.dev <ntfs3@lists.linux.dev>
> Subject: regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux) 
>  
> Hello,
> 
> A) Included below is I think a sure-way to reproduce the bug. (I wasn't sure if you had done reproduction yet, and were looking only at provided kernel traces) 
> B) Is there a specific kernel-bug ## I can cite and watch for progress/fix ? 
> 
> I ran into the same while doing similar rsync tests related to qemu bug 
> https://gitlab.com/qemu-project/qemu/-/issues/727 
> 
> 1) 
> create empty qcow2 file
> [root@sirius gana]# qemu-img create -f qcow2  /mnt/a16/gkpics01.qcow2 99723771904
> 
> 2) 
> create another qcow2/vhdx containing sgdata, Ensure that data is good/not be corrupt. (I prepare my vhdx on windows) 
> That issue filing mentioned earlier has scriptlets for "synthetic generated data"  (sgdata)
> https://gitlab.com/qemu-project/qemu/-/issues/727#note_739930694 
> 
> 3) download a fedora rawhide iso
> https://archive.fedoraproject.org/pub/fedora/linux/development/rawhide/Workstation/x86_64/iso/
> 
> 4)  
> [root@sirius gana]# qemu-system-x86_64 -cpu qemu64 -m 4096 -machine "type=q35" -accel "kvm" -smp "sockets=1,cores=8,threads=1" -boot "d" -cdrom "/vol/15KJ_Images/transcend/Fedora-Workstation-Live-x86_64-Rawhide-20211230.n.0.iso" -hda "/mnt/a16/gkpics01.qcow2" -hdb "/vol/15KJ_Images/test/sgdata.vhdx" -device "virtio-vga-gl" -display "gtk,gl=on" -rtc "base=utc" -net "user" -device "virtio-net,netdev=vmnic" -netdev "user,id=vmnic,net=192.168.20.0/24,dns=192.168.20.3,dhcpstart=192.168.20.15"
> 
> Inside VM, open a terminal tab
> 1) use gdisk make type-0700 partition /dev/sda1
> 2) format /dev/sda1 as  ntfs
> mkfs -t ntfs -Q -L fs_gkpics01 /dev/sda1
> 3) mount /dev/sda and /dev/sdb 
> mount /dev/sda1 /mnt/a
> mount /dev/sdb2 /mnt/b 
> 4) ( sdate=`date`; cd /mnt/b ; rsync -avH ./photos001 /mnt/a | tee /tmp/rsynclog.txt ; echo $sdate ; date ) 
> 
> monitor dmesg -w in a second tab
> 
> Basically, in the above test, I am trying to copy about 85GiB of data from /dev/sdb2 to /dev/sda1
> 
> The kernel trace happens within less than a couple of Gb of transfer, after which rsync bails out. 
> 
> -Gana

Re: regarding: Re: Bug using new ntfs3 file system driver (5.15.2 on Arch Linux)

[Ganapathi Kamath]

Rgd bug: "fs/ntfs3: page_cache_ra_unbounded on rsync from ntfs3 to ext4" https://bugzilla.kernel.org/show_bug.cgi?id=215460 
Above bug, first found in 5.15.x, doesn't surface in 5.17.0-0.rc2 . 
No tracebacks were emitted whether the virtual-disk used was dd-raw-image or qcow2(fixed)
Unsure if the bug will recur.
 
However, on 5.17.0-0.rc2, a different trace arises on using qcow2(dynamic) 
I filed this bug as "during rsync, vm guest kernel trace arising from memcg_kmem_charge_page alloc_pages" https://bugzilla.kernel.org/show_bug.cgi?id=215563
Traceback seems to come from closer to page allocater code

-Gana