[dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[kernel test robot]

Hello,

kernel test robot noticed "WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio" on:

commit: b4834f12a4df607aaedc627fa9b93f3b18f664ba ("cifs: Cut over to using netfslib")
https://git.kernel.org/cgit/linux/kernel/git/dhowells/linux-fs.git cifs-netfs

in testcase: filebench
version: filebench-x86_64-22620e6-1_20240224
with following parameters:

	disk: 1HDD
	fs: btrfs
	fs2: cifs
	test: filemicro_seqwriterandvargam.f
	cpufreq_governor: performance



compiler: gcc-13
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202404161031.468b84f-oliver.sang@intel.com


[   51.734544][  T654] ------------[ cut here ]------------
[   51.740181][  T654] R=00000012: folio 11 is not under writeback
[ 51.746415][ T654] WARNING: CPU: 34 PID: 654 at fs/netfs/write_collect.c:105 netfs_writeback_lookup_folio (fs/netfs/write_collect.c:105 (discriminator 1)) 
[   51.757394][  T654] Modules linked in: cmac nls_utf8 cifs cifs_arc4 nls_ucs2_utils cifs_md4 dns_resolver kmem intel_rapl_msr intel_rapl_common skx_edac x86_pkg_temp_thermal btrfs intel_powerclamp coretemp kvm_intel blake2b_generic xor sr_mod kvm cdrom raid6_pq crct10dif_pclmul sd_mod crc32_pclmul libcrc32c crc32c_intel ghash_clmulni_intel sg sha512_ssse3 ipmi_ssif binfmt_misc device_dax nvme nd_pmem rapl nvme_core nd_btt dax_pmem ahci ast t10_pi libahci intel_cstate acpi_ipmi ipmi_si crc64_rocksoft_generic mei_me drm_shmem_helper i2c_i801 crc64_rocksoft ipmi_devintf ioatdma libata drm_kms_helper intel_uncore mei nfit crc64 lpc_ich i2c_smbus intel_pch_thermal dca wmi ipmi_msghandler libnvdimm joydev drm fuse dm_mod loop ip_tables
[   51.775438][ T1342] Events disabled
[   51.821864][  T654] CPU: 34 PID: 654 Comm: kworker/u386:27 Tainted: G S                 6.9.0-rc3-00036-gb4834f12a4df #1
[   51.823577][ T1342]
[   51.827064][  T654] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5C620.86B.0D.01.0286.011120190816 01/11/2019
[   51.827066][  T654] Workqueue: events_unbound netfs_write_collection_worker
[   51.838243][ T1342] Events disabled
[   51.840127][  T654]
[ 51.840128][ T654] RIP: 0010:netfs_writeback_lookup_folio (fs/netfs/write_collect.c:105 (discriminator 1)) 
[   51.851254][ T1342]
[ 51.858206][ T654] Code: 48 89 de e8 c9 ad ff ff e9 62 ff ff ff 48 8b 53 20 8b b5 ac 01 00 00 48 c7 c7 d0 e0 97 82 c6 05 12 0b c0 01 01 e8 a7 f8 b9 ff <0f> 0b eb a8 e8 1e 99 ae 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
All code
========
   0:	48 89 de             	mov    %rbx,%rsi
   3:	e8 c9 ad ff ff       	callq  0xffffffffffffadd1
   8:	e9 62 ff ff ff       	jmpq   0xffffffffffffff6f
   d:	48 8b 53 20          	mov    0x20(%rbx),%rdx
  11:	8b b5 ac 01 00 00    	mov    0x1ac(%rbp),%esi
  17:	48 c7 c7 d0 e0 97 82 	mov    $0xffffffff8297e0d0,%rdi
  1e:	c6 05 12 0b c0 01 01 	movb   $0x1,0x1c00b12(%rip)        # 0x1c00b37
  25:	e8 a7 f8 b9 ff       	callq  0xffffffffffb9f8d1
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb a8                	jmp    0xffffffffffffffd6
  2e:	e8 1e 99 ae 00       	callq  0xae9951
  33:	66 66 2e 0f 1f 84 00 	data16 nopw %cs:0x0(%rax,%rax,1)
  3a:	00 00 00 00 
  3e:	0f                   	.byte 0xf
  3f:	1f                   	(bad)  

Code starting with the faulting instruction
===========================================
   0:	0f 0b                	ud2    
   2:	eb a8                	jmp    0xffffffffffffffac
   4:	e8 1e 99 ae 00       	callq  0xae9927
   9:	66 66 2e 0f 1f 84 00 	data16 nopw %cs:0x0(%rax,%rax,1)
  10:	00 00 00 00 
  14:	0f                   	.byte 0xf
  15:	1f                   	(bad)  
[   51.858208][  T654] RSP: 0018:ffffc90007f57d80 EFLAGS: 00010286
[   51.858209][  T654] RAX: 0000000000000000 RBX: ffffea00081f3340 RCX: 0000000000000000
[   51.909493][  T654] RDX: ffff88df016ae2c0 RSI: ffff88df016a0b00 RDI: ffff88df016a0b00
[   51.917619][  T654] RBP: ffff88816bbc8280 R08: 0000000000000000 R09: 0000000000000003
[   51.925740][  T654] R10: ffffc90007f57c18 R11: ffffffff82fd56a8 R12: 0000000000000011
[   51.933861][  T654] R13: ffff88cf8421af00 R14: ffff88816bbc8280 R15: ffff88816bbc83e0
[   51.941993][  T654] FS:  0000000000000000(0000) GS:ffff88df01680000(0000) knlGS:0000000000000000
[   51.951072][  T654] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   51.957817][  T654] CR2: 00007ffff7e0ed90 CR3: 0000005f7de1c002 CR4: 00000000007706f0
[   51.965946][  T654] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   51.974071][  T654] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   51.982190][  T654] PKRU: 55555554
[   51.985880][  T654] Call Trace:
[   51.989369][  T654]  <TASK>
[ 51.992447][ T654] ? __warn (kernel/panic.c:694) 
[ 51.996657][ T654] ? netfs_writeback_lookup_folio (fs/netfs/write_collect.c:105 (discriminator 1)) 
[ 52.002864][ T654] ? report_bug (lib/bug.c:180 lib/bug.c:219) 
[ 52.007512][ T654] ? handle_bug (arch/x86/kernel/traps.c:239) 
[ 52.011982][ T654] ? exc_invalid_op (arch/x86/kernel/traps.c:260 (discriminator 1)) 
[ 52.016793][ T654] ? asm_exc_invalid_op (arch/x86/include/asm/idtentry.h:621) 
[ 52.021948][ T654] ? netfs_writeback_lookup_folio (fs/netfs/write_collect.c:105 (discriminator 1)) 
[ 52.028140][ T654] netfs_collect_write_results (fs/netfs/write_collect.c:128 (discriminator 1) fs/netfs/write_collect.c:547 (discriminator 1)) 
[ 52.034062][ T654] netfs_write_collection_worker (arch/x86/include/asm/bitops.h:206 arch/x86/include/asm/bitops.h:238 include/asm-generic/bitops/instrumented-non-atomic.h:142 fs/netfs/write_collect.c:637) 
[ 52.040078][ T654] process_one_work (kernel/workqueue.c:3254) 
[ 52.045046][ T654] worker_thread (kernel/workqueue.c:3329 (discriminator 2) kernel/workqueue.c:3416 (discriminator 2)) 
[ 52.049749][ T654] ? __pfx_worker_thread (kernel/workqueue.c:3362) 
[ 52.054965][ T654] kthread (kernel/kthread.c:388) 
[ 52.058967][ T654] ? __pfx_kthread (kernel/kthread.c:341) 
[ 52.063652][ T654] ret_from_fork (arch/x86/kernel/process.c:147) 
[ 52.068171][ T654] ? __pfx_kthread (kernel/kthread.c:341) 
[ 52.072850][ T654] ret_from_fork_asm (arch/x86/entry/entry_64.S:256) 
[   52.077711][  T654]  </TASK>
[   52.080821][  T654] ---[ end trace 0000000000000000 ]---



The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240416/202404161031.468b84f-oliver.sang@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

When I try and do "sudo bin/lkp install job.yaml", I get an error because
Fedora 39 doesn't have a libarchive-tools package.

Also, I can't find a "filemicro_seqwriterandvargam.f" script, presumably
because it would get patched into existence by the failing install step above.

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[Oliver Sang]

hi, David,

On Wed, Apr 17, 2024 at 01:18:41PM +0100, David Howells wrote:
> When I try and do "sudo bin/lkp install job.yaml", I get an error because
> Fedora 39 doesn't have a libarchive-tools package.

sorry for late. we made some fix for fedora 39 recently, could you try again?
I can pass "sudo bin/lkp install job.yaml" on my local machine with fedora 39
now.

we found the similar issue
(WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio)
still exist upon below commit in latest branch: dhowells-fs/cifs-netfs

0e533e79c0edb cifs: Cut over to using netfslib

> 
> Also, I can't find a "filemicro_seqwriterandvargam.f" script, presumably
> because it would get patched into existence by the failing install step above.

yes, this is only generated after all previous steps pass.

$ bin/lkp split-job --compatible job.yaml
...
/tmp/split-job20240424-80542-6az47a => ./job-1HDD-btrfs-cifs-filemicro_seqwriterandvargam.f.yaml


BTW, you also need follow
https://github.com/intel/lkp-tests?tab=readme-ov-file#run-your-own-disk-partitions
to use your own disk since this test needs 1 HDD.

if you find further issues, could you capture full logs? thanks a lot!

> 
> David
> 

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

Oliver Sang <oliver.sang@intel.com> wrote:

> I can pass "sudo bin/lkp install job.yaml" on my local machine with fedora 39
> now.

Note that this causes:

systemd-sysv-generator[23561]: SysV service '/etc/rc.d/init.d/network' lacks a native systemd unit file. ♻️ Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it safe, robust and future-proof. ⚠️ This compatibility logic is deprecated, expect removal soon. ⚠️

to appear.  What's it doing to the networking settings?  It shouldn't be
touching those.

Also, does it have to install its own cifs server?  Can it not be directed to
my test server that's already set up on another machine?  And does it have to
build a kernel?  Can it not use the one that's already running on the machine?

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

The "lkp install" didn't complete:

==> Retrieving sources...
  -> Source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  -> Cloning linux git repo...
Cloning into bare repository '/root/lkp-tests/programs/turbostat/pkg/linux'...
remote: Enumerating objects: 10112942, done.
remote: Counting objects: 100% (889/889), done.
remote: Compressing objects: 100% (475/475), done.
remote: Total 10112942 (delta 554), reused 549 (delta 412), pack-reused 10112053
Receiving objects: 100% (10112942/10112942), 2.78 GiB | 4.16 MiB/s, done.
Resolving deltas: 100% (8300839/8300839), done.
==> WARNING: Skipping verification of source file PGP signatures.
==> Validating source files with md5sums...
    linux ... Skipped
==> Extracting sources...
  -> Source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  -> Creating working copy of linux git repo...
fatal: '/root/lkp-tests/pkg/turbostat/linux' does not appear to be a git repository
fatal: Could not read from remote repository.


I looked around under /root/lkp-tests and there's no pkg/ directory.  It seems
to be using tmp-pkg instead.

Is there a way to skip the cloning of the kernel?  I already have my test
kernel running on my test machine, booted by PXE/tftp from the build tree on
my desktop.  Just tell me what options I need to enable.

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

David Howells <dhowells@redhat.com> wrote:

> ==> Retrieving sources...
>   -> Source is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>   -> Cloning linux git repo...
> Cloning into bare repository '/root/lkp-tests/programs/turbostat/pkg/linux'...

Actually, it cloned the linux git repo twice by http, once into:

	programs/turbostat/pkg/linux/

which is a bare repo, and once into:

	tmp-pkg/turbostat/src/linux/

which has all the files checked out.

If it must clone linux, can it at least clone one from the other?

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

Okay I got it to install somehow by moving tmp-pkg to pkg, running lkp
install (which failed), then moving it back and running lkp install again,
which succeeded.

Running lkp split-job gives me:

/root/lkp-tests/lib/erb.rb:35: warning: Passing safe_level with the 2nd argument of ERB.new is deprecated. Do not use it, and specify other arguments as keyword arguments.
/root/lkp-tests/lib/erb.rb:35: warning: Passing trim_mode with the 3rd argument of ERB.new is deprecated. Use keyword argument like ERB.new(str, trim_mode: ...) instead.
/root/lkp-tests/lib/erb.rb:35: warning: Passing safe_level with the 2nd argument of ERB.new is deprecated. Do not use it, and specify other arguments as keyword arguments.
/root/lkp-tests/lib/erb.rb:35: warning: Passing trim_mode with the 3rd argument of ERB.new is deprecated. Use keyword argument like ERB.new(str, trim_mode: ...) instead.
job.yaml => ./job-performance-1HDD-btrfs-cifs-filemicro_seqwriterandvargam.f-b4834f12a4df607aaedc627fa9b93f3b18f664ba-debian-12-x86_64-20240206.cgz.yaml

It looks like some of the Ruby scripting is out of date.

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[David Howells]

Trying to do "lkp run job.yaml" doesn't work:

/root/lkp-tests/filters/need_kconfig.rb:20:in `read_kernel_kconfigs': .config doesn't exist: /pkg/linux/x86_64-rhel-8.3/gcc-13/b4834f12a4df607aaedc627fa9b93f3b18f664ba/.config (Job::ParamError)
        from /root/lkp-tests/filters/need_kconfig.rb:176:in `block in expand_expression'
        from /root/lkp-tests/lib/erb.rb:51:in `eval'
        from /root/lkp-tests/lib/erb.rb:51:in `expand_expression'
        from /root/lkp-tests/lib/job.rb:646:in `evaluate_param'
        from /root/lkp-tests/lib/job.rb:694:in `block in expand_params'
        from /root/lkp-tests/lib/job.rb:79:in `block in for_each_in'
        from /root/lkp-tests/lib/job.rb:78:in `each'
        from /root/lkp-tests/lib/job.rb:78:in `for_each_in'
        from /root/lkp-tests/lib/job.rb:691:in `expand_params'
        from /root/lkp-tests/bin/run-local:138:in `<main>'

I tried to run the filebench directly, but that only wants to hammer on
/tmp/bigfileset/ and also wants a file for SHM precreating in /tmp.  I was
able to get it to work with cifs by:

touch /tmp/filebench-shm-IF6uX8
truncate -s 184975240 /tmp/filebench-shm-IF6uX8
mkdir /tmp/bigfileset
mount //myserver/test /tmp/bigfileset/ -o user=shares,pass=...,cache=loose

/root/lkp-tests/programs/filebench/pkg/filebench-lkp/lkp/benchmarks/filebench/bin/filebench -f /lkp/benchmarks/filebench/share/filebench/workloads/filemicro_seqwriterandvargam.f

It tries to remove /tmp/bigfileset/, can't because it's mounted, and then
continues anyway.

It should be easier than this ;-)

David

Re: [dhowells-fs:cifs-netfs] [cifs] b4834f12a4: WARNING:at_fs/netfs/write_collect.c:#netfs_writeback_lookup_folio

[Philip Li]

On Fri, Apr 26, 2024 at 10:42:41AM +0100, David Howells wrote:
> Trying to do "lkp run job.yaml" doesn't work:
> 
> /root/lkp-tests/filters/need_kconfig.rb:20:in `read_kernel_kconfigs': .config doesn't exist: /pkg/linux/x86_64-rhel-8.3/gcc-13/b4834f12a4df607aaedc627fa9b93f3b18f664ba/.config (Job::ParamError)
>         from /root/lkp-tests/filters/need_kconfig.rb:176:in `block in expand_expression'
>         from /root/lkp-tests/lib/erb.rb:51:in `eval'
>         from /root/lkp-tests/lib/erb.rb:51:in `expand_expression'
>         from /root/lkp-tests/lib/job.rb:646:in `evaluate_param'
>         from /root/lkp-tests/lib/job.rb:694:in `block in expand_params'
>         from /root/lkp-tests/lib/job.rb:79:in `block in for_each_in'
>         from /root/lkp-tests/lib/job.rb:78:in `each'
>         from /root/lkp-tests/lib/job.rb:78:in `for_each_in'
>         from /root/lkp-tests/lib/job.rb:691:in `expand_params'
>         from /root/lkp-tests/bin/run-local:138:in `<main>'
> 
> I tried to run the filebench directly, but that only wants to hammer on
> /tmp/bigfileset/ and also wants a file for SHM precreating in /tmp.  I was
> able to get it to work with cifs by:
> 
> touch /tmp/filebench-shm-IF6uX8
> truncate -s 184975240 /tmp/filebench-shm-IF6uX8
> mkdir /tmp/bigfileset
> mount //myserver/test /tmp/bigfileset/ -o user=shares,pass=...,cache=loose
> 
> /root/lkp-tests/programs/filebench/pkg/filebench-lkp/lkp/benchmarks/filebench/bin/filebench -f /lkp/benchmarks/filebench/share/filebench/workloads/filemicro_seqwriterandvargam.f
> 
> It tries to remove /tmp/bigfileset/, can't because it's mounted, and then
> continues anyway.
> 
> It should be easier than this ;-)

Hi David, really apologize for all the problems you mentioned here and early mails
during the reproduce. We will address them one by one ASAP to see make the reproduce
useful instead of this kind of bad experience.

BTW: if you have any patch (like debugging) to try, we are pleased to test it directly in
our testbox.

Sorry again to cost your a lot extra time for this.

Thanks

> 
> David
> 
>