Re: [PATCH v7 bpf-next 0/9] introduce support for XDP programs in CPUMAP

From: Jakub Sitnicki <jakub@cloudflare.com>
To: Lorenzo Bianconi <lorenzo@kernel.org>
Cc: netdev@vger.kernel.org, davem@davemloft.net, ast@kernel.org,
	brouer@redhat.com, daniel@iogearbox.net, toke@redhat.com,
	lorenzo.bianconi@redhat.com, dsahern@kernel.org,
	andrii.nakryiko@gmail.com, bpf@vger.kernel.org
Subject: Re: [PATCH v7 bpf-next 0/9] introduce support for XDP programs in CPUMAP
Date: Fri, 17 Jul 2020 17:13:33 +0200	[thread overview]
Message-ID: <20200717171333.3fe979e6@toad> (raw)
In-Reply-To: <20200717110136.GA1683270@localhost.localdomain>

On Fri, 17 Jul 2020 13:01:36 +0200
Lorenzo Bianconi <lorenzo@kernel.org> wrote:

> [...]
> 
> > This started showing up with when running ./test_progs from recent
> > bpf-next (bfdfa51702de). Any chance it is related?
> > 
> > [ 2950.440613] =============================================
> > 
> > [ 3073.281578] INFO: task cpumap/0/map:26:536 blocked for more than 860 seconds.
> > [ 3073.285492]       Tainted: G        W         5.8.0-rc4-01471-g15d51f3a516b #814
> > [ 3073.289177] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > [ 3073.293021] cpumap/0/map:26 D    0   536      2 0x00004000
> > [ 3073.295755] Call Trace:
> > [ 3073.297143]  __schedule+0x5ad/0xf10
> > [ 3073.299032]  ? pci_mmcfg_check_reserved+0xd0/0xd0
> > [ 3073.301416]  ? static_obj+0x31/0x80
> > [ 3073.303277]  ? mark_held_locks+0x24/0x90
> > [ 3073.305313]  ? cpu_map_update_elem+0x6d0/0x6d0
> > [ 3073.307544]  schedule+0x6f/0x160
> > [ 3073.309282]  schedule_preempt_disabled+0x14/0x20
> > [ 3073.311593]  kthread+0x175/0x240
> > [ 3073.313299]  ? kthread_create_on_node+0xd0/0xd0
> > [ 3073.315106]  ret_from_fork+0x1f/0x30
> > [ 3073.316365]
> >                Showing all locks held in the system:
> > [ 3073.318423] 1 lock held by khungtaskd/33:
> > [ 3073.319642]  #0: ffffffff82d246a0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x28/0x1c3
> > 
> > [ 3073.322249] =============================================  
> 
> Hi Jakub,
> 
> can you please provide more info? can you please identify the test that trigger
> the issue? I run test_progs with bpf-next master branch and it works fine for me.
> I run the tests in a vm with 4 vcpus and 4G of memory.
> 
> Regards,
> Lorenzo
> 

Was able to trigger it running the newly added selftest:

virtme-init: console is ttyS0
bash-5.0# ./test_progs -n 100
#100/1 cpumap_with_progs:OK
#100 xdp_cpumap_attach:OK
Summary: 1/1 PASSED, 0 SKIPPED, 0 FAILED
bash-5.0# [  247.177168] INFO: task cpumap/0/map:3:198 blocked for more than 122 seconds.
[  247.181306]       Not tainted 5.8.0-rc4-01456-gbfdfa51702de #815
[  247.184487] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  247.188876] cpumap/0/map:3  D    0   198      2 0x00004000
[  247.192624] Call Trace:
[  247.194327]  __schedule+0x5ad/0xf10
[  247.196860]  ? pci_mmcfg_check_reserved+0xd0/0xd0
[  247.199853]  ? static_obj+0x31/0x80
[  247.201917]  ? mark_held_locks+0x24/0x90
[  247.204398]  ? cpu_map_update_elem+0x6d0/0x6d0
[  247.207098]  schedule+0x6f/0x160
[  247.209079]  schedule_preempt_disabled+0x14/0x20
[  247.211863]  kthread+0x175/0x240
[  247.213698]  ? kthread_create_on_node+0xd0/0xd0
[  247.216054]  ret_from_fork+0x1f/0x30
[  247.218363]
[  247.218363] Showing all locks held in the system:
[  247.222150] 1 lock held by khungtaskd/33:
[  247.224894]  #0: ffffffff82d246a0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x28/0x1c3
[  247.231113]
[  247.232335] =============================================
[  247.232335]

qemu running with 4 vCPUs, 4 GB of memory. .config uploaded at
https://paste.centos.org/view/0c14663d

HTH,
-jkbs