Re: Prog section rejected: Argument list too long (7)!

[Vincent Li <vincent.mc.li@gmail.com>]

On Sat, 24 Jul 2021, Vincent Li wrote:

> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote:
> >
> >
> > Hi BPF experts,
> >
> > I have a cilium PR https://github.com/cilium/cilium/pull/16916 that
> > failed to pass verifier in kernel 4.19, the error is like:
> >
> > level=warning msg="Prog section '2/7' rejected: Argument list too long
> > (7)!" subsys=datapath-loader
> > level=warning msg=" - Type:         3" subsys=datapath-loader
> > level=warning msg=" - Attach Type:  0" subsys=datapath-loader
> > level=warning msg=" - Instructions: 4578 (482 over limit)"
> > subsys=datapath-loader
> > level=warning msg=" - License:      GPL" subsys=datapath-loader
> > level=warning subsys=datapath-loader
> > level=warning msg="Verifier analysis:" subsys=datapath-loader
> > level=warning subsys=datapath-loader
> > level=warning msg="Error filling program arrays!" subsys=datapath-loader
> > level=warning msg="Unable to load program" subsys=datapath-loader
> >
> > then I tried to run the PR locally in my dev machine with custom upstream
> > kernel version, I narrowed the issue down to between upstream kernel
> > version 5.7 and 5.8, in 5.7, it failed with:
> 
> I further narrow it down to between 5.7 and 5.8-rc1 release, but still
> no clue which commits in 5.8-rc1 resolved the issue
> 
> >
> > level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn
> > 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader
> > level=warning subsys=datapath-loader
> > level=warning msg="Log buffer too small to dump verifier log 16777215
> > bytes (9 tries)!" subsys=datapath-loader
> > level=warning msg="Error filling program arrays!" subsys=datapath-loader
> > level=warning msg="Unable to load program" subsys=datapath-loader
> >
> > 5.8 works fine.
> >
> > What difference between 5.7 and 5.8 to cause this verifier problem, I
> > tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits
> > that would make the difference with my limited BPF knowledge. Any clue
> > would be appreciated!

I have git bisected to this commit:

# first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make 
verifier log more relevant by default

this commit looks only dealing with log, accidently fixed the PR issue I 
have? my PR use __bpf_memcpy_builtin() to rewrite the tunnel inner packet 
destination MAC address, somehow related?

[root@centos-dev bpf-next]# git bisect log
git bisect start '--term-new=fixed' '--term-old=unfixed'
# fixed: [b3a9e3b9622ae10064826dccb4f7a52bd88c7407] Linux 5.8-rc1
git bisect fixed b3a9e3b9622ae10064826dccb4f7a52bd88c7407
# unfixed: [3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162] Linux 5.7
git bisect unfixed 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162
# fixed: [ee01c4d72adffb7d424535adf630f2955748fa8b] Merge branch 'akpm' 
(patches from Andrew)
git bisect fixed ee01c4d72adffb7d424535adf630f2955748fa8b
# unfixed: [16d91548d1057691979de4686693f0ff92f46000] Merge tag 
'xfs-5.8-merge-8' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
git bisect unfixed 16d91548d1057691979de4686693f0ff92f46000
# fixed: [098205f3c688885394ed1f670a6a7cb4a58728a3] Merge branch '1GbE' of 
git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue
git bisect fixed 098205f3c688885394ed1f670a6a7cb4a58728a3
# fixed: [da1a782a7140fab22f2dfe8453d7b73c786d73de] net: ipa: kill 
ipa_cmd_dma_task_32b_addr_add()
git bisect fixed da1a782a7140fab22f2dfe8453d7b73c786d73de
# unfixed: [aa8a6ee3e3fc4001e952de37660fe71826da8189] docs: networking: 
convert team.txt to ReST
git bisect unfixed aa8a6ee3e3fc4001e952de37660fe71826da8189
# unfixed: [5b95dea31636ce93660930d16172fe75589b2e70] Merge branch 
'net-smc-extent-buffer-mapping-and-port-handling'
git bisect unfixed 5b95dea31636ce93660930d16172fe75589b2e70
# fixed: [3316d50905f0e551d4786767d827589960a8cb83] bnxt_en: Split HW ring 
statistics strings into RX and TX parts.
git bisect fixed 3316d50905f0e551d4786767d827589960a8cb83
# fixed: [c321022244708aec4675de4f032ef1ba9ff0c640] selftests/bpf: Test 
allowed maps for bpf_sk_select_reuseport
git bisect fixed c321022244708aec4675de4f032ef1ba9ff0c640
# fixed: [50325b1761e31ad17d252e795af72a9af8c5a7d7] bpftool: Expose 
attach_type-to-string array to non-cgroup code
git bisect fixed 50325b1761e31ad17d252e795af72a9af8c5a7d7
# fixed: [8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af] bpf, cgroup: Remove 
unused exports
git bisect fixed 8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af
# unfixed: [6f3f65d80dac8f2bafce2213005821fccdce194c] net: bpf: Allow TC 
programs to call BPF_FUNC_skb_change_head
git bisect unfixed 6f3f65d80dac8f2bafce2213005821fccdce194c
# fixed: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make verifier log 
more relevant by default
git bisect fixed 6f8a57ccf8511724e6f48d732cb2940889789ab2
# unfixed: [0a05861f80fe7d4dcfdabcc98d9854947573e072] xsk: Fix typo in 
xsk_umem_consume_tx and xsk_generic_xmit comments
git bisect unfixed 0a05861f80fe7d4dcfdabcc98d9854947573e072
# unfixed: [71d19214776e61b33da48f7c1b46e522c7f78221] bpf: add 
bpf_ktime_get_boot_ns()
git bisect unfixed 71d19214776e61b33da48f7c1b46e522c7f78221
# first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make 
verifier log more relevant by default


> >
> > Thanks
> >
> > Vincent
> >
>