* Prog section rejected: Argument list too long (7)! @ 2021-07-24 2:17 Vincent Li 2021-07-24 17:03 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-24 2:17 UTC (permalink / raw) To: bpf Hi BPF experts, I have a cilium PR https://github.com/cilium/cilium/pull/16916 that failed to pass verifier in kernel 4.19, the error is like: level=warning msg="Prog section '2/7' rejected: Argument list too long (7)!" subsys=datapath-loader level=warning msg=" - Type: 3" subsys=datapath-loader level=warning msg=" - Attach Type: 0" subsys=datapath-loader level=warning msg=" - Instructions: 4578 (482 over limit)" subsys=datapath-loader level=warning msg=" - License: GPL" subsys=datapath-loader level=warning subsys=datapath-loader level=warning msg="Verifier analysis:" subsys=datapath-loader level=warning subsys=datapath-loader level=warning msg="Error filling program arrays!" subsys=datapath-loader level=warning msg="Unable to load program" subsys=datapath-loader then I tried to run the PR locally in my dev machine with custom upstream kernel version, I narrowed the issue down to between upstream kernel version 5.7 and 5.8, in 5.7, it failed with: level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader level=warning subsys=datapath-loader level=warning msg="Log buffer too small to dump verifier log 16777215 bytes (9 tries)!" subsys=datapath-loader level=warning msg="Error filling program arrays!" subsys=datapath-loader level=warning msg="Unable to load program" subsys=datapath-loader 5.8 works fine. What difference between 5.7 and 5.8 to cause this verifier problem, I tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits that would make the difference with my limited BPF knowledge. Any clue would be appreciated! Thanks Vincent ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-24 2:17 Prog section rejected: Argument list too long (7)! Vincent Li @ 2021-07-24 17:03 ` Vincent Li 2021-07-25 6:16 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-24 17:03 UTC (permalink / raw) To: bpf On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > Hi BPF experts, > > I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > failed to pass verifier in kernel 4.19, the error is like: > > level=warning msg="Prog section '2/7' rejected: Argument list too long > (7)!" subsys=datapath-loader > level=warning msg=" - Type: 3" subsys=datapath-loader > level=warning msg=" - Attach Type: 0" subsys=datapath-loader > level=warning msg=" - Instructions: 4578 (482 over limit)" > subsys=datapath-loader > level=warning msg=" - License: GPL" subsys=datapath-loader > level=warning subsys=datapath-loader > level=warning msg="Verifier analysis:" subsys=datapath-loader > level=warning subsys=datapath-loader > level=warning msg="Error filling program arrays!" subsys=datapath-loader > level=warning msg="Unable to load program" subsys=datapath-loader > > then I tried to run the PR locally in my dev machine with custom upstream > kernel version, I narrowed the issue down to between upstream kernel > version 5.7 and 5.8, in 5.7, it failed with: I further narrow it down to between 5.7 and 5.8-rc1 release, but still no clue which commits in 5.8-rc1 resolved the issue > > level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > level=warning subsys=datapath-loader > level=warning msg="Log buffer too small to dump verifier log 16777215 > bytes (9 tries)!" subsys=datapath-loader > level=warning msg="Error filling program arrays!" subsys=datapath-loader > level=warning msg="Unable to load program" subsys=datapath-loader > > 5.8 works fine. > > What difference between 5.7 and 5.8 to cause this verifier problem, I > tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > that would make the difference with my limited BPF knowledge. Any clue > would be appreciated! > > Thanks > > Vincent > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-24 17:03 ` Vincent Li @ 2021-07-25 6:16 ` Vincent Li 2021-07-25 15:22 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-25 6:16 UTC (permalink / raw) To: Vincent Li; +Cc: bpf On Sat, 24 Jul 2021, Vincent Li wrote: > On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > > > > Hi BPF experts, > > > > I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > failed to pass verifier in kernel 4.19, the error is like: > > > > level=warning msg="Prog section '2/7' rejected: Argument list too long > > (7)!" subsys=datapath-loader > > level=warning msg=" - Type: 3" subsys=datapath-loader > > level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > level=warning msg=" - Instructions: 4578 (482 over limit)" > > subsys=datapath-loader > > level=warning msg=" - License: GPL" subsys=datapath-loader > > level=warning subsys=datapath-loader > > level=warning msg="Verifier analysis:" subsys=datapath-loader > > level=warning subsys=datapath-loader > > level=warning msg="Error filling program arrays!" subsys=datapath-loader > > level=warning msg="Unable to load program" subsys=datapath-loader > > > > then I tried to run the PR locally in my dev machine with custom upstream > > kernel version, I narrowed the issue down to between upstream kernel > > version 5.7 and 5.8, in 5.7, it failed with: > > I further narrow it down to between 5.7 and 5.8-rc1 release, but still > no clue which commits in 5.8-rc1 resolved the issue > > > > > level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > level=warning subsys=datapath-loader > > level=warning msg="Log buffer too small to dump verifier log 16777215 > > bytes (9 tries)!" subsys=datapath-loader > > level=warning msg="Error filling program arrays!" subsys=datapath-loader > > level=warning msg="Unable to load program" subsys=datapath-loader > > > > 5.8 works fine. > > > > What difference between 5.7 and 5.8 to cause this verifier problem, I > > tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > that would make the difference with my limited BPF knowledge. Any clue > > would be appreciated! I have git bisected to this commit: # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make verifier log more relevant by default this commit looks only dealing with log, accidently fixed the PR issue I have? my PR use __bpf_memcpy_builtin() to rewrite the tunnel inner packet destination MAC address, somehow related? [root@centos-dev bpf-next]# git bisect log git bisect start '--term-new=fixed' '--term-old=unfixed' # fixed: [b3a9e3b9622ae10064826dccb4f7a52bd88c7407] Linux 5.8-rc1 git bisect fixed b3a9e3b9622ae10064826dccb4f7a52bd88c7407 # unfixed: [3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162] Linux 5.7 git bisect unfixed 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 # fixed: [ee01c4d72adffb7d424535adf630f2955748fa8b] Merge branch 'akpm' (patches from Andrew) git bisect fixed ee01c4d72adffb7d424535adf630f2955748fa8b # unfixed: [16d91548d1057691979de4686693f0ff92f46000] Merge tag 'xfs-5.8-merge-8' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux git bisect unfixed 16d91548d1057691979de4686693f0ff92f46000 # fixed: [098205f3c688885394ed1f670a6a7cb4a58728a3] Merge branch '1GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue git bisect fixed 098205f3c688885394ed1f670a6a7cb4a58728a3 # fixed: [da1a782a7140fab22f2dfe8453d7b73c786d73de] net: ipa: kill ipa_cmd_dma_task_32b_addr_add() git bisect fixed da1a782a7140fab22f2dfe8453d7b73c786d73de # unfixed: [aa8a6ee3e3fc4001e952de37660fe71826da8189] docs: networking: convert team.txt to ReST git bisect unfixed aa8a6ee3e3fc4001e952de37660fe71826da8189 # unfixed: [5b95dea31636ce93660930d16172fe75589b2e70] Merge branch 'net-smc-extent-buffer-mapping-and-port-handling' git bisect unfixed 5b95dea31636ce93660930d16172fe75589b2e70 # fixed: [3316d50905f0e551d4786767d827589960a8cb83] bnxt_en: Split HW ring statistics strings into RX and TX parts. git bisect fixed 3316d50905f0e551d4786767d827589960a8cb83 # fixed: [c321022244708aec4675de4f032ef1ba9ff0c640] selftests/bpf: Test allowed maps for bpf_sk_select_reuseport git bisect fixed c321022244708aec4675de4f032ef1ba9ff0c640 # fixed: [50325b1761e31ad17d252e795af72a9af8c5a7d7] bpftool: Expose attach_type-to-string array to non-cgroup code git bisect fixed 50325b1761e31ad17d252e795af72a9af8c5a7d7 # fixed: [8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af] bpf, cgroup: Remove unused exports git bisect fixed 8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af # unfixed: [6f3f65d80dac8f2bafce2213005821fccdce194c] net: bpf: Allow TC programs to call BPF_FUNC_skb_change_head git bisect unfixed 6f3f65d80dac8f2bafce2213005821fccdce194c # fixed: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make verifier log more relevant by default git bisect fixed 6f8a57ccf8511724e6f48d732cb2940889789ab2 # unfixed: [0a05861f80fe7d4dcfdabcc98d9854947573e072] xsk: Fix typo in xsk_umem_consume_tx and xsk_generic_xmit comments git bisect unfixed 0a05861f80fe7d4dcfdabcc98d9854947573e072 # unfixed: [71d19214776e61b33da48f7c1b46e522c7f78221] bpf: add bpf_ktime_get_boot_ns() git bisect unfixed 71d19214776e61b33da48f7c1b46e522c7f78221 # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make verifier log more relevant by default > > > > Thanks > > > > Vincent > > > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-25 6:16 ` Vincent Li @ 2021-07-25 15:22 ` Vincent Li 2021-07-26 1:01 ` Yonghong Song 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-25 15:22 UTC (permalink / raw) To: Vincent Li; +Cc: bpf On Sat, 24 Jul 2021, Vincent Li wrote: > > > On Sat, 24 Jul 2021, Vincent Li wrote: > > > On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > > > > > > > Hi BPF experts, > > > > > > I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > > failed to pass verifier in kernel 4.19, the error is like: > > > > > > level=warning msg="Prog section '2/7' rejected: Argument list too long > > > (7)!" subsys=datapath-loader > > > level=warning msg=" - Type: 3" subsys=datapath-loader > > > level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > > level=warning msg=" - Instructions: 4578 (482 over limit)" > > > subsys=datapath-loader > > > level=warning msg=" - License: GPL" subsys=datapath-loader > > > level=warning subsys=datapath-loader > > > level=warning msg="Verifier analysis:" subsys=datapath-loader > > > level=warning subsys=datapath-loader > > > level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > level=warning msg="Unable to load program" subsys=datapath-loader > > > > > > then I tried to run the PR locally in my dev machine with custom upstream > > > kernel version, I narrowed the issue down to between upstream kernel > > > version 5.7 and 5.8, in 5.7, it failed with: > > > > I further narrow it down to between 5.7 and 5.8-rc1 release, but still > > no clue which commits in 5.8-rc1 resolved the issue > > > > > > > > level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > > 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > > level=warning subsys=datapath-loader > > > level=warning msg="Log buffer too small to dump verifier log 16777215 > > > bytes (9 tries)!" subsys=datapath-loader > > > level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > level=warning msg="Unable to load program" subsys=datapath-loader > > > > > > 5.8 works fine. > > > > > > What difference between 5.7 and 5.8 to cause this verifier problem, I > > > tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > > that would make the difference with my limited BPF knowledge. Any clue > > > would be appreciated! > > I have git bisected to this commit: > > # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make > verifier log more relevant by default both the cilium github PR test and my local dev machine PR test has the verbose set, for example, my local test has: diff --git a/pkg/datapath/loader/netlink.go b/pkg/datapath/loader/netlink.go index 381e1fbc8..00015eabc 100644 --- a/pkg/datapath/loader/netlink.go +++ b/pkg/datapath/loader/netlink.go @@ -106,7 +106,7 @@ func replaceDatapath(ctx context.Context, ifName, objPath, progSec, progDirectio loaderProg = "tc" args = []string{"filter", "replace", "dev", ifName, progDirection, "prio", "1", "handle", "1", "bpf", "da", "obj", objPath, - "sec", progSec, + "sec", progSec, "verbose", } } cmd = exec.CommandContext(ctx, loaderProg, args...).WithFilters(libbpfFixupMsg) if I remove the "verbose" change, and run the Cilium agent without kernel commit 6f8a57ccf8, the problem is gone, it seems commit 6f8a57ccf8 is related > > this commit looks only dealing with log, accidently fixed the PR issue I > have? my PR use __bpf_memcpy_builtin() to rewrite the tunnel inner packet > destination MAC address, somehow related? > > [root@centos-dev bpf-next]# git bisect log > git bisect start '--term-new=fixed' '--term-old=unfixed' > # fixed: [b3a9e3b9622ae10064826dccb4f7a52bd88c7407] Linux 5.8-rc1 > git bisect fixed b3a9e3b9622ae10064826dccb4f7a52bd88c7407 > # unfixed: [3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162] Linux 5.7 > git bisect unfixed 3d77e6a8804abcc0504c904bd6e5cdf3a5cf8162 > # fixed: [ee01c4d72adffb7d424535adf630f2955748fa8b] Merge branch 'akpm' > (patches from Andrew) > git bisect fixed ee01c4d72adffb7d424535adf630f2955748fa8b > # unfixed: [16d91548d1057691979de4686693f0ff92f46000] Merge tag > 'xfs-5.8-merge-8' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux > git bisect unfixed 16d91548d1057691979de4686693f0ff92f46000 > # fixed: [098205f3c688885394ed1f670a6a7cb4a58728a3] Merge branch '1GbE' of > git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/next-queue > git bisect fixed 098205f3c688885394ed1f670a6a7cb4a58728a3 > # fixed: [da1a782a7140fab22f2dfe8453d7b73c786d73de] net: ipa: kill > ipa_cmd_dma_task_32b_addr_add() > git bisect fixed da1a782a7140fab22f2dfe8453d7b73c786d73de > # unfixed: [aa8a6ee3e3fc4001e952de37660fe71826da8189] docs: networking: > convert team.txt to ReST > git bisect unfixed aa8a6ee3e3fc4001e952de37660fe71826da8189 > # unfixed: [5b95dea31636ce93660930d16172fe75589b2e70] Merge branch > 'net-smc-extent-buffer-mapping-and-port-handling' > git bisect unfixed 5b95dea31636ce93660930d16172fe75589b2e70 > # fixed: [3316d50905f0e551d4786767d827589960a8cb83] bnxt_en: Split HW ring > statistics strings into RX and TX parts. > git bisect fixed 3316d50905f0e551d4786767d827589960a8cb83 > # fixed: [c321022244708aec4675de4f032ef1ba9ff0c640] selftests/bpf: Test > allowed maps for bpf_sk_select_reuseport > git bisect fixed c321022244708aec4675de4f032ef1ba9ff0c640 > # fixed: [50325b1761e31ad17d252e795af72a9af8c5a7d7] bpftool: Expose > attach_type-to-string array to non-cgroup code > git bisect fixed 50325b1761e31ad17d252e795af72a9af8c5a7d7 > # fixed: [8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af] bpf, cgroup: Remove > unused exports > git bisect fixed 8c1b2bf16d5944cd5c3a8a72e24ed9e22360c1af > # unfixed: [6f3f65d80dac8f2bafce2213005821fccdce194c] net: bpf: Allow TC > programs to call BPF_FUNC_skb_change_head > git bisect unfixed 6f3f65d80dac8f2bafce2213005821fccdce194c > # fixed: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make verifier log > more relevant by default > git bisect fixed 6f8a57ccf8511724e6f48d732cb2940889789ab2 > # unfixed: [0a05861f80fe7d4dcfdabcc98d9854947573e072] xsk: Fix typo in > xsk_umem_consume_tx and xsk_generic_xmit comments > git bisect unfixed 0a05861f80fe7d4dcfdabcc98d9854947573e072 > # unfixed: [71d19214776e61b33da48f7c1b46e522c7f78221] bpf: add > bpf_ktime_get_boot_ns() > git bisect unfixed 71d19214776e61b33da48f7c1b46e522c7f78221 > # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make > verifier log more relevant by default > > > > > > > > Thanks > > > > > > Vincent > > > > > > ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-25 15:22 ` Vincent Li @ 2021-07-26 1:01 ` Yonghong Song 2021-07-26 1:14 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Yonghong Song @ 2021-07-26 1:01 UTC (permalink / raw) To: Vincent Li; +Cc: bpf On 7/25/21 8:22 AM, Vincent Li wrote: > > > > On Sat, 24 Jul 2021, Vincent Li wrote: > >> >> >> On Sat, 24 Jul 2021, Vincent Li wrote: >> >>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>> >>>> >>>> Hi BPF experts, >>>> >>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that >>>> failed to pass verifier in kernel 4.19, the error is like: >>>> >>>> level=warning msg="Prog section '2/7' rejected: Argument list too long >>>> (7)!" subsys=datapath-loader >>>> level=warning msg=" - Type: 3" subsys=datapath-loader >>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader >>>> level=warning msg=" - Instructions: 4578 (482 over limit)" >>>> subsys=datapath-loader >>>> level=warning msg=" - License: GPL" subsys=datapath-loader >>>> level=warning subsys=datapath-loader >>>> level=warning msg="Verifier analysis:" subsys=datapath-loader >>>> level=warning subsys=datapath-loader >>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader >>>> level=warning msg="Unable to load program" subsys=datapath-loader >>>> >>>> then I tried to run the PR locally in my dev machine with custom upstream >>>> kernel version, I narrowed the issue down to between upstream kernel >>>> version 5.7 and 5.8, in 5.7, it failed with: >>> >>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still >>> no clue which commits in 5.8-rc1 resolved the issue >>> >>>> >>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn >>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader >>>> level=warning subsys=datapath-loader >>>> level=warning msg="Log buffer too small to dump verifier log 16777215 >>>> bytes (9 tries)!" subsys=datapath-loader The error message is "Log buffer too small to dump verifier log 16777215 bytes (9 tries)!". Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log much shorter. So it fixed the above log buffer too small issue. >>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader >>>> level=warning msg="Unable to load program" subsys=datapath-loader >>>> >>>> 5.8 works fine. >>>> >>>> What difference between 5.7 and 5.8 to cause this verifier problem, I >>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits >>>> that would make the difference with my limited BPF knowledge. Any clue >>>> would be appreciated! >> >> I have git bisected to this commit: >> >> # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make >> verifier log more relevant by default > > both the cilium github PR test and my local dev machine PR test has the > verbose set, for example, my local test has: > > diff --git a/pkg/datapath/loader/netlink.go > b/pkg/datapath/loader/netlink.go > index 381e1fbc8..00015eabc 100644 > --- a/pkg/datapath/loader/netlink.go > +++ b/pkg/datapath/loader/netlink.go > @@ -106,7 +106,7 @@ func replaceDatapath(ctx context.Context, ifName, > objPath, progSec, progDirectio > loaderProg = "tc" > args = []string{"filter", "replace", "dev", ifName, > progDirection, > "prio", "1", "handle", "1", "bpf", "da", "obj", > objPath, > - "sec", progSec, > + "sec", progSec, "verbose", > } > } > cmd = exec.CommandContext(ctx, loaderProg, > args...).WithFilters(libbpfFixupMsg) > > if I remove the "verbose" change, and run the Cilium agent without > kernel commit 6f8a57ccf8, the problem is gone, it seems commit 6f8a57ccf8 > is related Remove "verbose" should work since the kernel won't do logging any more. > >> >> this commit looks only dealing with log, accidently fixed the PR issue I >> have? my PR use __bpf_memcpy_builtin() to rewrite the tunnel inner packet >> destination MAC address, somehow related? >> [...] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-26 1:01 ` Yonghong Song @ 2021-07-26 1:14 ` Vincent Li 2021-07-26 2:39 ` Yonghong Song 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-26 1:14 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote: > > > > On 7/25/21 8:22 AM, Vincent Li wrote: > > > > > > > > On Sat, 24 Jul 2021, Vincent Li wrote: > > > >> > >> > >> On Sat, 24 Jul 2021, Vincent Li wrote: > >> > >>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>> > >>>> > >>>> Hi BPF experts, > >>>> > >>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > >>>> failed to pass verifier in kernel 4.19, the error is like: > >>>> > >>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > >>>> (7)!" subsys=datapath-loader > >>>> level=warning msg=" - Type: 3" subsys=datapath-loader > >>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > >>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > >>>> subsys=datapath-loader > >>>> level=warning msg=" - License: GPL" subsys=datapath-loader > >>>> level=warning subsys=datapath-loader > >>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > >>>> level=warning subsys=datapath-loader > >>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>> > >>>> then I tried to run the PR locally in my dev machine with custom upstream > >>>> kernel version, I narrowed the issue down to between upstream kernel > >>>> version 5.7 and 5.8, in 5.7, it failed with: > >>> > >>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > >>> no clue which commits in 5.8-rc1 resolved the issue > >>> > >>>> > >>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > >>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > >>>> level=warning subsys=datapath-loader > >>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > >>>> bytes (9 tries)!" subsys=datapath-loader > > The error message is "Log buffer too small to dump verifier log 16777215 > bytes (9 tries)!". > > Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > much shorter. So it fixed the above log buffer too small issue. > Thank you for the confirmation, after I remove 'verbose' log, indeed the problem went away for kernel 5.x- 5.8, but the "Prog section '2/7' rejected: Argument list too long.." issue persisted even after I remove the "verbose" logging for kernel version 4.19, any clue on that? > >>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>> > >>>> 5.8 works fine. > >>>> > >>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > >>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > >>>> that would make the difference with my limited BPF knowledge. Any clue > >>>> would be appreciated! > >> > >> I have git bisected to this commit: > >> > >> # first fixed commit: [6f8a57ccf8511724e6f48d732cb2940889789ab2] bpf: Make > >> verifier log more relevant by default > > > > both the cilium github PR test and my local dev machine PR test has the > > verbose set, for example, my local test has: > > > > diff --git a/pkg/datapath/loader/netlink.go > > b/pkg/datapath/loader/netlink.go > > index 381e1fbc8..00015eabc 100644 > > --- a/pkg/datapath/loader/netlink.go > > +++ b/pkg/datapath/loader/netlink.go > > @@ -106,7 +106,7 @@ func replaceDatapath(ctx context.Context, ifName, > > objPath, progSec, progDirectio > > loaderProg = "tc" > > args = []string{"filter", "replace", "dev", ifName, > > progDirection, > > "prio", "1", "handle", "1", "bpf", "da", "obj", > > objPath, > > - "sec", progSec, > > + "sec", progSec, "verbose", > > } > > } > > cmd = exec.CommandContext(ctx, loaderProg, > > args...).WithFilters(libbpfFixupMsg) > > > > if I remove the "verbose" change, and run the Cilium agent without > > kernel commit 6f8a57ccf8, the problem is gone, it seems commit 6f8a57ccf8 > > is related > > Remove "verbose" should work since the kernel won't do logging any more. > > > > >> > >> this commit looks only dealing with log, accidently fixed the PR issue I > >> have? my PR use __bpf_memcpy_builtin() to rewrite the tunnel inner packet > >> destination MAC address, somehow related? > >> > > [...] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-26 1:14 ` Vincent Li @ 2021-07-26 2:39 ` Yonghong Song 2021-07-26 3:49 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Yonghong Song @ 2021-07-26 2:39 UTC (permalink / raw) To: Vincent Li; +Cc: bpf On 7/25/21 6:14 PM, Vincent Li wrote: > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote: >> >> >> >> On 7/25/21 8:22 AM, Vincent Li wrote: >>> >>> >>> >>> On Sat, 24 Jul 2021, Vincent Li wrote: >>> >>>> >>>> >>>> On Sat, 24 Jul 2021, Vincent Li wrote: >>>> >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: >>>>>> >>>>>> >>>>>> Hi BPF experts, >>>>>> >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that >>>>>> failed to pass verifier in kernel 4.19, the error is like: >>>>>> >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long >>>>>> (7)!" subsys=datapath-loader >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" >>>>>> subsys=datapath-loader >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader >>>>>> level=warning subsys=datapath-loader >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader >>>>>> level=warning subsys=datapath-loader >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader >>>>>> >>>>>> then I tried to run the PR locally in my dev machine with custom upstream >>>>>> kernel version, I narrowed the issue down to between upstream kernel >>>>>> version 5.7 and 5.8, in 5.7, it failed with: >>>>> >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still >>>>> no clue which commits in 5.8-rc1 resolved the issue >>>>> >>>>>> >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader >>>>>> level=warning subsys=datapath-loader >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 >>>>>> bytes (9 tries)!" subsys=datapath-loader >> >> The error message is "Log buffer too small to dump verifier log 16777215 >> bytes (9 tries)!". >> >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log >> much shorter. So it fixed the above log buffer too small issue. >> > > Thank you for the confirmation, after I remove 'verbose' log, indeed > the problem went away for kernel 5.x- 5.8, but the > "Prog section '2/7' rejected: Argument list too long.." issue > persisted even after I remove the "verbose" logging > for kernel version 4.19, any clue on that? No, I don't. You need to have detailed verifier log. In verifier, there are quite some places which returns -E2BIG. > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader >>>>>> >>>>>> 5.8 works fine. >>>>>> >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits >>>>>> that would make the difference with my limited BPF knowledge. Any clue >>>>>> would be appreciated! >>>> >>>> I have git bisected to this commit: [...] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-26 2:39 ` Yonghong Song @ 2021-07-26 3:49 ` Vincent Li 2021-07-26 17:35 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-26 3:49 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@fb.com> wrote: > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote: > >> > >> > >> > >> On 7/25/21 8:22 AM, Vincent Li wrote: > >>> > >>> > >>> > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > >>> > >>>> > >>>> > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > >>>> > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > >>>>>> > >>>>>> > >>>>>> Hi BPF experts, > >>>>>> > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > >>>>>> > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > >>>>>> (7)!" subsys=datapath-loader > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > >>>>>> subsys=datapath-loader > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>>>> > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > >>>>> > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > >>>>> no clue which commits in 5.8-rc1 resolved the issue > >>>>> > >>>>>> > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > >>>>>> level=warning subsys=datapath-loader > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > >>>>>> bytes (9 tries)!" subsys=datapath-loader > >> > >> The error message is "Log buffer too small to dump verifier log 16777215 > >> bytes (9 tries)!". > >> > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > >> much shorter. So it fixed the above log buffer too small issue. > >> > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > the problem went away for kernel 5.x- 5.8, but the > > "Prog section '2/7' rejected: Argument list too long.." issue > > persisted even after I remove the "verbose" logging > > for kernel version 4.19, any clue on that? > > No, I don't. > > You need to have detailed verifier log. In verifier, there are quite > some places which returns -E2BIG. > I will do another round of bisect, correct myself, the "The argument list too long" occurred in 5.1, but not in 5.2 > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > >>>>>> > >>>>>> 5.8 works fine. > >>>>>> > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > >>>>>> would be appreciated! > >>>> > >>>> I have git bisected to this commit: > [...] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-26 3:49 ` Vincent Li @ 2021-07-26 17:35 ` Vincent Li 2021-09-19 1:43 ` Vincent Li 0 siblings, 1 reply; 10+ messages in thread From: Vincent Li @ 2021-07-26 17:35 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Sun, Jul 25, 2021 at 8:49 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@fb.com> wrote: > > > > > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote: > > >> > > >> > > >> > > >> On 7/25/21 8:22 AM, Vincent Li wrote: > > >>> > > >>> > > >>> > > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > > >>> > > >>>> > > >>>> > > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > > >>>> > > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > >>>>>> > > >>>>>> > > >>>>>> Hi BPF experts, > > >>>>>> > > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > > >>>>>> > > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > > >>>>>> (7)!" subsys=datapath-loader > > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > > >>>>>> subsys=datapath-loader > > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > >>>>>> > > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > > >>>>> > > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > > >>>>> no clue which commits in 5.8-rc1 resolved the issue > > >>>>> > > >>>>>> > > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > >>>>>> level=warning subsys=datapath-loader > > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > > >>>>>> bytes (9 tries)!" subsys=datapath-loader > > >> > > >> The error message is "Log buffer too small to dump verifier log 16777215 > > >> bytes (9 tries)!". > > >> > > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > > >> much shorter. So it fixed the above log buffer too small issue. > > >> > > > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > > the problem went away for kernel 5.x- 5.8, but the > > > "Prog section '2/7' rejected: Argument list too long.." issue > > > persisted even after I remove the "verbose" logging > > > for kernel version 4.19, any clue on that? > > > > No, I don't. > > > > You need to have detailed verifier log. In verifier, there are quite > > some places which returns -E2BIG. > > > I will do another round of bisect, correct myself, the "The argument > list too long" occurred in 5.1, but not in 5.2 It looks to be this commit fixed the issue commit c04c0d2b968ac45d6ef020316808ef6c82325a82 (HEAD) Author: Alexei Starovoitov <ast@kernel.org> Date: Mon Apr 1 21:27:45 2019 -0700 bpf: increase complexity limit and maximum program size > > > > > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > >>>>>> > > >>>>>> 5.8 works fine. > > >>>>>> > > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > > >>>>>> would be appreciated! > > >>>> > > >>>> I have git bisected to this commit: > > [...] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Prog section rejected: Argument list too long (7)! 2021-07-26 17:35 ` Vincent Li @ 2021-09-19 1:43 ` Vincent Li 0 siblings, 0 replies; 10+ messages in thread From: Vincent Li @ 2021-09-19 1:43 UTC (permalink / raw) To: Yonghong Song; +Cc: bpf On Mon, Jul 26, 2021 at 10:35 AM Vincent Li <vincent.mc.li@gmail.com> wrote: > > On Sun, Jul 25, 2021 at 8:49 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > > On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@fb.com> wrote: > > > > > > > > > > > > On 7/25/21 6:14 PM, Vincent Li wrote: > > > > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote: > > > >> > > > >> > > > >> > > > >> On 7/25/21 8:22 AM, Vincent Li wrote: > > > >>> > > > >>> > > > >>> > > > >>> On Sat, 24 Jul 2021, Vincent Li wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On Sat, 24 Jul 2021, Vincent Li wrote: > > > >>>> > > > >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote: > > > >>>>>> > > > >>>>>> > > > >>>>>> Hi BPF experts, > > > >>>>>> > > > >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that > > > >>>>>> failed to pass verifier in kernel 4.19, the error is like: > > > >>>>>> > > > >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long > > > >>>>>> (7)!" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Type: 3" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Attach Type: 0" subsys=datapath-loader > > > >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)" > > > >>>>>> subsys=datapath-loader > > > >>>>>> level=warning msg=" - License: GPL" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > > >>>>>> > > > >>>>>> then I tried to run the PR locally in my dev machine with custom upstream > > > >>>>>> kernel version, I narrowed the issue down to between upstream kernel > > > >>>>>> version 5.7 and 5.8, in 5.7, it failed with: > > > >>>>> > > > >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still > > > >>>>> no clue which commits in 5.8-rc1 resolved the issue > > > >>>>> > > > >>>>>> > > > >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn > > > >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader > > > >>>>>> level=warning subsys=datapath-loader > > > >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215 > > > >>>>>> bytes (9 tries)!" subsys=datapath-loader > > > >> > > > >> The error message is "Log buffer too small to dump verifier log 16777215 > > > >> bytes (9 tries)!". > > > >> > > > >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log > > > >> much shorter. So it fixed the above log buffer too small issue. > > > >> > > > > > > > > Thank you for the confirmation, after I remove 'verbose' log, indeed > > > > the problem went away for kernel 5.x- 5.8, but the > > > > "Prog section '2/7' rejected: Argument list too long.." issue > > > > persisted even after I remove the "verbose" logging > > > > for kernel version 4.19, any clue on that? > > > > > > No, I don't. > > > > > > You need to have detailed verifier log. In verifier, there are quite > > > some places which returns -E2BIG. > > > > > I will do another round of bisect, correct myself, the "The argument > > list too long" occurred in 5.1, but not in 5.2 > > It looks to be this commit fixed the issue > commit c04c0d2b968ac45d6ef020316808ef6c82325a82 (HEAD) > Author: Alexei Starovoitov <ast@kernel.org> > Date: Mon Apr 1 21:27:45 2019 -0700 > bpf: increase complexity limit and maximum program size > > > > > > > > > > > > > > >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader > > > >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader > > > >>>>>> > > > >>>>>> 5.8 works fine. > > > >>>>>> > > > >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I > > > >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits > > > >>>>>> that would make the difference with my limited BPF knowledge. Any clue > > > >>>>>> would be appreciated! > > > >>>> > > > >>>> I have git bisected to this commit: > > > [...] Finally bring an end to my long time mystery issue of using eth_store_daddr() result in "invalid mem access", I need to initialize the mac address variable with 0, uninitialized variable also cause "invalid read from stack off" in kernel 4.9, but not version above 4.9. https://github.com/cilium/cilium/pull/17370#issuecomment-922396415 ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-09-19 1:44 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-07-24 2:17 Prog section rejected: Argument list too long (7)! Vincent Li 2021-07-24 17:03 ` Vincent Li 2021-07-25 6:16 ` Vincent Li 2021-07-25 15:22 ` Vincent Li 2021-07-26 1:01 ` Yonghong Song 2021-07-26 1:14 ` Vincent Li 2021-07-26 2:39 ` Yonghong Song 2021-07-26 3:49 ` Vincent Li 2021-07-26 17:35 ` Vincent Li 2021-09-19 1:43 ` Vincent Li
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).