Re: Prog section rejected: Argument list too long (7)!

From: Vincent Li <vincent.mc.li@gmail.com>
To: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Subject: Re: Prog section rejected: Argument list too long (7)!
Date: Sun, 25 Jul 2021 20:49:30 -0700	[thread overview]
Message-ID: <CAK3+h2z=qxzDm=-isjuM01n8Mt5NpoAHCkwHNzOWFXNMAczUdw@mail.gmail.com> (raw)
In-Reply-To: <e1a2904f-1b43-e1a8-e20d-0449798274bb@fb.com>

On Sun, Jul 25, 2021 at 7:39 PM Yonghong Song <yhs@fb.com> wrote:
>
>
>
> On 7/25/21 6:14 PM, Vincent Li wrote:
> > On Sun, Jul 25, 2021 at 6:01 PM Yonghong Song <yhs@fb.com> wrote:
> >>
> >>
> >>
> >> On 7/25/21 8:22 AM, Vincent Li wrote:
> >>>
> >>>
> >>>
> >>> On Sat, 24 Jul 2021, Vincent Li wrote:
> >>>
> >>>>
> >>>>
> >>>> On Sat, 24 Jul 2021, Vincent Li wrote:
> >>>>
> >>>>> On Fri, Jul 23, 2021 at 7:17 PM Vincent Li <vincent.mc.li@gmail.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Hi BPF experts,
> >>>>>>
> >>>>>> I have a cilium PR https://github.com/cilium/cilium/pull/16916 that
> >>>>>> failed to pass verifier in kernel 4.19, the error is like:
> >>>>>>
> >>>>>> level=warning msg="Prog section '2/7' rejected: Argument list too long
> >>>>>> (7)!" subsys=datapath-loader
> >>>>>> level=warning msg=" - Type:         3" subsys=datapath-loader
> >>>>>> level=warning msg=" - Attach Type:  0" subsys=datapath-loader
> >>>>>> level=warning msg=" - Instructions: 4578 (482 over limit)"
> >>>>>> subsys=datapath-loader
> >>>>>> level=warning msg=" - License:      GPL" subsys=datapath-loader
> >>>>>> level=warning subsys=datapath-loader
> >>>>>> level=warning msg="Verifier analysis:" subsys=datapath-loader
> >>>>>> level=warning subsys=datapath-loader
> >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader
> >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader
> >>>>>>
> >>>>>> then I tried to run the PR locally in my dev machine with custom upstream
> >>>>>> kernel version, I narrowed the issue down to between upstream kernel
> >>>>>> version 5.7 and 5.8, in 5.7, it failed with:
> >>>>>
> >>>>> I further narrow it down to between 5.7 and 5.8-rc1 release, but still
> >>>>> no clue which commits in 5.8-rc1 resolved the issue
> >>>>>
> >>>>>>
> >>>>>> level=warning msg="processed 50 insns (limit 1000000) max_states_per_insn
> >>>>>> 0 total_states 1 peak_states 1 mark_read 1" subsys=datapath-loader
> >>>>>> level=warning subsys=datapath-loader
> >>>>>> level=warning msg="Log buffer too small to dump verifier log 16777215
> >>>>>> bytes (9 tries)!" subsys=datapath-loader
> >>
> >> The error message is "Log buffer too small to dump verifier log 16777215
> >> bytes (9 tries)!".
> >>
> >> Commit 6f8a57ccf8511724e6f48d732cb2940889789ab2 made the default log
> >> much shorter. So it fixed the above log buffer too small issue.
> >>
> >
> > Thank you for the confirmation, after I remove 'verbose' log, indeed
> > the problem went away for kernel 5.x- 5.8, but the
> > "Prog section '2/7' rejected: Argument list too long.." issue
> > persisted even after I remove the "verbose" logging
> > for kernel version 4.19, any clue on that?
>
> No, I don't.
>
> You need to have detailed verifier log. In verifier, there are quite
> some places which returns -E2BIG.
>
I will do another round of bisect,  correct myself, the "The argument
list too long" occurred in 5.1, but not in 5.2

> >
> >
> >>>>>> level=warning msg="Error filling program arrays!" subsys=datapath-loader
> >>>>>> level=warning msg="Unable to load program" subsys=datapath-loader
> >>>>>>
> >>>>>> 5.8 works fine.
> >>>>>>
> >>>>>> What difference between 5.7 and 5.8 to cause this verifier problem, I
> >>>>>> tried to git log v5.7..v5.8 kernel/bpf/verifier, I could not see commits
> >>>>>> that would make the difference with my limited BPF knowledge. Any clue
> >>>>>> would be appreciated!
> >>>>
> >>>> I have git bisected to this commit:
> [...]