bpf selftest failed in 5.4.210 kernel

bpf selftest failed in 5.4.210 kernel

[RAJESH DASARI]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

Hi ,

We are running bpf selftests on 5.4.210 kernel version and we see that
test case 11 of  test_align failed. Please find the below error.

selftests: bpf: test_align
Test  11: pointer variable subtraction ... Failed to find match 16:
R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
0xfffffffc)
# func#0 @0
# 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
# 0: (61) r2 = *(u32 *)(r1 +76)
# 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
# 1: (61) r3 = *(u32 *)(r1 +80)

For complete errors please see the attached file. The same test case
execution was successful in the 5.4.209 version , could you please let
me know any known issue with the recent changes in 5.4.210 and how to
fix these errors.


Thanks,
Rajesh Dasari.

[-- Attachment #2: test_align_bpf_error --]
[-- Type: application/octet-stream, Size: 7435 bytes --]

selftests: bpf: test_align
Test  11: pointer variable subtraction ... Failed to find match 16: R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0xfffffffc)
# func#0 @0
# 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
# 0: (61) r2 = *(u32 *)(r1 +76)
# 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
# 1: (61) r3 = *(u32 *)(r1 +80)
# 2: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 2: (bf) r0 = r2
# 3: R0_w=pkt(id=0,off=0,r=0,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 3: (07) r0 += 8
# 4: R0_w=pkt(id=0,off=8,r=0,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 4: (3d) if r3 >= r0 goto pc+1
#  R0_w=pkt(id=0,off=8,r=0,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 5: R0_w=pkt(id=0,off=8,r=0,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 5: (95) exit
# 6: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R10=fp0
# 6: (71) r6 = *(u8 *)(r2 +0)
# 7: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 7: (bf) r7 = r6
# 8: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R7_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 8: (57) r6 &= 15
# 9: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=inv(id=0,umax_value=15,var_off=(0x0; 0xf)) R7_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 9: (67) r6 <<= 2
# 10: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R6_w=inv(id=0,umax_value=60,var_off=(0x0; 0x3c)) R7_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 10: (07) r6 += 14
11: (bf) r5 = r2
# 12: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R5_w=pkt(id=0,off=0,r=8,imm=0) R6_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 12: (1f) r5 -= r6
# last_idx 12 first_idx 0
# regs=40 stack=0 before 11: (bf) r5 = r2
# regs=40 stack=0 before 10: (07) r6 += 14
# regs=40 stack=0 before 9: (67) r6 <<= 2
# regs=40 stack=0 before 8: (57) r6 &= 15
# regs=40 stack=0 before 7: (bf) r7 = r6
# regs=40 stack=0 before 6: (71) r6 = *(u8 *)(r2 +0)
# 13: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 0x7c)) R6_w=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R10=fp0
# 13: (67) r7 <<= 2
# 14: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 0x7c)) R6_w=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=inv(id=0,umax_value=1020,var_off=(0x0; 0x3fc)) R10=fp0
# 14: (07) r7 += 76
# 15: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 0x7c)) R6_w=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=inv(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 15: (0f) r5 += r7
# last_idx 15 first_idx 0
# regs=80 stack=0 before 14: (07) r7 += 76
# regs=80 stack=0 before 13: (67) r7 <<= 2
# regs=80 stack=0 before 12: (1f) r5 -= r6
# regs=80 stack=0 before 11: (bf) r5 = r2
# regs=80 stack=0 before 10: (07) r6 += 14
# regs=80 stack=0 before 9: (67) r6 <<= 2
# regs=80 stack=0 before 8: (57) r6 &= 15
# regs=80 stack=0 before 7: (bf) r7 = r6
# regs=40 stack=0 before 6: (71) r6 = *(u8 *)(r2 +0)
6: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6_w=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 16: (bf) r4 = r5
# 17: R0_w=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=8,imm=0) R3_w=pkt_end(id=0,off=0,imm=0) R4_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6_w=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7_w=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 17: (07) r4 += 4
# 18: R0=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2=pkt(id=0,off=0,r=8,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=pkt(id=2,off=4,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 18: (3d) if r3 >= r4 goto pc+1
#  R0=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2=pkt(id=0,off=0,r=8,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=pkt(id=2,off=4,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 19: R0=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2=pkt(id=0,off=0,r=8,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=pkt(id=2,off=4,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 19: (95) exit
# 20: R0=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2=pkt(id=0,off=0,r=8,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=pkt(id=2,off=4,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6=invP(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c)) R7=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 20: (61) r6 = *(u32 *)(r5 +0)
# 21: R0=pkt(id=0,off=8,r=8,imm=0) R1=ctx(id=0,off=0,imm=0) R2=pkt(id=0,off=0,r=8,imm=0) R3=pkt_end(id=0,off=0,imm=0) R4=pkt(id=2,off=4,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc)) R6_w=inv(id=0,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R7=invP(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc)) R10=fp0
# 21: (95) exit

# processed 22 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
# FAIL
# Results: 11 pass 1 fail
not ok 7 selftests: bpf: test_align # exit=1

Re: bpf selftest failed in 5.4.210 kernel

[Greg KH]

On Wed, Aug 17, 2022 at 09:22:00PM +0300, RAJESH DASARI wrote:
> Hi ,
> 
> We are running bpf selftests on 5.4.210 kernel version and we see that
> test case 11 of  test_align failed. Please find the below error.
> 
> selftests: bpf: test_align
> Test  11: pointer variable subtraction ... Failed to find match 16:
> R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> 0xfffffffc)
> # func#0 @0
> # 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> # 0: (61) r2 = *(u32 *)(r1 +76)
> # 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
> # 1: (61) r3 = *(u32 *)(r1 +80)
> 
> For complete errors please see the attached file. The same test case
> execution was successful in the 5.4.209 version , could you please let
> me know any known issue with the recent changes in 5.4.210 and how to
> fix these errors.

Can you use 'git bisect' to find the offending commit?

thanks,

greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Jean-Philippe Brucker]

On Thu, Aug 18, 2022 at 07:24:03AM +0200, Greg KH wrote:
> On Wed, Aug 17, 2022 at 09:22:00PM +0300, RAJESH DASARI wrote:
> > Hi ,
> > 
> > We are running bpf selftests on 5.4.210 kernel version and we see that
> > test case 11 of  test_align failed. Please find the below error.
> > 
> > selftests: bpf: test_align
> > Test  11: pointer variable subtraction ... Failed to find match 16:
> > R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> > 0xfffffffc)
> > # func#0 @0
> > # 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> > # 0: (61) r2 = *(u32 *)(r1 +76)
> > # 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
> > # 1: (61) r3 = *(u32 *)(r1 +80)
> > 
> > For complete errors please see the attached file. The same test case
> > execution was successful in the 5.4.209 version , could you please let
> > me know any known issue with the recent changes in 5.4.210 and how to
> > fix these errors.
> 
> Can you use 'git bisect' to find the offending commit?

It probably is 6098562ed9df ("selftests/bpf: Fix "dubious pointer
arithmetic" test")
https://lore.kernel.org/all/20220803145005.2385039-6-ovidiu.panait@windriver.com/
Could you try reverting that?

The patch didn't have a Fixes: tags, because the bugfix it refers to was
merged at the same time. That bugfix is upstream commit b02709587ea3
("bpf: Fix propagation of 32-bit signed bounds from 64-bit bounds.")

Since b02709587ea3 was only backported down to 5.10, this fix shouldn't be
in 5.4. Sorry for not catching this earlier.

Thanks,
Jean

Re: bpf selftest failed in 5.4.210 kernel

[RAJESH DASARI]

I tried with git bisect between v5.4.210(bad) and v5.4.209(good).
Everytime I did bisect I compiled the kernel , booted my instance with
the new kernel and ran the selftests after trying out for 3 times ,
git bisect pointed to the  below commit as a first bad commit.

git bisect bad
9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit
commit 9d6f67365d9cdb389fbdac2bb5b00e59e345930e
Author: John Fastabend <john.fastabend@gmail.com>
Date:   Wed Aug 3 17:50:03 2022 +0300

    bpf: Test_verifier, #70 error message updates for 32-bit right shift

Thanks,
Rajesh Dasari.

On Thu, Aug 18, 2022 at 8:24 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Aug 17, 2022 at 09:22:00PM +0300, RAJESH DASARI wrote:
> > Hi ,
> >
> > We are running bpf selftests on 5.4.210 kernel version and we see that
> > test case 11 of  test_align failed. Please find the below error.
> >
> > selftests: bpf: test_align
> > Test  11: pointer variable subtraction ... Failed to find match 16:
> > R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> > 0xfffffffc)
> > # func#0 @0
> > # 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> > # 0: (61) r2 = *(u32 *)(r1 +76)
> > # 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
> > # 1: (61) r3 = *(u32 *)(r1 +80)
> >
> > For complete errors please see the attached file. The same test case
> > execution was successful in the 5.4.209 version , could you please let
> > me know any known issue with the recent changes in 5.4.210 and how to
> > fix these errors.
>
> Can you use 'git bisect' to find the offending commit?
>
> thanks,
>
> greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Greg KH]

On Thu, Aug 18, 2022 at 08:55:16AM +0100, Jean-Philippe Brucker wrote:
> On Thu, Aug 18, 2022 at 07:24:03AM +0200, Greg KH wrote:
> > On Wed, Aug 17, 2022 at 09:22:00PM +0300, RAJESH DASARI wrote:
> > > Hi ,
> > > 
> > > We are running bpf selftests on 5.4.210 kernel version and we see that
> > > test case 11 of  test_align failed. Please find the below error.
> > > 
> > > selftests: bpf: test_align
> > > Test  11: pointer variable subtraction ... Failed to find match 16:
> > > R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> > > 0xfffffffc)
> > > # func#0 @0
> > > # 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> > > # 0: (61) r2 = *(u32 *)(r1 +76)
> > > # 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
> > > # 1: (61) r3 = *(u32 *)(r1 +80)
> > > 
> > > For complete errors please see the attached file. The same test case
> > > execution was successful in the 5.4.209 version , could you please let
> > > me know any known issue with the recent changes in 5.4.210 and how to
> > > fix these errors.
> > 
> > Can you use 'git bisect' to find the offending commit?
> 
> It probably is 6098562ed9df ("selftests/bpf: Fix "dubious pointer
> arithmetic" test")
> https://lore.kernel.org/all/20220803145005.2385039-6-ovidiu.panait@windriver.com/
> Could you try reverting that?
> 
> The patch didn't have a Fixes: tags, because the bugfix it refers to was
> merged at the same time. That bugfix is upstream commit b02709587ea3
> ("bpf: Fix propagation of 32-bit signed bounds from 64-bit bounds.")
> 
> Since b02709587ea3 was only backported down to 5.10, this fix shouldn't be
> in 5.4. Sorry for not catching this earlier.

Can you send in a revert for this?

thanks,

greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[RAJESH DASARI]

[-- Attachment #1: Type: text/plain, Size: 2679 bytes --]

Hi ,

I did some more tests ,  Please find the observation below.

step 1:  On v5.4.210 kernel , I reverted  only commit  bpf: Verifer,
adjust_scalar_min_max_vals to always call update_reg_bounds()
7c1134c7da997523e2834dd516e2ddc51920699a , compiled the kernel and
booted the system with the new kernel.
step 2:  On system with newly compiled kernel , I clone the  v54.4.210
source code  and  reverted  commit  selftests/bpf: Fix test_align
verifier log patterns  and selftests/bpf: Fix "dubious pointer
arithmetic" test , then ran the selftests, test_align test cases
execution was successful.
step 3: If i revert only selftests/bpf: Fix "dubious pointer
arithmetic" test , test cases are still failing.



Please find the attached PDF for the other scenarios which I have executed.





Thanks,
Rajesh Dasari.

On Fri, Aug 19, 2022 at 1:57 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Thu, Aug 18, 2022 at 08:55:16AM +0100, Jean-Philippe Brucker wrote:
> > On Thu, Aug 18, 2022 at 07:24:03AM +0200, Greg KH wrote:
> > > On Wed, Aug 17, 2022 at 09:22:00PM +0300, RAJESH DASARI wrote:
> > > > Hi ,
> > > >
> > > > We are running bpf selftests on 5.4.210 kernel version and we see that
> > > > test case 11 of  test_align failed. Please find the below error.
> > > >
> > > > selftests: bpf: test_align
> > > > Test  11: pointer variable subtraction ... Failed to find match 16:
> > > > R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> > > > 0xfffffffc)
> > > > # func#0 @0
> > > > # 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
> > > > # 0: (61) r2 = *(u32 *)(r1 +76)
> > > > # 1: R1=ctx(id=0,off=0,imm=0) R2_w=pkt(id=0,off=0,r=0,imm=0) R10=fp0
> > > > # 1: (61) r3 = *(u32 *)(r1 +80)
> > > >
> > > > For complete errors please see the attached file. The same test case
> > > > execution was successful in the 5.4.209 version , could you please let
> > > > me know any known issue with the recent changes in 5.4.210 and how to
> > > > fix these errors.
> > >
> > > Can you use 'git bisect' to find the offending commit?
> >
> > It probably is 6098562ed9df ("selftests/bpf: Fix "dubious pointer
> > arithmetic" test")
> > https://lore.kernel.org/all/20220803145005.2385039-6-ovidiu.panait@windriver.com/
> > Could you try reverting that?
> >
> > The patch didn't have a Fixes: tags, because the bugfix it refers to was
> > merged at the same time. That bugfix is upstream commit b02709587ea3
> > ("bpf: Fix propagation of 32-bit signed bounds from 64-bit bounds.")
> >
> > Since b02709587ea3 was only backported down to 5.10, this fix shouldn't be
> > in 5.4. Sorry for not catching this earlier.
>
> Can you send in a revert for this?
>
> thanks,
>
> greg k-h

[-- Attachment #2: selftest-on-5.4.210.pdf --]
[-- Type: application/pdf, Size: 48577 bytes --]

Re: bpf selftest failed in 5.4.210 kernel

[Greg KH]

On Fri, Aug 19, 2022 at 07:20:11PM +0300, RAJESH DASARI wrote:
> Hi ,
> 
> I did some more tests ,  Please find the observation below.
> 
> step 1:  On v5.4.210 kernel , I reverted  only commit  bpf: Verifer,
> adjust_scalar_min_max_vals to always call update_reg_bounds()
> 7c1134c7da997523e2834dd516e2ddc51920699a , compiled the kernel and
> booted the system with the new kernel.
> step 2:  On system with newly compiled kernel , I clone the  v54.4.210
> source code  and  reverted  commit  selftests/bpf: Fix test_align
> verifier log patterns  and selftests/bpf: Fix "dubious pointer
> arithmetic" test , then ran the selftests, test_align test cases
> execution was successful.
> step 3: If i revert only selftests/bpf: Fix "dubious pointer
> arithmetic" test , test cases are still failing.
> 
> 
> 
> Please find the attached PDF for the other scenarios which I have executed.

For obvious reasons, we can't read random .pdf files sent to us.  Please
put it all in text.

thanks,

greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[RAJESH DASARI]

Hi,

Please find the test scenarios which I have tried.

Test 1:

Running system Kernel version (tag/commit) :  v5.4.210
Kernel source code checkout : v5.4.210
test_align test case execution status : Failure

Test 2:

Running system Kernel version (tag/commit) : v5.4.210
Kernel source code checkout : v5.4.209
test_align test case execution status : Failure

Test 3:

Running system Kernel version (tag/commit) : v5.4.209
Kernel source code checkout : v5.4.209
test_align test case execution status : Success

Test 4:

Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
avoid spamming the console with old error logs ( Kernel compiled at
this commit  and system is booted with this change)
Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
test_align verifier log patterns and selftests/bpf: Fix "dubious
pointer arithmetic" test. If I revert only the Fix "dubious pointer
arithmetic" test, the testcase still fails.
test_align test case execution status : Success

Test 5:

Running system Kernel version (tag/commit) :  v5.4.210 but reverted
commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
update_reg_bounds() )
Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
test_align verifier log patterns and selftests/bpf: Fix "dubious
pointer arithmetic" test.
test_align test case execution status : Success

Test 6 :

Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
error message updates for 32-bit right shift( Kernel compiled at this
commit  and system is booted with this change)
Kernel source code checkout : v5.4.209 or v5.4.210
test_align test case execution status : Failure

Thanks,
Rajesh Dasari.

On Sat, Aug 20, 2022 at 10:00 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Fri, Aug 19, 2022 at 07:20:11PM +0300, RAJESH DASARI wrote:
> > Hi ,
> >
> > I did some more tests ,  Please find the observation below.
> >
> > step 1:  On v5.4.210 kernel , I reverted  only commit  bpf: Verifer,
> > adjust_scalar_min_max_vals to always call update_reg_bounds()
> > 7c1134c7da997523e2834dd516e2ddc51920699a , compiled the kernel and
> > booted the system with the new kernel.
> > step 2:  On system with newly compiled kernel , I clone the  v54.4.210
> > source code  and  reverted  commit  selftests/bpf: Fix test_align
> > verifier log patterns  and selftests/bpf: Fix "dubious pointer
> > arithmetic" test , then ran the selftests, test_align test cases
> > execution was successful.
> > step 3: If i revert only selftests/bpf: Fix "dubious pointer
> > arithmetic" test , test cases are still failing.
> >
> >
> >
> > Please find the attached PDF for the other scenarios which I have executed.
>
> For obvious reasons, we can't read random .pdf files sent to us.  Please
> put it all in text.
>
> thanks,
>
> greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Greg KH]

On Mon, Aug 22, 2022 at 10:23:02PM +0300, RAJESH DASARI wrote:
> Hi,
> 
> Please find the test scenarios which I have tried.
> 
> Test 1:
> 
> Running system Kernel version (tag/commit) :  v5.4.210
> Kernel source code checkout : v5.4.210
> test_align test case execution status : Failure
> 
> Test 2:
> 
> Running system Kernel version (tag/commit) : v5.4.210
> Kernel source code checkout : v5.4.209
> test_align test case execution status : Failure
> 
> Test 3:
> 
> Running system Kernel version (tag/commit) : v5.4.209
> Kernel source code checkout : v5.4.209
> test_align test case execution status : Success
> 
> Test 4:
> 
> Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
> avoid spamming the console with old error logs ( Kernel compiled at
> this commit  and system is booted with this change)
> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> test_align verifier log patterns and selftests/bpf: Fix "dubious
> pointer arithmetic" test. If I revert only the Fix "dubious pointer
> arithmetic" test, the testcase still fails.
> test_align test case execution status : Success
> 
> Test 5:
> 
> Running system Kernel version (tag/commit) :  v5.4.210 but reverted
> commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
> update_reg_bounds() )
> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> test_align verifier log patterns and selftests/bpf: Fix "dubious
> pointer arithmetic" test.
> test_align test case execution status : Success
> 
> Test 6 :
> 
> Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
> error message updates for 32-bit right shift( Kernel compiled at this
> commit  and system is booted with this change)
> Kernel source code checkout : v5.4.209 or v5.4.210
> test_align test case execution status : Failure

I'm sorry, but I don't know what to do with this report at all.

Is there some failure somewhere?  If you use 'git bisect' do you find
the offending commit?

confused,

greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[RAJESH DASARI]

Sorry for the confusion, results are indeed confusing to me .
If I try with git bisect I get

git bisect bad
9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit

If I  try to test myself with multiple test scenarios(I have mentioned
in  the previous mails) for the bad commits , I see that bad commits
are
bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()
selftests/bpf: Fix test_align verifier log patterns
selftests/bpf: Fix "dubious pointer arithmetic" test

Thanks,
Rajesh Dasari.

On Tue, Aug 23, 2022 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Mon, Aug 22, 2022 at 10:23:02PM +0300, RAJESH DASARI wrote:
> > Hi,
> >
> > Please find the test scenarios which I have tried.
> >
> > Test 1:
> >
> > Running system Kernel version (tag/commit) :  v5.4.210
> > Kernel source code checkout : v5.4.210
> > test_align test case execution status : Failure
> >
> > Test 2:
> >
> > Running system Kernel version (tag/commit) : v5.4.210
> > Kernel source code checkout : v5.4.209
> > test_align test case execution status : Failure
> >
> > Test 3:
> >
> > Running system Kernel version (tag/commit) : v5.4.209
> > Kernel source code checkout : v5.4.209
> > test_align test case execution status : Success
> >
> > Test 4:
> >
> > Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
> > avoid spamming the console with old error logs ( Kernel compiled at
> > this commit  and system is booted with this change)
> > Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> > test_align verifier log patterns and selftests/bpf: Fix "dubious
> > pointer arithmetic" test. If I revert only the Fix "dubious pointer
> > arithmetic" test, the testcase still fails.
> > test_align test case execution status : Success
> >
> > Test 5:
> >
> > Running system Kernel version (tag/commit) :  v5.4.210 but reverted
> > commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
> > update_reg_bounds() )
> > Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> > test_align verifier log patterns and selftests/bpf: Fix "dubious
> > pointer arithmetic" test.
> > test_align test case execution status : Success
> >
> > Test 6 :
> >
> > Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
> > error message updates for 32-bit right shift( Kernel compiled at this
> > commit  and system is booted with this change)
> > Kernel source code checkout : v5.4.209 or v5.4.210
> > test_align test case execution status : Failure
>
> I'm sorry, but I don't know what to do with this report at all.
>
> Is there some failure somewhere?  If you use 'git bisect' do you find
> the offending commit?
>
> confused,
>
> greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Jean-Philippe Brucker]

On Tue, Aug 23, 2022 at 10:31:40AM +0300, RAJESH DASARI wrote:
> Sorry for the confusion, results are indeed confusing to me .
> If I try with git bisect I get
> 
> git bisect bad
> 9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit

For me bisecting points to:

(A)	7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()")

This changes the BPF verifier output and (as expected) breaks the
test_align selftest. That's why in the same series [1] another patch fixed
test_align. In v5.4.y, that patch is:

(B)	6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")

Unfortunately commit (B) addresses multiple verifier changes, not solely
(A). My guess is those changes were in series [1] and haven't been
backported to v5.4. So multiple solutions:

* Partially revert (B), only keeping the changes needed by (A)
* Revert (A) and (B)
* Add the missing commits that (B) also addresses

I don't know which, I suppose it depends on the intent behind backporting
(A). Ovidiu?

In any case 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic"
test") can be reverted, I can send that once we figure out the rest.

Thanks,
Jean

[1] https://lore.kernel.org/bpf/158507130343.15666.8018068546764556975.stgit@john-Precision-5820-Tower/

> 
> If I  try to test myself with multiple test scenarios(I have mentioned
> in  the previous mails) for the bad commits , I see that bad commits
> are
> bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()
> selftests/bpf: Fix test_align verifier log patterns
> selftests/bpf: Fix "dubious pointer arithmetic" test
> 
> Thanks,
> Rajesh Dasari.
> 
> On Tue, Aug 23, 2022 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Mon, Aug 22, 2022 at 10:23:02PM +0300, RAJESH DASARI wrote:
> > > Hi,
> > >
> > > Please find the test scenarios which I have tried.
> > >
> > > Test 1:
> > >
> > > Running system Kernel version (tag/commit) :  v5.4.210
> > > Kernel source code checkout : v5.4.210
> > > test_align test case execution status : Failure
> > >
> > > Test 2:
> > >
> > > Running system Kernel version (tag/commit) : v5.4.210
> > > Kernel source code checkout : v5.4.209
> > > test_align test case execution status : Failure
> > >
> > > Test 3:
> > >
> > > Running system Kernel version (tag/commit) : v5.4.209
> > > Kernel source code checkout : v5.4.209
> > > test_align test case execution status : Success
> > >
> > > Test 4:
> > >
> > > Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
> > > avoid spamming the console with old error logs ( Kernel compiled at
> > > this commit  and system is booted with this change)
> > > Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> > > test_align verifier log patterns and selftests/bpf: Fix "dubious
> > > pointer arithmetic" test. If I revert only the Fix "dubious pointer
> > > arithmetic" test, the testcase still fails.
> > > test_align test case execution status : Success
> > >
> > > Test 5:
> > >
> > > Running system Kernel version (tag/commit) :  v5.4.210 but reverted
> > > commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
> > > update_reg_bounds() )
> > > Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
> > > test_align verifier log patterns and selftests/bpf: Fix "dubious
> > > pointer arithmetic" test.
> > > test_align test case execution status : Success
> > >
> > > Test 6 :
> > >
> > > Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
> > > error message updates for 32-bit right shift( Kernel compiled at this
> > > commit  and system is booted with this change)
> > > Kernel source code checkout : v5.4.209 or v5.4.210
> > > test_align test case execution status : Failure
> >
> > I'm sorry, but I don't know what to do with this report at all.
> >
> > Is there some failure somewhere?  If you use 'git bisect' do you find
> > the offending commit?
> >
> > confused,
> >
> > greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Ovidiu Panait]

Hi Jean-Philippe,

On 8/23/22 21:34, Jean-Philippe Brucker wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Tue, Aug 23, 2022 at 10:31:40AM +0300, RAJESH DASARI wrote:
>> Sorry for the confusion, results are indeed confusing to me .
>> If I try with git bisect I get
>>
>> git bisect bad
>> 9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit
> For me bisecting points to:
>
> (A)     7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()")
>
> This changes the BPF verifier output and (as expected) breaks the
> test_align selftest. That's why in the same series [1] another patch fixed
> test_align. In v5.4.y, that patch is:
>
> (B)     6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")
>
> Unfortunately commit (B) addresses multiple verifier changes, not solely
> (A). My guess is those changes were in series [1] and haven't been
> backported to v5.4. So multiple solutions:
>
> * Partially revert (B), only keeping the changes needed by (A)
> * Revert (A) and (B)
> * Add the missing commits that (B) also addresses
>
> I don't know which, I suppose it depends on the intent behind backporting
> (A). Ovidiu?

The intent behind backporting 7c1134c7da99 ("bpf: Verifer, 
adjust_scalar_min_max_vals to always call update_reg_bounds()") was to 
fix CVE-2021-4159.

If we revert test 11 changes brought in by 6a9b3f0f3bad ("selftests/bpf: 
Fix test_align verifier log patterns") backport, all test_align 
testcases pass on my side:

diff --git a/tools/testing/selftests/bpf/test_align.c 
b/tools/testing/selftests/bpf/test_align.c
index c9c9bdce9d6d..4726e3eca9b2 100644
--- a/tools/testing/selftests/bpf/test_align.c
+++ b/tools/testing/selftests/bpf/test_align.c
@@ -580,18 +580,18 @@ static struct bpf_align_test tests[] = {
                         /* Adding 14 makes R6 be (4n+2) */
                         {11, 
"R6_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c))"},
                         /* Subtracting from packet pointer overflows 
ubounds */
-                       {13, 
"R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 
0x7c)"},
+                       {13, 
"R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 
0x7c))"},
                         /* New unknown value in R7 is (4n), >= 76 */
                         {15, 
"R7_w=inv(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc))"},
                         /* Adding it to packet pointer gives nice 
bounds again */
-                       {16, 
"R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 
0xfffffffc)"},
+                       {16, 
"R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 
0x7fc))"},
                         /* At the time the word size load is performed 
from R5,
                          * its total fixed offset is NET_IP_ALIGN + 
reg->off (0)
                          * which is 2.  Then the variable offset is 
(4n+2), so
                          * the total offset is 4-byte aligned and meets the
                          * load's requirements.
                          */
-                       {20, 
"R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 
0xfffffffc)"},
+                       {20, 
"R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc))"},
                 },
         },
  };

root@intel-x86-64:~/bpf# ./test_align
Test   0: mov ... PASS
Test   1: shift ... PASS
Test   2: addsub ... PASS
Test   3: mul ... PASS
Test   4: unknown shift ... PASS
Test   5: unknown mul ... PASS
Test   6: packet const offset ... PASS
Test   7: packet variable offset ... PASS
Test   8: packet variable offset 2 ... PASS
Test   9: dubious pointer arithmetic ... PASS
Test  10: variable subtraction ... PASS
Test  11: pointer variable subtraction ... PASS
Results: 12 pass 0 fail

> In any case 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic"
> test") can be reverted, I can send that once we figure out the rest.

In my testing, with [1] and [2] applied, but without [3], the following 
test_align selftest would still fail:

Test   9: dubious pointer arithmetic ... Failed to find match 9: 
R5=inv(id=0,umin_value=2,umax_value=9223372034707292158,var_off=(0x2; 
0x7fffffff7ffffffc)


[1] 7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always 
call update_reg_bounds()")

[2] 6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")

[3] 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic" test")

> Thanks,
> Jean
>
> [1] https://lore.kernel.org/bpf/158507130343.15666.8018068546764556975.stgit@john-Precision-5820-Tower/
>
>> If I  try to test myself with multiple test scenarios(I have mentioned
>> in  the previous mails) for the bad commits , I see that bad commits
>> are
>> bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()
>> selftests/bpf: Fix test_align verifier log patterns
>> selftests/bpf: Fix "dubious pointer arithmetic" test
>>
>> Thanks,
>> Rajesh Dasari.
>>
>> On Tue, Aug 23, 2022 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>>> On Mon, Aug 22, 2022 at 10:23:02PM +0300, RAJESH DASARI wrote:
>>>> Hi,
>>>>
>>>> Please find the test scenarios which I have tried.
>>>>
>>>> Test 1:
>>>>
>>>> Running system Kernel version (tag/commit) :  v5.4.210
>>>> Kernel source code checkout : v5.4.210
>>>> test_align test case execution status : Failure
>>>>
>>>> Test 2:
>>>>
>>>> Running system Kernel version (tag/commit) : v5.4.210
>>>> Kernel source code checkout : v5.4.209
>>>> test_align test case execution status : Failure
>>>>
>>>> Test 3:
>>>>
>>>> Running system Kernel version (tag/commit) : v5.4.209
>>>> Kernel source code checkout : v5.4.209
>>>> test_align test case execution status : Success
>>>>
>>>> Test 4:
>>>>
>>>> Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
>>>> avoid spamming the console with old error logs ( Kernel compiled at
>>>> this commit  and system is booted with this change)
>>>> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
>>>> test_align verifier log patterns and selftests/bpf: Fix "dubious
>>>> pointer arithmetic" test. If I revert only the Fix "dubious pointer
>>>> arithmetic" test, the testcase still fails.
>>>> test_align test case execution status : Success
>>>>
>>>> Test 5:
>>>>
>>>> Running system Kernel version (tag/commit) :  v5.4.210 but reverted
>>>> commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
>>>> update_reg_bounds() )
>>>> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
>>>> test_align verifier log patterns and selftests/bpf: Fix "dubious
>>>> pointer arithmetic" test.
>>>> test_align test case execution status : Success
>>>>
>>>> Test 6 :
>>>>
>>>> Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
>>>> error message updates for 32-bit right shift( Kernel compiled at this
>>>> commit  and system is booted with this change)
>>>> Kernel source code checkout : v5.4.209 or v5.4.210
>>>> test_align test case execution status : Failure
>>> I'm sorry, but I don't know what to do with this report at all.
>>>
>>> Is there some failure somewhere?  If you use 'git bisect' do you find
>>> the offending commit?
>>>
>>> confused,
>>>
>>> greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Ovidiu Panait]

Hi Jean-Philippe,

On 8/23/22 21:34, Jean-Philippe Brucker wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> On Tue, Aug 23, 2022 at 10:31:40AM +0300, RAJESH DASARI wrote:
>> Sorry for the confusion, results are indeed confusing to me .
>> If I try with git bisect I get
>>
>> git bisect bad
>> 9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit
> For me bisecting points to:
>
> (A)     7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()")
>
> This changes the BPF verifier output and (as expected) breaks the
> test_align selftest. That's why in the same series [1] another patch fixed
> test_align. In v5.4.y, that patch is:
>
> (B)     6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")
>
> Unfortunately commit (B) addresses multiple verifier changes, not solely
> (A). My guess is those changes were in series [1] and haven't been
> backported to v5.4. So multiple solutions:
>
> * Partially revert (B), only keeping the changes needed by (A)
> * Revert (A) and (B)
> * Add the missing commits that (B) also addresses
>
> I don't know which, I suppose it depends on the intent behind backporting
> (A). Ovidiu?
The intent behind backporting 7c1134c7da99 ("bpf: Verifer, 
adjust_scalar_min_max_vals to always call update_reg_bounds()") was to 
fix CVE-2021-4159.

If we revert test 11 changes brought in by 6a9b3f0f3bad ("selftests/bpf: 
Fix test_align verifier log patterns") backport, all test_align 
testcases pass on my side:

diff --git a/tools/testing/selftests/bpf/test_align.c 
b/tools/testing/selftests/bpf/test_align.c
index c9c9bdce9d6d..4726e3eca9b2 100644
--- a/tools/testing/selftests/bpf/test_align.c
+++ b/tools/testing/selftests/bpf/test_align.c
@@ -580,18 +580,18 @@ static struct bpf_align_test tests[] = {
                         /* Adding 14 makes R6 be (4n+2) */
                         {11, 
"R6_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c))"},
                         /* Subtracting from packet pointer overflows 
ubounds */
-                       {13, 
"R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 
0x7c)"},
+                       {13, 
"R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82; 
0x7c))"},
                         /* New unknown value in R7 is (4n), >= 76 */
                         {15, 
"R7_w=inv(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc))"},
                         /* Adding it to packet pointer gives nice 
bounds again */
-                       {16, 
"R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 
0xfffffffc)"},
+                       {16, 
"R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2; 
0x7fc))"},
                         /* At the time the word size load is performed 
from R5,
                          * its total fixed offset is NET_IP_ALIGN + 
reg->off (0)
                          * which is 2.  Then the variable offset is 
(4n+2), so
                          * the total offset is 4-byte aligned and meets 
the
                          * load's requirements.
                          */
-                       {20, 
"R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 
0xfffffffc)"},
+                       {20, 
"R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 
0x7fc))"},
                 },
         },
  };

root@intel-x86-64:~/bpf# ./test_align
Test   0: mov ... PASS
Test   1: shift ... PASS
Test   2: addsub ... PASS
Test   3: mul ... PASS
Test   4: unknown shift ... PASS
Test   5: unknown mul ... PASS
Test   6: packet const offset ... PASS
Test   7: packet variable offset ... PASS
Test   8: packet variable offset 2 ... PASS
Test   9: dubious pointer arithmetic ... PASS
Test  10: variable subtraction ... PASS
Test  11: pointer variable subtraction ... PASS
Results: 12 pass 0 fail
> In any case 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic"
> test") can be reverted, I can send that once we figure out the rest.

In my testing, with [1] and [2] applied, but without [3], the following 
test_align selftest would still fail:

Test   9: dubious pointer arithmetic ... Failed to find match 9: 
R5=inv(id=0,umin_value=2,umax_value=9223372034707292158,var_off=(0x2; 
0x7fffffff7ffffffc)


[1] 7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always 
call update_reg_bounds()")
[2] 6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")
[3] 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic" test")

Ovidiu

> Thanks,
> Jean
>
> [1] https://lore.kernel.org/bpf/158507130343.15666.8018068546764556975.stgit@john-Precision-5820-Tower/
>
>> If I  try to test myself with multiple test scenarios(I have mentioned
>> in  the previous mails) for the bad commits , I see that bad commits
>> are
>> bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()
>> selftests/bpf: Fix test_align verifier log patterns
>> selftests/bpf: Fix "dubious pointer arithmetic" test
>>
>> Thanks,
>> Rajesh Dasari.
>>
>> On Tue, Aug 23, 2022 at 10:04 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>>> On Mon, Aug 22, 2022 at 10:23:02PM +0300, RAJESH DASARI wrote:
>>>> Hi,
>>>>
>>>> Please find the test scenarios which I have tried.
>>>>
>>>> Test 1:
>>>>
>>>> Running system Kernel version (tag/commit) :  v5.4.210
>>>> Kernel source code checkout : v5.4.210
>>>> test_align test case execution status : Failure
>>>>
>>>> Test 2:
>>>>
>>>> Running system Kernel version (tag/commit) : v5.4.210
>>>> Kernel source code checkout : v5.4.209
>>>> test_align test case execution status : Failure
>>>>
>>>> Test 3:
>>>>
>>>> Running system Kernel version (tag/commit) : v5.4.209
>>>> Kernel source code checkout : v5.4.209
>>>> test_align test case execution status : Success
>>>>
>>>> Test 4:
>>>>
>>>> Running system Kernel version (tag/commit) : ACPI: APEI: Better fix to
>>>> avoid spamming the console with old error logs ( Kernel compiled at
>>>> this commit  and system is booted with this change)
>>>> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
>>>> test_align verifier log patterns and selftests/bpf: Fix "dubious
>>>> pointer arithmetic" test. If I revert only the Fix "dubious pointer
>>>> arithmetic" test, the testcase still fails.
>>>> test_align test case execution status : Success
>>>>
>>>> Test 5:
>>>>
>>>> Running system Kernel version (tag/commit) :  v5.4.210 but reverted
>>>> commit (bpf: Verifer, adjust_scalar_min_max_vals to always call
>>>> update_reg_bounds() )
>>>> Kernel source code checkout : v5.4.210 but reverted selftests/bpf: Fix
>>>> test_align verifier log patterns and selftests/bpf: Fix "dubious
>>>> pointer arithmetic" test.
>>>> test_align test case execution status : Success
>>>>
>>>> Test 6 :
>>>>
>>>> Running system Kernel version (tag/commit) : bpf: Test_verifier, #70
>>>> error message updates for 32-bit right shift( Kernel compiled at this
>>>> commit  and system is booted with this change)
>>>> Kernel source code checkout : v5.4.209 or v5.4.210
>>>> test_align test case execution status : Failure
>>> I'm sorry, but I don't know what to do with this report at all.
>>>
>>> Is there some failure somewhere?  If you use 'git bisect' do you find
>>> the offending commit?
>>>
>>> confused,
>>>
>>> greg k-h

Re: bpf selftest failed in 5.4.210 kernel

[Jean-Philippe Brucker]

On Tue, Aug 23, 2022 at 11:25:15PM +0300, Ovidiu Panait wrote:
> Hi Jean-Philippe,
> 
> On 8/23/22 21:34, Jean-Philippe Brucker wrote:
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> > 
> > On Tue, Aug 23, 2022 at 10:31:40AM +0300, RAJESH DASARI wrote:
> > > Sorry for the confusion, results are indeed confusing to me .
> > > If I try with git bisect I get
> > > 
> > > git bisect bad
> > > 9d6f67365d9cdb389fbdac2bb5b00e59e345930e is the first bad commit
> > For me bisecting points to:
> > 
> > (A)     7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always call update_reg_bounds()")
> > 
> > This changes the BPF verifier output and (as expected) breaks the
> > test_align selftest. That's why in the same series [1] another patch fixed
> > test_align. In v5.4.y, that patch is:
> > 
> > (B)     6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")
> > 
> > Unfortunately commit (B) addresses multiple verifier changes, not solely
> > (A). My guess is those changes were in series [1] and haven't been
> > backported to v5.4. So multiple solutions:
> > 
> > * Partially revert (B), only keeping the changes needed by (A)
> > * Revert (A) and (B)
> > * Add the missing commits that (B) also addresses
> > 
> > I don't know which, I suppose it depends on the intent behind backporting
> > (A). Ovidiu?
> The intent behind backporting 7c1134c7da99 ("bpf: Verifer,
> adjust_scalar_min_max_vals to always call update_reg_bounds()") was to fix
> CVE-2021-4159.
> 
> If we revert test 11 changes brought in by 6a9b3f0f3bad ("selftests/bpf: Fix
> test_align verifier log patterns") backport, all test_align testcases pass
> on my side:
> 
> diff --git a/tools/testing/selftests/bpf/test_align.c
> b/tools/testing/selftests/bpf/test_align.c
> index c9c9bdce9d6d..4726e3eca9b2 100644
> --- a/tools/testing/selftests/bpf/test_align.c
> +++ b/tools/testing/selftests/bpf/test_align.c
> @@ -580,18 +580,18 @@ static struct bpf_align_test tests[] = {
>                         /* Adding 14 makes R6 be (4n+2) */
>                         {11,
> "R6_w=inv(id=0,umin_value=14,umax_value=74,var_off=(0x2; 0x7c))"},
>                         /* Subtracting from packet pointer overflows ubounds
> */
> -                       {13, "R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82;
> 0x7c)"},
> +                       {13, "R5_w=pkt(id=1,off=0,r=8,umin_value=18446744073709551542,umax_value=18446744073709551602,var_off=(0xffffffffffffff82;
> 0x7c))"},
>                         /* New unknown value in R7 is (4n), >= 76 */
>                         {15,
> "R7_w=inv(id=0,umin_value=76,umax_value=1096,var_off=(0x0; 0x7fc))"},
>                         /* Adding it to packet pointer gives nice bounds
> again */
> -                       {16,
> "R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> 0xfffffffc)"},
> +                       {16,
> "R5_w=pkt(id=2,off=0,r=0,umin_value=2,umax_value=1082,var_off=(0x2;
> 0x7fc))"},
>                         /* At the time the word size load is performed from
> R5,
>                          * its total fixed offset is NET_IP_ALIGN + reg->off
> (0)
>                          * which is 2.  Then the variable offset is (4n+2),
> so
>                          * the total offset is 4-byte aligned and meets the
>                          * load's requirements.
>                          */
> -                       {20,
> "R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2;
> 0xfffffffc)"},
> +                       {20,
> "R5=pkt(id=2,off=0,r=4,umin_value=2,umax_value=1082,var_off=(0x2; 0x7fc))"},
>                 },
>         },
>  };
> 
> root@intel-x86-64:~/bpf# ./test_align
> Test   0: mov ... PASS
> Test   1: shift ... PASS
> Test   2: addsub ... PASS
> Test   3: mul ... PASS
> Test   4: unknown shift ... PASS
> Test   5: unknown mul ... PASS
> Test   6: packet const offset ... PASS
> Test   7: packet variable offset ... PASS
> Test   8: packet variable offset 2 ... PASS
> Test   9: dubious pointer arithmetic ... PASS
> Test  10: variable subtraction ... PASS
> Test  11: pointer variable subtraction ... PASS
> Results: 12 pass 0 fail
> > In any case 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic"
> > test") can be reverted, I can send that once we figure out the rest.
> 
> In my testing, with [1] and [2] applied, but without [3], the following
> test_align selftest would still fail:
> 
> Test   9: dubious pointer arithmetic ... Failed to find match 9:
> R5=inv(id=0,umin_value=2,umax_value=9223372034707292158,var_off=(0x2;
> 0x7fffffff7ffffffc)

Right thanks for the details, so I think the cleanest is to revert [3] and
partially revert [2], tests 11 and part of 9. I'll send that out

Thanks,
Jean

> 
> 
> [1] 7c1134c7da99 ("bpf: Verifer, adjust_scalar_min_max_vals to always call
> update_reg_bounds()")
> [2] 6a9b3f0f3bad ("selftests/bpf: Fix test_align verifier log patterns")
> [3] 6098562ed9df ("selftests/bpf: Fix "dubious pointer arithmetic" test")
> 
> Ovidiu