From: Yonghong Song <yhs@fb.com>
To: Weqaar Janjua <weqaar.janjua@gmail.com>
Cc: "Björn Töpel" <bjorn.topel@intel.com>,
bpf@vger.kernel.org, netdev@vger.kernel.org,
"Daniel Borkmann" <daniel@iogearbox.net>,
ast@kernel.org, "Magnus Karlsson" <magnus.karlsson@gmail.com>,
"Weqaar Janjua" <weqaar.a.janjua@intel.com>,
shuah@kernel.org, skhan@linuxfoundation.org,
linux-kselftest@vger.kernel.org,
"Anders Roxell" <anders.roxell@linaro.org>,
jonathan.lemon@gmail.com
Subject: Re: [PATCH bpf-next v3 1/5] selftests/bpf: xsk selftests framework
Date: Fri, 27 Nov 2020 19:13:00 -0800 [thread overview]
Message-ID: <b153b6af-6f75-d091-7022-999b01f553aa@fb.com> (raw)
In-Reply-To: <CAPLEeBZh+BEJp_k0bDQ8nmprMPqQ29JSEXCxscm5wAZQH81bAQ@mail.gmail.com>
On 11/27/20 9:54 AM, Weqaar Janjua wrote:
> On Fri, 27 Nov 2020 at 04:19, Yonghong Song <yhs@fb.com> wrote:
>>
>>
>>
>> On 11/26/20 1:22 PM, Weqaar Janjua wrote:
>>> On Thu, 26 Nov 2020 at 09:01, Björn Töpel <bjorn.topel@intel.com> wrote:
>>>>
>>>> On 2020-11-26 07:44, Yonghong Song wrote:
>>>>>
>>>> [...]
>>>>>
>>>>> What other configures I am missing?
>>>>>
>>>>> BTW, I cherry-picked the following pick from bpf tree in this experiment.
>>>>> commit e7f4a5919bf66e530e08ff352d9b78ed89574e6b (HEAD -> xsk)
>>>>> Author: Björn Töpel <bjorn.topel@intel.com>
>>>>> Date: Mon Nov 23 18:56:00 2020 +0100
>>>>>
>>>>> net, xsk: Avoid taking multiple skbuff references
>>>>>
>>>>
>>>> Hmm, I'm getting an oops, unless I cherry-pick:
>>>>
>>>> 36ccdf85829a ("net, xsk: Avoid taking multiple skbuff references")
>>>>
>>>> *AND*
>>>>
>>>> 537cf4e3cc2f ("xsk: Fix umem cleanup bug at socket destruct")
>>>>
>>>> from bpf/master.
>>>>
>>>
>>> Same as Bjorn's findings ^^^, additionally applying the second patch
>>> 537cf4e3cc2f [PASS] all tests for me
>>>
>>> PREREQUISITES: [ PASS ]
>>> SKB NOPOLL: [ PASS ]
>>> SKB POLL: [ PASS ]
>>> DRV NOPOLL: [ PASS ]
>>> DRV POLL: [ PASS ]
>>> SKB SOCKET TEARDOWN: [ PASS ]
>>> DRV SOCKET TEARDOWN: [ PASS ]
>>> SKB BIDIRECTIONAL SOCKETS: [ PASS ]
>>> DRV BIDIRECTIONAL SOCKETS: [ PASS ]
>>>
>>> With the first patch alone, as soon as we enter DRV/Native NOPOLL mode
>>> kernel panics, whereas in your case NOPOLL tests were falling with
>>> packets being *lost* as per seqnum mismatch.
>>>
>>> Can you please test this out with both patches and let us know?
>>
>> I applied both the above patches in bpf-next as well as this patch set,
>> I still see failures. I am attaching my config file. Maybe you can take
>> a look at what is the issue.
>>
> Thanks for the config, can you please confirm the compiler version,
> and resource limits i.e. stack size, memory, etc.?
root@arch-fb-vm1:~/net-next/net-next/tools/testing/selftests/bpf ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 15587
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 15587
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
compiler: gcc 8.2
>
> Only NOPOLL tests are failing for you as I see it, do the same tests
> fail every time?
In my case, with above two bpf patches applied as well, I got:
$ ./test_xsk.sh
setting up ve9127: root: 192.168.222.1/30
setting up ve4520: af_xdp4520: 192.168.222.2/30
Spec file created: veth.spec
PREREQUISITES: [ PASS ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [59], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB NOPOLL: [ FAIL ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
# NS switched: af_xdp4520
1..1
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
# End-of-tranmission frame received: PASS
# Received 10000 packets on interface ve4520
ok 1 PASS: SKB POLL
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
SKB POLL: [ PASS ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [153], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV NOPOLL: [ FAIL ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
# End-of-tranmission frame received: PASS
# Received 10000 packets on interface ve4520
ok 1 PASS: DRV POLL
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
DRV POLL: [ PASS ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Creating socket
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [54], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB SOCKET TEARDOWN: [ FAIL ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Creating socket
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [0], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV SOCKET TEARDOWN: [ FAIL ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Creating socket
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [64], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB BIDIRECTIONAL SOCKETS: [ FAIL ]
# Interface found: ve9127
# Interface found: ve4520
# NS switched: af_xdp4520
1..1
# Creating socket
# Interface [ve4520] vector [Rx]
# Interface [ve9127] vector [Tx]
# Sending 10000 packets on interface ve9127
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [83], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV BIDIRECTIONAL SOCKETS: [ FAIL ]
cleaning up...
removing link ve4520
removing ns af_xdp4520
removing spec file: veth.spec
Second runs have one previous success becoming failure.
./test_xsk.sh
setting up ve2458: root: 192.168.222.1/30
setting up ve4468: af_xdp4468: 192.168.222.2/30
[ 286.597111] IPv6: ADDRCONF(NETDEV_CHANGE): ve4468: link becomes ready
Spec file created: veth.spec
PREREQUISITES: [ PASS ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [67], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB NOPOLL: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
# End-of-tranmission frame received: PASS
# Received 10000 packets on interface ve4468
ok 1 PASS: SKB POLL
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
SKB POLL: [ PASS ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [191], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV NOPOLL: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [0], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV POLL: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Creating socket
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [0], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB SOCKET TEARDOWN: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Creating socket
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [171], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV SOCKET TEARDOWN: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Creating socket
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [124], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
SKB BIDIRECTIONAL SOCKETS: [ FAIL ]
# Interface found: ve2458
# Interface found: ve4468
# NS switched: af_xdp4468
1..1
# Creating socket
# Interface [ve4468] vector [Rx]
# Interface [ve2458] vector [Tx]
# Sending 10000 packets on interface ve2458
not ok 1 ERROR: [worker_pkt_validate] prev_pkt [195], payloadseqnum [0]
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
DRV BIDIRECTIONAL SOCKETS: [ FAIL ]
cleaning up...
removing link ve4468
removing ns af_xdp4468
removing spec file: veth.spec
>
> I will need to spend some time debugging this to have a fix.
Thanks.
>
> Thanks,
> /Weqaar
>
>>>
>>>> Can I just run test_xsk.sh at tools/testing/selftests/bpf/ directory?
>>>> This will be easier than the above for bpf developers. If it does not
>>>> work, I would like to recommend to make it work.
>>>>
>>> yes test_xsk.shis self contained, will update the instructions in there with v4.
>>
>> That will be great. Thanks!
>>
>>>
>>> Thanks,
>>> /Weqaar
>>>>
>>>> Björn
next prev parent reply other threads:[~2020-11-28 3:38 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-25 18:37 [PATCH bpf-next v3 0/5] selftests/bpf: xsk selftests Weqaar Janjua
2020-11-25 18:37 ` [PATCH bpf-next v3 1/5] selftests/bpf: xsk selftests framework Weqaar Janjua
2020-11-26 6:44 ` Yonghong Song
2020-11-26 9:01 ` Björn Töpel
2020-11-26 21:22 ` Weqaar Janjua
[not found] ` <9c73643f-0fdc-d867-6fe0-b3b8031a6cf2@fb.com>
2020-11-27 17:54 ` Weqaar Janjua
2020-11-28 3:13 ` Yonghong Song [this message]
2020-12-07 21:55 ` Weqaar Janjua
2020-12-08 3:48 ` Yonghong Song
2020-11-25 18:37 ` [PATCH bpf-next v3 2/5] selftests/bpf: xsk selftests - SKB POLL, NOPOLL Weqaar Janjua
2020-11-27 4:31 ` Yonghong Song
2020-11-27 9:01 ` Weqaar Janjua
2020-11-25 18:37 ` [PATCH bpf-next v3 3/5] selftests/bpf: xsk selftests - DRV " Weqaar Janjua
2020-11-25 18:37 ` [PATCH bpf-next v3 4/5] selftests/bpf: xsk selftests - Socket Teardown - SKB, DRV Weqaar Janjua
2020-11-25 18:37 ` [PATCH bpf-next v3 5/5] selftests/bpf: xsk selftests - Bi-directional Sockets " Weqaar Janjua
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b153b6af-6f75-d091-7022-999b01f553aa@fb.com \
--to=yhs@fb.com \
--cc=anders.roxell@linaro.org \
--cc=ast@kernel.org \
--cc=bjorn.topel@intel.com \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=jonathan.lemon@gmail.com \
--cc=linux-kselftest@vger.kernel.org \
--cc=magnus.karlsson@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=shuah@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=weqaar.a.janjua@intel.com \
--cc=weqaar.janjua@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).