Flaky bpf cg_storage_* tests

Flaky bpf cg_storage_* tests

[Martin KaFai Lau]

Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to 
time. A recent example is 
https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924

Could you help to take a look? may be run it under netns and also have better 
filtering by ip/port when counting packets?

Thanks!

Re: Flaky bpf cg_storage_* tests

[Stanislav Fomichev]

On Tue, Mar 28, 2023 at 10:40 AM Martin KaFai Lau <martin.lau@linux.dev> wrote:
>
> Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to
> time. A recent example is
> https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924
>
> Could you help to take a look? may be run it under netns and also have better
> filtering by ip/port when counting packets?

Error: #43/2 cg_storage_multi/isolated
test_isolated:PASS:skel-load 0 nsec
test_isolated:PASS:parent-egress1-cg-attach 0 nsec
test_isolated:PASS:parent-egress2-cg-attach 0 nsec
test_isolated:PASS:parent-ingress-cg-attach 0 nsec
test_isolated:PASS:first-connect-send 0 nsec
test_isolated:FAIL:first-invoke invocations=2

Error: #43/3 cg_storage_multi/shared
test_shared:PASS:skel-load 0 nsec
test_shared:PASS:parent-egress1-cg-attach 0 nsec
test_shared:PASS:parent-egress2-cg-attach 0 nsec
test_shared:PASS:parent-ingress-cg-attach 0 nsec
test_shared:PASS:first-connect-send 0 nsec
test_shared:FAIL:first-invoke invocations=2

Probably because we're using tcp? And race with syn vs syn+ack
(invocatoins=1 vs invocations=2)?
YiFei, maybe we should count only pure syns?

> Thanks!

Re: Flaky bpf cg_storage_* tests

[YiFei Zhu]

On Tue, Mar 28, 2023 at 11:08 AM Stanislav Fomichev <sdf@google.com> wrote:
>
> On Tue, Mar 28, 2023 at 10:40 AM Martin KaFai Lau <martin.lau@linux.dev> wrote:
> >
> > Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to
> > time. A recent example is
> > https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924
> >
> > Could you help to take a look? may be run it under netns and also have better
> > filtering by ip/port when counting packets?
>
> Error: #43/2 cg_storage_multi/isolated
> test_isolated:PASS:skel-load 0 nsec
> test_isolated:PASS:parent-egress1-cg-attach 0 nsec
> test_isolated:PASS:parent-egress2-cg-attach 0 nsec
> test_isolated:PASS:parent-ingress-cg-attach 0 nsec
> test_isolated:PASS:first-connect-send 0 nsec
> test_isolated:FAIL:first-invoke invocations=2
>
> Error: #43/3 cg_storage_multi/shared
> test_shared:PASS:skel-load 0 nsec
> test_shared:PASS:parent-egress1-cg-attach 0 nsec
> test_shared:PASS:parent-egress2-cg-attach 0 nsec
> test_shared:PASS:parent-ingress-cg-attach 0 nsec
> test_shared:PASS:first-connect-send 0 nsec
> test_shared:FAIL:first-invoke invocations=2
>
> Probably because we're using tcp? And race with syn vs syn+ack
> (invocatoins=1 vs invocations=2)?

I don't remember what I wrote in the test :)

Nope it's not TCP. I see line 65:
  server_fd = start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);

However I see line 169:
  * Assert that there is three runs, two with parent cgroup egress and
  * one with parent cgroup ingress, stored in separate parent storages.

Expected 3 got 2, is it possible we are racing against ingress?

> YiFei, maybe we should count only pure syns?
>
> > Thanks!

Re: Flaky bpf cg_storage_* tests

[Stanislav Fomichev]

On Tue, Mar 28, 2023 at 11:57 AM YiFei Zhu <zhuyifei@google.com> wrote:
>
> On Tue, Mar 28, 2023 at 11:08 AM Stanislav Fomichev <sdf@google.com> wrote:
> >
> > On Tue, Mar 28, 2023 at 10:40 AM Martin KaFai Lau <martin.lau@linux.dev> wrote:
> > >
> > > Hi YiFei and Stan, it is observed that the cg_stroage_* tests fail from time to
> > > time. A recent example is
> > > https://github.com/kernel-patches/bpf/actions/runs/4543867424/jobs/8009943115?pr=3924
> > >
> > > Could you help to take a look? may be run it under netns and also have better
> > > filtering by ip/port when counting packets?
> >
> > Error: #43/2 cg_storage_multi/isolated
> > test_isolated:PASS:skel-load 0 nsec
> > test_isolated:PASS:parent-egress1-cg-attach 0 nsec
> > test_isolated:PASS:parent-egress2-cg-attach 0 nsec
> > test_isolated:PASS:parent-ingress-cg-attach 0 nsec
> > test_isolated:PASS:first-connect-send 0 nsec
> > test_isolated:FAIL:first-invoke invocations=2
> >
> > Error: #43/3 cg_storage_multi/shared
> > test_shared:PASS:skel-load 0 nsec
> > test_shared:PASS:parent-egress1-cg-attach 0 nsec
> > test_shared:PASS:parent-egress2-cg-attach 0 nsec
> > test_shared:PASS:parent-ingress-cg-attach 0 nsec
> > test_shared:PASS:first-connect-send 0 nsec
> > test_shared:FAIL:first-invoke invocations=2
> >
> > Probably because we're using tcp? And race with syn vs syn+ack
> > (invocatoins=1 vs invocations=2)?
>
> I don't remember what I wrote in the test :)
>
> Nope it's not TCP. I see line 65:
>   server_fd = start_server(AF_INET, SOCK_DGRAM, NULL, 0, 0);
>
> However I see line 169:
>   * Assert that there is three runs, two with parent cgroup egress and
>   * one with parent cgroup ingress, stored in separate parent storages.
>
> Expected 3 got 2, is it possible we are racing against ingress?

Not sure. Can you try to see if you can reproduce locally?

> > YiFei, maybe we should count only pure syns?
> >
> > > Thanks!