BPF Archive on lore.kernel.org
 help / color / Atom feed
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
To: Stanislav Fomichev <sdf@fomichev.me>
Cc: Stanislav Fomichev <sdf@google.com>,
	Networking <netdev@vger.kernel.org>, bpf <bpf@vger.kernel.org>,
	"David S. Miller" <davem@davemloft.net>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Andrii Nakryiko <andriin@fb.com>
Subject: Re: [PATCH bpf-next 3/4] selftests/bpf: test_progs: remove global fail/success counts
Date: Wed, 14 Aug 2019 13:22:34 -0700
Message-ID: <CAEf4BzY=ZrS1+7RDPFyzLheZv-C4HqBEh_iLWgnUFPh95TvRFg@mail.gmail.com> (raw)
In-Reply-To: <20190814200751.GM2820@mini-arch>

On Wed, Aug 14, 2019 at 1:07 PM Stanislav Fomichev <sdf@fomichev.me> wrote:
>
> On 08/14, Andrii Nakryiko wrote:
> > On Wed, Aug 14, 2019 at 9:48 AM Stanislav Fomichev <sdf@google.com> wrote:
> > >
> > > Now that we have a global per-test/per-environment state, there
> > > is no longer the need to have global fail/success counters
> > > (and there is no need to save/get the diff before/after the
> > > test).
> > >
> > > Cc: Andrii Nakryiko <andriin@fb.com>
> > > Signed-off-by: Stanislav Fomichev <sdf@google.com>
> > > ---

[...]

> > > +void __test__fail(const char *file, int line)
> > > +{
> > > +       if (env.test->subtest_name)
> > > +               fprintf(stderr, "%s:%s failed at %s:%d, errno=%d\n",
> >
> > Nit: let's keep <test>/<subtest> convention here as well?
> >
> > Failure doesn't always set errno, this will be quite confusing
> > sometimes. Especially for higher-level libbpf APIs, which don't set
> > errno at all.
> > If test wants to log additional information, let it do it explicitly.
> SG. Maybe we can adapt log_err from cgroup_helpers.h for error reporting
> once I move sockopt tests into test_progs.
>
> > Also, _CHECK already logs error message, so this is going to be
> > double-verbose for typical case. I'd say let's drop these error
> > messages and instead slightly extend _CHECK one with line number (it
> > already logs func name).
> Not everything goes through the _CHECK macro unfortunately, see

Well, it should (at least eventually). If existing macro doesn't cover
typical use case, we can add one that does cover.

> all the cases where I did s/error_cnt++/test__fail/. How about I
> remove the error message from _CHECK and leave it in __test_fail?

I'd keep test__fail() and test__success() as a low-level building
block. And move all the logging into corresponding high-level macro.
This still gives flexibility to do one-off crazy tests, if necessary,
while having consistent approach for everything else.

>
> > > +                       env.test->test_name, env.test->subtest_name,
> > > +                       file, line, errno);
> > > +       else
> > > +               fprintf(stderr, "%s failed at %s:%d, errno=%d\n",
> > > +                       env.test->test_name, file, line, errno);
> > > +
> > > +       env.test->fail_cnt++;
> > > +}
> > > +
> > >  struct ipv4_packet pkt_v4 = {
> > >         .eth.h_proto = __bpf_constant_htons(ETH_P_IP),
> > >         .iph.ihl = 5,
> > > @@ -150,7 +145,7 @@ int bpf_find_map(const char *test, struct bpf_object *obj, const char *name)
> > >         map = bpf_object__find_map_by_name(obj, name);
> > >         if (!map) {
> > >                 printf("%s:FAIL:map '%s' not found\n", test, name);
> > > -               error_cnt++;
> > > +               test__fail();
> > >                 return -1;
> > >         }
> > >         return bpf_map__fd(map);
> > > @@ -509,8 +504,6 @@ int main(int argc, char **argv)
> > >         stdio_hijack();
> > >         for (i = 0; i < prog_test_cnt; i++) {
> > >                 struct prog_test_def *test = &prog_test_defs[i];
> > > -               int old_pass_cnt = pass_cnt;
> > > -               int old_error_cnt = error_cnt;
> > >
> > >                 env.test = test;
> > >                 test->test_num = i + 1;
> > > @@ -525,14 +518,12 @@ int main(int argc, char **argv)
> > >                         test__end_subtest();
> > >
> > >                 test->tested = true;
> > > -               test->pass_cnt = pass_cnt - old_pass_cnt;
> > > -               test->error_cnt = error_cnt - old_error_cnt;
> > > -               if (test->error_cnt)
> > > +               if (test->fail_cnt)
> > >                         env.fail_cnt++;
> > >                 else
> > >                         env.succ_cnt++;
> > >
> > > -               dump_test_log(test, test->error_cnt);
> > > +               dump_test_log(test, test->fail_cnt);
> > >
> > >                 fprintf(env.stdout, "#%3d     %4s %s\n",
> > >                         test->test_num,
> > > @@ -546,5 +537,5 @@ int main(int argc, char **argv)
> > >         free(env.test_selector.num_set);
> > >         free(env.subtest_selector.num_set);
> > >
> > > -       return error_cnt ? EXIT_FAILURE : EXIT_SUCCESS;
> > > +       return env.fail_cnt ? EXIT_FAILURE : EXIT_SUCCESS;
> > >  }
> > > diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h
> > > index 9defd35cb6c0..7b05921784a4 100644
> > > --- a/tools/testing/selftests/bpf/test_progs.h
> > > +++ b/tools/testing/selftests/bpf/test_progs.h
> > > @@ -38,7 +38,23 @@ typedef __u16 __sum16;
> > >  #include "trace_helpers.h"
> > >  #include "flow_dissector_load.h"
> > >
> > > -struct prog_test_def;
> > > +struct prog_test_def {
> > > +       const char *test_name;
> > > +       int test_num;
> > > +       void (*run_test)(void);
> > > +       bool force_log;
> > > +       bool tested;
> > > +
> > > +       const char *subtest_name;
> > > +       int subtest_num;
> > > +
> > > +       int succ_cnt;
> > > +       int fail_cnt;
> >
> > So I'm neutral on this rename, I even considered it myself initially.
> > But keep in mind, that succ/fail in env means number of tests, while
> > test->succ/fail means number of assertions. We don't report total
> > number of failed checks anymore, so it doesn't matter, but if we ever
> > want to keep track of that at env level, it will be very confusing and
> > inconvenient.
> Point taken, I didn't think about it, let me undo the rename. I'll
> try to add a comment instead to highlight the difference.
>
> > > +
> > > +       /* store counts before subtest started */
> > > +       int old_succ_cnt;
> > > +       int old_fail_cnt;
> > > +};
> >
> > Did you move it here just to access env.test->succ_cnt in _CHECK()?
> > Maybe add test__success() counterpart to test__fail() instead?
> Yeah, good point, will do.
>
> > >
> > >  struct test_selector {
> > >         const char *name;
> > > @@ -67,13 +83,13 @@ struct test_env {
> > >         int skip_cnt; /* skipped tests */
> > >  };
> > >
> > > -extern int error_cnt;
> > > -extern int pass_cnt;
> > >  extern struct test_env env;
> > >
> > >  extern void test__force_log();
> > >  extern bool test__start_subtest(const char *name);
> > >  extern void test__skip(void);
> > > +#define test__fail() __test__fail(__FILE__, __LINE__)
> > > +extern void __test__fail(const char *file, int line);
> >
> > Given my comment above about too verbose logging, I'd say let's keep
> > this simple and have just
> >
> > extern void test__fail()
> >
> > And let _CHECK log file:line.
> See above about test__fail without _CHECK. Maybe we should do QCHECK
> as you suggested in the other email.
>
> So those lonely:
>
>         if (err) {
>                 error_cnt++;
>                 return;
>         }
>
> checks can instead be converted to:
>
>         if (QCHECK(err))
>                 return;
>
> Let me play with it a bit and see how it goes.

Yeah, give it a go. Try keeping file:line logging in macro, where it's
more natural, IMO.

>
> > >
> > >  #define MAGIC_BYTES 123
> > >
> > > @@ -96,11 +112,11 @@ extern struct ipv6_packet pkt_v6;
> > >  #define _CHECK(condition, tag, duration, format...) ({                 \
> > >         int __ret = !!(condition);                                      \
> > >         if (__ret) {                                                    \
> > > -               error_cnt++;                                            \
> > > +               test__fail();                                           \
> > >                 printf("%s:FAIL:%s ", __func__, tag);                   \
> > >                 printf(format);                                         \
> > >         } else {                                                        \
> > > -               pass_cnt++;                                             \
> > > +               env.test->succ_cnt++;                                   \
> > >                 printf("%s:PASS:%s %d nsec\n",                          \
> > >                        __func__, tag, duration);                        \
> > >         }                                                               \
> > > --
> > > 2.23.0.rc1.153.gdeed80330f-goog
> > >

  reply index

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-14 16:47 [PATCH bpf-next 0/4] selftests/bpf: test_progs: misc fixes Stanislav Fomichev
2019-08-14 16:47 ` [PATCH bpf-next 1/4] selftests/bpf: test_progs: change formatting of the condenced output Stanislav Fomichev
2019-08-14 17:00   ` Alexei Starovoitov
2019-08-14 17:07     ` Stanislav Fomichev
2019-08-14 16:47 ` [PATCH bpf-next 2/4] selftests/bpf: test_progs: test__skip Stanislav Fomichev
2019-08-14 19:22   ` Andrii Nakryiko
2019-08-14 19:30     ` Andrii Nakryiko
2019-08-14 19:53       ` Stanislav Fomichev
2019-08-14 20:01         ` Andrii Nakryiko
2019-08-16  5:16           ` Alexei Starovoitov
2019-08-16  5:23             ` Andrii Nakryiko
2019-08-14 16:47 ` [PATCH bpf-next 3/4] selftests/bpf: test_progs: remove global fail/success counts Stanislav Fomichev
2019-08-14 19:47   ` Andrii Nakryiko
2019-08-14 20:07     ` Stanislav Fomichev
2019-08-14 20:22       ` Andrii Nakryiko [this message]
2019-08-14 16:47 ` [PATCH bpf-next 4/4] selftests/bpf: test_progs: remove asserts from subtests Stanislav Fomichev
2019-08-14 19:52   ` Andrii Nakryiko

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAEf4BzY=ZrS1+7RDPFyzLheZv-C4HqBEh_iLWgnUFPh95TvRFg@mail.gmail.com' \
    --to=andrii.nakryiko@gmail.com \
    --cc=andriin@fb.com \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=netdev@vger.kernel.org \
    --cc=sdf@fomichev.me \
    --cc=sdf@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org bpf@archiver.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/ public-inbox