Re: [PATCH 0/8] selftests/bpf: installation and out of tree build fixes

Re: [PATCH 0/8] selftests/bpf: installation and out of tree build fixes

[Yauheni Kaliuta]

Hi, Andrii!

>>>>> On Tue, 26 May 2020 15:32:10 -0700, Andrii Nakryiko  wrote:

 > On Thu, May 21, 2020 at 11:41 PM Yauheni Kaliuta
 > <yauheni.kaliuta@redhat.com> wrote:
 >> 
 >> 
 >> Actually, a bit more needed :)

 > From the other kselftest thread, it seems like selftests are not
 > supporting builds out-of-tree. With that, wouldn't it be simpler to
 > build in tree and then just copy selftests/bpf directory to wherever
 > you need to run tests from? It would be simple and reliable. Given I
 > and probably everyone else never build and run tests out-of-tree, it's
 > just too easy to break this and you'll be constantly chasing some
 > non-obvious breakages...

 > Is there some problem with such approach?

This is `make install` ;).

I personally do not need OOT build, but since it's in the code,
I'd prefer either fix it or remove it, otherwise it's
misleading. But I have not got reply from kselftest.

 >> 
 >> >>>>> On Fri, 22 May 2020 07:13:02 +0300, Yauheni Kaliuta  wrote:
 >> 
 >> > I had a look, here are some fixes.
 >> > Yauheni Kaliuta (8):
 >> >   selftests/bpf: remove test_align from Makefile
 >> >   selftests/bpf: build bench.o for any $(OUTPUT)
 >> >   selftests/bpf: install btf .c files
 >> >   selftests/bpf: fix object files installation
 >> >   selftests/bpf: add output dir to include list
 >> >   selftests/bpf: fix urandom_read installation
 >> >   selftests/bpf: fix test.h placing for out of tree build
 >> >   selftests/bpf: factor out MKDIR rule
 >> 
 >> >  tools/testing/selftests/bpf/Makefile | 77 ++++++++++++++++++++--------
 >> >  1 file changed, 55 insertions(+), 22 deletions(-)
 >> 
 >> > --
 >> > 2.26.2
 >> 
 >> 
 >> --
 >> WBR,
 >> Yauheni Kaliuta
 >> 


-- 
WBR,
Yauheni Kaliuta

Re: [PATCH 0/8] selftests/bpf: installation and out of tree build fixes

[Andrii Nakryiko]

On Tue, May 26, 2020 at 9:52 PM Yauheni Kaliuta
<yauheni.kaliuta@redhat.com> wrote:
>
> Hi, Andrii!
>
> >>>>> On Tue, 26 May 2020 15:32:10 -0700, Andrii Nakryiko  wrote:
>
>  > On Thu, May 21, 2020 at 11:41 PM Yauheni Kaliuta
>  > <yauheni.kaliuta@redhat.com> wrote:
>  >>
>  >>
>  >> Actually, a bit more needed :)
>
>  > From the other kselftest thread, it seems like selftests are not
>  > supporting builds out-of-tree. With that, wouldn't it be simpler to
>  > build in tree and then just copy selftests/bpf directory to wherever
>  > you need to run tests from? It would be simple and reliable. Given I
>  > and probably everyone else never build and run tests out-of-tree, it's
>  > just too easy to break this and you'll be constantly chasing some
>  > non-obvious breakages...
>
>  > Is there some problem with such approach?
>
> This is `make install` ;).

So patch #2, #5, and #7 is solving just `make install` problem?..

My point is that by building in tree and then just copying everything
under selftests/bpf directory to wherever you want to "install" it
would just work. And won't require complicating already complicated
Makefile. Any problem with such approach?


>
> I personally do not need OOT build, but since it's in the code,
> I'd prefer either fix it or remove it, otherwise it's
> misleading. But I have not got reply from kselftest.
>
>  >>
>  >> >>>>> On Fri, 22 May 2020 07:13:02 +0300, Yauheni Kaliuta  wrote:
>  >>
>  >> > I had a look, here are some fixes.
>  >> > Yauheni Kaliuta (8):
>  >> >   selftests/bpf: remove test_align from Makefile
>  >> >   selftests/bpf: build bench.o for any $(OUTPUT)
>  >> >   selftests/bpf: install btf .c files
>  >> >   selftests/bpf: fix object files installation
>  >> >   selftests/bpf: add output dir to include list
>  >> >   selftests/bpf: fix urandom_read installation
>  >> >   selftests/bpf: fix test.h placing for out of tree build
>  >> >   selftests/bpf: factor out MKDIR rule
>  >>
>  >> >  tools/testing/selftests/bpf/Makefile | 77 ++++++++++++++++++++--------
>  >> >  1 file changed, 55 insertions(+), 22 deletions(-)
>  >>
>  >> > --
>  >> > 2.26.2
>  >>
>  >>
>  >> --
>  >> WBR,
>  >> Yauheni Kaliuta
>  >>
>
>
> --
> WBR,
> Yauheni Kaliuta
>

Re: [PATCH 0/8] selftests/bpf: installation and out of tree build fixes

[Yauheni Kaliuta]

Hi, Andrii!

>>>>> On Tue, 26 May 2020 22:04:35 -0700, Andrii Nakryiko  wrote:

 > On Tue, May 26, 2020 at 9:52 PM Yauheni Kaliuta
 > <yauheni.kaliuta@redhat.com> wrote:
 >> 
 >> Hi, Andrii!
 >> 
 >> >>>>> On Tue, 26 May 2020 15:32:10 -0700, Andrii Nakryiko  wrote:
 >> 
 >> > On Thu, May 21, 2020 at 11:41 PM Yauheni Kaliuta
 >> > <yauheni.kaliuta@redhat.com> wrote:
 >> >>
 >> >>
 >> >> Actually, a bit more needed :)
 >> 
 >> > From the other kselftest thread, it seems like selftests are not
 >> > supporting builds out-of-tree. With that, wouldn't it be simpler to
 >> > build in tree and then just copy selftests/bpf directory to wherever
 >> > you need to run tests from? It would be simple and reliable. Given I
 >> > and probably everyone else never build and run tests out-of-tree, it's
 >> > just too easy to break this and you'll be constantly chasing some
 >> > non-obvious breakages...
 >> 
 >> > Is there some problem with such approach?
 >> 
 >> This is `make install` ;).

 > So patch #2, #5, and #7 is solving just `make install` problem?..

No, they are fixing OOT build problems. I should have probably
split the series, oot and install fixes.

 > My point is that by building in tree and then just copying
 > everything under selftests/bpf directory to wherever you want
 > to "install" it would just work. And won't require
 > complicating already complicated Makefile. Any problem with
 > such approach?

I understand. I see only wasting of space as a problem, but
should check.

 >> 
 >> I personally do not need OOT build, but since it's in the code,
 >> I'd prefer either fix it or remove it, otherwise it's
 >> misleading. But I have not got reply from kselftest.
 >> 
 >> >>
 >> >> >>>>> On Fri, 22 May 2020 07:13:02 +0300, Yauheni Kaliuta  wrote:
 >> >>
 >> >> > I had a look, here are some fixes.
 >> >> > Yauheni Kaliuta (8):
 >> >> >   selftests/bpf: remove test_align from Makefile
 >> >> >   selftests/bpf: build bench.o for any $(OUTPUT)
 >> >> >   selftests/bpf: install btf .c files
 >> >> >   selftests/bpf: fix object files installation
 >> >> >   selftests/bpf: add output dir to include list
 >> >> >   selftests/bpf: fix urandom_read installation
 >> >> >   selftests/bpf: fix test.h placing for out of tree build
 >> >> >   selftests/bpf: factor out MKDIR rule
 >> >>
 >> >> >  tools/testing/selftests/bpf/Makefile | 77 ++++++++++++++++++++--------
 >> >> >  1 file changed, 55 insertions(+), 22 deletions(-)
 >> >>
 >> >> > --
 >> >> > 2.26.2
 >> >>
 >> >>
 >> >> --
 >> >> WBR,
 >> >> Yauheni Kaliuta
 >> >>
 >> 
 >> 
 >> --
 >> WBR,
 >> Yauheni Kaliuta
 >> 


-- 
WBR,
Yauheni Kaliuta

Re: [PATCH 0/8] selftests/bpf: installation and out of tree build fixes

[Yauheni Kaliuta]

On Wed, May 27, 2020 at 10:25 AM Yauheni Kaliuta
<yauheni.kaliuta@redhat.com> wrote:

[...]

>  > My point is that by building in tree and then just copying
>  > everything under selftests/bpf directory to wherever you want
>  > to "install" it would just work. And won't require
>  > complicating already complicated Makefile. Any problem with
>  > such approach?
>
> I understand. I see only wasting of space as a problem, but
> should check.
>

Well, it messes up with the lib.mk functionality. There must be
explicit was for customization, like it's done with OVERRIDE_TARGETS.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Yauheni Kaliuta]

Hi, Alexei!

>>>>> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov  wrote:

 > On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta
 > <yauheni.kaliuta@redhat.com> wrote:
 >> 
 >> Hi, Alexei!
 >> 
 >> >>>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov  wrote:
 >> 
 >> > On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta
 >> > <yauheni.kaliuta@redhat.com> wrote:
 >> >>
 >> >> Hi, Andrii!
 >> >>
 >> >> >>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko  wrote:
 >> >>
 >> >> > On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta
 >> >> > <yauheni.kaliuta@redhat.com> wrote:
 >> >> >>
 >> >> >> There is difference in depoying static and generated extra resource
 >> >> >> files between in/out of tree build and flavors:
 >> >> >>
 >> >> >> - in case of unflavored out-of-tree build static files are not
 >> >> >> available and must be copied as well as both static and generated
 >> >> >> files for flavored build.
 >> >> >>
 >> >> >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES
 >> >> >> is chosen in analogy to TEST_GEN_* variants.
 >> >> >>
 >> >>
 >> >> > Can we keep them together but be smarter about what needs to
 >> >> > be copied based on source/target directories? I would really
 >> >> > like to not blow up all these rules.
 >> >>
 >> >> I can try, ok, I just find it a bit more clear. But it's good to
 >> >> get some input from kselftest about OOT build in general.
 >> 
 >> > I see no value in 'make install' of selftests/bpf
 >> > and since it's broken just remove that makefile target.
 >> 
 >> Some CI systems perform testing next stage after building were
 >> build tree is not available anymore. So it's in use at the
 >> moment.

 > such CI systems can do 'cp -r' then

It's a discussion for linux-kselftest@ (added).

At the moment `make install` is generic kselftest functionality
and since bpf is part of that infra it looks a bit strange to
break it intentionally.

-- 
WBR,
Yauheni Kaliuta

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta
<yauheni.kaliuta@redhat.com> wrote:
>
> Hi, Alexei!
>
> >>>>> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov  wrote:
>
>  > On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta
>  > <yauheni.kaliuta@redhat.com> wrote:
>  >>
>  >> Hi, Alexei!
>  >>
>  >> >>>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov  wrote:
>  >>
>  >> > On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta
>  >> > <yauheni.kaliuta@redhat.com> wrote:
>  >> >>
>  >> >> Hi, Andrii!
>  >> >>
>  >> >> >>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko  wrote:
>  >> >>
>  >> >> > On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta
>  >> >> > <yauheni.kaliuta@redhat.com> wrote:
>  >> >> >>
>  >> >> >> There is difference in depoying static and generated extra resource
>  >> >> >> files between in/out of tree build and flavors:
>  >> >> >>
>  >> >> >> - in case of unflavored out-of-tree build static files are not
>  >> >> >> available and must be copied as well as both static and generated
>  >> >> >> files for flavored build.
>  >> >> >>
>  >> >> >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES
>  >> >> >> is chosen in analogy to TEST_GEN_* variants.
>  >> >> >>
>  >> >>
>  >> >> > Can we keep them together but be smarter about what needs to
>  >> >> > be copied based on source/target directories? I would really
>  >> >> > like to not blow up all these rules.
>  >> >>
>  >> >> I can try, ok, I just find it a bit more clear. But it's good to
>  >> >> get some input from kselftest about OOT build in general.
>  >>
>  >> > I see no value in 'make install' of selftests/bpf
>  >> > and since it's broken just remove that makefile target.
>  >>
>  >> Some CI systems perform testing next stage after building were
>  >> build tree is not available anymore. So it's in use at the
>  >> moment.
>
>  > such CI systems can do 'cp -r' then
>
> It's a discussion for linux-kselftest@ (added).
>
> At the moment `make install` is generic kselftest functionality
> and since bpf is part of that infra it looks a bit strange to
> break it intentionally.

selftests/bpf is only historically part of selftests.
It probably should stop using kselftest build infra all together.
We had breakages in selftests/bpf in the past only because
of changes in kselftests bits.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[shuah]

On 5/27/20 11:05 AM, Alexei Starovoitov wrote:
> On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta
> <yauheni.kaliuta@redhat.com> wrote:
>>
>> Hi, Alexei!
>>
>>>>>>> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov  wrote:
>>
>>   > On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta
>>   > <yauheni.kaliuta@redhat.com> wrote:
>>   >>
>>   >> Hi, Alexei!
>>   >>
>>   >> >>>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov  wrote:
>>   >>
>>   >> > On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta
>>   >> > <yauheni.kaliuta@redhat.com> wrote:
>>   >> >>
>>   >> >> Hi, Andrii!
>>   >> >>
>>   >> >> >>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko  wrote:
>>   >> >>
>>   >> >> > On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta
>>   >> >> > <yauheni.kaliuta@redhat.com> wrote:
>>   >> >> >>
>>   >> >> >> There is difference in depoying static and generated extra resource
>>   >> >> >> files between in/out of tree build and flavors:
>>   >> >> >>
>>   >> >> >> - in case of unflavored out-of-tree build static files are not
>>   >> >> >> available and must be copied as well as both static and generated
>>   >> >> >> files for flavored build.
>>   >> >> >>
>>   >> >> >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES
>>   >> >> >> is chosen in analogy to TEST_GEN_* variants.
>>   >> >> >>
>>   >> >>
>>   >> >> > Can we keep them together but be smarter about what needs to
>>   >> >> > be copied based on source/target directories? I would really
>>   >> >> > like to not blow up all these rules.
>>   >> >>
>>   >> >> I can try, ok, I just find it a bit more clear. But it's good to
>>   >> >> get some input from kselftest about OOT build in general.
>>   >>
>>   >> > I see no value in 'make install' of selftests/bpf
>>   >> > and since it's broken just remove that makefile target.
>>   >>
>>   >> Some CI systems perform testing next stage after building were
>>   >> build tree is not available anymore. So it's in use at the
>>   >> moment.
>>
>>   > such CI systems can do 'cp -r' then
>> >> It's a discussion for linux-kselftest@ (added).
>>
>> At the moment `make install` is generic kselftest functionality
>> and since bpf is part of that infra it looks a bit strange to
>> break it intentionally.
> 
> selftests/bpf is only historically part of selftests.
> It probably should stop using kselftest build infra all together.
> We had breakages in selftests/bpf in the past only because
> of changes in kselftests bits.
> 

The question is whether or not the breakages addresses quickly.
Also, bpf keels breaking ksleftest builds and installs because
it has dependencies on bleeding edge tools and causes problems
for kselftest users.

You are pulling me into the discussion midway and I am missing the
context for the discussion. There is another thread on this topic
where Yauheni and I have been talking about bpf install.

I would say bpf install has never really worked from kselftest
install mechanism.

Ideally all tests use kselftest common install rule to leverage
the install and not have users do individual test installs.
It isn't productive and cooperative to say let's have bpf test
do its thing. It is part of selftests and we have to figure out
how to have it consistently build and run.

It isn't building for me on Linux 5.7-rc7 at the moment, leave
alone install.

The test Makefile has to handle OUTPUT directory. Please add me
to the entire patch series especially if it changes selftests
Makefile and lib.mk. I will review and try to see if we can make
bpf install work under kselftest common infrastructure.

I recently fixed bugs in kvm test Makefile to address the install
problems for cross-builds.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Wed, May 27, 2020 at 3:01 PM shuah <shuah@kernel.org> wrote:
>
> On 5/27/20 11:05 AM, Alexei Starovoitov wrote:
> > On Wed, May 27, 2020 at 10:02 AM Yauheni Kaliuta
> > <yauheni.kaliuta@redhat.com> wrote:
> >>
> >> Hi, Alexei!
> >>
> >>>>>>> On Wed, 27 May 2020 09:48:04 -0700, Alexei Starovoitov  wrote:
> >>
> >>   > On Wed, May 27, 2020 at 12:19 AM Yauheni Kaliuta
> >>   > <yauheni.kaliuta@redhat.com> wrote:
> >>   >>
> >>   >> Hi, Alexei!
> >>   >>
> >>   >> >>>>> On Tue, 26 May 2020 22:37:39 -0700, Alexei Starovoitov  wrote:
> >>   >>
> >>   >> > On Tue, May 26, 2020 at 10:31 PM Yauheni Kaliuta
> >>   >> > <yauheni.kaliuta@redhat.com> wrote:
> >>   >> >>
> >>   >> >> Hi, Andrii!
> >>   >> >>
> >>   >> >> >>>>> On Tue, 26 May 2020 17:19:18 -0700, Andrii Nakryiko  wrote:
> >>   >> >>
> >>   >> >> > On Fri, May 22, 2020 at 1:19 AM Yauheni Kaliuta
> >>   >> >> > <yauheni.kaliuta@redhat.com> wrote:
> >>   >> >> >>
> >>   >> >> >> There is difference in depoying static and generated extra resource
> >>   >> >> >> files between in/out of tree build and flavors:
> >>   >> >> >>
> >>   >> >> >> - in case of unflavored out-of-tree build static files are not
> >>   >> >> >> available and must be copied as well as both static and generated
> >>   >> >> >> files for flavored build.
> >>   >> >> >>
> >>   >> >> >> So split the rules and variables. The name TRUNNER_EXTRA_GEN_FILES
> >>   >> >> >> is chosen in analogy to TEST_GEN_* variants.
> >>   >> >> >>
> >>   >> >>
> >>   >> >> > Can we keep them together but be smarter about what needs to
> >>   >> >> > be copied based on source/target directories? I would really
> >>   >> >> > like to not blow up all these rules.
> >>   >> >>
> >>   >> >> I can try, ok, I just find it a bit more clear. But it's good to
> >>   >> >> get some input from kselftest about OOT build in general.
> >>   >>
> >>   >> > I see no value in 'make install' of selftests/bpf
> >>   >> > and since it's broken just remove that makefile target.
> >>   >>
> >>   >> Some CI systems perform testing next stage after building were
> >>   >> build tree is not available anymore. So it's in use at the
> >>   >> moment.
> >>
> >>   > such CI systems can do 'cp -r' then
> >> >> It's a discussion for linux-kselftest@ (added).
> >>
> >> At the moment `make install` is generic kselftest functionality
> >> and since bpf is part of that infra it looks a bit strange to
> >> break it intentionally.
> >
> > selftests/bpf is only historically part of selftests.
> > It probably should stop using kselftest build infra all together.
> > We had breakages in selftests/bpf in the past only because
> > of changes in kselftests bits.
> >
>
> The question is whether or not the breakages addresses quickly.
> Also, bpf keels breaking ksleftest builds and installs because
> it has dependencies on bleeding edge tools and causes problems
> for kselftest users.
>
> You are pulling me into the discussion midway and I am missing the
> context for the discussion. There is another thread on this topic
> where Yauheni and I have been talking about bpf install.
>
> I would say bpf install has never really worked from kselftest
> install mechanism.
>
> Ideally all tests use kselftest common install rule to leverage
> the install and not have users do individual test installs.
> It isn't productive and cooperative to say let's have bpf test
> do its thing. It is part of selftests and we have to figure out
> how to have it consistently build and run.
>
> It isn't building for me on Linux 5.7-rc7 at the moment, leave
> alone install.
>
> The test Makefile has to handle OUTPUT directory. Please add me
> to the entire patch series especially if it changes selftests
> Makefile and lib.mk. I will review and try to see if we can make
> bpf install work under kselftest common infrastructure.

I prefer to keep selftests/bpf install broken.
This forced marriage between kselftests and selftests/bpf
never worked well. I think it's a time to free them up from each other.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Jiri Benc]

On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> I prefer to keep selftests/bpf install broken.
> This forced marriage between kselftests and selftests/bpf
> never worked well. I think it's a time to free them up from each other.

Alexei, it would be great if you could cooperate with other people
instead of pushing your own way. The selftests infrastructure was put
to the kernel to have one place for testing. Inventing yet another way
to add tests does not help anyone. You don't own the kernel. We're
community, we should cooperate.

 Jiri

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Greg KH]

On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > I prefer to keep selftests/bpf install broken.
> > This forced marriage between kselftests and selftests/bpf
> > never worked well. I think it's a time to free them up from each other.
> 
> Alexei, it would be great if you could cooperate with other people
> instead of pushing your own way. The selftests infrastructure was put
> to the kernel to have one place for testing. Inventing yet another way
> to add tests does not help anyone. You don't own the kernel. We're
> community, we should cooperate.

I agree, we rely on the infrastructure of the kselftests framework so
that testing systems do not have to create "custom" frameworks to handle
all of the individual variants that could easily crop up here.

Let's keep it easy for people to run and use these tests, to not do so
is to ensure that they are not used, which is the exact opposite goal of
creating tests.

thanks,

greg k-h

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > > I prefer to keep selftests/bpf install broken.
> > > This forced marriage between kselftests and selftests/bpf
> > > never worked well. I think it's a time to free them up from each other.
> > 
> > Alexei, it would be great if you could cooperate with other people
> > instead of pushing your own way. The selftests infrastructure was put
> > to the kernel to have one place for testing. Inventing yet another way
> > to add tests does not help anyone. You don't own the kernel. We're
> > community, we should cooperate.
> 
> I agree, we rely on the infrastructure of the kselftests framework so
> that testing systems do not have to create "custom" frameworks to handle
> all of the individual variants that could easily crop up here.
> 
> Let's keep it easy for people to run and use these tests, to not do so
> is to ensure that they are not used, which is the exact opposite goal of
> creating tests.

Greg,

It is easy for people (bpf developers) to run and use the tests.
Every developer runs them before submitting patches.
New tests is a hard requirement for any new features.
Maintainers run them for every push.

What I was and will push back hard is when other people (not bpf developers)
come back with an excuse that some CI system has a hard time running these
tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
The example of this is that we already have github/libbpf CI that runs
selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
paste what already works instead of burdening people (bpf developers) who run
and use existing tests. I frankly have no sympathy to folks who put their own
interest of their CI development in front of bpf community of developers.
The main job of CI is to help developers and maintainers.
Where helping means to not impose new dumb rules on developers because CI
framework is dumb. Fix CI instead.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>> I prefer to keep selftests/bpf install broken.
>>>> This forced marriage between kselftests and selftests/bpf
>>>> never worked well. I think it's a time to free them up from each other.
>>>
>>> Alexei, it would be great if you could cooperate with other people
>>> instead of pushing your own way. The selftests infrastructure was put
>>> to the kernel to have one place for testing. Inventing yet another way
>>> to add tests does not help anyone. You don't own the kernel. We're
>>> community, we should cooperate.
>>
>> I agree, we rely on the infrastructure of the kselftests framework so
>> that testing systems do not have to create "custom" frameworks to handle
>> all of the individual variants that could easily crop up here.
>>
>> Let's keep it easy for people to run and use these tests, to not do so
>> is to ensure that they are not used, which is the exact opposite goal of
>> creating tests.
> 
> Greg,
> 
> It is easy for people (bpf developers) to run and use the tests.
> Every developer runs them before submitting patches.
> New tests is a hard requirement for any new features.
> Maintainers run them for every push.
> 
> What I was and will push back hard is when other people (not bpf developers)
> come back with an excuse that some CI system has a hard time running these
> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> The example of this is that we already have github/libbpf CI that runs
> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> paste what already works instead of burdening people (bpf developers) who run
> and use existing tests. I frankly have no sympathy to folks who put their own
> interest of their CI development in front of bpf community of developers.
> The main job of CI is to help developers and maintainers.
> Where helping means to not impose new dumb rules on developers because CI
> framework is dumb. Fix CI instead.
> 

Here is what CI users are requesting:

- ability to install bpf test with other selftests using kselftest
   install. The common framework is in place and with minor changes
   to bpf test Makefile, we can make this happen. Others and myself
   are willing to work on this, so we can get bpf test coverage in
   test rings.

- be able to build and run existing tests without breaking the test
   build when new tests are added that have hard dependency on new
   versions of tools (llvm etc.). This isn't such a novel idea. We
   don't break kernel builds every single release and even when we
   require newer compiler releases. Plan the new tests with the intent
   to not break existing users and add new tests at the same time.
   We use min rev and not bleeding edge as the requirement for kernel
   build.

Requiring test rings upgrade to new versions of llvm is unreasonable.
It places undue burden on the admins to do this every single release
(may be even every rc cycle)

What is dumb about these requests and why is it not acceptable to just
bpf when all other sub-systems keep adding tests continuously using the
selftests framework so we can test the kernel better and our releases
are of better quality.

If you check the volume of tests that get added every release, you can
easily see it isn't hard.

Calling the needs of CI dumb is detrimental to kernel quality as these
rings provide a very important function. Addressing their use-case helps
get better test coverage for bpf and kernel areas that use bpf.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Yauheni Kaliuta]

Hi, Alexei,

On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> > On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> > > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > > > I prefer to keep selftests/bpf install broken.
> > > > This forced marriage between kselftests and selftests/bpf
> > > > never worked well. I think it's a time to free them up from each other.
> > >
> > > Alexei, it would be great if you could cooperate with other people
> > > instead of pushing your own way. The selftests infrastructure was put
> > > to the kernel to have one place for testing. Inventing yet another way
> > > to add tests does not help anyone. You don't own the kernel. We're
> > > community, we should cooperate.
> >
> > I agree, we rely on the infrastructure of the kselftests framework so
> > that testing systems do not have to create "custom" frameworks to handle
> > all of the individual variants that could easily crop up here.
> >
> > Let's keep it easy for people to run and use these tests, to not do so
> > is to ensure that they are not used, which is the exact opposite goal of
> > creating tests.
>
> Greg,
>
> It is easy for people (bpf developers) to run and use the tests.
> Every developer runs them before submitting patches.
> New tests is a hard requirement for any new features.
> Maintainers run them for every push.
>
> What I was and will push back hard is when other people (not bpf developers)
> come back with an excuse that some CI system has a hard time running these
> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> The example of this is that we already have github/libbpf CI that runs
> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> paste what already works instead of burdening people (bpf developers) who run
> and use existing tests. I frankly have no sympathy to folks who put their own
> interest of their CI development in front of bpf community of developers.
> The main job of CI is to help developers and maintainers.
> Where helping means to not impose new dumb rules on developers because CI
> framework is dumb. Fix CI instead.
>

Any good reason why bpf selftests, residing under selftests/, should
be an exception?
"Breakages" is not, breakages are fixable.

-- 
WBR, Yauheni

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
> > On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> > > On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> > > > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > > > > I prefer to keep selftests/bpf install broken.
> > > > > This forced marriage between kselftests and selftests/bpf
> > > > > never worked well. I think it's a time to free them up from each other.
> > > > 
> > > > Alexei, it would be great if you could cooperate with other people
> > > > instead of pushing your own way. The selftests infrastructure was put
> > > > to the kernel to have one place for testing. Inventing yet another way
> > > > to add tests does not help anyone. You don't own the kernel. We're
> > > > community, we should cooperate.
> > > 
> > > I agree, we rely on the infrastructure of the kselftests framework so
> > > that testing systems do not have to create "custom" frameworks to handle
> > > all of the individual variants that could easily crop up here.
> > > 
> > > Let's keep it easy for people to run and use these tests, to not do so
> > > is to ensure that they are not used, which is the exact opposite goal of
> > > creating tests.
> > 
> > Greg,
> > 
> > It is easy for people (bpf developers) to run and use the tests.
> > Every developer runs them before submitting patches.
> > New tests is a hard requirement for any new features.
> > Maintainers run them for every push.
> > 
> > What I was and will push back hard is when other people (not bpf developers)
> > come back with an excuse that some CI system has a hard time running these
> > tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> > The example of this is that we already have github/libbpf CI that runs
> > selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> > paste what already works instead of burdening people (bpf developers) who run
> > and use existing tests. I frankly have no sympathy to folks who put their own
> > interest of their CI development in front of bpf community of developers.
> > The main job of CI is to help developers and maintainers.
> > Where helping means to not impose new dumb rules on developers because CI
> > framework is dumb. Fix CI instead.
> > 
> 
> Here is what CI users are requesting:
> 
> - ability to install bpf test with other selftests using kselftest
>   install. The common framework is in place and with minor changes
>   to bpf test Makefile, we can make this happen. Others and myself
>   are willing to work on this, so we can get bpf test coverage in
>   test rings.

so you're saying that bpf maintainers and all bpf developers now
would need to incorporate new 'make install' step to their workflow
because some unknown CI system that is not even functional decided
to do 'make install' ?
That's exactly my point about selfish CI developers who put their
needs in front of bpf community of developers.

> - be able to build and run existing tests without breaking the test
>   build when new tests are added that have hard dependency on new
>   versions of tools (llvm etc.). This isn't such a novel idea. We
>   don't break kernel builds every single release and even when we
>   require newer compiler releases. Plan the new tests with the intent
>   to not break existing users and add new tests at the same time.
>   We use min rev and not bleeding edge as the requirement for kernel
>   build.

'existing users'? CI is not a user. CI is machine that should _help_
developers. Above two things (forcing install on humans and
breaking day-to-day tests for bpf maintainers and developers)
is the opposite of helping developers.
Please do NOT use such useless CI as an excuse.
Such CI should not be built in the first place when it slows down
the development instead of helping it.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Thu, May 28, 2020 at 08:10:57PM +0300, Yauheni Kaliuta wrote:
> Hi, Alexei,
> 
> On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> > > On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> > > > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > > > > I prefer to keep selftests/bpf install broken.
> > > > > This forced marriage between kselftests and selftests/bpf
> > > > > never worked well. I think it's a time to free them up from each other.
> > > >
> > > > Alexei, it would be great if you could cooperate with other people
> > > > instead of pushing your own way. The selftests infrastructure was put
> > > > to the kernel to have one place for testing. Inventing yet another way
> > > > to add tests does not help anyone. You don't own the kernel. We're
> > > > community, we should cooperate.
> > >
> > > I agree, we rely on the infrastructure of the kselftests framework so
> > > that testing systems do not have to create "custom" frameworks to handle
> > > all of the individual variants that could easily crop up here.
> > >
> > > Let's keep it easy for people to run and use these tests, to not do so
> > > is to ensure that they are not used, which is the exact opposite goal of
> > > creating tests.
> >
> > Greg,
> >
> > It is easy for people (bpf developers) to run and use the tests.
> > Every developer runs them before submitting patches.
> > New tests is a hard requirement for any new features.
> > Maintainers run them for every push.
> >
> > What I was and will push back hard is when other people (not bpf developers)
> > come back with an excuse that some CI system has a hard time running these
> > tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> > The example of this is that we already have github/libbpf CI that runs
> > selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> > paste what already works instead of burdening people (bpf developers) who run
> > and use existing tests. I frankly have no sympathy to folks who put their own
> > interest of their CI development in front of bpf community of developers.
> > The main job of CI is to help developers and maintainers.
> > Where helping means to not impose new dumb rules on developers because CI
> > framework is dumb. Fix CI instead.
> >
> 
> Any good reason why bpf selftests, residing under selftests/, should
> be an exception?
> "Breakages" is not, breakages are fixable.

As I said early the location of bpf selftests in tools/testing/selftests/
was a historical mistake that needs to be corrected.
There is no value in residing in that directory.
kselftest are aiming to test the kernel whereas selftests/bpf are testing
kernel, libbpf, bpftool, llvm, pahole. These are the tests for bpf ecosystem.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Yauheni Kaliuta]

Hi!

On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:

[...]
> >
> > Here is what CI users are requesting:
> >
> > - ability to install bpf test with other selftests using kselftest
> >   install. The common framework is in place and with minor changes
> >   to bpf test Makefile, we can make this happen. Others and myself
> >   are willing to work on this, so we can get bpf test coverage in
> >   test rings.
>
> so you're saying that bpf maintainers and all bpf developers now
> would need to incorporate new 'make install' step to their workflow
> because some unknown CI system that is not even functional decided
> to do 'make install' ?
> That's exactly my point about selfish CI developers who put their
> needs in front of bpf community of developers.

May be, it can work both ways to make everybody happy :) (I haven't
seen yet fundamental problems why not).

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Thu, May 28, 2020 at 11:29 AM Yauheni Kaliuta
<yauheni.kaliuta@redhat.com> wrote:
>
> Hi!
>
> On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>
> [...]
> > >
> > > Here is what CI users are requesting:
> > >
> > > - ability to install bpf test with other selftests using kselftest
> > >   install. The common framework is in place and with minor changes
> > >   to bpf test Makefile, we can make this happen. Others and myself
> > >   are willing to work on this, so we can get bpf test coverage in
> > >   test rings.
> >
> > so you're saying that bpf maintainers and all bpf developers now
> > would need to incorporate new 'make install' step to their workflow
> > because some unknown CI system that is not even functional decided
> > to do 'make install' ?
> > That's exactly my point about selfish CI developers who put their
> > needs in front of bpf community of developers.
>
> May be, it can work both ways to make everybody happy :) (I haven't
> seen yet fundamental problems why not).

then stop pretending and do 'cp -r' for your CI as you were suggested
several times already.
It works just fine for libbpf CI. Feel free to copy those scripts.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>> never worked well. I think it's a time to free them up from each other.
>>>>>
>>>>> Alexei, it would be great if you could cooperate with other people
>>>>> instead of pushing your own way. The selftests infrastructure was put
>>>>> to the kernel to have one place for testing. Inventing yet another way
>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>> community, we should cooperate.
>>>>
>>>> I agree, we rely on the infrastructure of the kselftests framework so
>>>> that testing systems do not have to create "custom" frameworks to handle
>>>> all of the individual variants that could easily crop up here.
>>>>
>>>> Let's keep it easy for people to run and use these tests, to not do so
>>>> is to ensure that they are not used, which is the exact opposite goal of
>>>> creating tests.
>>>
>>> Greg,
>>>
>>> It is easy for people (bpf developers) to run and use the tests.
>>> Every developer runs them before submitting patches.
>>> New tests is a hard requirement for any new features.
>>> Maintainers run them for every push.
>>>
>>> What I was and will push back hard is when other people (not bpf developers)
>>> come back with an excuse that some CI system has a hard time running these
>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
>>> The example of this is that we already have github/libbpf CI that runs
>>> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
>>> paste what already works instead of burdening people (bpf developers) who run
>>> and use existing tests. I frankly have no sympathy to folks who put their own
>>> interest of their CI development in front of bpf community of developers.
>>> The main job of CI is to help developers and maintainers.
>>> Where helping means to not impose new dumb rules on developers because CI
>>> framework is dumb. Fix CI instead.
>>>
>>
>> Here is what CI users are requesting:
>>
>> - ability to install bpf test with other selftests using kselftest
>>    install. The common framework is in place and with minor changes
>>    to bpf test Makefile, we can make this happen. Others and myself
>>    are willing to work on this, so we can get bpf test coverage in
>>    test rings.
> 
> so you're saying that bpf maintainers and all bpf developers now
> would need to incorporate new 'make install' step to their workflow
> because some unknown CI system that is not even functional decided
> to do 'make install' ?
> That's exactly my point about selfish CI developers who put their
> needs in front of bpf community of developers.
> 

There is no need change bpf maintainer and developer workflow. You
don't have to use install option. Kselftest framework doesn't
require a specific workflow and you can:

1. Build and run your tests from bpf directory if you choose to
2. Install to run on different target.

Adding install install option requires a change to bpf Makefile
only to copy test that are built to install directory.

make kselftest-install from the main kernel Makefile in conjunction
with selftests Makefile and lib.mk will handle all of that.

Sounds like there is a misunderstanding that bpf maintainer/developer
workflow will have to change to support install. That is not the case.
The reason kselftest exists on the first place is to have common
framework to take care of build/run/install as a common layer so
individual test writers don't have to worry about these details
and write tests instead.

>> - be able to build and run existing tests without breaking the test
>>    build when new tests are added that have hard dependency on new
>>    versions of tools (llvm etc.). This isn't such a novel idea. We
>>    don't break kernel builds every single release and even when we
>>    require newer compiler releases. Plan the new tests with the intent
>>    to not break existing users and add new tests at the same time.
>>    We use min rev and not bleeding edge as the requirement for kernel
>>    build.
> 
> 'existing users'? 

I said existing tests not users. When you add new bpf tests, existing
tests should continue to build and run without dependency on new revs
of llvm.

CI is not a user. CI is machine that should _help_
> developers. Above two things (forcing install on humans and
> breaking day-to-day tests for bpf maintainers and developers)
> is the opposite of helping developers.
> Please do NOT use such useless CI as an excuse.
> Such CI should not be built in the first place when it slows down
> the development instead of helping it.
> 
LKFT test ring runs selftests now and Kernel CI will be soon. Various
users run selftests in their own environments. LKFT ring admins have
to build and install tests whether it is automated or not and look at
the results. If bpf test doesn't build and/or installed, it won't run
on test rings that qualify stable/next/main releases.

I don't understand why CI use-case is useless.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 12:34 PM, Alexei Starovoitov wrote:
> On Thu, May 28, 2020 at 11:29 AM Yauheni Kaliuta
> <yauheni.kaliuta@redhat.com> wrote:
>>
>> Hi!
>>
>> On Thu, May 28, 2020 at 9:16 PM Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>>
>>> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>>
>> [...]
>>>>
>>>> Here is what CI users are requesting:
>>>>
>>>> - ability to install bpf test with other selftests using kselftest
>>>>    install. The common framework is in place and with minor changes
>>>>    to bpf test Makefile, we can make this happen. Others and myself
>>>>    are willing to work on this, so we can get bpf test coverage in
>>>>    test rings.
>>>
>>> so you're saying that bpf maintainers and all bpf developers now
>>> would need to incorporate new 'make install' step to their workflow
>>> because some unknown CI system that is not even functional decided
>>> to do 'make install' ?
>>> That's exactly my point about selfish CI developers who put their
>>> needs in front of bpf community of developers.
>>
>> May be, it can work both ways to make everybody happy :) (I haven't
>> seen yet fundamental problems why not).
> 
> then stop pretending and do 'cp -r' for your CI as you were suggested
> several times already.
> It works just fine for libbpf CI. Feel free to copy those scripts.
> 

Yauheni,

Let's drop discussing this patch set. I don't have all the patches
and changes in this to weigh in. There is no need to add cp -r at all.

I want to take the discussion one level up to the use-cases.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
> Hi, Alexei,
> 
> On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>> I prefer to keep selftests/bpf install broken.
>>>>> This forced marriage between kselftests and selftests/bpf
>>>>> never worked well. I think it's a time to free them up from each other.
>>>>
>>>> Alexei, it would be great if you could cooperate with other people
>>>> instead of pushing your own way. The selftests infrastructure was put
>>>> to the kernel to have one place for testing. Inventing yet another way
>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>> community, we should cooperate.
>>>
>>> I agree, we rely on the infrastructure of the kselftests framework so
>>> that testing systems do not have to create "custom" frameworks to handle
>>> all of the individual variants that could easily crop up here.
>>>
>>> Let's keep it easy for people to run and use these tests, to not do so
>>> is to ensure that they are not used, which is the exact opposite goal of
>>> creating tests.
>>
>> Greg,
>>
>> It is easy for people (bpf developers) to run and use the tests.
>> Every developer runs them before submitting patches.
>> New tests is a hard requirement for any new features.
>> Maintainers run them for every push.
>>
>> What I was and will push back hard is when other people (not bpf developers)
>> come back with an excuse that some CI system has a hard time running these
>> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
>> The example of this is that we already have github/libbpf CI that runs
>> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
>> paste what already works instead of burdening people (bpf developers) who run
>> and use existing tests. I frankly have no sympathy to folks who put their own
>> interest of their CI development in front of bpf community of developers.
>> The main job of CI is to help developers and maintainers.
>> Where helping means to not impose new dumb rules on developers because CI
>> framework is dumb. Fix CI instead.
>>
> 
> Any good reason why bpf selftests, residing under selftests/, should
> be an exception?
> "Breakages" is not, breakages are fixable.
> 

Let's not talk about moving tests. I don't want to discuss that until
we are all on the same page on what is the problem in adding install
support to bpf Makefile.

It is possible that there is a misunderstanding that bpf maintainer
and developer workflow will change. Which is definitely not needed.
If this patch series requires it, it isn't correct and needs to be
reworked.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Alexei Starovoitov]

On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
> On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
> > On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
> > > On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
> > > > On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> > > > > On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> > > > > > On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> > > > > > > I prefer to keep selftests/bpf install broken.
> > > > > > > This forced marriage between kselftests and selftests/bpf
> > > > > > > never worked well. I think it's a time to free them up from each other.
> > > > > > 
> > > > > > Alexei, it would be great if you could cooperate with other people
> > > > > > instead of pushing your own way. The selftests infrastructure was put
> > > > > > to the kernel to have one place for testing. Inventing yet another way
> > > > > > to add tests does not help anyone. You don't own the kernel. We're
> > > > > > community, we should cooperate.
> > > > > 
> > > > > I agree, we rely on the infrastructure of the kselftests framework so
> > > > > that testing systems do not have to create "custom" frameworks to handle
> > > > > all of the individual variants that could easily crop up here.
> > > > > 
> > > > > Let's keep it easy for people to run and use these tests, to not do so
> > > > > is to ensure that they are not used, which is the exact opposite goal of
> > > > > creating tests.
> > > > 
> > > > Greg,
> > > > 
> > > > It is easy for people (bpf developers) to run and use the tests.
> > > > Every developer runs them before submitting patches.
> > > > New tests is a hard requirement for any new features.
> > > > Maintainers run them for every push.
> > > > 
> > > > What I was and will push back hard is when other people (not bpf developers)
> > > > come back with an excuse that some CI system has a hard time running these
> > > > tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> > > > The example of this is that we already have github/libbpf CI that runs
> > > > selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> > > > paste what already works instead of burdening people (bpf developers) who run
> > > > and use existing tests. I frankly have no sympathy to folks who put their own
> > > > interest of their CI development in front of bpf community of developers.
> > > > The main job of CI is to help developers and maintainers.
> > > > Where helping means to not impose new dumb rules on developers because CI
> > > > framework is dumb. Fix CI instead.
> > > > 
> > > 
> > > Here is what CI users are requesting:
> > > 
> > > - ability to install bpf test with other selftests using kselftest
> > >    install. The common framework is in place and with minor changes
> > >    to bpf test Makefile, we can make this happen. Others and myself
> > >    are willing to work on this, so we can get bpf test coverage in
> > >    test rings.
> > 
> > so you're saying that bpf maintainers and all bpf developers now
> > would need to incorporate new 'make install' step to their workflow
> > because some unknown CI system that is not even functional decided
> > to do 'make install' ?
> > That's exactly my point about selfish CI developers who put their
> > needs in front of bpf community of developers.
> > 
> 
> There is no need change bpf maintainer and developer workflow. You
> don't have to use install option. Kselftest framework doesn't
> require a specific workflow and you can:
> 
> 1. Build and run your tests from bpf directory if you choose to
> 2. Install to run on different target.
> 
> Adding install install option requires a change to bpf Makefile
> only to copy test that are built to install directory.
> 
> make kselftest-install from the main kernel Makefile in conjunction
> with selftests Makefile and lib.mk will handle all of that.
> 
> Sounds like there is a misunderstanding that bpf maintainer/developer
> workflow will have to change to support install. That is not the case.
> The reason kselftest exists on the first place is to have common
> framework to take care of build/run/install as a common layer so
> individual test writers don't have to worry about these details
> and write tests instead.

I don't think you understand the 'make install' implications.
Not doing 'make install' means that all the Makefile changes that
Yauheni is proposing will immediately bit rot.
People are constantly adding new tests and changing makefile.
'make install' _will_ be broken instantly unless _humans_ incorporate
it in their patch development process and maintainer incorporate
that step into their workflow as well.

Ohh, but don't worry about this broken 'make install' is not an answer.
It's broken now and I don't want to see patches that move it
from one broken state into another broken state and at the same time
add complexity to it.

That's very different from 'make install' doing 'cp -r' of the whole dir.
In such case the chances of it going stale and broken are much lower.

> > > - be able to build and run existing tests without breaking the test
> > >    build when new tests are added that have hard dependency on new
> > >    versions of tools (llvm etc.). This isn't such a novel idea. We
> > >    don't break kernel builds every single release and even when we
> > >    require newer compiler releases. Plan the new tests with the intent
> > >    to not break existing users and add new tests at the same time.
> > >    We use min rev and not bleeding edge as the requirement for kernel
> > >    build.
> > 
> > 'existing users'?
> 
> I said existing tests not users. When you add new bpf tests, existing
> tests should continue to build and run without dependency on new revs
> of llvm.

Who said that existing test stop building with new llvm ?
Please check your facts.

> If bpf test doesn't build and/or installed, it won't run
> on test rings that qualify stable/next/main releases.

That's the case today and I prefer to keep it this way.
stable folks ignored our recommendations on how selftests/bpf should
be run, so please don't come up with your ways of doing it and
complain that something doesn't work.

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Yauheni Kaliuta]

On Thu, May 28, 2020 at 10:09 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
> > Hi, Alexei,
> >
> > On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> >>
> >> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
> >>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
> >>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
> >>>>> I prefer to keep selftests/bpf install broken.
> >>>>> This forced marriage between kselftests and selftests/bpf
> >>>>> never worked well. I think it's a time to free them up from each other.
> >>>>
> >>>> Alexei, it would be great if you could cooperate with other people
> >>>> instead of pushing your own way. The selftests infrastructure was put
> >>>> to the kernel to have one place for testing. Inventing yet another way
> >>>> to add tests does not help anyone. You don't own the kernel. We're
> >>>> community, we should cooperate.
> >>>
> >>> I agree, we rely on the infrastructure of the kselftests framework so
> >>> that testing systems do not have to create "custom" frameworks to handle
> >>> all of the individual variants that could easily crop up here.
> >>>
> >>> Let's keep it easy for people to run and use these tests, to not do so
> >>> is to ensure that they are not used, which is the exact opposite goal of
> >>> creating tests.
> >>
> >> Greg,
> >>
> >> It is easy for people (bpf developers) to run and use the tests.
> >> Every developer runs them before submitting patches.
> >> New tests is a hard requirement for any new features.
> >> Maintainers run them for every push.
> >>
> >> What I was and will push back hard is when other people (not bpf developers)
> >> come back with an excuse that some CI system has a hard time running these
> >> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
> >> The example of this is that we already have github/libbpf CI that runs
> >> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
> >> paste what already works instead of burdening people (bpf developers) who run
> >> and use existing tests. I frankly have no sympathy to folks who put their own
> >> interest of their CI development in front of bpf community of developers.
> >> The main job of CI is to help developers and maintainers.
> >> Where helping means to not impose new dumb rules on developers because CI
> >> framework is dumb. Fix CI instead.
> >>
> >
> > Any good reason why bpf selftests, residing under selftests/, should
> > be an exception?
> > "Breakages" is not, breakages are fixable.
> >
>
> Let's not talk about moving tests. I don't want to discuss that until
> we are all on the same page on what is the problem in adding install
> support to bpf Makefile.
>
> It is possible that there is a misunderstanding that bpf maintainer
> and developer workflow will change. Which is definitely not needed.
> If this patch series requires it, it isn't correct and needs to be
> reworked.

patch series does not change it. Running bpf tests in-place is one of
my own usecases, I'm not going to break it :)
The series tried to fix what does not work (install and build in
$(OUTPUT) directory) but exists in the code. I'll include you in Cc
for the respin(s) if you are interested.

The discussion is pretty much not connected to the patchset, but about
bpf selftests and the infra in general.

-- 
WBR, Yauheni

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 1:20 PM, Yauheni Kaliuta wrote:
> On Thu, May 28, 2020 at 10:09 PM Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> On 5/28/20 11:10 AM, Yauheni Kaliuta wrote:
>>> Hi, Alexei,
>>>
>>> On Thu, May 28, 2020 at 7:14 PM Alexei Starovoitov
>>> <alexei.starovoitov@gmail.com> wrote:
>>>>
>>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>>> never worked well. I think it's a time to free them up from each other.
>>>>>>
>>>>>> Alexei, it would be great if you could cooperate with other people
>>>>>> instead of pushing your own way. The selftests infrastructure was put
>>>>>> to the kernel to have one place for testing. Inventing yet another way
>>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>>> community, we should cooperate.
>>>>>
>>>>> I agree, we rely on the infrastructure of the kselftests framework so
>>>>> that testing systems do not have to create "custom" frameworks to handle
>>>>> all of the individual variants that could easily crop up here.
>>>>>
>>>>> Let's keep it easy for people to run and use these tests, to not do so
>>>>> is to ensure that they are not used, which is the exact opposite goal of
>>>>> creating tests.
>>>>
>>>> Greg,
>>>>
>>>> It is easy for people (bpf developers) to run and use the tests.
>>>> Every developer runs them before submitting patches.
>>>> New tests is a hard requirement for any new features.
>>>> Maintainers run them for every push.
>>>>
>>>> What I was and will push back hard is when other people (not bpf developers)
>>>> come back with an excuse that some CI system has a hard time running these
>>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
>>>> The example of this is that we already have github/libbpf CI that runs
>>>> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
>>>> paste what already works instead of burdening people (bpf developers) who run
>>>> and use existing tests. I frankly have no sympathy to folks who put their own
>>>> interest of their CI development in front of bpf community of developers.
>>>> The main job of CI is to help developers and maintainers.
>>>> Where helping means to not impose new dumb rules on developers because CI
>>>> framework is dumb. Fix CI instead.
>>>>
>>>
>>> Any good reason why bpf selftests, residing under selftests/, should
>>> be an exception?
>>> "Breakages" is not, breakages are fixable.
>>>
>>
>> Let's not talk about moving tests. I don't want to discuss that until
>> we are all on the same page on what is the problem in adding install
>> support to bpf Makefile.
>>
>> It is possible that there is a misunderstanding that bpf maintainer
>> and developer workflow will change. Which is definitely not needed.
>> If this patch series requires it, it isn't correct and needs to be
>> reworked.
> 
> patch series does not change it. Running bpf tests in-place is one of
> my own usecases, I'm not going to break it :)

Great. I wasn't suggesting you would. So there is no need for the
concern that bpf developer/maintainer workflow will have to change
to add install support.

> The series tried to fix what does not work (install and build in
> $(OUTPUT) directory) but exists in the code. I'll include you in Cc
> for the respin(s) if you are interested.

Please do.

> 
> The discussion is pretty much not connected to the patchset, but about
> bpf selftests and the infra in general.
> 

Yeah. We have to get a better understanding and be on the same page on
this anyway.

thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
> On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
>> On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
>>> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>>>> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
>>>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>>>> never worked well. I think it's a time to free them up from each other.
>>>>>>>
>>>>>>> Alexei, it would be great if you could cooperate with other people
>>>>>>> instead of pushing your own way. The selftests infrastructure was put
>>>>>>> to the kernel to have one place for testing. Inventing yet another way
>>>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>>>> community, we should cooperate.
>>>>>>
>>>>>> I agree, we rely on the infrastructure of the kselftests framework so
>>>>>> that testing systems do not have to create "custom" frameworks to handle
>>>>>> all of the individual variants that could easily crop up here.
>>>>>>
>>>>>> Let's keep it easy for people to run and use these tests, to not do so
>>>>>> is to ensure that they are not used, which is the exact opposite goal of
>>>>>> creating tests.
>>>>>
>>>>> Greg,
>>>>>
>>>>> It is easy for people (bpf developers) to run and use the tests.
>>>>> Every developer runs them before submitting patches.
>>>>> New tests is a hard requirement for any new features.
>>>>> Maintainers run them for every push.
>>>>>
>>>>> What I was and will push back hard is when other people (not bpf developers)
>>>>> come back with an excuse that some CI system has a hard time running these
>>>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not the tests.
>>>>> The example of this is that we already have github/libbpf CI that runs
>>>>> selftests/bpf just fine. Anyone who wants to do another CI are welcome to copy
>>>>> paste what already works instead of burdening people (bpf developers) who run
>>>>> and use existing tests. I frankly have no sympathy to folks who put their own
>>>>> interest of their CI development in front of bpf community of developers.
>>>>> The main job of CI is to help developers and maintainers.
>>>>> Where helping means to not impose new dumb rules on developers because CI
>>>>> framework is dumb. Fix CI instead.
>>>>>
>>>>
>>>> Here is what CI users are requesting:
>>>>
>>>> - ability to install bpf test with other selftests using kselftest
>>>>     install. The common framework is in place and with minor changes
>>>>     to bpf test Makefile, we can make this happen. Others and myself
>>>>     are willing to work on this, so we can get bpf test coverage in
>>>>     test rings.
>>>
>>> so you're saying that bpf maintainers and all bpf developers now
>>> would need to incorporate new 'make install' step to their workflow
>>> because some unknown CI system that is not even functional decided
>>> to do 'make install' ?
>>> That's exactly my point about selfish CI developers who put their
>>> needs in front of bpf community of developers.
>>>
>>
>> There is no need change bpf maintainer and developer workflow. You
>> don't have to use install option. Kselftest framework doesn't
>> require a specific workflow and you can:
>>
>> 1. Build and run your tests from bpf directory if you choose to
>> 2. Install to run on different target.
>>
>> Adding install install option requires a change to bpf Makefile
>> only to copy test that are built to install directory.
>>
>> make kselftest-install from the main kernel Makefile in conjunction
>> with selftests Makefile and lib.mk will handle all of that.
>>
>> Sounds like there is a misunderstanding that bpf maintainer/developer
>> workflow will have to change to support install. That is not the case.
>> The reason kselftest exists on the first place is to have common
>> framework to take care of build/run/install as a common layer so
>> individual test writers don't have to worry about these details
>> and write tests instead.
> 
> I don't think you understand the 'make install' implications.
> Not doing 'make install' means that all the Makefile changes that
> Yauheni is proposing will immediately bit rot.
> People are constantly adding new tests and changing makefile.
> 'make install' _will_ be broken instantly unless _humans_ incorporate
> it in their patch development process and maintainer incorporate
> that step into their workflow as well.
> 

I don't think so. I think if you want to work with us on this, we can
find a way. bpf isn't such a unique test that adding install will break
it.

> Ohh, but don't worry about this broken 'make install' is not an answer.
> It's broken now and I don't want to see patches that move it
> from one broken state into another broken state and at the same time
> add complexity to it.
> 

Well! What you are saying "I don;t want to collaborate to find a
solution to the problem".

> That's very different from 'make install' doing 'cp -r' of the whole dir.
> In such case the chances of it going stale and broken are much lower.
> 
>>>> - be able to build and run existing tests without breaking the test
>>>>     build when new tests are added that have hard dependency on new
>>>>     versions of tools (llvm etc.). This isn't such a novel idea. We
>>>>     don't break kernel builds every single release and even when we
>>>>     require newer compiler releases. Plan the new tests with the intent
>>>>     to not break existing users and add new tests at the same time.
>>>>     We use min rev and not bleeding edge as the requirement for kernel
>>>>     build.
>>>
>>> 'existing users'?
>>
>> I said existing tests not users. When you add new bpf tests, existing
>> tests should continue to build and run without dependency on new revs
>> of llvm.
> 
> Who said that existing test stop building with new llvm ?
> Please check your facts.
> 

I am basing this on a previous discussion on this topic. bpf test(s)
that build stop building and the solution always has been upgrade
to new tool chain. If this changed now and there is no hard tie to
bpf test building and llvm release, great.

>> If bpf test doesn't build and/or installed, it won't run
>> on test rings that qualify stable/next/main releases.
> 
> That's the case today and I prefer to keep it this way.

Why is that? Are you making a decision for the rest of the kernel
with this approach? If bpf doesn't get tested in stable test rings,
this isn't bpf problem alone. This is a kernel release issue.

> stable folks ignored our recommendations on how selftests/bpf should
> be run, so please don't come up with your ways of doing it and
> complain that something doesn't work.
> 

If you want to talk about history, bpf test was added in Oct 2016 and
by then kselftest was in use with its run_tests and install support.
So it is misleading to suggest that the framework came after the bpf
test. bpf test was added to existing kselftest framework used  by all
other tests. Expecting the existing framework to change and adapt to
a new test isn't reasonable.

When a new test is added, it has to work within the framework. Framework
can be improved and changed, however not the way you are attempting to
do in this thread. You can do this like others in the community to do
by making changes and improvements.

Sadly, it doesn't appear we are getting anywhere productive with this
thread. :(


thanks,
-- Shuah

Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan]

On 5/28/20 2:09 PM, Shuah Khan wrote:
> On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
>> On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
>>> On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
>>>> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>>>>> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
>>>>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>>>>> never worked well. I think it's a time to free them up from 
>>>>>>>>> each other.
>>>>>>>>
>>>>>>>> Alexei, it would be great if you could cooperate with other people
>>>>>>>> instead of pushing your own way. The selftests infrastructure 
>>>>>>>> was put
>>>>>>>> to the kernel to have one place for testing. Inventing yet 
>>>>>>>> another way
>>>>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>>>>> community, we should cooperate.
>>>>>>>
>>>>>>> I agree, we rely on the infrastructure of the kselftests 
>>>>>>> framework so
>>>>>>> that testing systems do not have to create "custom" frameworks to 
>>>>>>> handle
>>>>>>> all of the individual variants that could easily crop up here.
>>>>>>>
>>>>>>> Let's keep it easy for people to run and use these tests, to not 
>>>>>>> do so
>>>>>>> is to ensure that they are not used, which is the exact opposite 
>>>>>>> goal of
>>>>>>> creating tests.
>>>>>>
>>>>>> Greg,
>>>>>>
>>>>>> It is easy for people (bpf developers) to run and use the tests.
>>>>>> Every developer runs them before submitting patches.
>>>>>> New tests is a hard requirement for any new features.
>>>>>> Maintainers run them for every push.
>>>>>>
>>>>>> What I was and will push back hard is when other people (not bpf 
>>>>>> developers)
>>>>>> come back with an excuse that some CI system has a hard time 
>>>>>> running these
>>>>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not 
>>>>>> the tests.
>>>>>> The example of this is that we already have github/libbpf CI that 
>>>>>> runs
>>>>>> selftests/bpf just fine. Anyone who wants to do another CI are 
>>>>>> welcome to copy
>>>>>> paste what already works instead of burdening people (bpf 
>>>>>> developers) who run
>>>>>> and use existing tests. I frankly have no sympathy to folks who 
>>>>>> put their own
>>>>>> interest of their CI development in front of bpf community of 
>>>>>> developers.
>>>>>> The main job of CI is to help developers and maintainers.
>>>>>> Where helping means to not impose new dumb rules on developers 
>>>>>> because CI
>>>>>> framework is dumb. Fix CI instead.
>>>>>>
>>>>>
>>>>> Here is what CI users are requesting:
>>>>>
>>>>> - ability to install bpf test with other selftests using kselftest
>>>>>     install. The common framework is in place and with minor changes
>>>>>     to bpf test Makefile, we can make this happen. Others and myself
>>>>>     are willing to work on this, so we can get bpf test coverage in
>>>>>     test rings.
>>>>
>>>> so you're saying that bpf maintainers and all bpf developers now
>>>> would need to incorporate new 'make install' step to their workflow
>>>> because some unknown CI system that is not even functional decided
>>>> to do 'make install' ?
>>>> That's exactly my point about selfish CI developers who put their
>>>> needs in front of bpf community of developers.
>>>>
>>>
>>> There is no need change bpf maintainer and developer workflow. You
>>> don't have to use install option. Kselftest framework doesn't
>>> require a specific workflow and you can:
>>>
>>> 1. Build and run your tests from bpf directory if you choose to
>>> 2. Install to run on different target.
>>>
>>> Adding install install option requires a change to bpf Makefile
>>> only to copy test that are built to install directory.
>>>
>>> make kselftest-install from the main kernel Makefile in conjunction
>>> with selftests Makefile and lib.mk will handle all of that.
>>>
>>> Sounds like there is a misunderstanding that bpf maintainer/developer
>>> workflow will have to change to support install. That is not the case.
>>> The reason kselftest exists on the first place is to have common
>>> framework to take care of build/run/install as a common layer so
>>> individual test writers don't have to worry about these details
>>> and write tests instead.
>>
>> I don't think you understand the 'make install' implications.
>> Not doing 'make install' means that all the Makefile changes that
>> Yauheni is proposing will immediately bit rot.
>> People are constantly adding new tests and changing makefile.
>> 'make install' _will_ be broken instantly unless _humans_ incorporate
>> it in their patch development process and maintainer incorporate
>> that step into their workflow as well.
>>
> 
> I don't think so. I think if you want to work with us on this, we can
> find a way. bpf isn't such a unique test that adding install will break
> it.
> 
>> Ohh, but don't worry about this broken 'make install' is not an answer.
>> It's broken now and I don't want to see patches that move it
>> from one broken state into another broken state and at the same time
>> add complexity to it.
>>
> 
> Well! What you are saying "I don;t want to collaborate to find a
> solution to the problem".
> 
>> That's very different from 'make install' doing 'cp -r' of the whole dir.
>> In such case the chances of it going stale and broken are much lower.
>>
>>>>> - be able to build and run existing tests without breaking the test
>>>>>     build when new tests are added that have hard dependency on new
>>>>>     versions of tools (llvm etc.). This isn't such a novel idea. We
>>>>>     don't break kernel builds every single release and even when we
>>>>>     require newer compiler releases. Plan the new tests with the 
>>>>> intent
>>>>>     to not break existing users and add new tests at the same time.
>>>>>     We use min rev and not bleeding edge as the requirement for kernel
>>>>>     build.
>>>>
>>>> 'existing users'?
>>>
>>> I said existing tests not users. When you add new bpf tests, existing
>>> tests should continue to build and run without dependency on new revs
>>> of llvm.
>>
>> Who said that existing test stop building with new llvm ?
>> Please check your facts.
>>
> 
> I am basing this on a previous discussion on this topic. bpf test(s)
> that build stop building and the solution always has been upgrade
> to new tool chain. If this changed now and there is no hard tie to
> bpf test building and llvm release, great.
> 
>>> If bpf test doesn't build and/or installed, it won't run
>>> on test rings that qualify stable/next/main releases.
>>
>> That's the case today and I prefer to keep it this way.
> 
> Why is that? Are you making a decision for the rest of the kernel
> with this approach? If bpf doesn't get tested in stable test rings,
> this isn't bpf problem alone. This is a kernel release issue.
> 
>> stable folks ignored our recommendations on how selftests/bpf should
>> be run, so please don't come up with your ways of doing it and
>> complain that something doesn't work.
>>
> 
> If you want to talk about history, bpf test was added in Oct 2016 and
> by then kselftest was in use with its run_tests and install support.
> So it is misleading to suggest that the framework came after the bpf
> test. bpf test was added to existing kselftest framework used  by all
> other tests. Expecting the existing framework to change and adapt to
> a new test isn't reasonable.
> 
> When a new test is added, it has to work within the framework. Framework
> can be improved and changed, however not the way you are attempting to
> do in this thread. You can do this like others in the community to do
> by making changes and improvements.
> 

In any case, kselftest framework lets you override INSTALL_RULE and
define your own. Same is true for other rules: RUN_TESTS, EMIT_TESTS,
CLEAN, INSTALL

bpf does this already for CLEAN.

# override lib.mk's default rules
OVERRIDE_TARGETS := 1
override define CLEAN

If generic install rule doesn't work for you do override so bpf so
users can install it.

thanks,
-- Shuah