Re: [PATCH] selftests/bpf: split -extras target to -static and -gen

[Shuah Khan <skhan@linuxfoundation.org>]

On 5/28/20 2:09 PM, Shuah Khan wrote:
> On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
>> On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
>>> On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
>>>> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>>>>> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
>>>>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>>>>> never worked well. I think it's a time to free them up from 
>>>>>>>>> each other.
>>>>>>>>
>>>>>>>> Alexei, it would be great if you could cooperate with other people
>>>>>>>> instead of pushing your own way. The selftests infrastructure 
>>>>>>>> was put
>>>>>>>> to the kernel to have one place for testing. Inventing yet 
>>>>>>>> another way
>>>>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>>>>> community, we should cooperate.
>>>>>>>
>>>>>>> I agree, we rely on the infrastructure of the kselftests 
>>>>>>> framework so
>>>>>>> that testing systems do not have to create "custom" frameworks to 
>>>>>>> handle
>>>>>>> all of the individual variants that could easily crop up here.
>>>>>>>
>>>>>>> Let's keep it easy for people to run and use these tests, to not 
>>>>>>> do so
>>>>>>> is to ensure that they are not used, which is the exact opposite 
>>>>>>> goal of
>>>>>>> creating tests.
>>>>>>
>>>>>> Greg,
>>>>>>
>>>>>> It is easy for people (bpf developers) to run and use the tests.
>>>>>> Every developer runs them before submitting patches.
>>>>>> New tests is a hard requirement for any new features.
>>>>>> Maintainers run them for every push.
>>>>>>
>>>>>> What I was and will push back hard is when other people (not bpf 
>>>>>> developers)
>>>>>> come back with an excuse that some CI system has a hard time 
>>>>>> running these
>>>>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not 
>>>>>> the tests.
>>>>>> The example of this is that we already have github/libbpf CI that 
>>>>>> runs
>>>>>> selftests/bpf just fine. Anyone who wants to do another CI are 
>>>>>> welcome to copy
>>>>>> paste what already works instead of burdening people (bpf 
>>>>>> developers) who run
>>>>>> and use existing tests. I frankly have no sympathy to folks who 
>>>>>> put their own
>>>>>> interest of their CI development in front of bpf community of 
>>>>>> developers.
>>>>>> The main job of CI is to help developers and maintainers.
>>>>>> Where helping means to not impose new dumb rules on developers 
>>>>>> because CI
>>>>>> framework is dumb. Fix CI instead.
>>>>>>
>>>>>
>>>>> Here is what CI users are requesting:
>>>>>
>>>>> - ability to install bpf test with other selftests using kselftest
>>>>>     install. The common framework is in place and with minor changes
>>>>>     to bpf test Makefile, we can make this happen. Others and myself
>>>>>     are willing to work on this, so we can get bpf test coverage in
>>>>>     test rings.
>>>>
>>>> so you're saying that bpf maintainers and all bpf developers now
>>>> would need to incorporate new 'make install' step to their workflow
>>>> because some unknown CI system that is not even functional decided
>>>> to do 'make install' ?
>>>> That's exactly my point about selfish CI developers who put their
>>>> needs in front of bpf community of developers.
>>>>
>>>
>>> There is no need change bpf maintainer and developer workflow. You
>>> don't have to use install option. Kselftest framework doesn't
>>> require a specific workflow and you can:
>>>
>>> 1. Build and run your tests from bpf directory if you choose to
>>> 2. Install to run on different target.
>>>
>>> Adding install install option requires a change to bpf Makefile
>>> only to copy test that are built to install directory.
>>>
>>> make kselftest-install from the main kernel Makefile in conjunction
>>> with selftests Makefile and lib.mk will handle all of that.
>>>
>>> Sounds like there is a misunderstanding that bpf maintainer/developer
>>> workflow will have to change to support install. That is not the case.
>>> The reason kselftest exists on the first place is to have common
>>> framework to take care of build/run/install as a common layer so
>>> individual test writers don't have to worry about these details
>>> and write tests instead.
>>
>> I don't think you understand the 'make install' implications.
>> Not doing 'make install' means that all the Makefile changes that
>> Yauheni is proposing will immediately bit rot.
>> People are constantly adding new tests and changing makefile.
>> 'make install' _will_ be broken instantly unless _humans_ incorporate
>> it in their patch development process and maintainer incorporate
>> that step into their workflow as well.
>>
> 
> I don't think so. I think if you want to work with us on this, we can
> find a way. bpf isn't such a unique test that adding install will break
> it.
> 
>> Ohh, but don't worry about this broken 'make install' is not an answer.
>> It's broken now and I don't want to see patches that move it
>> from one broken state into another broken state and at the same time
>> add complexity to it.
>>
> 
> Well! What you are saying "I don;t want to collaborate to find a
> solution to the problem".
> 
>> That's very different from 'make install' doing 'cp -r' of the whole dir.
>> In such case the chances of it going stale and broken are much lower.
>>
>>>>> - be able to build and run existing tests without breaking the test
>>>>>     build when new tests are added that have hard dependency on new
>>>>>     versions of tools (llvm etc.). This isn't such a novel idea. We
>>>>>     don't break kernel builds every single release and even when we
>>>>>     require newer compiler releases. Plan the new tests with the 
>>>>> intent
>>>>>     to not break existing users and add new tests at the same time.
>>>>>     We use min rev and not bleeding edge as the requirement for kernel
>>>>>     build.
>>>>
>>>> 'existing users'?
>>>
>>> I said existing tests not users. When you add new bpf tests, existing
>>> tests should continue to build and run without dependency on new revs
>>> of llvm.
>>
>> Who said that existing test stop building with new llvm ?
>> Please check your facts.
>>
> 
> I am basing this on a previous discussion on this topic. bpf test(s)
> that build stop building and the solution always has been upgrade
> to new tool chain. If this changed now and there is no hard tie to
> bpf test building and llvm release, great.
> 
>>> If bpf test doesn't build and/or installed, it won't run
>>> on test rings that qualify stable/next/main releases.
>>
>> That's the case today and I prefer to keep it this way.
> 
> Why is that? Are you making a decision for the rest of the kernel
> with this approach? If bpf doesn't get tested in stable test rings,
> this isn't bpf problem alone. This is a kernel release issue.
> 
>> stable folks ignored our recommendations on how selftests/bpf should
>> be run, so please don't come up with your ways of doing it and
>> complain that something doesn't work.
>>
> 
> If you want to talk about history, bpf test was added in Oct 2016 and
> by then kselftest was in use with its run_tests and install support.
> So it is misleading to suggest that the framework came after the bpf
> test. bpf test was added to existing kselftest framework used  by all
> other tests. Expecting the existing framework to change and adapt to
> a new test isn't reasonable.
> 
> When a new test is added, it has to work within the framework. Framework
> can be improved and changed, however not the way you are attempting to
> do in this thread. You can do this like others in the community to do
> by making changes and improvements.
> 

In any case, kselftest framework lets you override INSTALL_RULE and
define your own. Same is true for other rules: RUN_TESTS, EMIT_TESTS,
CLEAN, INSTALL

bpf does this already for CLEAN.

# override lib.mk's default rules
OVERRIDE_TARGETS := 1
override define CLEAN

If generic install rule doesn't work for you do override so bpf so
users can install it.

thanks,
-- Shuah