BPF Archive on lore.kernel.org
 help / color / Atom feed
From: Shuah Khan <skhan@linuxfoundation.org>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>,
	Yauheni Kaliuta <yauheni.kaliuta@redhat.com>
Cc: Greg KH <gregkh@linuxfoundation.org>,
	Jiri Benc <jbenc@redhat.com>, shuah <shuah@kernel.org>,
	Andrii Nakryiko <andrii.nakryiko@gmail.com>,
	bpf <bpf@vger.kernel.org>, Jiri Olsa <jolsa@redhat.com>,
	Andrii Nakryiko <andriin@fb.com>,
	Daniel Borkmann <daniel@iogearbox.net>,
Subject: Re: [PATCH] selftests/bpf: split -extras target to -static and -gen
Date: Thu, 28 May 2020 16:47:33 -0600
Message-ID: <1ae97f52-fda0-ab1a-a027-d191e5a22858@linuxfoundation.org> (raw)
In-Reply-To: <91af28bd-6f86-97d7-49bc-5b3e1be448e1@linuxfoundation.org>

On 5/28/20 2:09 PM, Shuah Khan wrote:
> On 5/28/20 1:18 PM, Alexei Starovoitov wrote:
>> On Thu, May 28, 2020 at 12:59:06PM -0600, Shuah Khan wrote:
>>> On 5/28/20 12:15 PM, Alexei Starovoitov wrote:
>>>> On Thu, May 28, 2020 at 11:07:09AM -0600, Shuah Khan wrote:
>>>>> On 5/28/20 10:14 AM, Alexei Starovoitov wrote:
>>>>>> On Thu, May 28, 2020 at 12:56:31PM +0200, Greg KH wrote:
>>>>>>> On Thu, May 28, 2020 at 10:05:57AM +0200, Jiri Benc wrote:
>>>>>>>> On Wed, 27 May 2020 15:23:13 -0700, Alexei Starovoitov wrote:
>>>>>>>>> I prefer to keep selftests/bpf install broken.
>>>>>>>>> This forced marriage between kselftests and selftests/bpf
>>>>>>>>> never worked well. I think it's a time to free them up from 
>>>>>>>>> each other.
>>>>>>>> Alexei, it would be great if you could cooperate with other people
>>>>>>>> instead of pushing your own way. The selftests infrastructure 
>>>>>>>> was put
>>>>>>>> to the kernel to have one place for testing. Inventing yet 
>>>>>>>> another way
>>>>>>>> to add tests does not help anyone. You don't own the kernel. We're
>>>>>>>> community, we should cooperate.
>>>>>>> I agree, we rely on the infrastructure of the kselftests 
>>>>>>> framework so
>>>>>>> that testing systems do not have to create "custom" frameworks to 
>>>>>>> handle
>>>>>>> all of the individual variants that could easily crop up here.
>>>>>>> Let's keep it easy for people to run and use these tests, to not 
>>>>>>> do so
>>>>>>> is to ensure that they are not used, which is the exact opposite 
>>>>>>> goal of
>>>>>>> creating tests.
>>>>>> Greg,
>>>>>> It is easy for people (bpf developers) to run and use the tests.
>>>>>> Every developer runs them before submitting patches.
>>>>>> New tests is a hard requirement for any new features.
>>>>>> Maintainers run them for every push.
>>>>>> What I was and will push back hard is when other people (not bpf 
>>>>>> developers)
>>>>>> come back with an excuse that some CI system has a hard time 
>>>>>> running these
>>>>>> tests. It's the problem of weak CI. That CI needs to be fixed. Not 
>>>>>> the tests.
>>>>>> The example of this is that we already have github/libbpf CI that 
>>>>>> runs
>>>>>> selftests/bpf just fine. Anyone who wants to do another CI are 
>>>>>> welcome to copy
>>>>>> paste what already works instead of burdening people (bpf 
>>>>>> developers) who run
>>>>>> and use existing tests. I frankly have no sympathy to folks who 
>>>>>> put their own
>>>>>> interest of their CI development in front of bpf community of 
>>>>>> developers.
>>>>>> The main job of CI is to help developers and maintainers.
>>>>>> Where helping means to not impose new dumb rules on developers 
>>>>>> because CI
>>>>>> framework is dumb. Fix CI instead.
>>>>> Here is what CI users are requesting:
>>>>> - ability to install bpf test with other selftests using kselftest
>>>>>     install. The common framework is in place and with minor changes
>>>>>     to bpf test Makefile, we can make this happen. Others and myself
>>>>>     are willing to work on this, so we can get bpf test coverage in
>>>>>     test rings.
>>>> so you're saying that bpf maintainers and all bpf developers now
>>>> would need to incorporate new 'make install' step to their workflow
>>>> because some unknown CI system that is not even functional decided
>>>> to do 'make install' ?
>>>> That's exactly my point about selfish CI developers who put their
>>>> needs in front of bpf community of developers.
>>> There is no need change bpf maintainer and developer workflow. You
>>> don't have to use install option. Kselftest framework doesn't
>>> require a specific workflow and you can:
>>> 1. Build and run your tests from bpf directory if you choose to
>>> 2. Install to run on different target.
>>> Adding install install option requires a change to bpf Makefile
>>> only to copy test that are built to install directory.
>>> make kselftest-install from the main kernel Makefile in conjunction
>>> with selftests Makefile and lib.mk will handle all of that.
>>> Sounds like there is a misunderstanding that bpf maintainer/developer
>>> workflow will have to change to support install. That is not the case.
>>> The reason kselftest exists on the first place is to have common
>>> framework to take care of build/run/install as a common layer so
>>> individual test writers don't have to worry about these details
>>> and write tests instead.
>> I don't think you understand the 'make install' implications.
>> Not doing 'make install' means that all the Makefile changes that
>> Yauheni is proposing will immediately bit rot.
>> People are constantly adding new tests and changing makefile.
>> 'make install' _will_ be broken instantly unless _humans_ incorporate
>> it in their patch development process and maintainer incorporate
>> that step into their workflow as well.
> I don't think so. I think if you want to work with us on this, we can
> find a way. bpf isn't such a unique test that adding install will break
> it.
>> Ohh, but don't worry about this broken 'make install' is not an answer.
>> It's broken now and I don't want to see patches that move it
>> from one broken state into another broken state and at the same time
>> add complexity to it.
> Well! What you are saying "I don;t want to collaborate to find a
> solution to the problem".
>> That's very different from 'make install' doing 'cp -r' of the whole dir.
>> In such case the chances of it going stale and broken are much lower.
>>>>> - be able to build and run existing tests without breaking the test
>>>>>     build when new tests are added that have hard dependency on new
>>>>>     versions of tools (llvm etc.). This isn't such a novel idea. We
>>>>>     don't break kernel builds every single release and even when we
>>>>>     require newer compiler releases. Plan the new tests with the 
>>>>> intent
>>>>>     to not break existing users and add new tests at the same time.
>>>>>     We use min rev and not bleeding edge as the requirement for kernel
>>>>>     build.
>>>> 'existing users'?
>>> I said existing tests not users. When you add new bpf tests, existing
>>> tests should continue to build and run without dependency on new revs
>>> of llvm.
>> Who said that existing test stop building with new llvm ?
>> Please check your facts.
> I am basing this on a previous discussion on this topic. bpf test(s)
> that build stop building and the solution always has been upgrade
> to new tool chain. If this changed now and there is no hard tie to
> bpf test building and llvm release, great.
>>> If bpf test doesn't build and/or installed, it won't run
>>> on test rings that qualify stable/next/main releases.
>> That's the case today and I prefer to keep it this way.
> Why is that? Are you making a decision for the rest of the kernel
> with this approach? If bpf doesn't get tested in stable test rings,
> this isn't bpf problem alone. This is a kernel release issue.
>> stable folks ignored our recommendations on how selftests/bpf should
>> be run, so please don't come up with your ways of doing it and
>> complain that something doesn't work.
> If you want to talk about history, bpf test was added in Oct 2016 and
> by then kselftest was in use with its run_tests and install support.
> So it is misleading to suggest that the framework came after the bpf
> test. bpf test was added to existing kselftest framework used  by all
> other tests. Expecting the existing framework to change and adapt to
> a new test isn't reasonable.
> When a new test is added, it has to work within the framework. Framework
> can be improved and changed, however not the way you are attempting to
> do in this thread. You can do this like others in the community to do
> by making changes and improvements.

In any case, kselftest framework lets you override INSTALL_RULE and
define your own. Same is true for other rules: RUN_TESTS, EMIT_TESTS,

bpf does this already for CLEAN.

# override lib.mk's default rules
override define CLEAN

If generic install rule doesn't work for you do override so bpf so
users can install it.

-- Shuah

  reply index

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-22  4:13 [PATCH 0/8] selftests/bpf: installation and out of tree build fixes Yauheni Kaliuta
2020-05-22  4:13 ` [PATCH 1/8] selftests/bpf: remove test_align from Makefile Yauheni Kaliuta
2020-05-26 22:13   ` Andrii Nakryiko
2020-05-22  4:13 ` [PATCH 2/8] selftests/bpf: build bench.o for any $(OUTPUT) Yauheni Kaliuta
2020-05-26 22:13   ` Andrii Nakryiko
2020-05-27  4:54     ` Yauheni Kaliuta
2020-05-22  4:13 ` [PATCH 3/8] selftests/bpf: install btf .c files Yauheni Kaliuta
2020-05-22  4:13 ` [PATCH 4/8] selftests/bpf: fix object files installation Yauheni Kaliuta
2020-05-26 22:30   ` Andrii Nakryiko
2020-05-27  5:17     ` Yauheni Kaliuta
2020-05-28 18:39       ` Andrii Nakryiko
2020-05-28 18:46         ` Yauheni Kaliuta
2020-05-22  4:13 ` [PATCH 5/8] selftests/bpf: add output dir to include list Yauheni Kaliuta
2020-05-26 22:13   ` Andrii Nakryiko
2020-05-22  4:13 ` [PATCH 6/8] selftests/bpf: fix urandom_read installation Yauheni Kaliuta
2020-05-26 22:13   ` Andrii Nakryiko
2020-05-22  4:13 ` [PATCH 7/8] selftests/bpf: fix test.h placing for out of tree build Yauheni Kaliuta
2020-05-26 22:26   ` Andrii Nakryiko
2020-05-27  5:06     ` Yauheni Kaliuta
2020-05-22  4:13 ` [PATCH 8/8] selftests/bpf: factor out MKDIR rule Yauheni Kaliuta
2020-05-26 22:29   ` Andrii Nakryiko
2020-05-27  5:07     ` Yauheni Kaliuta
2020-05-22  6:40 ` [PATCH 0/8] selftests/bpf: installation and out of tree build fixes Yauheni Kaliuta
2020-05-22  8:19   ` [PATCH] selftests/bpf: split -extras target to -static and -gen Yauheni Kaliuta
2020-05-27  0:19     ` Andrii Nakryiko
2020-05-27  5:21       ` Yauheni Kaliuta
2020-05-27  5:37         ` Alexei Starovoitov
2020-05-27  7:19           ` Yauheni Kaliuta
2020-05-27 16:48             ` Alexei Starovoitov
2020-05-27 17:02               ` Yauheni Kaliuta
2020-05-27 17:05                 ` Alexei Starovoitov
2020-05-27 22:01                   ` shuah
2020-05-27 22:23                     ` Alexei Starovoitov
2020-05-28  8:05                       ` Jiri Benc
2020-05-28 10:56                         ` Greg KH
2020-05-28 16:14                           ` Alexei Starovoitov
2020-05-28 17:07                             ` Shuah Khan
2020-05-28 18:15                               ` Alexei Starovoitov
2020-05-28 18:29                                 ` Yauheni Kaliuta
2020-05-28 18:34                                   ` Alexei Starovoitov
2020-05-28 19:05                                     ` Shuah Khan
2020-05-28 18:59                                 ` Shuah Khan
2020-05-28 19:18                                   ` Alexei Starovoitov
2020-05-28 20:09                                     ` Shuah Khan
2020-05-28 22:47                                       ` Shuah Khan [this message]
2020-05-28 17:10                             ` Yauheni Kaliuta
2020-05-28 18:17                               ` Alexei Starovoitov
2020-05-28 19:09                               ` Shuah Khan
2020-05-28 19:20                                 ` Yauheni Kaliuta
2020-05-28 19:34                                   ` Shuah Khan
2020-05-26 21:48   ` [PATCH 0/8] selftests/bpf: installation and out of tree build fixes Daniel Borkmann
2020-05-27  4:45     ` Yauheni Kaliuta
2020-05-26 22:32   ` Andrii Nakryiko
2020-05-27  4:52     ` Yauheni Kaliuta
2020-05-27  5:04       ` Andrii Nakryiko
2020-05-27  7:25         ` Yauheni Kaliuta
2020-05-27  8:05           ` Yauheni Kaliuta

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1ae97f52-fda0-ab1a-a027-d191e5a22858@linuxfoundation.org \
    --to=skhan@linuxfoundation.org \
    --cc=alexei.starovoitov@gmail.com \
    --cc=andrii.nakryiko@gmail.com \
    --cc=andriin@fb.com \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=gregkh@linuxfoundation.org \
    --cc=jbenc@redhat.com \
    --cc=jolsa@redhat.com \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=shuah@kernel.org \
    --cc=yauheni.kaliuta@redhat.com \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git