[LTP] [RFC PATCH] API: Allow testing of kernel features in development

[LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe via ltp]

Add an unstable kernel ABI flag and a runtest file for unstable
tests. This means we can add tests which are likely to be broken by
changes in the kernel ABI. Without disrupting LTP releases which are
required to be stable.

Users who require stability can filter the tests with this flag
or not schedule the unstable runtest file(s).

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
---

How about adding this to the fanotify22 patch?

 include/tst_test.h        | 12 ++++++++++++
 lib/tst_test.c            |  6 ++++++
 runtest/syscalls-unstable |  3 +++
 3 files changed, 21 insertions(+)
 create mode 100644 runtest/syscalls-unstable

diff --git a/include/tst_test.h b/include/tst_test.h
index 450ddf086..ff31e972e 100644
--- a/include/tst_test.h
+++ b/include/tst_test.h
@@ -174,6 +174,18 @@ struct tst_test {
 	int skip_in_lockdown:1;
 	int skip_in_compat:1;
 
+	/*
+	 * Test is for a feature that has not been added to the stable
+	 * kernel ABI. That is, it's for a feature only available in
+	 * linux-next, an RC or some other development branch.
+	 *
+	 * This string should be set to some text describing the
+	 * kernel branch and version the test was developed
+	 * against. e.g. "5.16 RC2", "linux-next-20211220",
+	 * "net-next".
+	 */
+	const char *const unstable_abi_version;
+
 	/*
 	 * The skip_filesystem is a NULL terminated list of filesystems the
 	 * test does not support. It can also be used to disable whole class of
diff --git a/lib/tst_test.c b/lib/tst_test.c
index 9b51bb5be..babb250d0 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -1013,6 +1013,12 @@ static void do_setup(int argc, char *argv[])
 	if (!tst_test)
 		tst_brk(TBROK, "No tests to run");
 
+	if (tst_test->unstable_abi_version) {
+		tst_res(TINFO,
+			"This test was developed against pre-release kernel: %s",
+			tst_test->unstable_abi_version);
+	}
+
 	if (tst_test->tconf_msg)
 		tst_brk(TCONF, "%s", tst_test->tconf_msg);
 
diff --git a/runtest/syscalls-unstable b/runtest/syscalls-unstable
new file mode 100644
index 000000000..a87284afe
--- /dev/null
+++ b/runtest/syscalls-unstable
@@ -0,0 +1,3 @@
+# Tests for kernel features which are not finalized
+
+fanotify22 fanotify22
-- 
2.34.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Li Wang]

Hi Richard,

On Tue, Dec 21, 2021 at 7:31 PM Richard Palethorpe via ltp
<ltp@lists.linux.it> wrote:
>
> Add an unstable kernel ABI flag and a runtest file for unstable
> tests. This means we can add tests which are likely to be broken by
> changes in the kernel ABI. Without disrupting LTP releases which are
> required to be stable.
>
> Users who require stability can filter the tests with this flag
> or not schedule the unstable runtest file(s).
>
> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
> ---
>
> How about adding this to the fanotify22 patch?
>
>  include/tst_test.h        | 12 ++++++++++++
>  lib/tst_test.c            |  6 ++++++
>  runtest/syscalls-unstable |  3 +++
>  3 files changed, 21 insertions(+)
>  create mode 100644 runtest/syscalls-unstable
>
> diff --git a/include/tst_test.h b/include/tst_test.h
> index 450ddf086..ff31e972e 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -174,6 +174,18 @@ struct tst_test {
>         int skip_in_lockdown:1;
>         int skip_in_compat:1;
>
> +       /*
> +        * Test is for a feature that has not been added to the stable
> +        * kernel ABI. That is, it's for a feature only available in
> +        * linux-next, an RC or some other development branch.
> +        *
> +        * This string should be set to some text describing the
> +        * kernel branch and version the test was developed
> +        * against. e.g. "5.16 RC2", "linux-next-20211220",
> +        * "net-next".
> +        */
> +       const char *const unstable_abi_version;

I'm not sure if this will bring convenience or trouble to LTP.

If a new syscall is in development (or merged in linux-next) but
drop/change in mainline-kernel. We have to adjust the test case
accordingly.

And after the feature is go into the mainline kernel finally, should
we move that from 'syscalls-unstable' to 'syscalls' and remove the
tst_test->unstable_abi_version field at the same time?

Btw, why not we just keep the unstable-syscall test case stay
in the review phase until it gets merged in mailline-kernel?
I guess that wouldn't block anything or make anyone unhappy.

> +
>         /*
>          * The skip_filesystem is a NULL terminated list of filesystems the
>          * test does not support. It can also be used to disable whole class of
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 9b51bb5be..babb250d0 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -1013,6 +1013,12 @@ static void do_setup(int argc, char *argv[])
>         if (!tst_test)
>                 tst_brk(TBROK, "No tests to run");
>
> +       if (tst_test->unstable_abi_version) {
> +               tst_res(TINFO,
> +                       "This test was developed against pre-release kernel: %s",
> +                       tst_test->unstable_abi_version);
> +       }
> +
>         if (tst_test->tconf_msg)
>                 tst_brk(TCONF, "%s", tst_test->tconf_msg);
>
> diff --git a/runtest/syscalls-unstable b/runtest/syscalls-unstable
> new file mode 100644
> index 000000000..a87284afe
> --- /dev/null
> +++ b/runtest/syscalls-unstable
> @@ -0,0 +1,3 @@
> +# Tests for kernel features which are not finalized
> +
> +fanotify22 fanotify22
> --
> 2.34.0
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>


-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe]

Hello Li,

Li Wang <liwang@redhat.com> writes:

> Hi Richard,
>
> On Tue, Dec 21, 2021 at 7:31 PM Richard Palethorpe via ltp
> <ltp@lists.linux.it> wrote:
>>
>> Add an unstable kernel ABI flag and a runtest file for unstable
>> tests. This means we can add tests which are likely to be broken by
>> changes in the kernel ABI. Without disrupting LTP releases which are
>> required to be stable.
>>
>> Users who require stability can filter the tests with this flag
>> or not schedule the unstable runtest file(s).
>>
>> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
>> ---
>>
>> How about adding this to the fanotify22 patch?
>>
>>  include/tst_test.h        | 12 ++++++++++++
>>  lib/tst_test.c            |  6 ++++++
>>  runtest/syscalls-unstable |  3 +++
>>  3 files changed, 21 insertions(+)
>>  create mode 100644 runtest/syscalls-unstable
>>
>> diff --git a/include/tst_test.h b/include/tst_test.h
>> index 450ddf086..ff31e972e 100644
>> --- a/include/tst_test.h
>> +++ b/include/tst_test.h
>> @@ -174,6 +174,18 @@ struct tst_test {
>>         int skip_in_lockdown:1;
>>         int skip_in_compat:1;
>>
>> +       /*
>> +        * Test is for a feature that has not been added to the stable
>> +        * kernel ABI. That is, it's for a feature only available in
>> +        * linux-next, an RC or some other development branch.
>> +        *
>> +        * This string should be set to some text describing the
>> +        * kernel branch and version the test was developed
>> +        * against. e.g. "5.16 RC2", "linux-next-20211220",
>> +        * "net-next".
>> +        */
>> +       const char *const unstable_abi_version;
>
> I'm not sure if this will bring convenience or trouble to LTP.
>
> If a new syscall is in development (or merged in linux-next) but
> drop/change in mainline-kernel. We have to adjust the test case
> accordingly.

If someone is motivated enough to contribute tests for an unreleased
feature. There is a high chance they will make the changes. In the worst
case we just delete the test.

>
> And after the feature is go into the mainline kernel finally, should
> we move that from 'syscalls-unstable' to 'syscalls' and remove the
> tst_test->unstable_abi_version field at the same time?

Yes.

>
> Btw, why not we just keep the unstable-syscall test case stay
> in the review phase until it gets merged in mailline-kernel?
> I guess that wouldn't block anything or make anyone unhappy.

It's a lot of work to merge patches into your own branch for testing RCs
and linux-next. It makes collaboration a lot harder. It's easy for
people just to enable the unstable tests. Then new features will get
tested on a much wider range of setups.

It increases the chances of finding bugs before release.

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Petr Vorel]

i all,

[ Cc automated-testing and people who might be interested ]

> Add an unstable kernel ABI flag and a runtest file for unstable
> tests. This means we can add tests which are likely to be broken by
> changes in the kernel ABI. Without disrupting LTP releases which are
> required to be stable.

> Users who require stability can filter the tests with this flag
> or not schedule the unstable runtest file(s).

I'm ok for this from a long term perspective, because agree Richie it can help
people to run tests on kernel from next tree or mainline rc kernel).

It's not much work to maintain this.

It'd also help people writing tests for  fanotify and IMA not having wait
several weeks.

Yes, we could add it to fanotify22 [1], but not to ima_conditionals.sh [2],
which is shell. But adding support to shell is trivial.

Acked-by: Petr Vorel <petr.vorel@gmail.com>

....
> +++ b/runtest/syscalls-unstable
How about having name syscalls-next? Although there can be tests which are from
some kernel maintainer tree (it does not have to be limited to next tree),
unstable can mean "tests which aren't fixed yet and thus buggy".
> @@ -0,0 +1,3 @@
> +# Tests for kernel features which are not finalized
> +
> +fanotify22 fanotify22

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/list/?series=272782
[2] https://patchwork.ozlabs.org/project/ltp/list/?series=265664

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Jan Stancek]

On Tue, Dec 21, 2021 at 6:56 PM Petr Vorel <pvorel@suse.cz> wrote:
>
> i all,
>
> [ Cc automated-testing and people who might be interested ]
>
> > Add an unstable kernel ABI flag and a runtest file for unstable
> > tests. This means we can add tests which are likely to be broken by
> > changes in the kernel ABI. Without disrupting LTP releases which are
> > required to be stable.
>
> > Users who require stability can filter the tests with this flag
> > or not schedule the unstable runtest file(s).
>
> I'm ok for this from a long term perspective, because agree Richie it can help
> people to run tests on kernel from next tree or mainline rc kernel).
>
> It's not much work to maintain this.
>
> It'd also help people writing tests for  fanotify and IMA not having wait
> several weeks.
>
> Yes, we could add it to fanotify22 [1], but not to ima_conditionals.sh [2],
> which is shell. But adding support to shell is trivial.
>
> Acked-by: Petr Vorel <petr.vorel@gmail.com>
>
> ....
> > +++ b/runtest/syscalls-unstable
> How about having name syscalls-next? Although there can be tests which are from
> some kernel maintainer tree (it does not have to be limited to next tree),
> unstable can mean "tests which aren't fixed yet and thus buggy".

staging?

IMO separate runtest would be enough, any notes why and how test was developed
could be in comments in code, where people can find it (less metadata
to maintain),
and those comments could stay there after feature is accepted to
mainline, we just
move test between runtest files.

> > @@ -0,0 +1,3 @@
> > +# Tests for kernel features which are not finalized
> > +
> > +fanotify22 fanotify22
>
> Kind regards,
> Petr
>
> [1] https://patchwork.ozlabs.org/project/ltp/list/?series=272782
> [2] https://patchwork.ozlabs.org/project/ltp/list/?series=265664
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe]

Hello,

Jan Stancek <jstancek@redhat.com> writes:

> On Tue, Dec 21, 2021 at 6:56 PM Petr Vorel <pvorel@suse.cz> wrote:
>>
>> i all,
>>
>> [ Cc automated-testing and people who might be interested ]
>>
>> > Add an unstable kernel ABI flag and a runtest file for unstable
>> > tests. This means we can add tests which are likely to be broken by
>> > changes in the kernel ABI. Without disrupting LTP releases which are
>> > required to be stable.
>>
>> > Users who require stability can filter the tests with this flag
>> > or not schedule the unstable runtest file(s).
>>
>> I'm ok for this from a long term perspective, because agree Richie it can help
>> people to run tests on kernel from next tree or mainline rc kernel).
>>
>> It's not much work to maintain this.
>>
>> It'd also help people writing tests for  fanotify and IMA not having wait
>> several weeks.
>>
>> Yes, we could add it to fanotify22 [1], but not to ima_conditionals.sh [2],
>> which is shell. But adding support to shell is trivial.
>>
>> Acked-by: Petr Vorel <petr.vorel@gmail.com>
>>
>> ....
>> > +++ b/runtest/syscalls-unstable
>> How about having name syscalls-next? Although there can be tests which are from
>> some kernel maintainer tree (it does not have to be limited to next tree),
>> unstable can mean "tests which aren't fixed yet and thus buggy".
>
> staging?

Staging and unstable could equally mean the test itself is not fininshed
IMO. I didn't suggest next for exactly the reason mentioned, but it
might be the better choice.

>
> IMO separate runtest would be enough, any notes why and how test was developed
> could be in comments in code, where people can find it (less metadata
> to maintain),
> and those comments could stay there after feature is accepted to
> mainline, we just
> move test between runtest files.

Then the test has a useless or misleading comment saying it was
developed against a feature still in development. It's trivial to remove
such comments or meta-data. I expect test authors will do it themselves
and if they don't we can rethink accepting such tests.

Also the patch uses the meta-data to print a hint. That way we do not
need to look at the source code, runtest file and LTP version before
deciding on the severity of a problem. Doing extra work upstream saves a
lot of work downstream.

Finally note that the plan is to schedule tests without runtest files
for parallel execution. That requires meta-data.

>
>> > @@ -0,0 +1,3 @@
>> > +# Tests for kernel features which are not finalized
>> > +
>> > +fanotify22 fanotify22
>>
>> Kind regards,
>> Petr
>>
>> [1] https://patchwork.ozlabs.org/project/ltp/list/?series=272782
>> [2] https://patchwork.ozlabs.org/project/ltp/list/?series=265664
>>
>> --
>> Mailing list info: https://lists.linux.it/listinfo/ltp
>>


-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Petr Vorel]

Hi Jan, all,

> > ....
> > > +++ b/runtest/syscalls-unstable
> > How about having name syscalls-next? Although there can be tests which are from
> > some kernel maintainer tree (it does not have to be limited to next tree),
> > unstable can mean "tests which aren't fixed yet and thus buggy".

> staging?
IMHO better than unstable, just staging is in kernel for not yet ready drivers
which need to cleanup, thus it could be confusing similar way as unstable.

But this should be ok, we document the purpose.

> IMO separate runtest would be enough, any notes why and how test was developed
> could be in comments in code, where people can find it (less metadata
> to maintain),
> and those comments could stay there after feature is accepted to
> mainline, we just
> move test between runtest files.

+1, actually the simplest solution.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Cyril Hrubis]

Hi!
> Add an unstable kernel ABI flag and a runtest file for unstable
> tests. This means we can add tests which are likely to be broken by
> changes in the kernel ABI. Without disrupting LTP releases which are
> required to be stable.
> 
> Users who require stability can filter the tests with this flag
> or not schedule the unstable runtest file(s).
> 
> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
> ---
> 
> How about adding this to the fanotify22 patch?
> 
>  include/tst_test.h        | 12 ++++++++++++
>  lib/tst_test.c            |  6 ++++++
>  runtest/syscalls-unstable |  3 +++
>  3 files changed, 21 insertions(+)
>  create mode 100644 runtest/syscalls-unstable
> 
> diff --git a/include/tst_test.h b/include/tst_test.h
> index 450ddf086..ff31e972e 100644
> --- a/include/tst_test.h
> +++ b/include/tst_test.h
> @@ -174,6 +174,18 @@ struct tst_test {
>  	int skip_in_lockdown:1;
>  	int skip_in_compat:1;
>  
> +	/*
> +	 * Test is for a feature that has not been added to the stable
> +	 * kernel ABI. That is, it's for a feature only available in
> +	 * linux-next, an RC or some other development branch.
> +	 *
> +	 * This string should be set to some text describing the
> +	 * kernel branch and version the test was developed
> +	 * against. e.g. "5.16 RC2", "linux-next-20211220",
> +	 * "net-next".
> +	 */
> +	const char *const unstable_abi_version;

I would rather call this 'remove_after_release' or 'remove_after_kernel_release' but that is very minor.

>  	/*
>  	 * The skip_filesystem is a NULL terminated list of filesystems the
>  	 * test does not support. It can also be used to disable whole class of
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 9b51bb5be..babb250d0 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -1013,6 +1013,12 @@ static void do_setup(int argc, char *argv[])
>  	if (!tst_test)
>  		tst_brk(TBROK, "No tests to run");
>  
> +	if (tst_test->unstable_abi_version) {
> +		tst_res(TINFO,
> +			"This test was developed against pre-release kernel: %s",
> +			tst_test->unstable_abi_version);
> +	}
> +
>  	if (tst_test->tconf_msg)
>  		tst_brk(TCONF, "%s", tst_test->tconf_msg);
>  
> diff --git a/runtest/syscalls-unstable b/runtest/syscalls-unstable
> new file mode 100644
> index 000000000..a87284afe
> --- /dev/null
> +++ b/runtest/syscalls-unstable
> @@ -0,0 +1,3 @@
> +# Tests for kernel features which are not finalized
> +
> +fanotify22 fanotify22

I would be even tempted to add this to the default syscalls runtest file
but render the test resultless, e.g. replace the tst_res() and tst_brk()
handlers in tst_res.c so that it will print only INFO messages.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Cyril Hrubis]

Hi!
> I would be even tempted to add this to the default syscalls runtest file
> but render the test resultless, e.g. replace the tst_res() and tst_brk()
> handlers in tst_res.c so that it will print only INFO messages.

And that would also allow us to have env variable switch that would
enable all tests for unrelased features, something as
LTP_ENABLE_UNRELASED or so. Also with that we can really just TCONF if
that variable is not set which would be much simpler.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Richard Palethorpe]

Hello,

Cyril Hrubis <chrubis@suse.cz> writes:

> Hi!
>> I would be even tempted to add this to the default syscalls runtest file
>> but render the test resultless, e.g. replace the tst_res() and tst_brk()
>> handlers in tst_res.c so that it will print only INFO messages.
>
> And that would also allow us to have env variable switch that would
> enable all tests for unrelased features, something as
> LTP_ENABLE_UNRELASED or so. Also with that we can really just TCONF if
> that variable is not set which would be much simpler.

I worry this will cause extra work for people reviewing
TCONFs. Obviously the meta-data can be used to avoid any issues. However
few people have integrated that into their tooling and workflow yet.

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] API: Allow testing of kernel features in development

[Petr Vorel]

Hi all,

> Hello,

> Cyril Hrubis <chrubis@suse.cz> writes:

> > Hi!
> >> I would be even tempted to add this to the default syscalls runtest file
> >> but render the test resultless, e.g. replace the tst_res() and tst_brk()
> >> handlers in tst_res.c so that it will print only INFO messages.

> > And that would also allow us to have env variable switch that would
> > enable all tests for unrelased features, something as
> > LTP_ENABLE_UNRELASED or so. Also with that we can really just TCONF if
> > that variable is not set which would be much simpler.

> I worry this will cause extra work for people reviewing
> TCONFs. Obviously the meta-data can be used to avoid any issues. However
> few people have integrated that into their tooling and workflow yet.

Can we solve this to have the policy? I don't have preference between having
special foo-unstable (or foo-staging) runtest files or TCONF unless
LTP_ENABLE_UNRELASED

(If we chose the latter, we could add this info into -h, thus people could run
./foo -h | grep ... to catch this before running; there could be even --json to
print metadata in json, which would include this info).

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH] Create policy for testing unstable kernel features

[Richard Palethorpe via ltp]

It's not clear if anything more than this is required for now.

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
---

 doc/test-writing-guidelines.txt | 10 ++++++++++
 runtest/staging                 |  1 +
 2 files changed, 11 insertions(+)
 create mode 100644 runtest/staging

diff --git a/doc/test-writing-guidelines.txt b/doc/test-writing-guidelines.txt
index 1fa751ed5..8ca014d45 100644
--- a/doc/test-writing-guidelines.txt
+++ b/doc/test-writing-guidelines.txt
@@ -334,3 +334,13 @@ It's easier to maintain a '.gitignore' file per directory with tests, rather
 than having single file in the project root directory. This way, we don't have
 to update all the gitignore files when moving directories, and they get deleted
 automatically when a directory with tests is removed.
+
+7 Testing pre-release kernel features
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Tests for features not yet in a mainline kernel release are accepted. However
+they must only be added to the +staging+ runtest file. Once a feature is part
+of the stable kernel ABI the associated test must be moved out of staging.
+
+This is primarily to help test kernel RCs by avoiding the need to download
+separate LTP patchsets.
diff --git a/runtest/staging b/runtest/staging
new file mode 100644
index 000000000..ef1cdea15
--- /dev/null
+++ b/runtest/staging
@@ -0,0 +1 @@
+# Tests for features that are not yet in the stable kernel ABI
-- 
2.34.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Petr Vorel]

Hi Richie,

Thanks!
I'd consider name next, but does not really matter that much.

+1 for single file, because not only syscalls have tests for RC
(at least IMA tests also do).

Acked-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Jan Stancek]

On Thu, Feb 3, 2022 at 11:18 AM Richard Palethorpe via ltp
<ltp@lists.linux.it> wrote:
>
> It's not clear if anything more than this is required for now.
>
> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>

Acked-by: Jan Stancek <jstancek@redhat.com>

> ---
>
>  doc/test-writing-guidelines.txt | 10 ++++++++++
>  runtest/staging                 |  1 +
>  2 files changed, 11 insertions(+)
>  create mode 100644 runtest/staging
>
> diff --git a/doc/test-writing-guidelines.txt b/doc/test-writing-guidelines.txt
> index 1fa751ed5..8ca014d45 100644
> --- a/doc/test-writing-guidelines.txt
> +++ b/doc/test-writing-guidelines.txt
> @@ -334,3 +334,13 @@ It's easier to maintain a '.gitignore' file per directory with tests, rather
>  than having single file in the project root directory. This way, we don't have
>  to update all the gitignore files when moving directories, and they get deleted
>  automatically when a directory with tests is removed.
> +
> +7 Testing pre-release kernel features
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Tests for features not yet in a mainline kernel release are accepted. However
> +they must only be added to the +staging+ runtest file. Once a feature is part
> +of the stable kernel ABI the associated test must be moved out of staging.
> +
> +This is primarily to help test kernel RCs by avoiding the need to download
> +separate LTP patchsets.
> diff --git a/runtest/staging b/runtest/staging
> new file mode 100644
> index 000000000..ef1cdea15
> --- /dev/null
> +++ b/runtest/staging
> @@ -0,0 +1 @@
> +# Tests for features that are not yet in the stable kernel ABI
> --
> 2.34.1
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Li Wang]

[-- Attachment #1.1: Type: text/plain, Size: 1663 bytes --]

On Thu, Feb 3, 2022 at 6:20 PM Richard Palethorpe <rpalethorpe@suse.com>
wrote:

> It's not clear if anything more than this is required for now.
>
> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
>

Reviewed-by: Li Wang <liwang@redhat.com>

> ---
>
>  doc/test-writing-guidelines.txt | 10 ++++++++++
>  runtest/staging                 |  1 +
>  2 files changed, 11 insertions(+)
>  create mode 100644 runtest/staging
>
> diff --git a/doc/test-writing-guidelines.txt
> b/doc/test-writing-guidelines.txt
> index 1fa751ed5..8ca014d45 100644
> --- a/doc/test-writing-guidelines.txt
> +++ b/doc/test-writing-guidelines.txt
> @@ -334,3 +334,13 @@ It's easier to maintain a '.gitignore' file per
> directory with tests, rather
>  than having single file in the project root directory. This way, we don't
> have
>  to update all the gitignore files when moving directories, and they get
> deleted
>  automatically when a directory with tests is removed.
> +
> +7 Testing pre-release kernel features
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Tests for features not yet in a mainline kernel release are accepted.
> However
> +they must only be added to the +staging+ runtest file. Once a feature is
> part
> +of the stable kernel ABI the associated test must be moved out of staging.
> +
> +This is primarily to help test kernel RCs by avoiding the need to download
> +separate LTP patchsets.
> diff --git a/runtest/staging b/runtest/staging
> new file mode 100644
> index 000000000..ef1cdea15
> --- /dev/null
> +++ b/runtest/staging
> @@ -0,0 +1 @@
> +# Tests for features that are not yet in the stable kernel ABI
> --
> 2.34.1
>
>

-- 
Regards,
Li Wang

[-- Attachment #1.2: Type: text/html, Size: 2679 bytes --]

[-- Attachment #2: Type: text/plain, Size: 60 bytes --]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Petr Vorel]

Hi all,

...
> > +7 Testing pre-release kernel features
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Tests for features not yet in a mainline kernel release are accepted.
> > However
> > +they must only be added to the +staging+ runtest file. Once a feature is
> > part
> > +of the stable kernel ABI the associated test must be moved out of staging.
> > +
> > +This is primarily to help test kernel RCs by avoiding the need to download
> > +separate LTP patchsets.
> > diff --git a/runtest/staging b/runtest/staging
...

FYI keyctl09 [1] patch is a candidate for merging under this policy.

Cyril wasn't sure [2] that we haven't agreed yet about how this should be
handled. Not sure what exactly is he missing.

I guess the description is quite open to allow testing both tests in next tree
or even in maintainer tree (before endup in next tree). From my point it should
be obvious that not *anything* out of mainline can be added, just tests which
sooner or later endup in mainline (and if not, they should be deleted).

Also procedure after release is pretty clear: just move everything out of
staging to expected runtest files (mostly syscalls).

=> I'd merge this.

Kind regards,
Petr

[1] https://patchwork.ozlabs.org/project/ltp/patch/20220223200731.1859670-1-yaelt@google.com/
[2] https://lore.kernel.org/ltp/YiC4Pj1sH8UIHY7k@yuki/

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Petr Vorel]

Hi all,

as there is another set of fanotify tests for rc kernel, I'd really like to have
some policy. Therefore I suggest to stick policy suggested by Richie in this
patchset [1], i.e. adding tests into runtest/staging until mainline kernel with
this functionality is released (feature is part of the stable kernel ABI).
i.e. each kernel release content of this file must be revised and (ideally)
moved to particular runtest file where it should belong or fixed (in rare
situations when feature changed) or removed (feature was reverted).

I volunteer to maintain runtest/staging.

This feature got already ack from Mike Frysinger, Li Wang, Jan Stancek (and me).
I'll wait little longer for ack from others (Yang Xu, Cyril Hrubis).

Anybody against it?

"remove_after_release" solution [2] suggested by Cyril and Richie, which would
keep kernel version could be quite easily implemented. But IMHO it's not worth
with current number of tests which need it (< 5, often 1 or 0).

Obviously I suggest to drop my original suggestion [2] (it didn't include
runtest/staging).

Kind regards,
Petr

[1] https://lore.kernel.org/ltp/20220203101803.10204-1-rpalethorpe@suse.com/
[2] https://lore.kernel.org/ltp/YdW5WEXgrotentzM@yuki/
[3] https://lore.kernel.org/ltp/20211210134556.26091-1-pvorel@suse.cz/

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Cyril Hrubis]

Hi!
> as there is another set of fanotify tests for rc kernel, I'd really like to have
> some policy. Therefore I suggest to stick policy suggested by Richie in this
> patchset [1], i.e. adding tests into runtest/staging until mainline kernel with
> this functionality is released (feature is part of the stable kernel ABI).
> i.e. each kernel release content of this file must be revised and (ideally)
> moved to particular runtest file where it should belong or fixed (in rare
> situations when feature changed) or removed (feature was reverted).
> 
> I volunteer to maintain runtest/staging.

As long as you commit to maintain the staging you have my ack as well.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH] Create policy for testing unstable kernel features

[Petr Vorel]

Hi all,

> Hi!
> > as there is another set of fanotify tests for rc kernel, I'd really like to have
> > some policy. Therefore I suggest to stick policy suggested by Richie in this
> > patchset [1], i.e. adding tests into runtest/staging until mainline kernel with
> > this functionality is released (feature is part of the stable kernel ABI).
> > i.e. each kernel release content of this file must be revised and (ideally)
> > moved to particular runtest file where it should belong or fixed (in rare
> > situations when feature changed) or removed (feature was reverted).

> > I volunteer to maintain runtest/staging.

> As long as you commit to maintain the staging you have my ack as well.
Thx! I merged this patch then and I'll merge Amir's fanotify{10,23} tests for
5.19-rc1 kernel.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp