[Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Shuah Khan]

As a first step towards a larger goal to enable developer
friendly kernel testing framework, a new make target is
planned for 3.17. In addition, 3.17 includes work done to
fix tools/testing/sefltests to run without failures.

Short summary of work done so far for 3.17:

- fix compile errors and warnings in various tests
- fix run-time errors when tests aren't run as root
- enhance and improve cpu and memory hot-plug tests
   to run in limited scope mode by default. A new make
   target to select full-scope testing. Prior to this
   change, cpu and memory hot-plug tests hung trying to
   hot-plug all but cpu0 and a large portion of the memory.
- add a new kselftest target to run existing selftests
   to start with.

What's planned for 3.18 and beyond:

- get feedback on the new kselftest target from the community
- add more tests to be run under kselftest umbrella
- identify existing tests under /lib and other areas that
   make a good candidate to be included under kselftest
- Some of these could be run as a tool and/or a independent
   test with a few changes and some probably aren't like the
   /lib/locking tests.
- As a goal, try to leverage existing tests and modify them
   as needed to run them as a black-box test (e.g: look into
   ways to make it run as a tool)
- Greg KH sparked the kernel selftest idea, has been in the loop
   for the work done so far, and reviewed the plan for 3.18.

thanks,
-- Shuah

Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Bird, Tim]

On Thursday, August 07, 2014 7:36 AM,  Shuah Khan wrote:
> 
> As a first step towards a larger goal to enable developer
> friendly kernel testing framework, a new make target is
> planned for 3.17. In addition, 3.17 includes work done to
> fix tools/testing/sefltests to run without failures.
> 
> Short summary of work done so far for 3.17:
> 
> - fix compile errors and warnings in various tests
> - fix run-time errors when tests aren't run as root
> - enhance and improve cpu and memory hot-plug tests
>    to run in limited scope mode by default. A new make
>    target to select full-scope testing. Prior to this
>    change, cpu and memory hot-plug tests hung trying to
>    hot-plug all but cpu0 and a large portion of the memory.
> - add a new kselftest target to run existing selftests
>    to start with.
> 
> What's planned for 3.18 and beyond:
> 
> - get feedback on the new kselftest target from the community
> - add more tests to be run under kselftest umbrella
> - identify existing tests under /lib and other areas that
>    make a good candidate to be included under kselftest
> - Some of these could be run as a tool and/or a independent
>    test with a few changes and some probably aren't like the
>    /lib/locking tests.
> - As a goal, try to leverage existing tests and modify them
>    as needed to run them as a black-box test (e.g: look into
>    ways to make it run as a tool)
> - Greg KH sparked the kernel selftest idea, has been in the loop
>    for the work done so far, and reviewed the plan for 3.18.

I'm quite interested in this (as is the CEWG), and I have lots of questions.
Will this be discussed/presented at KS or LinuxCon?  Should I ask my
questions on this list or wait until Chicago?  Are there any LKML threads
with info I can look at in the meantime?

Thanks.  This sounds like great work.
 -- Tim

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Shuah Khan]

On 08/07/2014 12:24 PM, Bird, Tim wrote:
>
> I'm quite interested in this (as is the CEWG), and I have lots of questions.
> Will this be discussed/presented at KS or LinuxCon?  Should I ask my
> questions on this list or wait until Chicago?  Are there any LKML threads
> with info I can look at in the meantime?
>
> Thanks.  This sounds like great work.

The plan to discuss this at the KS. Please continue this thread with
questions you have. I can point you to links to the patches for the
work done so far in 3.17 if you are interested.

-- Shuah

-- 
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Masami Hiramatsu]

Hello,

I'm also interested in the selftests, especially its
framework (top-level script).

http://article.gmane.org/gmane.linux.kernel/1763766

(2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
> friendly kernel testing framework, a new make target is
> planned for 3.17. In addition, 3.17 includes work done to
> fix tools/testing/sefltests to run without failures.
>
> Short summary of work done so far for 3.17:
>
> - fix compile errors and warnings in various tests
> - fix run-time errors when tests aren't run as root
> - enhance and improve cpu and memory hot-plug tests
>    to run in limited scope mode by default. A new make
>    target to select full-scope testing. Prior to this
>    change, cpu and memory hot-plug tests hung trying to
>    hot-plug all but cpu0 and a large portion of the memory.
> - add a new kselftest target to run existing selftests
>    to start with.

Instead of running the selftests, can we build the testcases and
install it as a tool? I think running tests on the tree is not a
good idea...
Also, as I said in above mail, I'd like to suggest to add at least
log management and statistics to the test. Those are good to automate
regression tests. :)

> What's planned for 3.18 and beyond:
>
> - get feedback on the new kselftest target from the community
> - add more tests to be run under kselftest umbrella
> - identify existing tests under /lib and other areas that
>    make a good candidate to be included under kselftest
> - Some of these could be run as a tool and/or a independent
>    test with a few changes and some probably aren't like the
>    /lib/locking tests.
> - As a goal, try to leverage existing tests and modify them
>    as needed to run them as a black-box test (e.g: look into
>    ways to make it run as a tool)
> - Greg KH sparked the kernel selftest idea, has been in the loop
>    for the work done so far, and reviewed the plan for 3.18.
>

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Shuah Khan]

On 08/07/2014 08:18 PM, Masami Hiramatsu wrote:
> Hello,
>
> I'm also interested in the selftests, especially its
> framework (top-level script).
>
> http://article.gmane.org/gmane.linux.kernel/1763766

Great.

>
> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>> friendly kernel testing framework, a new make target is
>> planned for 3.17. In addition, 3.17 includes work done to
>> fix tools/testing/sefltests to run without failures.
>>
>> Short summary of work done so far for 3.17:
>>
>> - fix compile errors and warnings in various tests
>> - fix run-time errors when tests aren't run as root
>> - enhance and improve cpu and memory hot-plug tests
>>     to run in limited scope mode by default. A new make
>>     target to select full-scope testing. Prior to this
>>     change, cpu and memory hot-plug tests hung trying to
>>     hot-plug all but cpu0 and a large portion of the memory.
>> - add a new kselftest target to run existing selftests
>>     to start with.
>
> Instead of running the selftests, can we build the testcases and
> install it as a tool? I think running tests on the tree is not a
> good idea...

One of the goals is to leverage developer tests that we already have.
When a developer makes a kernel change and wants to see if that change
lead to any regression, having the ability to buidl and run selftests on 
the newly installed kernel withe the same source tree is very useful.
That is the reason behind adding this new target.

Existing selfests are a collection of tools that exercise several
testcases that are specific each area they target. This is a good
start and we can definitely start enhancing the tools and build
testcases.

> Also, as I said in above mail, I'd like to suggest to add at least
> log management and statistics to the test. Those are good to automate
> regression tests. :)
>

Right. Some existing selftests do that, but not all. It would be
helpful to add that to all.

-- Shuah


-- 
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Masami Hiramatsu]

(2014/08/11 23:11), Shuah Khan wrote:
>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>>> friendly kernel testing framework, a new make target is
>>> planned for 3.17. In addition, 3.17 includes work done to
>>> fix tools/testing/sefltests to run without failures.
>>>
>>> Short summary of work done so far for 3.17:
>>>
>>> - fix compile errors and warnings in various tests
>>> - fix run-time errors when tests aren't run as root
>>> - enhance and improve cpu and memory hot-plug tests
>>>     to run in limited scope mode by default. A new make
>>>     target to select full-scope testing. Prior to this
>>>     change, cpu and memory hot-plug tests hung trying to
>>>     hot-plug all but cpu0 and a large portion of the memory.
>>> - add a new kselftest target to run existing selftests
>>>     to start with.
>>
>> Instead of running the selftests, can we build the testcases and
>> install it as a tool? I think running tests on the tree is not a
>> good idea...
> 
> One of the goals is to leverage developer tests that we already have.
> When a developer makes a kernel change and wants to see if that change
> lead to any regression, having the ability to buidl and run selftests on 
> the newly installed kernel withe the same source tree is very useful.
> That is the reason behind adding this new target.

I see, for that purpose, installing testcase may not fit.
BTW, how would it cover cross-build?

> Existing selfests are a collection of tools that exercise several
> testcases that are specific each area they target. This is a good
> start and we can definitely start enhancing the tools and build
> testcases.

Agreed.

> 
>> Also, as I said in above mail, I'd like to suggest to add at least
>> log management and statistics to the test. Those are good to automate
>> regression tests. :)
>>
> 
> Right. Some existing selftests do that, but not all. It would be
> helpful to add that to all.

Yeah, and I hope the selftest top-level script to provide such
functions to each test, because those should be common, and easy
to manage.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Grant Likely]

On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> (2014/08/11 23:11), Shuah Khan wrote:
> >> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
> >>> friendly kernel testing framework, a new make target is
> >>> planned for 3.17. In addition, 3.17 includes work done to
> >>> fix tools/testing/sefltests to run without failures.
> >>>
> >>> Short summary of work done so far for 3.17:
> >>>
> >>> - fix compile errors and warnings in various tests
> >>> - fix run-time errors when tests aren't run as root
> >>> - enhance and improve cpu and memory hot-plug tests
> >>>     to run in limited scope mode by default. A new make
> >>>     target to select full-scope testing. Prior to this
> >>>     change, cpu and memory hot-plug tests hung trying to
> >>>     hot-plug all but cpu0 and a large portion of the memory.
> >>> - add a new kselftest target to run existing selftests
> >>>     to start with.
> >>
> >> Instead of running the selftests, can we build the testcases and
> >> install it as a tool? I think running tests on the tree is not a
> >> good idea...
> > 
> > One of the goals is to leverage developer tests that we already have.
> > When a developer makes a kernel change and wants to see if that change
> > lead to any regression, having the ability to buidl and run selftests on 
> > the newly installed kernel withe the same source tree is very useful.
> > That is the reason behind adding this new target.
> 
> I see, for that purpose, installing testcase may not fit.
> BTW, how would it cover cross-build?

I'm interested in this as well. I'm working on a tool that crossbuilds a
very simple busybox rootfs and boots in QEMU for as many architectures
as possible. I want to make it easy to sanity test all the major
architectures. Right now it does little more than boot to a login
prompt, but I'd like to get the kselftests into it also.

g.

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Guenter Roeck]

On 08/12/2014 06:00 AM, Grant Likely wrote:
> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2014/08/11 23:11), Shuah Khan wrote:
>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>>>>> friendly kernel testing framework, a new make target is
>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>> fix tools/testing/sefltests to run without failures.
>>>>>
>>>>> Short summary of work done so far for 3.17:
>>>>>
>>>>> - fix compile errors and warnings in various tests
>>>>> - fix run-time errors when tests aren't run as root
>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>      to run in limited scope mode by default. A new make
>>>>>      target to select full-scope testing. Prior to this
>>>>>      change, cpu and memory hot-plug tests hung trying to
>>>>>      hot-plug all but cpu0 and a large portion of the memory.
>>>>> - add a new kselftest target to run existing selftests
>>>>>      to start with.
>>>>
>>>> Instead of running the selftests, can we build the testcases and
>>>> install it as a tool? I think running tests on the tree is not a
>>>> good idea...
>>>
>>> One of the goals is to leverage developer tests that we already have.
>>> When a developer makes a kernel change and wants to see if that change
>>> lead to any regression, having the ability to buidl and run selftests on
>>> the newly installed kernel withe the same source tree is very useful.
>>> That is the reason behind adding this new target.
>>
>> I see, for that purpose, installing testcase may not fit.
>> BTW, how would it cover cross-build?
>
> I'm interested in this as well. I'm working on a tool that crossbuilds a
> very simple busybox rootfs and boots in QEMU for as many architectures
> as possible. I want to make it easy to sanity test all the major
> architectures. Right now it does little more than boot to a login
> prompt, but I'd like to get the kselftests into it also.
>

Do you have that public yet ? I might want to use that for my -stable sanity tests.

Thanks,
Guenter

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Masami Hiramatsu]

(2014/08/13 1:15), Guenter Roeck wrote:
> On 08/12/2014 06:00 AM, Grant Likely wrote:
>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>>> (2014/08/11 23:11), Shuah Khan wrote:
>>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>>>>>> friendly kernel testing framework, a new make target is
>>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>>> fix tools/testing/sefltests to run without failures.
>>>>>>
>>>>>> Short summary of work done so far for 3.17:
>>>>>>
>>>>>> - fix compile errors and warnings in various tests
>>>>>> - fix run-time errors when tests aren't run as root
>>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>>      to run in limited scope mode by default. A new make
>>>>>>      target to select full-scope testing. Prior to this
>>>>>>      change, cpu and memory hot-plug tests hung trying to
>>>>>>      hot-plug all but cpu0 and a large portion of the memory.
>>>>>> - add a new kselftest target to run existing selftests
>>>>>>      to start with.
>>>>>
>>>>> Instead of running the selftests, can we build the testcases and
>>>>> install it as a tool? I think running tests on the tree is not a
>>>>> good idea...
>>>>
>>>> One of the goals is to leverage developer tests that we already have.
>>>> When a developer makes a kernel change and wants to see if that change
>>>> lead to any regression, having the ability to buidl and run selftests on
>>>> the newly installed kernel withe the same source tree is very useful.
>>>> That is the reason behind adding this new target.
>>>
>>> I see, for that purpose, installing testcase may not fit.
>>> BTW, how would it cover cross-build?
>>
>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> very simple busybox rootfs and boots in QEMU for as many architectures
>> as possible. I want to make it easy to sanity test all the major
>> architectures. Right now it does little more than boot to a login
>> prompt, but I'd like to get the kselftests into it also.
>>
> 
> Do you have that public yet ? I might want to use that for my -stable sanity tests.

IMHO, We'd better share those testing tools/environments as much as
possible so that each developer can ensure no regressions in their
patches before sending it.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Geert Uytterhoeven]

On Tue, Aug 12, 2014 at 6:21 PM, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:
>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> as possible. I want to make it easy to sanity test all the major
>>> architectures. Right now it does little more than boot to a login
>>> prompt, but I'd like to get the kselftests into it also.

Like Aboriginal Linux (landley.net/aboriginal/)?

>> Do you have that public yet ? I might want to use that for my -stable sanity tests.
>
> IMHO, We'd better share those testing tools/environments as much as
> possible so that each developer can ensure no regressions in their
> patches before sending it.

Yes!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Theodore Ts'o]

In case it's useful, this is the tool I use to build a root_fs for
ext4 testing:

https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/test-appliance/gen-image

There's an automated test running and xfstests results parser in there
as well, but that's probably a bit less useful for non-file-system developers.

      	     	    	       	   - Ted

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Grant Likely]

On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote:
> On 08/12/2014 06:00 AM, Grant Likely wrote:
>>
>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu
>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>>
>>> (2014/08/11 23:11), Shuah Khan wrote:
>>>>>
>>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger
>>>>> goal to enable developer
>>>>>>
>>>>>> friendly kernel testing framework, a new make target is
>>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>>> fix tools/testing/sefltests to run without failures.
>>>>>>
>>>>>> Short summary of work done so far for 3.17:
>>>>>>
>>>>>> - fix compile errors and warnings in various tests
>>>>>> - fix run-time errors when tests aren't run as root
>>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>>      to run in limited scope mode by default. A new make
>>>>>>      target to select full-scope testing. Prior to this
>>>>>>      change, cpu and memory hot-plug tests hung trying to
>>>>>>      hot-plug all but cpu0 and a large portion of the memory.
>>>>>> - add a new kselftest target to run existing selftests
>>>>>>      to start with.
>>>>>
>>>>>
>>>>> Instead of running the selftests, can we build the testcases and
>>>>> install it as a tool? I think running tests on the tree is not a
>>>>> good idea...
>>>>
>>>>
>>>> One of the goals is to leverage developer tests that we already have.
>>>> When a developer makes a kernel change and wants to see if that change
>>>> lead to any regression, having the ability to buidl and run selftests on
>>>> the newly installed kernel withe the same source tree is very useful.
>>>> That is the reason behind adding this new target.
>>>
>>>
>>> I see, for that purpose, installing testcase may not fit.
>>> BTW, how would it cover cross-build?
>>
>>
>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> very simple busybox rootfs and boots in QEMU for as many architectures
>> as possible. I want to make it easy to sanity test all the major
>> architectures. Right now it does little more than boot to a login
>> prompt, but I'd like to get the kselftests into it also.
>>
>
> Do you have that public yet ? I might want to use that for my -stable sanity
> tests.

Nothing yet. It's currently hacked into my build environment script.
I'm working on getting it into a form usable by others. Ideally I
think it should go into the kernel tools/testing directory.

g.

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

On Tue, Aug 12, 2014 at 05:23:28PM +0100, Grant Likely wrote:
> On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote:

> > Do you have that public yet ? I might want to use that for my -stable sanity
> > tests.

> Nothing yet. It's currently hacked into my build environment script.
> I'm working on getting it into a form usable by others. Ideally I
> think it should go into the kernel tools/testing directory.

While we're working on things like this it's probably worth getting the
scripts Kevin, Olof and I use for build and boot test reporting (I only
run them, I've not contributed anything) or something like them into
scripts/ - they're pretty good and having a common format makes things
easier when looking at multiple reports.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Grant Likely]

On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, Aug 12, 2014 at 05:23:28PM +0100, Grant Likely wrote:
>> On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>
>> > Do you have that public yet ? I might want to use that for my -stable sanity
>> > tests.
>
>> Nothing yet. It's currently hacked into my build environment script.
>> I'm working on getting it into a form usable by others. Ideally I
>> think it should go into the kernel tools/testing directory.
>
> While we're working on things like this it's probably worth getting the
> scripts Kevin, Olof and I use for build and boot test reporting (I only
> run them, I've not contributed anything) or something like them into
> scripts/ - they're pretty good and having a common format makes things
> easier when looking at multiple reports.

Where is the current copy of those scripts?

g.

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 610 bytes --]

On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote:
> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote:

> > While we're working on things like this it's probably worth getting the
> > scripts Kevin, Olof and I use for build and boot test reporting (I only
> > run them, I've not contributed anything) or something like them into
> > scripts/ - they're pretty good and having a common format makes things
> > easier when looking at multiple reports.

> Where is the current copy of those scripts?

Kevin's copy is here:

   git://git.linaro.org/people/khilman/build-scripts.git

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Geert Uytterhoeven]

On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote:
> On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote:
>> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote:
>
>> > While we're working on things like this it's probably worth getting the
>> > scripts Kevin, Olof and I use for build and boot test reporting (I only
>> > run them, I've not contributed anything) or something like them into
>> > scripts/ - they're pretty good and having a common format makes things
>> > easier when looking at multiple reports.
>
>> Where is the current copy of those scripts?
>
> Kevin's copy is here:
>
>    git://git.linaro.org/people/khilman/build-scripts.git

404

Kevin's name got expanded:

git://git.linaro.org/people/kevin.hilman/build-scripts.git

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 721 bytes --]

On Wed, Aug 13, 2014 at 01:12:28PM +0200, Geert Uytterhoeven wrote:
> On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote:

> > Kevin's copy is here:

> >    git://git.linaro.org/people/khilman/build-scripts.git

> 404

> Kevin's name got expanded:

> git://git.linaro.org/people/kevin.hilman/build-scripts.git

Interesting...  how are you cloning?  Both work for me:

$ git clone git://git.linaro.org/people/khilman/build-scripts.git
Cloning into 'build-scripts'...
remote: Counting objects: 489, done.
remote: Compressing objects: 100% (307/307), done.
remote: Total 489 (delta 302), reused 295 (delta 181)
Receiving objects: 100% (489/489), 73.90 KiB, done.
Resolving deltas: 100% (302/302), done.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Geert Uytterhoeven]

Hi Mark,

On Wed, Aug 13, 2014 at 2:42 PM, Mark Brown <broonie@sirena.org.uk> wrote:
> On Wed, Aug 13, 2014 at 01:12:28PM +0200, Geert Uytterhoeven wrote:
>> On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote:
>
>> > Kevin's copy is here:
>
>> >    git://git.linaro.org/people/khilman/build-scripts.git
>
>> 404
>
>> Kevin's name got expanded:
>
>> git://git.linaro.org/people/kevin.hilman/build-scripts.git
>
> Interesting...  how are you cloning?  Both work for me:
>
> $ git clone git://git.linaro.org/people/khilman/build-scripts.git
> Cloning into 'build-scripts'...
> remote: Counting objects: 489, done.
> remote: Compressing objects: 100% (307/307), done.
> remote: Total 489 (delta 302), reused 295 (delta 181)
> Receiving objects: 100% (489/489), 73.90 KiB, done.
> Resolving deltas: 100% (302/302), done.

Bummer, I didn't clone, but opened
http://git.linaro.org/people/khilman/build-scripts.git in my web browser
by clicking on the link.
So Linaro gitweb supports the full names only, while real git can
access both (cloning indeed works for both).

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Kevin Hilman]

On Wed, Aug 13, 2014 at 3:40 AM, Mark Brown <broonie@kernel.org> wrote:
> On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote:
>> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote:
>
>> > While we're working on things like this it's probably worth getting the
>> > scripts Kevin, Olof and I use for build and boot test reporting (I only
>> > run them, I've not contributed anything) or something like them into
>> > scripts/ - they're pretty good and having a common format makes things
>> > easier when looking at multiple reports.
>
>> Where is the current copy of those scripts?
>
> Kevin's copy is here:
>
>    git://git.linaro.org/people/khilman/build-scripts.git

FWIW, that repo is just a hodge-podge collection of various build/boot
scripts I'm using.  I just cleaned it up a little and removed a bunch
of them that I no longer use.

They get the job done for me, but are certainly not pretty.

Also for boot testing, the main tool I'm  using is pyboot, which is
essentially a tool to automate u-boot/fastboot over the serial
console.  It relies heavily on conmux to abstract the console and
power cycling.

    https://git.linaro.org/people/kevin.hilman/pyboot.git

In any case, the combination of those build-scripts and pyboot are
what I'm using to generated the automated build/boot reports I'm
sending to the kernel-build-reports list (e.g.
http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html)

Kevin

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Olof Johansson]

On Wed, Aug 13, 2014 at 8:00 AM, Kevin Hilman <khilman@linaro.org> wrote:
> On Wed, Aug 13, 2014 at 3:40 AM, Mark Brown <broonie@kernel.org> wrote:
>> On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote:
>>> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote:
>>
>>> > While we're working on things like this it's probably worth getting the
>>> > scripts Kevin, Olof and I use for build and boot test reporting (I only
>>> > run them, I've not contributed anything) or something like them into
>>> > scripts/ - they're pretty good and having a common format makes things
>>> > easier when looking at multiple reports.
>>
>>> Where is the current copy of those scripts?
>>
>> Kevin's copy is here:
>>
>>    git://git.linaro.org/people/khilman/build-scripts.git
>
> FWIW, that repo is just a hodge-podge collection of various build/boot
> scripts I'm using.  I just cleaned it up a little and removed a bunch
> of them that I no longer use.
>
> They get the job done for me, but are certainly not pretty.
>
> Also for boot testing, the main tool I'm  using is pyboot, which is
> essentially a tool to automate u-boot/fastboot over the serial
> console.  It relies heavily on conmux to abstract the console and
> power cycling.
>
>     https://git.linaro.org/people/kevin.hilman/pyboot.git
>
> In any case, the combination of those build-scripts and pyboot are
> what I'm using to generated the automated build/boot reports I'm
> sending to the kernel-build-reports list (e.g.
> http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html)

Yeah, I'm in the same boat (but I don't have a public git repo with my
scripts). They're all just big hacks to get things going, and I'm not
so sure it makes sense to collaborate on them too much. pyboot has
been useful to share between us, but even there it's been more work to
share a code base than having separate copies and borrowing ideas.

If things get turned too generic then they just become more cumbersome
to modify and improve (or even use).


-Olof

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 932 bytes --]

On Wed, Aug 13, 2014 at 09:40:34AM -0700, Olof Johansson wrote:

> Yeah, I'm in the same boat (but I don't have a public git repo with my
> scripts). They're all just big hacks to get things going, and I'm not
> so sure it makes sense to collaborate on them too much. pyboot has
> been useful to share between us, but even there it's been more work to
> share a code base than having separate copies and borrowing ideas.

> If things get turned too generic then they just become more cumbersome
> to modify and improve (or even use).

I think the reporting side is a lot more tractable than the execution
side here - there's enough going on with crunching a set of build and
boot logs into a useful report to be useful and it's a pretty common
need.  The execution side is different, that's a lot more likely to be
tied into the environment especially once you start running actual
runtime tests, but there's still common elements.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Tim Bird]

On 08/12/2014 09:23 AM, Grant Likely wrote:
> On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>> On 08/12/2014 06:00 AM, Grant Likely wrote:
>>>
>>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu
>>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>>>
>>>> (2014/08/11 23:11), Shuah Khan wrote:
>>>>>>
>>>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger
>>>>>> goal to enable developer
>>>>>>>
>>>>>>> friendly kernel testing framework, a new make target is
>>>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>>>> fix tools/testing/sefltests to run without failures.
>>>>>>>
>>>>>>> Short summary of work done so far for 3.17:
>>>>>>>
>>>>>>> - fix compile errors and warnings in various tests
>>>>>>> - fix run-time errors when tests aren't run as root
>>>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>>>       to run in limited scope mode by default. A new make
>>>>>>>       target to select full-scope testing. Prior to this
>>>>>>>       change, cpu and memory hot-plug tests hung trying to
>>>>>>>       hot-plug all but cpu0 and a large portion of the memory.
>>>>>>> - add a new kselftest target to run existing selftests
>>>>>>>       to start with.
>>>>>>
>>>>>>
>>>>>> Instead of running the selftests, can we build the testcases and
>>>>>> install it as a tool? I think running tests on the tree is not a
>>>>>> good idea...
>>>>>
>>>>>
>>>>> One of the goals is to leverage developer tests that we already have.
>>>>> When a developer makes a kernel change and wants to see if that change
>>>>> lead to any regression, having the ability to buidl and run selftests on
>>>>> the newly installed kernel withe the same source tree is very useful.
>>>>> That is the reason behind adding this new target.
>>>>
>>>>
>>>> I see, for that purpose, installing testcase may not fit.
>>>> BTW, how would it cover cross-build?
>>>
>>>
>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> as possible. I want to make it easy to sanity test all the major
>>> architectures. Right now it does little more than boot to a login
>>> prompt, but I'd like to get the kselftests into it also.
>>>
>>
>> Do you have that public yet ? I might want to use that for my -stable sanity
>> tests.
>
> Nothing yet. It's currently hacked into my build environment script.
> I'm working on getting it into a form usable by others. Ideally I
> think it should go into the kernel tools/testing directory.

I've been doing the same thing, focused on using Aboriginal.  I have
some scripts that I've used for years for abstracting the 
build/boot/test cycle for cross-platform hardware, that I can share. 
But my qemu foo is not so good.

I thought my scripts might overlap with the ktest stuff Steve Rostedt is
doing, and wanted to chat with him at the summit about what parts might
make sense in the kernel tree, and how I might integrate my higher-level
scripts to work with ktest.pl.

As for tests, one of the ones I'd really like to see in mainline is a 
basic size test.  I have one based on my scripts I can dust off.  It 
generates the "size" for all config options by doing on vs. off 
build/boot/size comparisons for each config option.

  -- Tim

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Steven Rostedt]

On Tue, 12 Aug 2014 10:46:40 -0700
Tim Bird <tim.bird@sonymobile.com> wrote:


> I've been doing the same thing, focused on using Aboriginal.  I have
> some scripts that I've used for years for abstracting the 
> build/boot/test cycle for cross-platform hardware, that I can share. 
> But my qemu foo is not so good.
> 
> I thought my scripts might overlap with the ktest stuff Steve Rostedt is
> doing, and wanted to chat with him at the summit about what parts might
> make sense in the kernel tree, and how I might integrate my higher-level
> scripts to work with ktest.pl.

Unfortunately, I wont be at KS this year. I have a family commitment
that conflicts for the date.

I will be, though, at ELCEU. I'm suspecting that you will be there too.
Can it wait till October?

-- Steve

> 
> As for tests, one of the ones I'd really like to see in mainline is a 
> basic size test.  I have one based on my scripts I can dust off.  It 
> generates the "size" for all config options by doing on vs. off 
> build/boot/size comparisons for each config option.
> 
>   -- Tim

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Tim Bird]

On 08/12/2014 11:06 AM, Steven Rostedt wrote:
> On Tue, 12 Aug 2014 10:46:40 -0700
> Tim Bird <tim.bird@sonymobile.com> wrote:
>
>
>> I've been doing the same thing, focused on using Aboriginal.  I have
>> some scripts that I've used for years for abstracting the
>> build/boot/test cycle for cross-platform hardware, that I can share.
>> But my qemu foo is not so good.
>>
>> I thought my scripts might overlap with the ktest stuff Steve Rostedt is
>> doing, and wanted to chat with him at the summit about what parts might
>> make sense in the kernel tree, and how I might integrate my higher-level
>> scripts to work with ktest.pl.
>
> Unfortunately, I wont be at KS this year. I have a family commitment
> that conflicts for the date.
>
> I will be, though, at ELCEU. I'm suspecting that you will be there too.
> Can it wait till October?

If I feel like pushing stuff sooner, we can always chat on LKML over 
patches. :-)  But basicallly, yeah, we can chat at ELCE.
  -- Tim

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Grant Likely]

On Tue, 12 Aug 2014 14:06:37 -0400, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 12 Aug 2014 10:46:40 -0700
> Tim Bird <tim.bird@sonymobile.com> wrote:
> 
> 
> > I've been doing the same thing, focused on using Aboriginal.  I have
> > some scripts that I've used for years for abstracting the 
> > build/boot/test cycle for cross-platform hardware, that I can share. 
> > But my qemu foo is not so good.
> > 
> > I thought my scripts might overlap with the ktest stuff Steve Rostedt is
> > doing, and wanted to chat with him at the summit about what parts might
> > make sense in the kernel tree, and how I might integrate my higher-level
> > scripts to work with ktest.pl.
> 
> Unfortunately, I wont be at KS this year. I have a family commitment
> that conflicts for the date.
> 
> I will be, though, at ELCEU. I'm suspecting that you will be there too.
> Can it wait till October?

I'm sure we'll talk about it more than once.  :-)

g.

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Andy Lutomirski]

On Aug 12, 2014 9:22 AM, "Guenter Roeck" <linux@roeck-us.net> wrote:
>
> On 08/12/2014 06:00 AM, Grant Likely wrote:
>>
>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>>>
>>> (2014/08/11 23:11), Shuah Khan wrote:
>>>>>
>>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>>>>>>
>>>>>> friendly kernel testing framework, a new make target is
>>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>>> fix tools/testing/sefltests to run without failures.
>>>>>>
>>>>>> Short summary of work done so far for 3.17:
>>>>>>
>>>>>> - fix compile errors and warnings in various tests
>>>>>> - fix run-time errors when tests aren't run as root
>>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>>      to run in limited scope mode by default. A new make
>>>>>>      target to select full-scope testing. Prior to this
>>>>>>      change, cpu and memory hot-plug tests hung trying to
>>>>>>      hot-plug all but cpu0 and a large portion of the memory.
>>>>>> - add a new kselftest target to run existing selftests
>>>>>>      to start with.
>>>>>
>>>>>
>>>>> Instead of running the selftests, can we build the testcases and
>>>>> install it as a tool? I think running tests on the tree is not a
>>>>> good idea...
>>>>
>>>>
>>>> One of the goals is to leverage developer tests that we already have.
>>>> When a developer makes a kernel change and wants to see if that change
>>>> lead to any regression, having the ability to buidl and run selftests on
>>>> the newly installed kernel withe the same source tree is very useful.
>>>> That is the reason behind adding this new target.
>>>
>>>
>>> I see, for that purpose, installing testcase may not fit.
>>> BTW, how would it cover cross-build?
>>
>>
>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> very simple busybox rootfs and boots in QEMU for as many architectures
>> as possible. I want to make it easy to sanity test all the major
>> architectures. Right now it does little more than boot to a login
>> prompt, but I'd like to get the kselftests into it also.
>>
>
> Do you have that public yet ? I might want to use that for my -stable sanity tests.

I've been working on a somewhat orthogonal approach: I have a script
that boots a kernel from a bunch of files, including, optionally, the
host system.  So far, it supports x86 and arm, but adding
architectures is very simple -- it just needs some hints for how to
invoke QEMU.

https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git

One of virtme's explicit goals is to make it easy to boot a kernel,
run a script, and shut down.  The syntax for that is a bit awkward
right now, but it works.

It still needs a rootfs for non-native boots, so your tool could help with that.

--Andy

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Shuah Khan]

On 08/12/2014 07:00 AM, Grant Likely wrote:
> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2014/08/11 23:11), Shuah Khan wrote:
>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>>>>> friendly kernel testing framework, a new make target is
>>>>> planned for 3.17. In addition, 3.17 includes work done to
>>>>> fix tools/testing/sefltests to run without failures.
>>>>>
>>>>> Short summary of work done so far for 3.17:
>>>>>
>>>>> - fix compile errors and warnings in various tests
>>>>> - fix run-time errors when tests aren't run as root
>>>>> - enhance and improve cpu and memory hot-plug tests
>>>>>      to run in limited scope mode by default. A new make
>>>>>      target to select full-scope testing. Prior to this
>>>>>      change, cpu and memory hot-plug tests hung trying to
>>>>>      hot-plug all but cpu0 and a large portion of the memory.
>>>>> - add a new kselftest target to run existing selftests
>>>>>      to start with.
>>>>
>>>> Instead of running the selftests, can we build the testcases and
>>>> install it as a tool? I think running tests on the tree is not a
>>>> good idea...
>>>
>>> One of the goals is to leverage developer tests that we already have.
>>> When a developer makes a kernel change and wants to see if that change
>>> lead to any regression, having the ability to buidl and run selftests on
>>> the newly installed kernel withe the same source tree is very useful.
>>> That is the reason behind adding this new target.
>>
>> I see, for that purpose, installing testcase may not fit.
>> BTW, how would it cover cross-build?

I haven't given this too much thought yet. It is at the back on
my mind. A couple of issues that will need to be worked to get
kselftest working in cross-build env.:

- tests (Makefiles especially) probably need work to pick the right
   compiler and so on. This would be first step.
- some tests probably aren't suited for all cross env. The goal
   should be though to keep this set to minimum.

>
> I'm interested in this as well. I'm working on a tool that crossbuilds a
> very simple busybox rootfs and boots in QEMU for as many architectures
> as possible. I want to make it easy to sanity test all the major
> architectures. Right now it does little more than boot to a login
> prompt, but I'd like to get the kselftests into it also.
>

Great. Why not? The first step would be get kselftest compiling in
cross-builds. I haven't tried it yet, however in a private
response to this thread, one developer mentioned that some tests
don't pick the correct compiler result of hard-coded gcc instead
of using $(CC) and assumptions on libraries. So as we go forward
we have to get these things fixed along the way.

-- Shuah



-- 
Shuah Khan
Senior Linux Kernel Developer - Open Source Group
Samsung Research America(Silicon Valley)
shuah.kh@samsung.com | (970) 672-0658

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Linus Walleij]

On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

>> I see, for that purpose, installing testcase may not fit.
>> BTW, how would it cover cross-build?
>
> I'm interested in this as well. I'm working on a tool that crossbuilds a
> very simple busybox rootfs and boots in QEMU for as many architectures
> as possible. I want to make it easy to sanity test all the major
> architectures. Right now it does little more than boot to a login
> prompt, but I'd like to get the kselftests into it also.

Hm that sounds like a goal similar to what Rob Landley has
described as one goal for Aboriginal Linux as well.
http://landley.net/aboriginal/about.html

If such cross builds are triggered from git pushes akin to how
Fengguang's 0-day is already working we are in a positive
flow I think.

Yours,
Linus Walleij

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Guenter Roeck]

On 08/13/2014 01:35 AM, Linus Walleij wrote:
> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>>> I see, for that purpose, installing testcase may not fit.
>>> BTW, how would it cover cross-build?
>>
>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> very simple busybox rootfs and boots in QEMU for as many architectures
>> as possible. I want to make it easy to sanity test all the major
>> architectures. Right now it does little more than boot to a login
>> prompt, but I'd like to get the kselftests into it also.
>
> Hm that sounds like a goal similar to what Rob Landley has
> described as one goal for Aboriginal Linux as well.
> http://landley.net/aboriginal/about.html
>

Yes, and to some degree buildroot.

Rob's attempts to support multiple architectures also shows its limitations.
For example, his m68k images don't work, at least not for me, because the
machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
I have been unable to find a working combination of kernel configuration,
qemu version, qemu command line, and root file system for m68k. Presumably
that must exist, because qemu supports m68k, I just have not been able
to figure out how to make it work.

For my own qemu runtime tests, I ended up collecting root file systems and
kernel configurations from all over the place. And then there is the problem
of qemu command line parameters, where each target and architecture requires
its own set of options, and it is sometimes all but impossible to find a
working set of parameters for a given target/architecture combination.

Guenter

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Andy Lutomirski]

On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> On 08/13/2014 01:35 AM, Linus Walleij wrote:
>>
>> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca>
>> wrote:
>>>
>>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu
>>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>
>>
>>>> I see, for that purpose, installing testcase may not fit.
>>>> BTW, how would it cover cross-build?
>>>
>>>
>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> as possible. I want to make it easy to sanity test all the major
>>> architectures. Right now it does little more than boot to a login
>>> prompt, but I'd like to get the kselftests into it also.
>>
>>
>> Hm that sounds like a goal similar to what Rob Landley has
>> described as one goal for Aboriginal Linux as well.
>> http://landley.net/aboriginal/about.html
>>
>
> Yes, and to some degree buildroot.
>
> Rob's attempts to support multiple architectures also shows its limitations.
> For example, his m68k images don't work, at least not for me, because the
> machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
> I have been unable to find a working combination of kernel configuration,
> qemu version, qemu command line, and root file system for m68k. Presumably
> that must exist, because qemu supports m68k, I just have not been able
> to figure out how to make it work.
>
> For my own qemu runtime tests, I ended up collecting root file systems and
> kernel configurations from all over the place. And then there is the problem
> of qemu command line parameters, where each target and architecture requires
> its own set of options, and it is sometimes all but impossible to find a
> working set of parameters for a given target/architecture combination.
>

virtme has exactly this problem (except for the root image part --
virtme can use debootstrap output directly).  In virtme, I'm trying to
solve it by just collecting known-working QEMU arguments and
documenting the corresponding kernel config requirements.

--Andy

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Bird, Tim]

On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
> 
> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> > On 08/13/2014 01:35 AM, Linus Walleij wrote:
> >>
> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote:
> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a
> >>> very simple busybox rootfs and boots in QEMU for as many architectures
> >>> as possible. I want to make it easy to sanity test all the major
> >>> architectures. Right now it does little more than boot to a login
> >>> prompt, but I'd like to get the kselftests into it also.
> >>
> >>
> >> Hm that sounds like a goal similar to what Rob Landley has
> >> described as one goal for Aboriginal Linux as well.
> >> http://landley.net/aboriginal/about.html
> >>
> >
> > Yes, and to some degree buildroot.
> >
> > Rob's attempts to support multiple architectures also shows its limitations.
> > For example, his m68k images don't work, at least not for me, because the
> > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
> > I have been unable to find a working combination of kernel configuration,
> > qemu version, qemu command line, and root file system for m68k. Presumably
> > that must exist, because qemu supports m68k, I just have not been able
> > to figure out how to make it work.
> >
> > For my own qemu runtime tests, I ended up collecting root file systems and
> > kernel configurations from all over the place. And then there is the problem
> > of qemu command line parameters, where each target and architecture requires
> > its own set of options, and it is sometimes all but impossible to find a
> > working set of parameters for a given target/architecture combination.
> >
> 
> virtme has exactly this problem (except for the root image part --
> virtme can use debootstrap output directly).  In virtme, I'm trying to
> solve it by just collecting known-working QEMU arguments and
> documenting the corresponding kernel config requirements.

I'm wondering if the kernel test tree might be a good place to keep such 
virtual machine/QEMU configurations.  They should be only about 1 line
(or a few lines)  per machine, and they would be useful for automated testing.
I also think they won't change every kernel release, so it shouldn't lead to the churn
problem we had with defconfigs.

Would there be objections to hosting this information in 
the kernel source tree?
 -- Tim

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Grant Likely]

On Wed, Aug 13, 2014 at 5:44 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote:
> On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
>>
>> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
>> > On 08/13/2014 01:35 AM, Linus Walleij wrote:
>> >>
>> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote:
>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> >>> very simple busybox rootfs and boots in QEMU for as many architectures
>> >>> as possible. I want to make it easy to sanity test all the major
>> >>> architectures. Right now it does little more than boot to a login
>> >>> prompt, but I'd like to get the kselftests into it also.
>> >>
>> >>
>> >> Hm that sounds like a goal similar to what Rob Landley has
>> >> described as one goal for Aboriginal Linux as well.
>> >> http://landley.net/aboriginal/about.html
>> >>
>> >
>> > Yes, and to some degree buildroot.
>> >
>> > Rob's attempts to support multiple architectures also shows its limitations.
>> > For example, his m68k images don't work, at least not for me, because the
>> > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
>> > I have been unable to find a working combination of kernel configuration,
>> > qemu version, qemu command line, and root file system for m68k. Presumably
>> > that must exist, because qemu supports m68k, I just have not been able
>> > to figure out how to make it work.
>> >
>> > For my own qemu runtime tests, I ended up collecting root file systems and
>> > kernel configurations from all over the place. And then there is the problem
>> > of qemu command line parameters, where each target and architecture requires
>> > its own set of options, and it is sometimes all but impossible to find a
>> > working set of parameters for a given target/architecture combination.
>> >
>>
>> virtme has exactly this problem (except for the root image part --
>> virtme can use debootstrap output directly).  In virtme, I'm trying to
>> solve it by just collecting known-working QEMU arguments and
>> documenting the corresponding kernel config requirements.
>
> I'm wondering if the kernel test tree might be a good place to keep such
> virtual machine/QEMU configurations.  They should be only about 1 line
> (or a few lines)  per machine, and they would be useful for automated testing.
> I also think they won't change every kernel release, so it shouldn't lead to the churn
> problem we had with defconfigs.
>
> Would there be objections to hosting this information in
> the kernel source tree?

This is exactly what I'm trying to put together.

g.

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Andy Lutomirski]

On Wed, Aug 13, 2014 at 10:07 AM, Grant Likely
<grant.likely@secretlab.ca> wrote:
> On Wed, Aug 13, 2014 at 5:44 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote:
>> On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
>>>
>>> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
>>> > On 08/13/2014 01:35 AM, Linus Walleij wrote:
>>> >>
>>> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote:
>>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> >>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> >>> as possible. I want to make it easy to sanity test all the major
>>> >>> architectures. Right now it does little more than boot to a login
>>> >>> prompt, but I'd like to get the kselftests into it also.
>>> >>
>>> >>
>>> >> Hm that sounds like a goal similar to what Rob Landley has
>>> >> described as one goal for Aboriginal Linux as well.
>>> >> http://landley.net/aboriginal/about.html
>>> >>
>>> >
>>> > Yes, and to some degree buildroot.
>>> >
>>> > Rob's attempts to support multiple architectures also shows its limitations.
>>> > For example, his m68k images don't work, at least not for me, because the
>>> > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
>>> > I have been unable to find a working combination of kernel configuration,
>>> > qemu version, qemu command line, and root file system for m68k. Presumably
>>> > that must exist, because qemu supports m68k, I just have not been able
>>> > to figure out how to make it work.
>>> >
>>> > For my own qemu runtime tests, I ended up collecting root file systems and
>>> > kernel configurations from all over the place. And then there is the problem
>>> > of qemu command line parameters, where each target and architecture requires
>>> > its own set of options, and it is sometimes all but impossible to find a
>>> > working set of parameters for a given target/architecture combination.
>>> >
>>>
>>> virtme has exactly this problem (except for the root image part --
>>> virtme can use debootstrap output directly).  In virtme, I'm trying to
>>> solve it by just collecting known-working QEMU arguments and
>>> documenting the corresponding kernel config requirements.
>>
>> I'm wondering if the kernel test tree might be a good place to keep such
>> virtual machine/QEMU configurations.  They should be only about 1 line
>> (or a few lines)  per machine, and they would be useful for automated testing.
>> I also think they won't change every kernel release, so it shouldn't lead to the churn
>> problem we had with defconfigs.
>>
>> Would there be objections to hosting this information in
>> the kernel source tree?
>
> This is exactly what I'm trying to put together.

If this ends up in machine-readable form, it'll be extremely useful.  Thanks!

--Andy

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Guenter Roeck]

On 08/13/2014 09:44 AM, Bird, Tim wrote:
> On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote:
>>
>> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
>>> On 08/13/2014 01:35 AM, Linus Walleij wrote:
>>>>
>>>> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote:
>>>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>>>> as possible. I want to make it easy to sanity test all the major
>>>>> architectures. Right now it does little more than boot to a login
>>>>> prompt, but I'd like to get the kselftests into it also.
>>>>
>>>>
>>>> Hm that sounds like a goal similar to what Rob Landley has
>>>> described as one goal for Aboriginal Linux as well.
>>>> http://landley.net/aboriginal/about.html
>>>>
>>>
>>> Yes, and to some degree buildroot.
>>>
>>> Rob's attempts to support multiple architectures also shows its limitations.
>>> For example, his m68k images don't work, at least not for me, because the
>>> machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.
>>> I have been unable to find a working combination of kernel configuration,
>>> qemu version, qemu command line, and root file system for m68k. Presumably
>>> that must exist, because qemu supports m68k, I just have not been able
>>> to figure out how to make it work.
>>>
>>> For my own qemu runtime tests, I ended up collecting root file systems and
>>> kernel configurations from all over the place. And then there is the problem
>>> of qemu command line parameters, where each target and architecture requires
>>> its own set of options, and it is sometimes all but impossible to find a
>>> working set of parameters for a given target/architecture combination.
>>>
>>
>> virtme has exactly this problem (except for the root image part --
>> virtme can use debootstrap output directly).  In virtme, I'm trying to
>> solve it by just collecting known-working QEMU arguments and
>> documenting the corresponding kernel config requirements.
>
> I'm wondering if the kernel test tree might be a good place to keep such
> virtual machine/QEMU configurations.  They should be only about 1 line
> (or a few lines)  per machine, and they would be useful for automated testing.
> I also think they won't change every kernel release, so it shouldn't lead to the churn
> problem we had with defconfigs.
>

You would also need working configurations. There may be a few exceptions,
where one of the defconfigs can be used as is, but in most cases at least
some minor tweaks are needed to create a kernel that is working with qemu.
Or at least that is my experience.

Guenter

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Rob Landley]

On 08/13/2014 11:16 AM, Andy Lutomirski wrote:
> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote:
>> For my own qemu runtime tests, I ended up collecting root file systems and
>> kernel configurations from all over the place. And then there is the problem
>> of qemu command line parameters, where each target and architecture requires
>> its own set of options, and it is sometimes all but impossible to find a
>> working set of parameters for a given target/architecture combination.
>>
> 
> virtme has exactly this problem (except for the root image part --
> virtme can use debootstrap output directly).  In virtme, I'm trying to
> solve it by just collecting known-working QEMU arguments and
> documenting the corresponding kernel config requirements.

My aboriginal linux system is designed around the assumption that nobody
will ever actually bother to use it. (That's why I ship prebuilt binary
tarballs.)

I also write extensive documentation nobody will ever read:

http://landley.net/aboriginal/about.html

Each system-image tarball has a "run-emulator.sh" script that's just the
qemu invocation. If you want to know how to launch qemu for the target:
there you go.

The kernel config is using the "miniconfig" technique:

http://landley.net/aboriginal/FAQ.html#dev_miniconfig

You assemble the miniconfig from the target-independent parts:

http://landley.net/hg/aboriginal/file/tip/sources/baseconfig-linux

And then concatenate the target-specific bits from the relevant
sources/targets file. ala:

http://landley.net/hg/aboriginal/file/tip/sources/targets/i686#l19

(All the target-specific information is in a single file under
sources/targets. There are various patches in sources/patches some of
which are target specific. All of them should go upstream, but kernel
development is so insular it's just not worth the effort.)

If you don't want to use miniconfig, the expanded configs are in the
root-filesystem tarballs (in the src directory).

> --Andy
> .
> 

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Rob Landley]

The most interesting email conversations occur when the machine with all
my email filters is in the shop...

On 08/13/2014 11:11 AM, Guenter Roeck wrote:
> On 08/13/2014 01:35 AM, Linus Walleij wrote:
>> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely
>> <grant.likely@secretlab.ca> wrote:
>>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu
>>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>
>>>> I see, for that purpose, installing testcase may not fit.
>>>> BTW, how would it cover cross-build?
>>>
>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> as possible. I want to make it easy to sanity test all the major
>>> architectures. Right now it does little more than boot to a login
>>> prompt, but I'd like to get the kselftests into it also.
>>
>> Hm that sounds like a goal similar to what Rob Landley has
>> described as one goal for Aboriginal Linux as well.
>> http://landley.net/aboriginal/about.html
>>
> 
> Yes, and to some degree buildroot.

Aboriginal is intentionally building the simplest system capable of
rebuilding itself under itself, with an emphasis on native compiling
from there.

(It's sort of in pieces on the floor at the moment as I switch from
uClibc to musl, which breaks rather a lot of subtle assumptions and
requires everything to be debugged again. Oh well.)

I'd happily make it work with other people's toolchains, but I've never
managed to beat a working _native_ toolchain out of buildroot, it cross
compiles everything and then the resulting system is not expected to be
a development environment. Building on an emulated target system is
apparently not most people's idea of fun.

> Rob's attempts to support multiple architectures also shows its
> limitations.

Some of this is the limitations of the toolchain I'm using, which is the
last GPLv2 release of gcc and binutils. I don't mess with GPLv3 code
unless it's for work, it is _never_ fun.

I keep hoping http://ellcc.org or similar will approach reasonably
functional, but so far that one isn't even using http://lld.llvm.org
instead of binutils, so...

(Also, the llvm build is checking for gcc 4.7 at compile time and
refuses to build on anything older. Yes, llvm has incestuous
dependencies on gcc.)

> For example, his m68k images don't work, at least not for me, because the
> machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1.

Laurent Vivier is working on a qemu fork to add support for that, but
neither he nor I have had time to put into it in forever.

It's something like git://gitorious.org/qemu-m68k/qemu-m68k branch
refs/heads/q800

(My dayjob has nothing to do with anything interesting, I'm paid to work
on a a company's fork of a vendor's bsp fork of a really old kernel. For
the new board, we just upgraded _to_ a version that's only 4 years old.
When they finally ship there will be a nominal compliance tarball put on
a website somewhere that nobody will ever look at or care about. I keep
meaning to put up a patreon to see if people would be willing to sponsor
me to spend all my time on actually _interesting_ stuff like this, but
the chances are low enough I haven't bothered.)

> I have been unable to find a working combination of kernel configuration,
> qemu version, qemu command line, and root file system for m68k. Presumably
> that must exist, because qemu supports m68k, I just have not been able
> to figure out how to make it work.

QEMU does not support m68k, qemu supports coldfire which is a nommu
subset of m68k. But people have used the aranym emulator to fake an
atari machine that _has_ run the root filesystems I've been building.
(With a different kernel config, and a largeish patch for aranym devices
that since went upstream. But it showed the basics were right.)

The problem with using aranym is my setup expects a certain pile of
devices. If /dev/console goes to stdin/stdout than it's easy to script
the sucker with tcl/expect and log the output with "tee". If you have a
virtual network card you can move the heavy lifting of compilation
outside the emulator with distcc hooked up to the cross compiler
(_without_ reintroducing the horrible complexity of cross compiling
because configure and make and linking and header preprocessing and such
all run natively inside the emulator where there's only a single build
context.) You need at least 256 megs of ram for 64 bit gcc to function.
It's really convenient to have three emulated hard drives (one for the
root filesystem, one for writeable space on /home, and one for the
source control and build scripts on /mnt. I described that setup in
http://landley.net/aboriginal/control-images and there are example build
control images there.)

Now that my initmpfs patches are in, I'm moving the
simple-root-filesystem to initmpfs so I can use network mounts instead
of filesystem images for all that... but the emulated network card
becomes a bit of a bottleneck the. The emulated hard drives tend to
fling data around faster. (Also I wanted to use v9fs but the v9fs
developers went ABSOLUTELY INSANE and accepted an IBM patch that broke
things so badly Red Hat yanked the 9p source code out of their kernel.)

http://sourceforge.net/p/v9fs/mailman/message/31629549/

http://scientificlinuxforum.org/index.php?showtopic=2858

> For my own qemu runtime tests, I ended up collecting root file systems and
> kernel configurations from all over the place. And then there is the
> problem
> of qemu command line parameters, where each target and architecture
> requires
> its own set of options, and it is sometimes all but impossible to find a
> working set of parameters for a given target/architecture combination.

Tell me about it. I collect them, and my system images document them. I
was in the process of trying to get Alpha to work when musl got close
enough to 1.0 that switching over became a higher priority, and now
there's the chicken and egg problem of adding musl support and getting
an emulated development enviornment working...

Mostly it's just "dayjob eats my time/energy, I do what I can in the
time I have left".

> Guenter

Rob

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Geert Uytterhoeven]

Hi Rob,

On Mon, Aug 18, 2014 at 5:08 AM, Rob Landley <rob@landley.net> wrote:
> (My dayjob has nothing to do with anything interesting, I'm paid to work
> on a a company's fork of a vendor's bsp fork of a really old kernel. For
> the new board, we just upgraded _to_ a version that's only 4 years old.
> When they finally ship there will be a nominal compliance tarball put on
> a website somewhere that nobody will ever look at or care about. I keep
> meaning to put up a patreon to see if people would be willing to sponsor
> me to spend all my time on actually _interesting_ stuff like this, but
> the chances are low enough I haven't bothered.)
>
>> I have been unable to find a working combination of kernel configuration,
>> qemu version, qemu command line, and root file system for m68k. Presumably
>> that must exist, because qemu supports m68k, I just have not been able
>> to figure out how to make it work.
>
> QEMU does not support m68k, qemu supports coldfire which is a nommu
> subset of m68k. But people have used the aranym emulator to fake an
> atari machine that _has_ run the root filesystems I've been building.
> (With a different kernel config, and a largeish patch for aranym devices
> that since went upstream. But it showed the basics were right.)

These days atari_defconfig in upstream should run fine on ARAnyM.

> The problem with using aranym is my setup expects a certain pile of
> devices. If /dev/console goes to stdin/stdout than it's easy to script
> the sucker with tcl/expect and log the output with "tee". If you have a

Ah, ARAnyM's nfcon only does stdout, not stdin.

> virtual network card you can move the heavy lifting of compilation
> outside the emulator with distcc hooked up to the cross compiler

But nfeth works fine.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Masami Hiramatsu]

(2014/08/13 17:35), Linus Walleij wrote:
> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>>> I see, for that purpose, installing testcase may not fit.
>>> BTW, how would it cover cross-build?
>>
>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>> very simple busybox rootfs and boots in QEMU for as many architectures
>> as possible. I want to make it easy to sanity test all the major
>> architectures. Right now it does little more than boot to a login
>> prompt, but I'd like to get the kselftests into it also.
> 
> Hm that sounds like a goal similar to what Rob Landley has
> described as one goal for Aboriginal Linux as well.
> http://landley.net/aboriginal/about.html

Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal
distro may not have "make" on their rootfs. In that case, we need to make self-
executable binaries and install it with testing "ash" scripts so that it can
run on busybox. It may be just a technical issue.

> If such cross builds are triggered from git pushes akin to how
> Fengguang's 0-day is already working we are in a positive
> flow I think.

Agreed.

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Rob Landley]

On 08/13/2014 11:45 AM, Masami Hiramatsu wrote:
> (2014/08/13 17:35), Linus Walleij wrote:
>> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>>
>>>> I see, for that purpose, installing testcase may not fit.
>>>> BTW, how would it cover cross-build?
>>>
>>> I'm interested in this as well. I'm working on a tool that crossbuilds a
>>> very simple busybox rootfs and boots in QEMU for as many architectures
>>> as possible. I want to make it easy to sanity test all the major
>>> architectures. Right now it does little more than boot to a login
>>> prompt, but I'd like to get the kselftests into it also.
>>
>> Hm that sounds like a goal similar to what Rob Landley has
>> described as one goal for Aboriginal Linux as well.
>> http://landley.net/aboriginal/about.html
> 
> Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal
> distro may not have "make" on their rootfs. In that case, we need to make self-
> executable binaries and install it with testing "ash" scripts so that it can
> run on busybox. It may be just a technical issue.

Aboriginal Linux has make in the native root filesystem. (Again, last
gplv2 version so it's a touch stale now. Back when I did busybox stuff I
planned to write a make for that, and now I plan to write one for
toybox, but it's a post 1.0 todo item.)

There's also a native-compiler tarball that's just the stuff it adds on
top of the simple-root-filesystem to make the development environment.
(This includes make and bash 2.05b.)

Now that my initmpfs patches are upstream, I'm refactoring things to put
the simple-root-filesystem in initramfs and splice the native-compiler
stuff from hda into the $PATH at runtime, in the init script. But I'm
two kernel releases behind, and should probably ship a 3.15 version
before doing much more development...

Rob

(P.S. Making a stripped down linux from scratch build based on busybox
and uclibc is why I got into busybox development in the first place.
That's why I wound up doing so much work on it I got the maintainership
handed to me for a bit. Now I'm doing it all again in toybox because
Android won't use busybox because it's under the GPL.)

Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond

[Aneesh Kumar K.V]

Grant Likely <grant.likely@secretlab.ca> writes:

> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>> (2014/08/11 23:11), Shuah Khan wrote:
>> >> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer
>> >>> friendly kernel testing framework, a new make target is
>> >>> planned for 3.17. In addition, 3.17 includes work done to
>> >>> fix tools/testing/sefltests to run without failures.
>> >>>
>> >>> Short summary of work done so far for 3.17:
>> >>>
>> >>> - fix compile errors and warnings in various tests
>> >>> - fix run-time errors when tests aren't run as root
>> >>> - enhance and improve cpu and memory hot-plug tests
>> >>>     to run in limited scope mode by default. A new make
>> >>>     target to select full-scope testing. Prior to this
>> >>>     change, cpu and memory hot-plug tests hung trying to
>> >>>     hot-plug all but cpu0 and a large portion of the memory.
>> >>> - add a new kselftest target to run existing selftests
>> >>>     to start with.
>> >>
>> >> Instead of running the selftests, can we build the testcases and
>> >> install it as a tool? I think running tests on the tree is not a
>> >> good idea...
>> > 
>> > One of the goals is to leverage developer tests that we already have.
>> > When a developer makes a kernel change and wants to see if that change
>> > lead to any regression, having the ability to buidl and run selftests on 
>> > the newly installed kernel withe the same source tree is very useful.
>> > That is the reason behind adding this new target.
>> 
>> I see, for that purpose, installing testcase may not fit.
>> BTW, how would it cover cross-build?
>
> I'm interested in this as well. I'm working on a tool that crossbuilds a
> very simple busybox rootfs and boots in QEMU for as many architectures
> as possible. I want to make it easy to sanity test all the major
> architectures. Right now it does little more than boot to a login
> prompt, but I'd like to get the kselftests into it also.
>

I am interested in this as well, I will reach Chicago only on 18th
afternoon (due to flight availability issues). I have been using
https://github.com/autotest/autotest-client-tests for running sanity
tests on different kernel versions. It can be driven with sandboxed setup
using https://github.com/autotest/virt-test

-aneesh