* [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond @ 2014-08-07 14:36 Shuah Khan 2014-08-07 18:24 ` Bird, Tim 2014-08-08 2:18 ` Masami Hiramatsu 0 siblings, 2 replies; 40+ messages in thread From: Shuah Khan @ 2014-08-07 14:36 UTC (permalink / raw) To: ksummit-discuss; +Cc: Greg Kroah-Hartman As a first step towards a larger goal to enable developer friendly kernel testing framework, a new make target is planned for 3.17. In addition, 3.17 includes work done to fix tools/testing/sefltests to run without failures. Short summary of work done so far for 3.17: - fix compile errors and warnings in various tests - fix run-time errors when tests aren't run as root - enhance and improve cpu and memory hot-plug tests to run in limited scope mode by default. A new make target to select full-scope testing. Prior to this change, cpu and memory hot-plug tests hung trying to hot-plug all but cpu0 and a large portion of the memory. - add a new kselftest target to run existing selftests to start with. What's planned for 3.18 and beyond: - get feedback on the new kselftest target from the community - add more tests to be run under kselftest umbrella - identify existing tests under /lib and other areas that make a good candidate to be included under kselftest - Some of these could be run as a tool and/or a independent test with a few changes and some probably aren't like the /lib/locking tests. - As a goal, try to leverage existing tests and modify them as needed to run them as a black-box test (e.g: look into ways to make it run as a tool) - Greg KH sparked the kernel selftest idea, has been in the loop for the work done so far, and reviewed the plan for 3.18. thanks, -- Shuah Shuah Khan Senior Linux Kernel Developer - Open Source Group Samsung Research America(Silicon Valley) shuah.kh@samsung.com | (970) 672-0658 ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-07 14:36 [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond Shuah Khan @ 2014-08-07 18:24 ` Bird, Tim 2014-08-07 19:59 ` Shuah Khan 2014-08-08 2:18 ` Masami Hiramatsu 1 sibling, 1 reply; 40+ messages in thread From: Bird, Tim @ 2014-08-07 18:24 UTC (permalink / raw) To: shuah.kh, ksummit-discuss; +Cc: Greg Kroah-Hartman On Thursday, August 07, 2014 7:36 AM, Shuah Khan wrote: > > As a first step towards a larger goal to enable developer > friendly kernel testing framework, a new make target is > planned for 3.17. In addition, 3.17 includes work done to > fix tools/testing/sefltests to run without failures. > > Short summary of work done so far for 3.17: > > - fix compile errors and warnings in various tests > - fix run-time errors when tests aren't run as root > - enhance and improve cpu and memory hot-plug tests > to run in limited scope mode by default. A new make > target to select full-scope testing. Prior to this > change, cpu and memory hot-plug tests hung trying to > hot-plug all but cpu0 and a large portion of the memory. > - add a new kselftest target to run existing selftests > to start with. > > What's planned for 3.18 and beyond: > > - get feedback on the new kselftest target from the community > - add more tests to be run under kselftest umbrella > - identify existing tests under /lib and other areas that > make a good candidate to be included under kselftest > - Some of these could be run as a tool and/or a independent > test with a few changes and some probably aren't like the > /lib/locking tests. > - As a goal, try to leverage existing tests and modify them > as needed to run them as a black-box test (e.g: look into > ways to make it run as a tool) > - Greg KH sparked the kernel selftest idea, has been in the loop > for the work done so far, and reviewed the plan for 3.18. I'm quite interested in this (as is the CEWG), and I have lots of questions. Will this be discussed/presented at KS or LinuxCon? Should I ask my questions on this list or wait until Chicago? Are there any LKML threads with info I can look at in the meantime? Thanks. This sounds like great work. -- Tim ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-07 18:24 ` Bird, Tim @ 2014-08-07 19:59 ` Shuah Khan 0 siblings, 0 replies; 40+ messages in thread From: Shuah Khan @ 2014-08-07 19:59 UTC (permalink / raw) To: Bird, Tim, ksummit-discuss; +Cc: Greg Kroah-Hartman, Shuah Khan On 08/07/2014 12:24 PM, Bird, Tim wrote: > > I'm quite interested in this (as is the CEWG), and I have lots of questions. > Will this be discussed/presented at KS or LinuxCon? Should I ask my > questions on this list or wait until Chicago? Are there any LKML threads > with info I can look at in the meantime? > > Thanks. This sounds like great work. The plan to discuss this at the KS. Please continue this thread with questions you have. I can point you to links to the patches for the work done so far in 3.17 if you are interested. -- Shuah -- Shuah Khan Senior Linux Kernel Developer - Open Source Group Samsung Research America(Silicon Valley) shuah.kh@samsung.com | (970) 672-0658 ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-07 14:36 [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond Shuah Khan 2014-08-07 18:24 ` Bird, Tim @ 2014-08-08 2:18 ` Masami Hiramatsu 2014-08-11 14:11 ` Shuah Khan 1 sibling, 1 reply; 40+ messages in thread From: Masami Hiramatsu @ 2014-08-08 2:18 UTC (permalink / raw) To: shuah.kh; +Cc: Greg Kroah-Hartman, ksummit-discuss Hello, I'm also interested in the selftests, especially its framework (top-level script). http://article.gmane.org/gmane.linux.kernel/1763766 (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer > friendly kernel testing framework, a new make target is > planned for 3.17. In addition, 3.17 includes work done to > fix tools/testing/sefltests to run without failures. > > Short summary of work done so far for 3.17: > > - fix compile errors and warnings in various tests > - fix run-time errors when tests aren't run as root > - enhance and improve cpu and memory hot-plug tests > to run in limited scope mode by default. A new make > target to select full-scope testing. Prior to this > change, cpu and memory hot-plug tests hung trying to > hot-plug all but cpu0 and a large portion of the memory. > - add a new kselftest target to run existing selftests > to start with. Instead of running the selftests, can we build the testcases and install it as a tool? I think running tests on the tree is not a good idea... Also, as I said in above mail, I'd like to suggest to add at least log management and statistics to the test. Those are good to automate regression tests. :) > What's planned for 3.18 and beyond: > > - get feedback on the new kselftest target from the community > - add more tests to be run under kselftest umbrella > - identify existing tests under /lib and other areas that > make a good candidate to be included under kselftest > - Some of these could be run as a tool and/or a independent > test with a few changes and some probably aren't like the > /lib/locking tests. > - As a goal, try to leverage existing tests and modify them > as needed to run them as a black-box test (e.g: look into > ways to make it run as a tool) > - Greg KH sparked the kernel selftest idea, has been in the loop > for the work done so far, and reviewed the plan for 3.18. > Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-08 2:18 ` Masami Hiramatsu @ 2014-08-11 14:11 ` Shuah Khan 2014-08-11 16:13 ` Masami Hiramatsu 0 siblings, 1 reply; 40+ messages in thread From: Shuah Khan @ 2014-08-11 14:11 UTC (permalink / raw) To: Masami Hiramatsu; +Cc: Greg Kroah-Hartman, ksummit-discuss, shuah Khan On 08/07/2014 08:18 PM, Masami Hiramatsu wrote: > Hello, > > I'm also interested in the selftests, especially its > framework (top-level script). > > http://article.gmane.org/gmane.linux.kernel/1763766 Great. > > (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >> friendly kernel testing framework, a new make target is >> planned for 3.17. In addition, 3.17 includes work done to >> fix tools/testing/sefltests to run without failures. >> >> Short summary of work done so far for 3.17: >> >> - fix compile errors and warnings in various tests >> - fix run-time errors when tests aren't run as root >> - enhance and improve cpu and memory hot-plug tests >> to run in limited scope mode by default. A new make >> target to select full-scope testing. Prior to this >> change, cpu and memory hot-plug tests hung trying to >> hot-plug all but cpu0 and a large portion of the memory. >> - add a new kselftest target to run existing selftests >> to start with. > > Instead of running the selftests, can we build the testcases and > install it as a tool? I think running tests on the tree is not a > good idea... One of the goals is to leverage developer tests that we already have. When a developer makes a kernel change and wants to see if that change lead to any regression, having the ability to buidl and run selftests on the newly installed kernel withe the same source tree is very useful. That is the reason behind adding this new target. Existing selfests are a collection of tools that exercise several testcases that are specific each area they target. This is a good start and we can definitely start enhancing the tools and build testcases. > Also, as I said in above mail, I'd like to suggest to add at least > log management and statistics to the test. Those are good to automate > regression tests. :) > Right. Some existing selftests do that, but not all. It would be helpful to add that to all. -- Shuah -- Shuah Khan Senior Linux Kernel Developer - Open Source Group Samsung Research America(Silicon Valley) shuah.kh@samsung.com | (970) 672-0658 ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-11 14:11 ` Shuah Khan @ 2014-08-11 16:13 ` Masami Hiramatsu 2014-08-12 13:00 ` Grant Likely 0 siblings, 1 reply; 40+ messages in thread From: Masami Hiramatsu @ 2014-08-11 16:13 UTC (permalink / raw) To: shuah.kh; +Cc: Greg Kroah-Hartman, ksummit-discuss (2014/08/11 23:11), Shuah Khan wrote: >> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >>> friendly kernel testing framework, a new make target is >>> planned for 3.17. In addition, 3.17 includes work done to >>> fix tools/testing/sefltests to run without failures. >>> >>> Short summary of work done so far for 3.17: >>> >>> - fix compile errors and warnings in various tests >>> - fix run-time errors when tests aren't run as root >>> - enhance and improve cpu and memory hot-plug tests >>> to run in limited scope mode by default. A new make >>> target to select full-scope testing. Prior to this >>> change, cpu and memory hot-plug tests hung trying to >>> hot-plug all but cpu0 and a large portion of the memory. >>> - add a new kselftest target to run existing selftests >>> to start with. >> >> Instead of running the selftests, can we build the testcases and >> install it as a tool? I think running tests on the tree is not a >> good idea... > > One of the goals is to leverage developer tests that we already have. > When a developer makes a kernel change and wants to see if that change > lead to any regression, having the ability to buidl and run selftests on > the newly installed kernel withe the same source tree is very useful. > That is the reason behind adding this new target. I see, for that purpose, installing testcase may not fit. BTW, how would it cover cross-build? > Existing selfests are a collection of tools that exercise several > testcases that are specific each area they target. This is a good > start and we can definitely start enhancing the tools and build > testcases. Agreed. > >> Also, as I said in above mail, I'd like to suggest to add at least >> log management and statistics to the test. Those are good to automate >> regression tests. :) >> > > Right. Some existing selftests do that, but not all. It would be > helpful to add that to all. Yeah, and I hope the selftest top-level script to provide such functions to each test, because those should be common, and easy to manage. Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-11 16:13 ` Masami Hiramatsu @ 2014-08-12 13:00 ` Grant Likely 2014-08-12 16:15 ` Guenter Roeck ` (3 more replies) 0 siblings, 4 replies; 40+ messages in thread From: Grant Likely @ 2014-08-12 13:00 UTC (permalink / raw) To: Masami Hiramatsu, shuah.kh; +Cc: Greg Kroah-Hartman, ksummit-discuss On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: > (2014/08/11 23:11), Shuah Khan wrote: > >> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer > >>> friendly kernel testing framework, a new make target is > >>> planned for 3.17. In addition, 3.17 includes work done to > >>> fix tools/testing/sefltests to run without failures. > >>> > >>> Short summary of work done so far for 3.17: > >>> > >>> - fix compile errors and warnings in various tests > >>> - fix run-time errors when tests aren't run as root > >>> - enhance and improve cpu and memory hot-plug tests > >>> to run in limited scope mode by default. A new make > >>> target to select full-scope testing. Prior to this > >>> change, cpu and memory hot-plug tests hung trying to > >>> hot-plug all but cpu0 and a large portion of the memory. > >>> - add a new kselftest target to run existing selftests > >>> to start with. > >> > >> Instead of running the selftests, can we build the testcases and > >> install it as a tool? I think running tests on the tree is not a > >> good idea... > > > > One of the goals is to leverage developer tests that we already have. > > When a developer makes a kernel change and wants to see if that change > > lead to any regression, having the ability to buidl and run selftests on > > the newly installed kernel withe the same source tree is very useful. > > That is the reason behind adding this new target. > > I see, for that purpose, installing testcase may not fit. > BTW, how would it cover cross-build? I'm interested in this as well. I'm working on a tool that crossbuilds a very simple busybox rootfs and boots in QEMU for as many architectures as possible. I want to make it easy to sanity test all the major architectures. Right now it does little more than boot to a login prompt, but I'd like to get the kselftests into it also. g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 13:00 ` Grant Likely @ 2014-08-12 16:15 ` Guenter Roeck 2014-08-12 16:21 ` Masami Hiramatsu ` (2 more replies) 2014-08-12 16:34 ` Shuah Khan ` (2 subsequent siblings) 3 siblings, 3 replies; 40+ messages in thread From: Guenter Roeck @ 2014-08-12 16:15 UTC (permalink / raw) To: Grant Likely, Masami Hiramatsu, shuah.kh Cc: Greg Kroah-Hartman, ksummit-discuss On 08/12/2014 06:00 AM, Grant Likely wrote: > On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >> (2014/08/11 23:11), Shuah Khan wrote: >>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >>>>> friendly kernel testing framework, a new make target is >>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>> fix tools/testing/sefltests to run without failures. >>>>> >>>>> Short summary of work done so far for 3.17: >>>>> >>>>> - fix compile errors and warnings in various tests >>>>> - fix run-time errors when tests aren't run as root >>>>> - enhance and improve cpu and memory hot-plug tests >>>>> to run in limited scope mode by default. A new make >>>>> target to select full-scope testing. Prior to this >>>>> change, cpu and memory hot-plug tests hung trying to >>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>> - add a new kselftest target to run existing selftests >>>>> to start with. >>>> >>>> Instead of running the selftests, can we build the testcases and >>>> install it as a tool? I think running tests on the tree is not a >>>> good idea... >>> >>> One of the goals is to leverage developer tests that we already have. >>> When a developer makes a kernel change and wants to see if that change >>> lead to any regression, having the ability to buidl and run selftests on >>> the newly installed kernel withe the same source tree is very useful. >>> That is the reason behind adding this new target. >> >> I see, for that purpose, installing testcase may not fit. >> BTW, how would it cover cross-build? > > I'm interested in this as well. I'm working on a tool that crossbuilds a > very simple busybox rootfs and boots in QEMU for as many architectures > as possible. I want to make it easy to sanity test all the major > architectures. Right now it does little more than boot to a login > prompt, but I'd like to get the kselftests into it also. > Do you have that public yet ? I might want to use that for my -stable sanity tests. Thanks, Guenter ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:15 ` Guenter Roeck @ 2014-08-12 16:21 ` Masami Hiramatsu 2014-08-12 16:51 ` Geert Uytterhoeven 2014-08-12 17:15 ` Theodore Ts'o 2014-08-12 16:23 ` Grant Likely 2014-08-12 17:30 ` Andy Lutomirski 2 siblings, 2 replies; 40+ messages in thread From: Masami Hiramatsu @ 2014-08-12 16:21 UTC (permalink / raw) To: Guenter Roeck; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman (2014/08/13 1:15), Guenter Roeck wrote: > On 08/12/2014 06:00 AM, Grant Likely wrote: >> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >>> (2014/08/11 23:11), Shuah Khan wrote: >>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >>>>>> friendly kernel testing framework, a new make target is >>>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>>> fix tools/testing/sefltests to run without failures. >>>>>> >>>>>> Short summary of work done so far for 3.17: >>>>>> >>>>>> - fix compile errors and warnings in various tests >>>>>> - fix run-time errors when tests aren't run as root >>>>>> - enhance and improve cpu and memory hot-plug tests >>>>>> to run in limited scope mode by default. A new make >>>>>> target to select full-scope testing. Prior to this >>>>>> change, cpu and memory hot-plug tests hung trying to >>>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>>> - add a new kselftest target to run existing selftests >>>>>> to start with. >>>>> >>>>> Instead of running the selftests, can we build the testcases and >>>>> install it as a tool? I think running tests on the tree is not a >>>>> good idea... >>>> >>>> One of the goals is to leverage developer tests that we already have. >>>> When a developer makes a kernel change and wants to see if that change >>>> lead to any regression, having the ability to buidl and run selftests on >>>> the newly installed kernel withe the same source tree is very useful. >>>> That is the reason behind adding this new target. >>> >>> I see, for that purpose, installing testcase may not fit. >>> BTW, how would it cover cross-build? >> >> I'm interested in this as well. I'm working on a tool that crossbuilds a >> very simple busybox rootfs and boots in QEMU for as many architectures >> as possible. I want to make it easy to sanity test all the major >> architectures. Right now it does little more than boot to a login >> prompt, but I'd like to get the kselftests into it also. >> > > Do you have that public yet ? I might want to use that for my -stable sanity tests. IMHO, We'd better share those testing tools/environments as much as possible so that each developer can ensure no regressions in their patches before sending it. Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:21 ` Masami Hiramatsu @ 2014-08-12 16:51 ` Geert Uytterhoeven 2014-08-12 17:15 ` Theodore Ts'o 1 sibling, 0 replies; 40+ messages in thread From: Geert Uytterhoeven @ 2014-08-12 16:51 UTC (permalink / raw) To: Masami Hiramatsu; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman On Tue, Aug 12, 2014 at 6:21 PM, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> as possible. I want to make it easy to sanity test all the major >>> architectures. Right now it does little more than boot to a login >>> prompt, but I'd like to get the kselftests into it also. Like Aboriginal Linux (landley.net/aboriginal/)? >> Do you have that public yet ? I might want to use that for my -stable sanity tests. > > IMHO, We'd better share those testing tools/environments as much as > possible so that each developer can ensure no regressions in their > patches before sending it. Yes! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:21 ` Masami Hiramatsu 2014-08-12 16:51 ` Geert Uytterhoeven @ 2014-08-12 17:15 ` Theodore Ts'o 1 sibling, 0 replies; 40+ messages in thread From: Theodore Ts'o @ 2014-08-12 17:15 UTC (permalink / raw) To: Masami Hiramatsu; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman In case it's useful, this is the tool I use to build a root_fs for ext4 testing: https://git.kernel.org/cgit/fs/ext2/xfstests-bld.git/tree/kvm-xfstests/test-appliance/gen-image There's an automated test running and xfstests results parser in there as well, but that's probably a bit less useful for non-file-system developers. - Ted ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:15 ` Guenter Roeck 2014-08-12 16:21 ` Masami Hiramatsu @ 2014-08-12 16:23 ` Grant Likely 2014-08-12 16:49 ` Mark Brown 2014-08-12 17:46 ` Tim Bird 2014-08-12 17:30 ` Andy Lutomirski 2 siblings, 2 replies; 40+ messages in thread From: Grant Likely @ 2014-08-12 16:23 UTC (permalink / raw) To: Guenter Roeck; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote: > On 08/12/2014 06:00 AM, Grant Likely wrote: >> >> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu >> <masami.hiramatsu.pt@hitachi.com> wrote: >>> >>> (2014/08/11 23:11), Shuah Khan wrote: >>>>> >>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger >>>>> goal to enable developer >>>>>> >>>>>> friendly kernel testing framework, a new make target is >>>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>>> fix tools/testing/sefltests to run without failures. >>>>>> >>>>>> Short summary of work done so far for 3.17: >>>>>> >>>>>> - fix compile errors and warnings in various tests >>>>>> - fix run-time errors when tests aren't run as root >>>>>> - enhance and improve cpu and memory hot-plug tests >>>>>> to run in limited scope mode by default. A new make >>>>>> target to select full-scope testing. Prior to this >>>>>> change, cpu and memory hot-plug tests hung trying to >>>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>>> - add a new kselftest target to run existing selftests >>>>>> to start with. >>>>> >>>>> >>>>> Instead of running the selftests, can we build the testcases and >>>>> install it as a tool? I think running tests on the tree is not a >>>>> good idea... >>>> >>>> >>>> One of the goals is to leverage developer tests that we already have. >>>> When a developer makes a kernel change and wants to see if that change >>>> lead to any regression, having the ability to buidl and run selftests on >>>> the newly installed kernel withe the same source tree is very useful. >>>> That is the reason behind adding this new target. >>> >>> >>> I see, for that purpose, installing testcase may not fit. >>> BTW, how would it cover cross-build? >> >> >> I'm interested in this as well. I'm working on a tool that crossbuilds a >> very simple busybox rootfs and boots in QEMU for as many architectures >> as possible. I want to make it easy to sanity test all the major >> architectures. Right now it does little more than boot to a login >> prompt, but I'd like to get the kselftests into it also. >> > > Do you have that public yet ? I might want to use that for my -stable sanity > tests. Nothing yet. It's currently hacked into my build environment script. I'm working on getting it into a form usable by others. Ideally I think it should go into the kernel tools/testing directory. g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:23 ` Grant Likely @ 2014-08-12 16:49 ` Mark Brown 2014-08-13 6:26 ` Grant Likely 2014-08-12 17:46 ` Tim Bird 1 sibling, 1 reply; 40+ messages in thread From: Mark Brown @ 2014-08-12 16:49 UTC (permalink / raw) To: Grant Likely; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman [-- Attachment #1: Type: text/plain, Size: 760 bytes --] On Tue, Aug 12, 2014 at 05:23:28PM +0100, Grant Likely wrote: > On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote: > > Do you have that public yet ? I might want to use that for my -stable sanity > > tests. > Nothing yet. It's currently hacked into my build environment script. > I'm working on getting it into a form usable by others. Ideally I > think it should go into the kernel tools/testing directory. While we're working on things like this it's probably worth getting the scripts Kevin, Olof and I use for build and boot test reporting (I only run them, I've not contributed anything) or something like them into scripts/ - they're pretty good and having a common format makes things easier when looking at multiple reports. [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:49 ` Mark Brown @ 2014-08-13 6:26 ` Grant Likely 2014-08-13 10:40 ` Mark Brown 0 siblings, 1 reply; 40+ messages in thread From: Grant Likely @ 2014-08-13 6:26 UTC (permalink / raw) To: Mark Brown; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote: > On Tue, Aug 12, 2014 at 05:23:28PM +0100, Grant Likely wrote: >> On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote: > >> > Do you have that public yet ? I might want to use that for my -stable sanity >> > tests. > >> Nothing yet. It's currently hacked into my build environment script. >> I'm working on getting it into a form usable by others. Ideally I >> think it should go into the kernel tools/testing directory. > > While we're working on things like this it's probably worth getting the > scripts Kevin, Olof and I use for build and boot test reporting (I only > run them, I've not contributed anything) or something like them into > scripts/ - they're pretty good and having a common format makes things > easier when looking at multiple reports. Where is the current copy of those scripts? g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 6:26 ` Grant Likely @ 2014-08-13 10:40 ` Mark Brown 2014-08-13 11:12 ` Geert Uytterhoeven 2014-08-13 15:00 ` Kevin Hilman 0 siblings, 2 replies; 40+ messages in thread From: Mark Brown @ 2014-08-13 10:40 UTC (permalink / raw) To: Grant Likely; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman [-- Attachment #1: Type: text/plain, Size: 610 bytes --] On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote: > On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote: > > While we're working on things like this it's probably worth getting the > > scripts Kevin, Olof and I use for build and boot test reporting (I only > > run them, I've not contributed anything) or something like them into > > scripts/ - they're pretty good and having a common format makes things > > easier when looking at multiple reports. > Where is the current copy of those scripts? Kevin's copy is here: git://git.linaro.org/people/khilman/build-scripts.git [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 10:40 ` Mark Brown @ 2014-08-13 11:12 ` Geert Uytterhoeven 2014-08-13 12:42 ` Mark Brown 2014-08-13 15:00 ` Kevin Hilman 1 sibling, 1 reply; 40+ messages in thread From: Geert Uytterhoeven @ 2014-08-13 11:12 UTC (permalink / raw) To: Mark Brown; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote: > On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote: >> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote: > >> > While we're working on things like this it's probably worth getting the >> > scripts Kevin, Olof and I use for build and boot test reporting (I only >> > run them, I've not contributed anything) or something like them into >> > scripts/ - they're pretty good and having a common format makes things >> > easier when looking at multiple reports. > >> Where is the current copy of those scripts? > > Kevin's copy is here: > > git://git.linaro.org/people/khilman/build-scripts.git 404 Kevin's name got expanded: git://git.linaro.org/people/kevin.hilman/build-scripts.git Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 11:12 ` Geert Uytterhoeven @ 2014-08-13 12:42 ` Mark Brown 2014-08-13 13:08 ` Geert Uytterhoeven 0 siblings, 1 reply; 40+ messages in thread From: Mark Brown @ 2014-08-13 12:42 UTC (permalink / raw) To: Geert Uytterhoeven; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman [-- Attachment #1: Type: text/plain, Size: 721 bytes --] On Wed, Aug 13, 2014 at 01:12:28PM +0200, Geert Uytterhoeven wrote: > On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote: > > Kevin's copy is here: > > git://git.linaro.org/people/khilman/build-scripts.git > 404 > Kevin's name got expanded: > git://git.linaro.org/people/kevin.hilman/build-scripts.git Interesting... how are you cloning? Both work for me: $ git clone git://git.linaro.org/people/khilman/build-scripts.git Cloning into 'build-scripts'... remote: Counting objects: 489, done. remote: Compressing objects: 100% (307/307), done. remote: Total 489 (delta 302), reused 295 (delta 181) Receiving objects: 100% (489/489), 73.90 KiB, done. Resolving deltas: 100% (302/302), done. [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 12:42 ` Mark Brown @ 2014-08-13 13:08 ` Geert Uytterhoeven 0 siblings, 0 replies; 40+ messages in thread From: Geert Uytterhoeven @ 2014-08-13 13:08 UTC (permalink / raw) To: Mark Brown; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman Hi Mark, On Wed, Aug 13, 2014 at 2:42 PM, Mark Brown <broonie@sirena.org.uk> wrote: > On Wed, Aug 13, 2014 at 01:12:28PM +0200, Geert Uytterhoeven wrote: >> On Wed, Aug 13, 2014 at 12:40 PM, Mark Brown <broonie@kernel.org> wrote: > >> > Kevin's copy is here: > >> > git://git.linaro.org/people/khilman/build-scripts.git > >> 404 > >> Kevin's name got expanded: > >> git://git.linaro.org/people/kevin.hilman/build-scripts.git > > Interesting... how are you cloning? Both work for me: > > $ git clone git://git.linaro.org/people/khilman/build-scripts.git > Cloning into 'build-scripts'... > remote: Counting objects: 489, done. > remote: Compressing objects: 100% (307/307), done. > remote: Total 489 (delta 302), reused 295 (delta 181) > Receiving objects: 100% (489/489), 73.90 KiB, done. > Resolving deltas: 100% (302/302), done. Bummer, I didn't clone, but opened http://git.linaro.org/people/khilman/build-scripts.git in my web browser by clicking on the link. So Linaro gitweb supports the full names only, while real git can access both (cloning indeed works for both). Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 10:40 ` Mark Brown 2014-08-13 11:12 ` Geert Uytterhoeven @ 2014-08-13 15:00 ` Kevin Hilman 2014-08-13 16:40 ` Olof Johansson 1 sibling, 1 reply; 40+ messages in thread From: Kevin Hilman @ 2014-08-13 15:00 UTC (permalink / raw) To: Mark Brown; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On Wed, Aug 13, 2014 at 3:40 AM, Mark Brown <broonie@kernel.org> wrote: > On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote: >> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote: > >> > While we're working on things like this it's probably worth getting the >> > scripts Kevin, Olof and I use for build and boot test reporting (I only >> > run them, I've not contributed anything) or something like them into >> > scripts/ - they're pretty good and having a common format makes things >> > easier when looking at multiple reports. > >> Where is the current copy of those scripts? > > Kevin's copy is here: > > git://git.linaro.org/people/khilman/build-scripts.git FWIW, that repo is just a hodge-podge collection of various build/boot scripts I'm using. I just cleaned it up a little and removed a bunch of them that I no longer use. They get the job done for me, but are certainly not pretty. Also for boot testing, the main tool I'm using is pyboot, which is essentially a tool to automate u-boot/fastboot over the serial console. It relies heavily on conmux to abstract the console and power cycling. https://git.linaro.org/people/kevin.hilman/pyboot.git In any case, the combination of those build-scripts and pyboot are what I'm using to generated the automated build/boot reports I'm sending to the kernel-build-reports list (e.g. http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html) Kevin ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 15:00 ` Kevin Hilman @ 2014-08-13 16:40 ` Olof Johansson 2014-08-13 17:11 ` Mark Brown 0 siblings, 1 reply; 40+ messages in thread From: Olof Johansson @ 2014-08-13 16:40 UTC (permalink / raw) To: Kevin Hilman; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman On Wed, Aug 13, 2014 at 8:00 AM, Kevin Hilman <khilman@linaro.org> wrote: > On Wed, Aug 13, 2014 at 3:40 AM, Mark Brown <broonie@kernel.org> wrote: >> On Wed, Aug 13, 2014 at 07:26:28AM +0100, Grant Likely wrote: >>> On Tue, Aug 12, 2014 at 5:49 PM, Mark Brown <broonie@kernel.org> wrote: >> >>> > While we're working on things like this it's probably worth getting the >>> > scripts Kevin, Olof and I use for build and boot test reporting (I only >>> > run them, I've not contributed anything) or something like them into >>> > scripts/ - they're pretty good and having a common format makes things >>> > easier when looking at multiple reports. >> >>> Where is the current copy of those scripts? >> >> Kevin's copy is here: >> >> git://git.linaro.org/people/khilman/build-scripts.git > > FWIW, that repo is just a hodge-podge collection of various build/boot > scripts I'm using. I just cleaned it up a little and removed a bunch > of them that I no longer use. > > They get the job done for me, but are certainly not pretty. > > Also for boot testing, the main tool I'm using is pyboot, which is > essentially a tool to automate u-boot/fastboot over the serial > console. It relies heavily on conmux to abstract the console and > power cycling. > > https://git.linaro.org/people/kevin.hilman/pyboot.git > > In any case, the combination of those build-scripts and pyboot are > what I'm using to generated the automated build/boot reports I'm > sending to the kernel-build-reports list (e.g. > http://lists.linaro.org/pipermail/kernel-build-reports/2014-August/004862.html) Yeah, I'm in the same boat (but I don't have a public git repo with my scripts). They're all just big hacks to get things going, and I'm not so sure it makes sense to collaborate on them too much. pyboot has been useful to share between us, but even there it's been more work to share a code base than having separate copies and borrowing ideas. If things get turned too generic then they just become more cumbersome to modify and improve (or even use). -Olof ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:40 ` Olof Johansson @ 2014-08-13 17:11 ` Mark Brown 0 siblings, 0 replies; 40+ messages in thread From: Mark Brown @ 2014-08-13 17:11 UTC (permalink / raw) To: Olof Johansson; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman [-- Attachment #1: Type: text/plain, Size: 932 bytes --] On Wed, Aug 13, 2014 at 09:40:34AM -0700, Olof Johansson wrote: > Yeah, I'm in the same boat (but I don't have a public git repo with my > scripts). They're all just big hacks to get things going, and I'm not > so sure it makes sense to collaborate on them too much. pyboot has > been useful to share between us, but even there it's been more work to > share a code base than having separate copies and borrowing ideas. > If things get turned too generic then they just become more cumbersome > to modify and improve (or even use). I think the reporting side is a lot more tractable than the execution side here - there's enough going on with crunching a set of build and boot logs into a useful report to be useful and it's a pretty common need. The execution side is different, that's a lot more likely to be tied into the environment especially once you start running actual runtime tests, but there's still common elements. [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:23 ` Grant Likely 2014-08-12 16:49 ` Mark Brown @ 2014-08-12 17:46 ` Tim Bird 2014-08-12 18:06 ` Steven Rostedt 1 sibling, 1 reply; 40+ messages in thread From: Tim Bird @ 2014-08-12 17:46 UTC (permalink / raw) To: Grant Likely, Guenter Roeck; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On 08/12/2014 09:23 AM, Grant Likely wrote: > On Tue, Aug 12, 2014 at 5:15 PM, Guenter Roeck <linux@roeck-us.net> wrote: >> On 08/12/2014 06:00 AM, Grant Likely wrote: >>> >>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu >>> <masami.hiramatsu.pt@hitachi.com> wrote: >>>> >>>> (2014/08/11 23:11), Shuah Khan wrote: >>>>>> >>>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger >>>>>> goal to enable developer >>>>>>> >>>>>>> friendly kernel testing framework, a new make target is >>>>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>>>> fix tools/testing/sefltests to run without failures. >>>>>>> >>>>>>> Short summary of work done so far for 3.17: >>>>>>> >>>>>>> - fix compile errors and warnings in various tests >>>>>>> - fix run-time errors when tests aren't run as root >>>>>>> - enhance and improve cpu and memory hot-plug tests >>>>>>> to run in limited scope mode by default. A new make >>>>>>> target to select full-scope testing. Prior to this >>>>>>> change, cpu and memory hot-plug tests hung trying to >>>>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>>>> - add a new kselftest target to run existing selftests >>>>>>> to start with. >>>>>> >>>>>> >>>>>> Instead of running the selftests, can we build the testcases and >>>>>> install it as a tool? I think running tests on the tree is not a >>>>>> good idea... >>>>> >>>>> >>>>> One of the goals is to leverage developer tests that we already have. >>>>> When a developer makes a kernel change and wants to see if that change >>>>> lead to any regression, having the ability to buidl and run selftests on >>>>> the newly installed kernel withe the same source tree is very useful. >>>>> That is the reason behind adding this new target. >>>> >>>> >>>> I see, for that purpose, installing testcase may not fit. >>>> BTW, how would it cover cross-build? >>> >>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> as possible. I want to make it easy to sanity test all the major >>> architectures. Right now it does little more than boot to a login >>> prompt, but I'd like to get the kselftests into it also. >>> >> >> Do you have that public yet ? I might want to use that for my -stable sanity >> tests. > > Nothing yet. It's currently hacked into my build environment script. > I'm working on getting it into a form usable by others. Ideally I > think it should go into the kernel tools/testing directory. I've been doing the same thing, focused on using Aboriginal. I have some scripts that I've used for years for abstracting the build/boot/test cycle for cross-platform hardware, that I can share. But my qemu foo is not so good. I thought my scripts might overlap with the ktest stuff Steve Rostedt is doing, and wanted to chat with him at the summit about what parts might make sense in the kernel tree, and how I might integrate my higher-level scripts to work with ktest.pl. As for tests, one of the ones I'd really like to see in mainline is a basic size test. I have one based on my scripts I can dust off. It generates the "size" for all config options by doing on vs. off build/boot/size comparisons for each config option. -- Tim ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 17:46 ` Tim Bird @ 2014-08-12 18:06 ` Steven Rostedt 2014-08-12 20:52 ` Tim Bird 2014-08-14 16:35 ` Grant Likely 0 siblings, 2 replies; 40+ messages in thread From: Steven Rostedt @ 2014-08-12 18:06 UTC (permalink / raw) To: Tim Bird; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On Tue, 12 Aug 2014 10:46:40 -0700 Tim Bird <tim.bird@sonymobile.com> wrote: > I've been doing the same thing, focused on using Aboriginal. I have > some scripts that I've used for years for abstracting the > build/boot/test cycle for cross-platform hardware, that I can share. > But my qemu foo is not so good. > > I thought my scripts might overlap with the ktest stuff Steve Rostedt is > doing, and wanted to chat with him at the summit about what parts might > make sense in the kernel tree, and how I might integrate my higher-level > scripts to work with ktest.pl. Unfortunately, I wont be at KS this year. I have a family commitment that conflicts for the date. I will be, though, at ELCEU. I'm suspecting that you will be there too. Can it wait till October? -- Steve > > As for tests, one of the ones I'd really like to see in mainline is a > basic size test. I have one based on my scripts I can dust off. It > generates the "size" for all config options by doing on vs. off > build/boot/size comparisons for each config option. > > -- Tim ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 18:06 ` Steven Rostedt @ 2014-08-12 20:52 ` Tim Bird 2014-08-14 16:35 ` Grant Likely 1 sibling, 0 replies; 40+ messages in thread From: Tim Bird @ 2014-08-12 20:52 UTC (permalink / raw) To: Steven Rostedt; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On 08/12/2014 11:06 AM, Steven Rostedt wrote: > On Tue, 12 Aug 2014 10:46:40 -0700 > Tim Bird <tim.bird@sonymobile.com> wrote: > > >> I've been doing the same thing, focused on using Aboriginal. I have >> some scripts that I've used for years for abstracting the >> build/boot/test cycle for cross-platform hardware, that I can share. >> But my qemu foo is not so good. >> >> I thought my scripts might overlap with the ktest stuff Steve Rostedt is >> doing, and wanted to chat with him at the summit about what parts might >> make sense in the kernel tree, and how I might integrate my higher-level >> scripts to work with ktest.pl. > > Unfortunately, I wont be at KS this year. I have a family commitment > that conflicts for the date. > > I will be, though, at ELCEU. I'm suspecting that you will be there too. > Can it wait till October? If I feel like pushing stuff sooner, we can always chat on LKML over patches. :-) But basicallly, yeah, we can chat at ELCE. -- Tim ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 18:06 ` Steven Rostedt 2014-08-12 20:52 ` Tim Bird @ 2014-08-14 16:35 ` Grant Likely 1 sibling, 0 replies; 40+ messages in thread From: Grant Likely @ 2014-08-14 16:35 UTC (permalink / raw) To: Steven Rostedt, Tim Bird; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On Tue, 12 Aug 2014 14:06:37 -0400, Steven Rostedt <rostedt@goodmis.org> wrote: > On Tue, 12 Aug 2014 10:46:40 -0700 > Tim Bird <tim.bird@sonymobile.com> wrote: > > > > I've been doing the same thing, focused on using Aboriginal. I have > > some scripts that I've used for years for abstracting the > > build/boot/test cycle for cross-platform hardware, that I can share. > > But my qemu foo is not so good. > > > > I thought my scripts might overlap with the ktest stuff Steve Rostedt is > > doing, and wanted to chat with him at the summit about what parts might > > make sense in the kernel tree, and how I might integrate my higher-level > > scripts to work with ktest.pl. > > Unfortunately, I wont be at KS this year. I have a family commitment > that conflicts for the date. > > I will be, though, at ELCEU. I'm suspecting that you will be there too. > Can it wait till October? I'm sure we'll talk about it more than once. :-) g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 16:15 ` Guenter Roeck 2014-08-12 16:21 ` Masami Hiramatsu 2014-08-12 16:23 ` Grant Likely @ 2014-08-12 17:30 ` Andy Lutomirski 2 siblings, 0 replies; 40+ messages in thread From: Andy Lutomirski @ 2014-08-12 17:30 UTC (permalink / raw) To: Guenter Roeck; +Cc: Greg KH, ksummit-discuss, shuah.kh On Aug 12, 2014 9:22 AM, "Guenter Roeck" <linux@roeck-us.net> wrote: > > On 08/12/2014 06:00 AM, Grant Likely wrote: >> >> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >>> >>> (2014/08/11 23:11), Shuah Khan wrote: >>>>> >>>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >>>>>> >>>>>> friendly kernel testing framework, a new make target is >>>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>>> fix tools/testing/sefltests to run without failures. >>>>>> >>>>>> Short summary of work done so far for 3.17: >>>>>> >>>>>> - fix compile errors and warnings in various tests >>>>>> - fix run-time errors when tests aren't run as root >>>>>> - enhance and improve cpu and memory hot-plug tests >>>>>> to run in limited scope mode by default. A new make >>>>>> target to select full-scope testing. Prior to this >>>>>> change, cpu and memory hot-plug tests hung trying to >>>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>>> - add a new kselftest target to run existing selftests >>>>>> to start with. >>>>> >>>>> >>>>> Instead of running the selftests, can we build the testcases and >>>>> install it as a tool? I think running tests on the tree is not a >>>>> good idea... >>>> >>>> >>>> One of the goals is to leverage developer tests that we already have. >>>> When a developer makes a kernel change and wants to see if that change >>>> lead to any regression, having the ability to buidl and run selftests on >>>> the newly installed kernel withe the same source tree is very useful. >>>> That is the reason behind adding this new target. >>> >>> >>> I see, for that purpose, installing testcase may not fit. >>> BTW, how would it cover cross-build? >> >> >> I'm interested in this as well. I'm working on a tool that crossbuilds a >> very simple busybox rootfs and boots in QEMU for as many architectures >> as possible. I want to make it easy to sanity test all the major >> architectures. Right now it does little more than boot to a login >> prompt, but I'd like to get the kselftests into it also. >> > > Do you have that public yet ? I might want to use that for my -stable sanity tests. I've been working on a somewhat orthogonal approach: I have a script that boots a kernel from a bunch of files, including, optionally, the host system. So far, it supports x86 and arm, but adding architectures is very simple -- it just needs some hints for how to invoke QEMU. https://git.kernel.org/cgit/utils/kernel/virtme/virtme.git One of virtme's explicit goals is to make it easy to boot a kernel, run a script, and shut down. The syntax for that is a bit awkward right now, but it works. It still needs a rootfs for non-native boots, so your tool could help with that. --Andy ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 13:00 ` Grant Likely 2014-08-12 16:15 ` Guenter Roeck @ 2014-08-12 16:34 ` Shuah Khan 2014-08-13 8:35 ` Linus Walleij 2014-08-13 15:06 ` Aneesh Kumar K.V 3 siblings, 0 replies; 40+ messages in thread From: Shuah Khan @ 2014-08-12 16:34 UTC (permalink / raw) To: Grant Likely, Masami Hiramatsu Cc: Greg Kroah-Hartman, ksummit-discuss, shuah Khan On 08/12/2014 07:00 AM, Grant Likely wrote: > On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >> (2014/08/11 23:11), Shuah Khan wrote: >>>> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >>>>> friendly kernel testing framework, a new make target is >>>>> planned for 3.17. In addition, 3.17 includes work done to >>>>> fix tools/testing/sefltests to run without failures. >>>>> >>>>> Short summary of work done so far for 3.17: >>>>> >>>>> - fix compile errors and warnings in various tests >>>>> - fix run-time errors when tests aren't run as root >>>>> - enhance and improve cpu and memory hot-plug tests >>>>> to run in limited scope mode by default. A new make >>>>> target to select full-scope testing. Prior to this >>>>> change, cpu and memory hot-plug tests hung trying to >>>>> hot-plug all but cpu0 and a large portion of the memory. >>>>> - add a new kselftest target to run existing selftests >>>>> to start with. >>>> >>>> Instead of running the selftests, can we build the testcases and >>>> install it as a tool? I think running tests on the tree is not a >>>> good idea... >>> >>> One of the goals is to leverage developer tests that we already have. >>> When a developer makes a kernel change and wants to see if that change >>> lead to any regression, having the ability to buidl and run selftests on >>> the newly installed kernel withe the same source tree is very useful. >>> That is the reason behind adding this new target. >> >> I see, for that purpose, installing testcase may not fit. >> BTW, how would it cover cross-build? I haven't given this too much thought yet. It is at the back on my mind. A couple of issues that will need to be worked to get kselftest working in cross-build env.: - tests (Makefiles especially) probably need work to pick the right compiler and so on. This would be first step. - some tests probably aren't suited for all cross env. The goal should be though to keep this set to minimum. > > I'm interested in this as well. I'm working on a tool that crossbuilds a > very simple busybox rootfs and boots in QEMU for as many architectures > as possible. I want to make it easy to sanity test all the major > architectures. Right now it does little more than boot to a login > prompt, but I'd like to get the kselftests into it also. > Great. Why not? The first step would be get kselftest compiling in cross-builds. I haven't tried it yet, however in a private response to this thread, one developer mentioned that some tests don't pick the correct compiler result of hard-coded gcc instead of using $(CC) and assumptions on libraries. So as we go forward we have to get these things fixed along the way. -- Shuah -- Shuah Khan Senior Linux Kernel Developer - Open Source Group Samsung Research America(Silicon Valley) shuah.kh@samsung.com | (970) 672-0658 ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 13:00 ` Grant Likely 2014-08-12 16:15 ` Guenter Roeck 2014-08-12 16:34 ` Shuah Khan @ 2014-08-13 8:35 ` Linus Walleij 2014-08-13 16:11 ` Guenter Roeck 2014-08-13 16:45 ` Masami Hiramatsu 2014-08-13 15:06 ` Aneesh Kumar K.V 3 siblings, 2 replies; 40+ messages in thread From: Linus Walleij @ 2014-08-13 8:35 UTC (permalink / raw) To: Grant Likely; +Cc: shuah.kh, Rob Landley, ksummit-discuss, Greg Kroah-Hartman On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote: > On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >> I see, for that purpose, installing testcase may not fit. >> BTW, how would it cover cross-build? > > I'm interested in this as well. I'm working on a tool that crossbuilds a > very simple busybox rootfs and boots in QEMU for as many architectures > as possible. I want to make it easy to sanity test all the major > architectures. Right now it does little more than boot to a login > prompt, but I'd like to get the kselftests into it also. Hm that sounds like a goal similar to what Rob Landley has described as one goal for Aboriginal Linux as well. http://landley.net/aboriginal/about.html If such cross builds are triggered from git pushes akin to how Fengguang's 0-day is already working we are in a positive flow I think. Yours, Linus Walleij ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 8:35 ` Linus Walleij @ 2014-08-13 16:11 ` Guenter Roeck 2014-08-13 16:16 ` Andy Lutomirski 2014-08-18 3:08 ` Rob Landley 2014-08-13 16:45 ` Masami Hiramatsu 1 sibling, 2 replies; 40+ messages in thread From: Guenter Roeck @ 2014-08-13 16:11 UTC (permalink / raw) To: Linus Walleij, Grant Likely Cc: shuah.kh, ksummit-discuss, Rob Landley, Greg Kroah-Hartman On 08/13/2014 01:35 AM, Linus Walleij wrote: > On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote: >> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: > >>> I see, for that purpose, installing testcase may not fit. >>> BTW, how would it cover cross-build? >> >> I'm interested in this as well. I'm working on a tool that crossbuilds a >> very simple busybox rootfs and boots in QEMU for as many architectures >> as possible. I want to make it easy to sanity test all the major >> architectures. Right now it does little more than boot to a login >> prompt, but I'd like to get the kselftests into it also. > > Hm that sounds like a goal similar to what Rob Landley has > described as one goal for Aboriginal Linux as well. > http://landley.net/aboriginal/about.html > Yes, and to some degree buildroot. Rob's attempts to support multiple architectures also shows its limitations. For example, his m68k images don't work, at least not for me, because the machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. I have been unable to find a working combination of kernel configuration, qemu version, qemu command line, and root file system for m68k. Presumably that must exist, because qemu supports m68k, I just have not been able to figure out how to make it work. For my own qemu runtime tests, I ended up collecting root file systems and kernel configurations from all over the place. And then there is the problem of qemu command line parameters, where each target and architecture requires its own set of options, and it is sometimes all but impossible to find a working set of parameters for a given target/architecture combination. Guenter ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:11 ` Guenter Roeck @ 2014-08-13 16:16 ` Andy Lutomirski 2014-08-13 16:44 ` Bird, Tim 2014-08-18 3:10 ` Rob Landley 2014-08-18 3:08 ` Rob Landley 1 sibling, 2 replies; 40+ messages in thread From: Andy Lutomirski @ 2014-08-13 16:16 UTC (permalink / raw) To: Guenter Roeck; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman, Rob Landley On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: > On 08/13/2014 01:35 AM, Linus Walleij wrote: >> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> >> wrote: >>> >>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu >>> <masami.hiramatsu.pt@hitachi.com> wrote: >> >> >>>> I see, for that purpose, installing testcase may not fit. >>>> BTW, how would it cover cross-build? >>> >>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> as possible. I want to make it easy to sanity test all the major >>> architectures. Right now it does little more than boot to a login >>> prompt, but I'd like to get the kselftests into it also. >> >> >> Hm that sounds like a goal similar to what Rob Landley has >> described as one goal for Aboriginal Linux as well. >> http://landley.net/aboriginal/about.html >> > > Yes, and to some degree buildroot. > > Rob's attempts to support multiple architectures also shows its limitations. > For example, his m68k images don't work, at least not for me, because the > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. > I have been unable to find a working combination of kernel configuration, > qemu version, qemu command line, and root file system for m68k. Presumably > that must exist, because qemu supports m68k, I just have not been able > to figure out how to make it work. > > For my own qemu runtime tests, I ended up collecting root file systems and > kernel configurations from all over the place. And then there is the problem > of qemu command line parameters, where each target and architecture requires > its own set of options, and it is sometimes all but impossible to find a > working set of parameters for a given target/architecture combination. > virtme has exactly this problem (except for the root image part -- virtme can use debootstrap output directly). In virtme, I'm trying to solve it by just collecting known-working QEMU arguments and documenting the corresponding kernel config requirements. --Andy ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:16 ` Andy Lutomirski @ 2014-08-13 16:44 ` Bird, Tim 2014-08-13 17:07 ` Grant Likely 2014-08-13 17:10 ` Guenter Roeck 2014-08-18 3:10 ` Rob Landley 1 sibling, 2 replies; 40+ messages in thread From: Bird, Tim @ 2014-08-13 16:44 UTC (permalink / raw) To: Andy Lutomirski, Guenter Roeck Cc: shuah.kh, Rob Landley, ksummit-discuss, Greg Kroah-Hartman On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote: > > On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: > > On 08/13/2014 01:35 AM, Linus Walleij wrote: > >> > >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote: > >>> I'm interested in this as well. I'm working on a tool that crossbuilds a > >>> very simple busybox rootfs and boots in QEMU for as many architectures > >>> as possible. I want to make it easy to sanity test all the major > >>> architectures. Right now it does little more than boot to a login > >>> prompt, but I'd like to get the kselftests into it also. > >> > >> > >> Hm that sounds like a goal similar to what Rob Landley has > >> described as one goal for Aboriginal Linux as well. > >> http://landley.net/aboriginal/about.html > >> > > > > Yes, and to some degree buildroot. > > > > Rob's attempts to support multiple architectures also shows its limitations. > > For example, his m68k images don't work, at least not for me, because the > > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. > > I have been unable to find a working combination of kernel configuration, > > qemu version, qemu command line, and root file system for m68k. Presumably > > that must exist, because qemu supports m68k, I just have not been able > > to figure out how to make it work. > > > > For my own qemu runtime tests, I ended up collecting root file systems and > > kernel configurations from all over the place. And then there is the problem > > of qemu command line parameters, where each target and architecture requires > > its own set of options, and it is sometimes all but impossible to find a > > working set of parameters for a given target/architecture combination. > > > > virtme has exactly this problem (except for the root image part -- > virtme can use debootstrap output directly). In virtme, I'm trying to > solve it by just collecting known-working QEMU arguments and > documenting the corresponding kernel config requirements. I'm wondering if the kernel test tree might be a good place to keep such virtual machine/QEMU configurations. They should be only about 1 line (or a few lines) per machine, and they would be useful for automated testing. I also think they won't change every kernel release, so it shouldn't lead to the churn problem we had with defconfigs. Would there be objections to hosting this information in the kernel source tree? -- Tim ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:44 ` Bird, Tim @ 2014-08-13 17:07 ` Grant Likely 2014-08-13 17:10 ` Andy Lutomirski 2014-08-13 17:10 ` Guenter Roeck 1 sibling, 1 reply; 40+ messages in thread From: Grant Likely @ 2014-08-13 17:07 UTC (permalink / raw) To: Bird, Tim; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman, Rob Landley On Wed, Aug 13, 2014 at 5:44 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote: > On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote: >> >> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: >> > On 08/13/2014 01:35 AM, Linus Walleij wrote: >> >> >> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote: >> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >> >>> very simple busybox rootfs and boots in QEMU for as many architectures >> >>> as possible. I want to make it easy to sanity test all the major >> >>> architectures. Right now it does little more than boot to a login >> >>> prompt, but I'd like to get the kselftests into it also. >> >> >> >> >> >> Hm that sounds like a goal similar to what Rob Landley has >> >> described as one goal for Aboriginal Linux as well. >> >> http://landley.net/aboriginal/about.html >> >> >> > >> > Yes, and to some degree buildroot. >> > >> > Rob's attempts to support multiple architectures also shows its limitations. >> > For example, his m68k images don't work, at least not for me, because the >> > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. >> > I have been unable to find a working combination of kernel configuration, >> > qemu version, qemu command line, and root file system for m68k. Presumably >> > that must exist, because qemu supports m68k, I just have not been able >> > to figure out how to make it work. >> > >> > For my own qemu runtime tests, I ended up collecting root file systems and >> > kernel configurations from all over the place. And then there is the problem >> > of qemu command line parameters, where each target and architecture requires >> > its own set of options, and it is sometimes all but impossible to find a >> > working set of parameters for a given target/architecture combination. >> > >> >> virtme has exactly this problem (except for the root image part -- >> virtme can use debootstrap output directly). In virtme, I'm trying to >> solve it by just collecting known-working QEMU arguments and >> documenting the corresponding kernel config requirements. > > I'm wondering if the kernel test tree might be a good place to keep such > virtual machine/QEMU configurations. They should be only about 1 line > (or a few lines) per machine, and they would be useful for automated testing. > I also think they won't change every kernel release, so it shouldn't lead to the churn > problem we had with defconfigs. > > Would there be objections to hosting this information in > the kernel source tree? This is exactly what I'm trying to put together. g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 17:07 ` Grant Likely @ 2014-08-13 17:10 ` Andy Lutomirski 0 siblings, 0 replies; 40+ messages in thread From: Andy Lutomirski @ 2014-08-13 17:10 UTC (permalink / raw) To: Grant Likely; +Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman, Rob Landley On Wed, Aug 13, 2014 at 10:07 AM, Grant Likely <grant.likely@secretlab.ca> wrote: > On Wed, Aug 13, 2014 at 5:44 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote: >> On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote: >>> >>> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: >>> > On 08/13/2014 01:35 AM, Linus Walleij wrote: >>> >> >>> >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote: >>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> >>> as possible. I want to make it easy to sanity test all the major >>> >>> architectures. Right now it does little more than boot to a login >>> >>> prompt, but I'd like to get the kselftests into it also. >>> >> >>> >> >>> >> Hm that sounds like a goal similar to what Rob Landley has >>> >> described as one goal for Aboriginal Linux as well. >>> >> http://landley.net/aboriginal/about.html >>> >> >>> > >>> > Yes, and to some degree buildroot. >>> > >>> > Rob's attempts to support multiple architectures also shows its limitations. >>> > For example, his m68k images don't work, at least not for me, because the >>> > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. >>> > I have been unable to find a working combination of kernel configuration, >>> > qemu version, qemu command line, and root file system for m68k. Presumably >>> > that must exist, because qemu supports m68k, I just have not been able >>> > to figure out how to make it work. >>> > >>> > For my own qemu runtime tests, I ended up collecting root file systems and >>> > kernel configurations from all over the place. And then there is the problem >>> > of qemu command line parameters, where each target and architecture requires >>> > its own set of options, and it is sometimes all but impossible to find a >>> > working set of parameters for a given target/architecture combination. >>> > >>> >>> virtme has exactly this problem (except for the root image part -- >>> virtme can use debootstrap output directly). In virtme, I'm trying to >>> solve it by just collecting known-working QEMU arguments and >>> documenting the corresponding kernel config requirements. >> >> I'm wondering if the kernel test tree might be a good place to keep such >> virtual machine/QEMU configurations. They should be only about 1 line >> (or a few lines) per machine, and they would be useful for automated testing. >> I also think they won't change every kernel release, so it shouldn't lead to the churn >> problem we had with defconfigs. >> >> Would there be objections to hosting this information in >> the kernel source tree? > > This is exactly what I'm trying to put together. If this ends up in machine-readable form, it'll be extremely useful. Thanks! --Andy ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:44 ` Bird, Tim 2014-08-13 17:07 ` Grant Likely @ 2014-08-13 17:10 ` Guenter Roeck 1 sibling, 0 replies; 40+ messages in thread From: Guenter Roeck @ 2014-08-13 17:10 UTC (permalink / raw) To: Bird, Tim, Andy Lutomirski Cc: shuah.kh, Rob Landley, ksummit-discuss, Greg Kroah-Hartman On 08/13/2014 09:44 AM, Bird, Tim wrote: > On Wednesday, August 13, 2014 9:16 AM,Andy Lutomirski wrote: >> >> On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: >>> On 08/13/2014 01:35 AM, Linus Walleij wrote: >>>> >>>> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely wrote: >>>>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>>>> very simple busybox rootfs and boots in QEMU for as many architectures >>>>> as possible. I want to make it easy to sanity test all the major >>>>> architectures. Right now it does little more than boot to a login >>>>> prompt, but I'd like to get the kselftests into it also. >>>> >>>> >>>> Hm that sounds like a goal similar to what Rob Landley has >>>> described as one goal for Aboriginal Linux as well. >>>> http://landley.net/aboriginal/about.html >>>> >>> >>> Yes, and to some degree buildroot. >>> >>> Rob's attempts to support multiple architectures also shows its limitations. >>> For example, his m68k images don't work, at least not for me, because the >>> machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. >>> I have been unable to find a working combination of kernel configuration, >>> qemu version, qemu command line, and root file system for m68k. Presumably >>> that must exist, because qemu supports m68k, I just have not been able >>> to figure out how to make it work. >>> >>> For my own qemu runtime tests, I ended up collecting root file systems and >>> kernel configurations from all over the place. And then there is the problem >>> of qemu command line parameters, where each target and architecture requires >>> its own set of options, and it is sometimes all but impossible to find a >>> working set of parameters for a given target/architecture combination. >>> >> >> virtme has exactly this problem (except for the root image part -- >> virtme can use debootstrap output directly). In virtme, I'm trying to >> solve it by just collecting known-working QEMU arguments and >> documenting the corresponding kernel config requirements. > > I'm wondering if the kernel test tree might be a good place to keep such > virtual machine/QEMU configurations. They should be only about 1 line > (or a few lines) per machine, and they would be useful for automated testing. > I also think they won't change every kernel release, so it shouldn't lead to the churn > problem we had with defconfigs. > You would also need working configurations. There may be a few exceptions, where one of the defconfigs can be used as is, but in most cases at least some minor tweaks are needed to create a kernel that is working with qemu. Or at least that is my experience. Guenter ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:16 ` Andy Lutomirski 2014-08-13 16:44 ` Bird, Tim @ 2014-08-18 3:10 ` Rob Landley 1 sibling, 0 replies; 40+ messages in thread From: Rob Landley @ 2014-08-18 3:10 UTC (permalink / raw) To: Andy Lutomirski, Guenter Roeck Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On 08/13/2014 11:16 AM, Andy Lutomirski wrote: > On Wed, Aug 13, 2014 at 9:11 AM, Guenter Roeck <linux@roeck-us.net> wrote: >> For my own qemu runtime tests, I ended up collecting root file systems and >> kernel configurations from all over the place. And then there is the problem >> of qemu command line parameters, where each target and architecture requires >> its own set of options, and it is sometimes all but impossible to find a >> working set of parameters for a given target/architecture combination. >> > > virtme has exactly this problem (except for the root image part -- > virtme can use debootstrap output directly). In virtme, I'm trying to > solve it by just collecting known-working QEMU arguments and > documenting the corresponding kernel config requirements. My aboriginal linux system is designed around the assumption that nobody will ever actually bother to use it. (That's why I ship prebuilt binary tarballs.) I also write extensive documentation nobody will ever read: http://landley.net/aboriginal/about.html Each system-image tarball has a "run-emulator.sh" script that's just the qemu invocation. If you want to know how to launch qemu for the target: there you go. The kernel config is using the "miniconfig" technique: http://landley.net/aboriginal/FAQ.html#dev_miniconfig You assemble the miniconfig from the target-independent parts: http://landley.net/hg/aboriginal/file/tip/sources/baseconfig-linux And then concatenate the target-specific bits from the relevant sources/targets file. ala: http://landley.net/hg/aboriginal/file/tip/sources/targets/i686#l19 (All the target-specific information is in a single file under sources/targets. There are various patches in sources/patches some of which are target specific. All of them should go upstream, but kernel development is so insular it's just not worth the effort.) If you don't want to use miniconfig, the expanded configs are in the root-filesystem tarballs (in the src directory). > --Andy > . > ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:11 ` Guenter Roeck 2014-08-13 16:16 ` Andy Lutomirski @ 2014-08-18 3:08 ` Rob Landley 2014-08-18 7:16 ` Geert Uytterhoeven 1 sibling, 1 reply; 40+ messages in thread From: Rob Landley @ 2014-08-18 3:08 UTC (permalink / raw) To: Guenter Roeck, Linus Walleij, Grant Likely Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman The most interesting email conversations occur when the machine with all my email filters is in the shop... On 08/13/2014 11:11 AM, Guenter Roeck wrote: > On 08/13/2014 01:35 AM, Linus Walleij wrote: >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely >> <grant.likely@secretlab.ca> wrote: >>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu >>> <masami.hiramatsu.pt@hitachi.com> wrote: >> >>>> I see, for that purpose, installing testcase may not fit. >>>> BTW, how would it cover cross-build? >>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> as possible. I want to make it easy to sanity test all the major >>> architectures. Right now it does little more than boot to a login >>> prompt, but I'd like to get the kselftests into it also. >> >> Hm that sounds like a goal similar to what Rob Landley has >> described as one goal for Aboriginal Linux as well. >> http://landley.net/aboriginal/about.html >> > > Yes, and to some degree buildroot. Aboriginal is intentionally building the simplest system capable of rebuilding itself under itself, with an emphasis on native compiling from there. (It's sort of in pieces on the floor at the moment as I switch from uClibc to musl, which breaks rather a lot of subtle assumptions and requires everything to be debugged again. Oh well.) I'd happily make it work with other people's toolchains, but I've never managed to beat a working _native_ toolchain out of buildroot, it cross compiles everything and then the resulting system is not expected to be a development environment. Building on an emulated target system is apparently not most people's idea of fun. > Rob's attempts to support multiple architectures also shows its > limitations. Some of this is the limitations of the toolchain I'm using, which is the last GPLv2 release of gcc and binutils. I don't mess with GPLv3 code unless it's for work, it is _never_ fun. I keep hoping http://ellcc.org or similar will approach reasonably functional, but so far that one isn't even using http://lld.llvm.org instead of binutils, so... (Also, the llvm build is checking for gcc 4.7 at compile time and refuses to build on anything older. Yes, llvm has incestuous dependencies on gcc.) > For example, his m68k images don't work, at least not for me, because the > machine he uses (q800) is not supported in qemu 1.6 or 2.0 or 2.1. Laurent Vivier is working on a qemu fork to add support for that, but neither he nor I have had time to put into it in forever. It's something like git://gitorious.org/qemu-m68k/qemu-m68k branch refs/heads/q800 (My dayjob has nothing to do with anything interesting, I'm paid to work on a a company's fork of a vendor's bsp fork of a really old kernel. For the new board, we just upgraded _to_ a version that's only 4 years old. When they finally ship there will be a nominal compliance tarball put on a website somewhere that nobody will ever look at or care about. I keep meaning to put up a patreon to see if people would be willing to sponsor me to spend all my time on actually _interesting_ stuff like this, but the chances are low enough I haven't bothered.) > I have been unable to find a working combination of kernel configuration, > qemu version, qemu command line, and root file system for m68k. Presumably > that must exist, because qemu supports m68k, I just have not been able > to figure out how to make it work. QEMU does not support m68k, qemu supports coldfire which is a nommu subset of m68k. But people have used the aranym emulator to fake an atari machine that _has_ run the root filesystems I've been building. (With a different kernel config, and a largeish patch for aranym devices that since went upstream. But it showed the basics were right.) The problem with using aranym is my setup expects a certain pile of devices. If /dev/console goes to stdin/stdout than it's easy to script the sucker with tcl/expect and log the output with "tee". If you have a virtual network card you can move the heavy lifting of compilation outside the emulator with distcc hooked up to the cross compiler (_without_ reintroducing the horrible complexity of cross compiling because configure and make and linking and header preprocessing and such all run natively inside the emulator where there's only a single build context.) You need at least 256 megs of ram for 64 bit gcc to function. It's really convenient to have three emulated hard drives (one for the root filesystem, one for writeable space on /home, and one for the source control and build scripts on /mnt. I described that setup in http://landley.net/aboriginal/control-images and there are example build control images there.) Now that my initmpfs patches are in, I'm moving the simple-root-filesystem to initmpfs so I can use network mounts instead of filesystem images for all that... but the emulated network card becomes a bit of a bottleneck the. The emulated hard drives tend to fling data around faster. (Also I wanted to use v9fs but the v9fs developers went ABSOLUTELY INSANE and accepted an IBM patch that broke things so badly Red Hat yanked the 9p source code out of their kernel.) http://sourceforge.net/p/v9fs/mailman/message/31629549/ http://scientificlinuxforum.org/index.php?showtopic=2858 > For my own qemu runtime tests, I ended up collecting root file systems and > kernel configurations from all over the place. And then there is the > problem > of qemu command line parameters, where each target and architecture > requires > its own set of options, and it is sometimes all but impossible to find a > working set of parameters for a given target/architecture combination. Tell me about it. I collect them, and my system images document them. I was in the process of trying to get Alpha to work when musl got close enough to 1.0 that switching over became a higher priority, and now there's the chicken and egg problem of adding musl support and getting an emulated development enviornment working... Mostly it's just "dayjob eats my time/energy, I do what I can in the time I have left". > Guenter Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-18 3:08 ` Rob Landley @ 2014-08-18 7:16 ` Geert Uytterhoeven 0 siblings, 0 replies; 40+ messages in thread From: Geert Uytterhoeven @ 2014-08-18 7:16 UTC (permalink / raw) To: Rob Landley; +Cc: Shuah Khan, ksummit-discuss, Greg Kroah-Hartman Hi Rob, On Mon, Aug 18, 2014 at 5:08 AM, Rob Landley <rob@landley.net> wrote: > (My dayjob has nothing to do with anything interesting, I'm paid to work > on a a company's fork of a vendor's bsp fork of a really old kernel. For > the new board, we just upgraded _to_ a version that's only 4 years old. > When they finally ship there will be a nominal compliance tarball put on > a website somewhere that nobody will ever look at or care about. I keep > meaning to put up a patreon to see if people would be willing to sponsor > me to spend all my time on actually _interesting_ stuff like this, but > the chances are low enough I haven't bothered.) > >> I have been unable to find a working combination of kernel configuration, >> qemu version, qemu command line, and root file system for m68k. Presumably >> that must exist, because qemu supports m68k, I just have not been able >> to figure out how to make it work. > > QEMU does not support m68k, qemu supports coldfire which is a nommu > subset of m68k. But people have used the aranym emulator to fake an > atari machine that _has_ run the root filesystems I've been building. > (With a different kernel config, and a largeish patch for aranym devices > that since went upstream. But it showed the basics were right.) These days atari_defconfig in upstream should run fine on ARAnyM. > The problem with using aranym is my setup expects a certain pile of > devices. If /dev/console goes to stdin/stdout than it's easy to script > the sucker with tcl/expect and log the output with "tee". If you have a Ah, ARAnyM's nfcon only does stdout, not stdin. > virtual network card you can move the heavy lifting of compilation > outside the emulator with distcc hooked up to the cross compiler But nfeth works fine. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 8:35 ` Linus Walleij 2014-08-13 16:11 ` Guenter Roeck @ 2014-08-13 16:45 ` Masami Hiramatsu 2014-08-18 3:18 ` Rob Landley 1 sibling, 1 reply; 40+ messages in thread From: Masami Hiramatsu @ 2014-08-13 16:45 UTC (permalink / raw) To: Linus Walleij; +Cc: shuah.kh, Rob Landley, ksummit-discuss, Greg Kroah-Hartman (2014/08/13 17:35), Linus Walleij wrote: > On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote: >> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: > >>> I see, for that purpose, installing testcase may not fit. >>> BTW, how would it cover cross-build? >> >> I'm interested in this as well. I'm working on a tool that crossbuilds a >> very simple busybox rootfs and boots in QEMU for as many architectures >> as possible. I want to make it easy to sanity test all the major >> architectures. Right now it does little more than boot to a login >> prompt, but I'd like to get the kselftests into it also. > > Hm that sounds like a goal similar to what Rob Landley has > described as one goal for Aboriginal Linux as well. > http://landley.net/aboriginal/about.html Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal distro may not have "make" on their rootfs. In that case, we need to make self- executable binaries and install it with testing "ash" scripts so that it can run on busybox. It may be just a technical issue. > If such cross builds are triggered from git pushes akin to how > Fengguang's 0-day is already working we are in a positive > flow I think. Agreed. Thank you, -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu.pt@hitachi.com ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-13 16:45 ` Masami Hiramatsu @ 2014-08-18 3:18 ` Rob Landley 0 siblings, 0 replies; 40+ messages in thread From: Rob Landley @ 2014-08-18 3:18 UTC (permalink / raw) To: Masami Hiramatsu, Linus Walleij Cc: shuah.kh, ksummit-discuss, Greg Kroah-Hartman On 08/13/2014 11:45 AM, Masami Hiramatsu wrote: > (2014/08/13 17:35), Linus Walleij wrote: >> On Tue, Aug 12, 2014 at 3:00 PM, Grant Likely <grant.likely@secretlab.ca> wrote: >>> On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >> >>>> I see, for that purpose, installing testcase may not fit. >>>> BTW, how would it cover cross-build? >>> >>> I'm interested in this as well. I'm working on a tool that crossbuilds a >>> very simple busybox rootfs and boots in QEMU for as many architectures >>> as possible. I want to make it easy to sanity test all the major >>> architectures. Right now it does little more than boot to a login >>> prompt, but I'd like to get the kselftests into it also. >> >> Hm that sounds like a goal similar to what Rob Landley has >> described as one goal for Aboriginal Linux as well. >> http://landley.net/aboriginal/about.html > > Hmm, I also consider that the Aboriginal Linux and other busybox-based minimal > distro may not have "make" on their rootfs. In that case, we need to make self- > executable binaries and install it with testing "ash" scripts so that it can > run on busybox. It may be just a technical issue. Aboriginal Linux has make in the native root filesystem. (Again, last gplv2 version so it's a touch stale now. Back when I did busybox stuff I planned to write a make for that, and now I plan to write one for toybox, but it's a post 1.0 todo item.) There's also a native-compiler tarball that's just the stuff it adds on top of the simple-root-filesystem to make the development environment. (This includes make and bash 2.05b.) Now that my initmpfs patches are upstream, I'm refactoring things to put the simple-root-filesystem in initramfs and splice the native-compiler stuff from hda into the $PATH at runtime, in the init script. But I'm two kernel releases behind, and should probably ship a 3.15 version before doing much more development... Rob (P.S. Making a stripped down linux from scratch build based on busybox and uclibc is why I got into busybox development in the first place. That's why I wound up doing so much work on it I got the maintainership handed to me for a bit. Now I'm doing it all again in toybox because Android won't use busybox because it's under the GPL.) ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond 2014-08-12 13:00 ` Grant Likely ` (2 preceding siblings ...) 2014-08-13 8:35 ` Linus Walleij @ 2014-08-13 15:06 ` Aneesh Kumar K.V 3 siblings, 0 replies; 40+ messages in thread From: Aneesh Kumar K.V @ 2014-08-13 15:06 UTC (permalink / raw) To: Grant Likely, Masami Hiramatsu, shuah.kh Cc: Greg Kroah-Hartman, ksummit-discuss Grant Likely <grant.likely@secretlab.ca> writes: > On Tue, 12 Aug 2014 01:13:07 +0900, Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote: >> (2014/08/11 23:11), Shuah Khan wrote: >> >> (2014/08/07 23:36), Shuah Khan wrote:> As a first step towards a larger goal to enable developer >> >>> friendly kernel testing framework, a new make target is >> >>> planned for 3.17. In addition, 3.17 includes work done to >> >>> fix tools/testing/sefltests to run without failures. >> >>> >> >>> Short summary of work done so far for 3.17: >> >>> >> >>> - fix compile errors and warnings in various tests >> >>> - fix run-time errors when tests aren't run as root >> >>> - enhance and improve cpu and memory hot-plug tests >> >>> to run in limited scope mode by default. A new make >> >>> target to select full-scope testing. Prior to this >> >>> change, cpu and memory hot-plug tests hung trying to >> >>> hot-plug all but cpu0 and a large portion of the memory. >> >>> - add a new kselftest target to run existing selftests >> >>> to start with. >> >> >> >> Instead of running the selftests, can we build the testcases and >> >> install it as a tool? I think running tests on the tree is not a >> >> good idea... >> > >> > One of the goals is to leverage developer tests that we already have. >> > When a developer makes a kernel change and wants to see if that change >> > lead to any regression, having the ability to buidl and run selftests on >> > the newly installed kernel withe the same source tree is very useful. >> > That is the reason behind adding this new target. >> >> I see, for that purpose, installing testcase may not fit. >> BTW, how would it cover cross-build? > > I'm interested in this as well. I'm working on a tool that crossbuilds a > very simple busybox rootfs and boots in QEMU for as many architectures > as possible. I want to make it easy to sanity test all the major > architectures. Right now it does little more than boot to a login > prompt, but I'd like to get the kselftests into it also. > I am interested in this as well, I will reach Chicago only on 18th afternoon (due to flight availability issues). I have been using https://github.com/autotest/autotest-client-tests for running sanity tests on different kernel versions. It can be driven with sandboxed setup using https://github.com/autotest/virt-test -aneesh ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2014-08-18 7:16 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-08-07 14:36 [Ksummit-discuss] kselftest - What's in 3.17 and plans for 3.18 and beyond Shuah Khan 2014-08-07 18:24 ` Bird, Tim 2014-08-07 19:59 ` Shuah Khan 2014-08-08 2:18 ` Masami Hiramatsu 2014-08-11 14:11 ` Shuah Khan 2014-08-11 16:13 ` Masami Hiramatsu 2014-08-12 13:00 ` Grant Likely 2014-08-12 16:15 ` Guenter Roeck 2014-08-12 16:21 ` Masami Hiramatsu 2014-08-12 16:51 ` Geert Uytterhoeven 2014-08-12 17:15 ` Theodore Ts'o 2014-08-12 16:23 ` Grant Likely 2014-08-12 16:49 ` Mark Brown 2014-08-13 6:26 ` Grant Likely 2014-08-13 10:40 ` Mark Brown 2014-08-13 11:12 ` Geert Uytterhoeven 2014-08-13 12:42 ` Mark Brown 2014-08-13 13:08 ` Geert Uytterhoeven 2014-08-13 15:00 ` Kevin Hilman 2014-08-13 16:40 ` Olof Johansson 2014-08-13 17:11 ` Mark Brown 2014-08-12 17:46 ` Tim Bird 2014-08-12 18:06 ` Steven Rostedt 2014-08-12 20:52 ` Tim Bird 2014-08-14 16:35 ` Grant Likely 2014-08-12 17:30 ` Andy Lutomirski 2014-08-12 16:34 ` Shuah Khan 2014-08-13 8:35 ` Linus Walleij 2014-08-13 16:11 ` Guenter Roeck 2014-08-13 16:16 ` Andy Lutomirski 2014-08-13 16:44 ` Bird, Tim 2014-08-13 17:07 ` Grant Likely 2014-08-13 17:10 ` Andy Lutomirski 2014-08-13 17:10 ` Guenter Roeck 2014-08-18 3:10 ` Rob Landley 2014-08-18 3:08 ` Rob Landley 2014-08-18 7:16 ` Geert Uytterhoeven 2014-08-13 16:45 ` Masami Hiramatsu 2014-08-18 3:18 ` Rob Landley 2014-08-13 15:06 ` Aneesh Kumar K.V
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.