* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <576cea07-770a-4864-c3f5-0832ff211e94-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> @ 2017-07-03 16:30 ` Steven Rostedt [not found] ` <20170703123025.7479702e-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-03 16:30 UTC (permalink / raw) To: Thorsten Leemhuis Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Sun, 2 Jul 2017 19:51:43 +0200 Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote: > Hi! Sorry, I know I'm late -- real life (travel, day job, ...) kept me > away from spending time on Linux kernel regression work :-/ > > Maybe I'm taking it a bit to far for the new kid in town, but I think I > want to propose two sessions. One for the maintainer summit, that deals > with a the most critical issues relevant to regression tracking. And one > technical session to deal with all the other stuff. Obviously we can > move below mentioned topics from one to the other or talk about them at > both if we want. > > = [MAINTAINERS SUMMIT] Improve regression tracking = > > * Follow up from last year: What to do about bugzilla.kernel.org? > Reporters still get stranded there. > * How to get subsystems maintainer involved more in regression tracking > to better make sure that reported regressions are tracked and not > forgotten accidentally. We should push harder for all reproducer tests to be put into selftests. I try to do that myself (although I admit, I forget to do it myself here and there. But I'm pushing myself to be better) > * Frustrations with regression tracking aka. how to establish > regression tracking properly to make sure it will never go away again. By adding reproducing tests to selftests, we can easily see what regressions are still there. > > = [TECH TOPIC] Improve the kernels quality by getting more people > involved in regression testing and reporting = Again, this can be answered by placing more reproducers into selftests. > > * A short report from the outcome of the maintainer summit discussion; > also pick up and topics here that where not properly discussed on the > maintainer summit or were postponed to this session. > * How to get distros more involved in regression tracking; especially > those that have a technical aware user base or normally ship up2date > kernel images (and thus have an greater interest in avoiding > regressions). I'm mainly thinking about Arch Linux, Debian, Fedora, and > openSUSE Tumbleweed here; having Ubuntu in the boat would be good, too! > (might be wise to talk about this on the maintainers summit as well, if > the right people are there) > * How to make it more easy to (ideally automatically!) track the > current status and the progress of each regression? Are there any tools > that could make regression tracking easier for all of us while not > introducing much overhead for maintainers? What is selftests? (Jeopardy answer for all of the above ;-) > > = Details = > > Below you'll find few more words about some points mentioned above; > there are a few other topics as well we could discuss if we want. But > first, a few general words on regression tracking from my point of view: > > * There are a lot of areas in regression tracking where things are far > from good (read: in a bad state). That makes it easy to discuss current > problems and their solutions for hours -- and at the same time forget > that discussing itself doesn't get us much forward (the old bugzilla > issue mentioned in this mail is a good example). We thus IMHO should > focus on the most important issues and lay the groundwork to establish > regression tracking properly again, then we move on to solve things that > are harder to solve. > > * Regression tracking currently is quite boring and exhausting (read: > high burn-out risk), as it involves quite a lot of manual work finding > regressions and keeping track of their progress (and at the end of the > day it does not feel like you achieved much). Some of that work can not > be automated. But quite a bit can and that would help a great deal to > establish regression tracking properly (currently I'm the only one doing > it and some development cycles I simply don't find spare time for it). > > I currently don't see any existing solutions that fit well with our > mail focused workflow and at the same time do not introduce much > overhead for subsystem maintainers (which I assume is what everyone > wants, as I fear solutions with much overhead won't fly at all). Ideas > how to solve this tricky problem area are highly welcomed. It's > something that can be discussed when the aforementioned points > "establish regression tracking properly" and "make it more easy to > manually or automatically track the current status of a regression" come up. > > == What to do about bugzilla.kernel.org = > > Discussed last year already; see https://lwn.net/Articles/705245/ for > details. Situation didn't change much since then: the bugzilla instance > was updated, but people still get stranded there as most subsystems > ignore it. That afaics frustrates people and makes them stop testing or > reporting bugs. > > Discuss how to improve things. [my2cent] Maybe a short term solution > like this could work: Serve a static page on bugzilla.kernel.org that > tells people where regressions/bugs for certain subsystems can be > reported, as it most of the time is some mailing list anyway. Such a > page could get compiled from MAINTAINERS (there is the "B:" field now > that points to bugzilla; if its not there point to a mailing lists; also > explain get_maintainers.pl). > > Leave our bugzilla reachable via bugzilla.kernel.org/frontpage (or > something like that) for those few subsystems that use it; that's afaics > ACPI and PM (including Cpufreq, Cpuidle, Hibernation, Suspend, ...) and > maybe PCI (not sure) -- or should we tell them to move to > bugzilla.freedesktop.org (or somewhere else) to get rid of our bugzilla > in the long etrm and make Konstantins life easier? Anyway: Make sure > bugs for other subsystems can't get filed in bugzilla.kernel.org anymore > to make sure they get lost there. [/my2cent] > > == How to get subsystems maintainer more involved in regression tracking > to […] == > > One reasons why I put this up is: It would help me a lot if people let > regressions-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org (side note: might be wise to make a > mailing-list that replaces this address) get told about regressions -- > simply CCing it on reports or answers to regressions reports is enough; > forwarding/bouncing mails there (even without additional text) is fine, > too. > > The other reason I included it: This came up in last years discussion on > this list and it seemed some people thought we can get the subsystems > maintainers more involved; so I thought it might be wise to discuss it. > Might also be a good idea to discuss here how to get distro kernel > maintainer more involved if enough are around. > > == How to establish regression tracking properly […] == > > This is a pretty vague topic on purpose. People seem to agree that > regression tracking is important, but for years nobody did it (it > stopped a little while after Rafael had to move on) and the little bit > that I can do in my rare spare time won't help much (and I have no idea > how long I can continue to find time for it). > > == Make it easier to track the progress of regression == > > One of the main reasons that makes regression tracking hard currently: > getting aware or regressions and tracking their progress is a lot of > manual work. I plan one step that hopefully makes the job a little > easier and at the same time might allow some automation in the long > term: ask people to include a certain keyword in their regressions > reports. Maybe something like "Linux-Regression" that doesn't get too > much false positives when searching for it on lists and via Google > (suggestions for a better tag welcome). > > In addition, I plan to hand out some form of ID for each regressions I > track and ask people to include it -- especially when they post patches > that fix said regression or move the discussion to a new place (like > "Corrects: Linux-Regression-d2afd"; again: suggestions welcome! Maybe I > should just use a URL where people find details?). > > That way I can notice more easy when a fix for a regression hits > linux-next or master; I also get aware if a discussion moves from > bugzilla to LKML or from one thread to another (fingers crossed). > Obviously it depends on cooperation of those involved. > > If this works out we could write a script or something that watches > mailing lists, bug trackers and git trees for the tag in question. That > script could file a database and automatically do some of the tracking job. > > == get distros more involved == > > I assume at least Ben (Debian), Laura (Fedora), and Takashi (openSUSE) > are around, so it might be a good idea to sit together and talk > regression tracking in general and how we could get the distros kernel > maintainers more involved. Even better would be to sit down before to > maybe come up with some ideas/plans we could talk during this session. > > One topic could be: How to make it easier for users of popular distros > to get involved in testing. The "Kernel of the day" (KOTD) from > SUSE/openSUSE was mentioned recently on this list already, but I got the > impression that the existence of this repo is not well known; guess it's > the same for my own Kernel Vanilla Repositories for Fedora (those > contain packages with a quite recent mainline version; see > https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories ) or the fact > that Fedora rawhide ships a recent mainline snapshot all the time. But > should distros also offer Linux-next somewhere? Or anything else? And > should the distros send experienced users upstream when they found a > regression? Or will subsystem maintainers send those users away because > they assume those kernels are not vanilla? > > > == Topics or vague ideas I left out on purpose == > > Here is a list of other things we could talk about, but I think better > left for a later time: > > * Kerneloops (http://oops.kernel.org/): It was discussed last year on > this list. I have no idea what the current status is. Is someone > watching & analysing it? And poking the right people when needed? (I > doubt it) > > * Regression tracking for stable kernels (many bugs only get noticed > once a new mainline version got released; at that time it might still be > easy to revert a certain patch in mainline and stable) > > * statistics: I didn't spend time to create statistics, like Rafael did > in the past. They'd be nice to have, but for now I think my time is > better spend elsewhere. > > * work towards growing the number of tester by making it easier for > them (better documentation, easier configuration, bisection scripts, ...) > > * maybe document a few some procedures for those that are not regular > kernel developers (like the "When users report bugs on the Fedora > tracker that look like actual upstream bugs, what's the best way to have > those reported?" thing that Laura mentioned earlier this month in the > mail "Bug reporting feedback loop" > > * provide better services than only a plain text list of regression on > a mailing list? > > * better documentation? for example explain the difference between bugs > and regressions somewhere to make people understand why their bugs might > get ignored, but as the same time know that we handle regressions more > seriously. > > * Should the regression tracker nag subsystem maintainers (and > reporters) more often if they are inactive? How do people for example > feel about (Semi-)Automatic nagging mails for regressions where there is > no progress? > > * Is the data and the format of the current reports show useful at all? > If not: How to improve it? > > * regression tracking is a fair amount of work, and it's frustrating, > and people burn out. How to avoid that? Can we maybe get regression > tracking on solid ground by somehow building a healthy community around > it (containing kernel developers, Distro maintainers and people that are > willing to help in their spare time) that work on regressions > testing/tracking and other QA stuff? > > * how to make the Linux kernel development so good that the mainstream > distros stop their kernel forks and do what they do with Firefox: Ship > the latest stable version (users get a new version with new features > every few weeks) or a longterm branch (makes a big version jump about > once a year; see Firefox ESR). This wont ever happen (famous last words). Distros want "stable kernels" with new features. That's not what stable is about. > > Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked > so closely into LWN.net articles about regression tracking and older > discussions about it. Anyway, I know that selftests are not the answer for everything, but anything that has a way to reproduce a bug should be added to it. Sure, it may depend on various hardware and/or file systems and different configs, but if we have a central location to place all bug reproducing tests (which we do have), then we should utilize it. When it's in the kernel tree, it will be used much more often. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170703123025.7479702e-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170703123025.7479702e-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-03 18:50 ` Dan Williams 2017-07-04 19:03 ` Thorsten Leemhuis 1 sibling, 0 replies; 71+ messages in thread From: Dan Williams @ 2017-07-03 18:50 UTC (permalink / raw) To: Steven Rostedt; +Cc: Thorsten Leemhuis, Linux API, Shuah Khan, ksummit On Mon, Jul 3, 2017 at 9:30 AM, Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> wrote: > On Sun, 2 Jul 2017 19:51:43 +0200 [..] >> >> Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked >> so closely into LWN.net articles about regression tracking and older >> discussions about it. > > Anyway, I know that selftests are not the answer for everything, but > anything that has a way to reproduce a bug should be added to it. Sure, > it may depend on various hardware and/or file systems and different > configs, but if we have a central location to place all bug reproducing > tests (which we do have), then we should utilize it. I agree with Steven, and I would add that you don't necessarily need specific hardware to write a test for a driver regression, see examples in tools/testing/nvdimm. I also tend to think that back-stopping regressions with new tests helps with the burn-out problem of tracking regressions. Where building tools and tests is potentially more fulfilling than just bug tracking. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170703123025.7479702e-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-03 18:50 ` Dan Williams @ 2017-07-04 19:03 ` Thorsten Leemhuis [not found] ` <ad94dc65-cc9c-f4f1-27c1-5a48603c7f59-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Thorsten Leemhuis @ 2017-07-04 19:03 UTC (permalink / raw) To: Steven Rostedt Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 03.07.2017 18:30, Steven Rostedt wrote: > On Sun, 2 Jul 2017 19:51:43 +0200 > Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote: >> * How to get subsystems maintainer involved more in regression tracking >> to better make sure that reported regressions are tracked and not >> forgotten accidentally. > We should push harder for all reproducer tests to be put into > selftests. I try to do that myself [...] > [...] > By adding reproducing tests to selftests, we can easily see what > regressions are still there. > [...] > What is selftests? (Jeopardy answer for all of the above ;-) Sure, writing and running selftests is a good idea. But as you said yourself in the later part of your mail: it won't help much in situations where the kernel (or a selftest) needs to run on a certain hardware or a specific (and maybe rare or complex) configuration. Sadly a lot of the regressions in my recent reports were of this kind afaics :-/ In fact I got the impression that most of the regressions that might get caught by selftests were directly handled by the subsystem maintainer and never made it to me or my reports -- and thus I can't ask maintainers to write selftests. *If* I got better aware of those problems I (a) could make sure they are not forgotten and (b) sooner or later could publicly state something like "hey, you had ten regressions recently in your subsystem where writing a selftest might have been a good idea, but you didn't even write one -- why?" (if we want something like that). > […] >> * how to make the Linux kernel development so good that the mainstream >> distros stop their kernel forks and do what they do with Firefox: Ship >> the latest stable version (users get a new version with new features >> every few weeks) or a longterm branch (makes a big version jump about >> once a year; see Firefox ESR). Hehe, I maybe left the field "regression tracking" to much here and wandered too far into QA territory. > This wont ever happen (famous last words). Distros want "stable > kernels" with new features. Ha, yes, it's a long shot (and maybe more a vague idea to work towards to). And maybe Debian stable and RHEL will always use the model they use today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and some others are updating to the latest Linux kernel release every few weeks already and it works fine for them. Maybe we can get Ubuntu and others to follow sooner or later. Sure, for some people a version jump to a major new kernel release will sound crazy, but when Linus introduced the current development scheme a lot of people also said "that will never fly" -- that was 13 years ago now and it works quite well. The situation was similar with Firefox as well. > That's not what stable is about. That afaics (disclaimer: English is not my mother tongue) depends on the interpretation of the word, as it can mean "nothing changes" or "rock solid/reliable" (even when two people have a "stable relationship" it does not mean that nothing changes between them...). >> Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked >> so closely into LWN.net articles about regression tracking and older >> discussions about it. > Anyway, I know that selftests are not the answer for everything, but > anything that has a way to reproduce a bug should be added to it. Sure, > it may depend on various hardware and/or file systems and different > configs, but if we have a central location to place all bug reproducing > tests (which we do have), then we should utilize it. > When it's in the kernel tree, it will be used much more often. +1 Ciao, Thorsten ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <ad94dc65-cc9c-f4f1-27c1-5a48603c7f59-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <ad94dc65-cc9c-f4f1-27c1-5a48603c7f59-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> @ 2017-07-05 12:45 ` Steven Rostedt [not found] ` <20170705084528.67499f8c-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 12:45 UTC (permalink / raw) To: Thorsten Leemhuis Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Tue, 4 Jul 2017 21:03:22 +0200 Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote: > On 03.07.2017 18:30, Steven Rostedt wrote: > > On Sun, 2 Jul 2017 19:51:43 +0200 > > Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote: > >> * How to get subsystems maintainer involved more in regression tracking > >> to better make sure that reported regressions are tracked and not > >> forgotten accidentally. > > We should push harder for all reproducer tests to be put into > > selftests. I try to do that myself [...] > > [...] > > By adding reproducing tests to selftests, we can easily see what > > regressions are still there. > > [...] > > What is selftests? (Jeopardy answer for all of the above ;-) > > Sure, writing and running selftests is a good idea. But as you said > yourself in the later part of your mail: it won't help much in > situations where the kernel (or a selftest) needs to run on a certain > hardware or a specific (and maybe rare or complex) configuration. Sadly > a lot of the regressions in my recent reports were of this kind afaics :-/ > > In fact I got the impression that most of the regressions that might get > caught by selftests were directly handled by the subsystem maintainer > and never made it to me or my reports -- and thus I can't ask > maintainers to write selftests. *If* I got better aware of those > problems I (a) could make sure they are not forgotten and (b) sooner or > later could publicly state something like "hey, you had ten regressions > recently in your subsystem where writing a selftest might have been a > good idea, but you didn't even write one -- why?" (if we want something > like that). I'm betting there's a lot of reproducer code that never makes it into a test. How do we solve that? Perhaps we need people looking at LKML for any signs "I did this, and it caused a bug" or "Here's a test case which can trigger the bug". Each of these instances should end up in selftests, and I'm sure they are not. We can't do much for special hardware, even though those tests should still be in the selftests for those that have the hardware, but we can do something about special configs. Perhaps selfttests should have a "config test" section. I have that in my own tests, but I use ktest to build them. > > > […] > >> * how to make the Linux kernel development so good that the mainstream > >> distros stop their kernel forks and do what they do with Firefox: Ship > >> the latest stable version (users get a new version with new features > >> every few weeks) or a longterm branch (makes a big version jump about > >> once a year; see Firefox ESR). > > Hehe, I maybe left the field "regression tracking" to much here and > wandered too far into QA territory. > > > This wont ever happen (famous last words). Distros want "stable > > kernels" with new features. > > Ha, yes, it's a long shot (and maybe more a vague idea to work towards > to). And maybe Debian stable and RHEL will always use the model they use > today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and > some others are updating to the latest Linux kernel release every few > weeks already and it works fine for them. Maybe we can get Ubuntu and > others to follow sooner or later. > > Sure, for some people a version jump to a major new kernel release will > sound crazy, but when Linus introduced the current development scheme a > lot of people also said "that will never fly" -- that was 13 years ago > now and it works quite well. The situation was similar with Firefox as well. > > > That's not what stable is about. > > That afaics (disclaimer: English is not my mother tongue) depends on the > interpretation of the word, as it can mean "nothing changes" or "rock > solid/reliable" (even when two people have a "stable relationship" it > does not mean that nothing changes between them...). Nothing to do with what language your mother tongue is ;-) When the stable releases were created, there was some pretty strict requirements for what should go into stable. Of course the requirements have changed throughout the years. But there are big differences in what Red Hat considers something "stable" and what the Linux stable releases consider to be stable. That is where I meant that things wont change. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705084528.67499f8c-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705084528.67499f8c-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 13:09 ` Carlos O'Donell [not found] ` <4080ecc7-1aa8-2940-f230-1b79d656cdb4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Carlos O'Donell @ 2017-07-05 13:09 UTC (permalink / raw) To: Steven Rostedt, Thorsten Leemhuis Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 07/05/2017 08:45 AM, Steven Rostedt wrote: > I'm betting there's a lot of reproducer code that never makes it into a > test. How do we solve that? Perhaps we need people looking at LKML for > any signs "I did this, and it caused a bug" or "Here's a test case > which can trigger the bug". Each of these instances should end up in > selftests, and I'm sure they are not. > > We can't do much for special hardware, even though those tests should > still be in the selftests for those that have the hardware, but we can > do something about special configs. Perhaps selfttests should have a > "config test" section. I have that in my own tests, but I use ktest to > build them. This problem is a reflection of our own explicit or implicit priorities. The priorities of developers and reviewers needs to change to make an impact on the problem. This is a hard problem. As a concrete action item, glibc core developers took a harder stance on (a) all user-visible bugs need a bug # (forces people to think about the problem and file a coherent public bug about it) (b) all bugs needs a regression test if possible, (c) and if not possible we need to extend the testing framework to make it possible (we've started using kernel namespaces to create isolated test configurations). This change in reviewer priorities has had a noticeable impact on developer priorities over the last 5 years. Timelines for this problem will be measured in years. -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <4080ecc7-1aa8-2940-f230-1b79d656cdb4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <4080ecc7-1aa8-2940-f230-1b79d656cdb4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> @ 2017-07-05 13:27 ` Steven Rostedt [not found] ` <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:47 ` Mark Brown 1 sibling, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 13:27 UTC (permalink / raw) To: Carlos O'Donell Cc: Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 5 Jul 2017 09:09:51 -0400 Carlos O'Donell <carlos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: > This problem is a reflection of our own explicit or implicit priorities. > The priorities of developers and reviewers needs to change to make an > impact on the problem. This is a hard problem. I 100% agree. > > As a concrete action item, glibc core developers took a harder stance on > (a) all user-visible bugs need a bug # (forces people to think about the Unfortunately, we don't have a good system for a "bug #". Most kernel developers hate bugzilla, and I think that includes Linus ;-) Which means, unless Linus builds us a new bug tracking system, there wont be any mandate for it. > problem and file a coherent public bug about it) (b) all bugs needs a > regression test if possible, (c) and if not possible we need to extend I would love all bug fixes to come with a test (when possible). > the testing framework to make it possible (we've started using kernel > namespaces to create isolated test configurations). Well, we have a selftest directory that should include all of these. And most people run them on either a test box or a VM. > > This change in reviewer priorities has had a noticeable impact on developer > priorities over the last 5 years. Timelines for this problem will be > measured in years. > Your "b" above is what I would like to push. But who's going to enforce this? With 10,000 changes per release, and a lot of them are fixes, the best we can do is the honor system. Start shaming people that don't have a regression test along with a Fixes tag (but we don't want people to fix bugs without adding that tag either). There is a fine line one must walk between getting people to change their approaches to bugs and regression tests, and pissing them off where they start doing the opposite of what would be best for the community. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 14:06 ` Greg KH [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 14:06 ` Carlos O'Donell 1 sibling, 1 reply; 71+ messages in thread From: Greg KH @ 2017-07-05 14:06 UTC (permalink / raw) To: Steven Rostedt Cc: Carlos O'Donell, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote: > Your "b" above is what I would like to push. But who's going to enforce > this? With 10,000 changes per release, and a lot of them are fixes, the > best we can do is the honor system. Start shaming people that don't > have a regression test along with a Fixes tag (but we don't want people > to fix bugs without adding that tag either). There is a fine line one > must walk between getting people to change their approaches to bugs and > regression tests, and pissing them off where they start doing the > opposite of what would be best for the community. I would bet, for the huge majority of our fixes, they are fixes for specific hardware, or workarounds for specific hardware issues. Now writing tests for those is not an impossible task (look at what the i915 developers have), but it is very very hard overall, especially if the base infrastructure isn't there to do it. For specific examples, here's the shortlog for fixes that went into drivers/usb/host/ for 4.12 after 4.12-rc1 came out. Do you know of a way to write a test for these types of things? usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk usb: xhci: Fix USB 3.1 supported protocol parsing usb: host: xhci-plat: propagate return value of platform_get_irq() xhci: Fix command ring stop regression in 4.11 xhci: remove GFP_DMA flag from allocation USB: xhci: fix lock-inversion problem usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd usb: host: xhci-mem: allocate zeroed Scratchpad Buffer xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton usb: xhci: trace URB before giving it back instead of after USB: host: xhci: use max-port define USB: ehci-platform: fix companion-device leak usb: r8a66597-hcd: select a different endpoint on timeout usb: r8a66597-hcd: decrease timeout And look at the commits with the "Fixes:" tag in it, I do, I read every one of them. See if writing a test for the majority of them would even be possible... I don't mean to poo-poo the idea, but please realize that around 75% of the kernel is hardware/arch support, so that means that 75% of the changes/fixes deal with hardware things (yes, change is in direct correlation to size of the codebase in the tree, strange but true). If only I had a subsystem that didn't have to deal with hardware, that must be so easy to work with :) thanks, greg k-h ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> @ 2017-07-05 14:28 ` Carlos O'Donell 2017-07-05 14:33 ` Steven Rostedt ` (3 subsequent siblings) 4 siblings, 0 replies; 71+ messages in thread From: Carlos O'Donell @ 2017-07-05 14:28 UTC (permalink / raw) To: Greg KH, Steven Rostedt Cc: Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 07/05/2017 10:06 AM, Greg KH wrote: > I don't mean to poo-poo the idea, but please realize that around 75% of > the kernel is hardware/arch support, so that means that 75% of the > changes/fixes deal with hardware things (yes, change is in direct > correlation to size of the codebase in the tree, strange but true). We should distinguish between the reviewer reviewing the regression test and running the regression test. As long as the submitter ran the regression test on their hardware, and it passed, the reviewer need only review the test for logical consistency and correctness? Lack of test infrastructure was a serious problem for us in glibc. We are relying on namespaces for more complex network and filesystem testing. Without namespaces we would have needed a much more complex setup that might never have seen developer adoption. When I attended LPC 2016 I prioritized listening in on namespaces discussions to make sure nothing was changing that might break our testing framework. This conversation is going to lead down the path of driver HAL or emulation in order to provide regression testing for code above the actual hardware, and that's another hard problem, but one need not go there. Starting with real hardware tests can have benefit. In glibc we test SSE, AVX, AVX512, TSX etc. but if you don't have the extensions you get a bunch of UNSUPPORTED tests. While upstream kernel may have a more limited set of available hardware per-person, the collective set of developers has hardware to cover all configurations, and they should run the regression tests for hardware they care about ... and *must* do so if they submit a patch to fix a bug! :-) -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 14:28 ` Carlos O'Donell @ 2017-07-05 14:33 ` Steven Rostedt [not found] ` <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:33 ` Mark Brown ` (2 subsequent siblings) 4 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 14:33 UTC (permalink / raw) To: Greg KH Cc: Carlos O'Donell, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 5 Jul 2017 16:06:07 +0200 Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote: > > Your "b" above is what I would like to push. But who's going to enforce > > this? With 10,000 changes per release, and a lot of them are fixes, the > > best we can do is the honor system. Start shaming people that don't > > have a regression test along with a Fixes tag (but we don't want people > > to fix bugs without adding that tag either). There is a fine line one > > must walk between getting people to change their approaches to bugs and > > regression tests, and pissing them off where they start doing the > > opposite of what would be best for the community. > > I would bet, for the huge majority of our fixes, they are fixes for > specific hardware, or workarounds for specific hardware issues. Now > writing tests for those is not an impossible task (look at what the i915 > developers have), but it is very very hard overall, especially if the > base infrastructure isn't there to do it. > > For specific examples, here's the shortlog for fixes that went into > drivers/usb/host/ for 4.12 after 4.12-rc1 came out. Do you know of a > way to write a test for these types of things? > usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk > usb: xhci: Fix USB 3.1 supported protocol parsing > usb: host: xhci-plat: propagate return value of platform_get_irq() > xhci: Fix command ring stop regression in 4.11 > xhci: remove GFP_DMA flag from allocation > USB: xhci: fix lock-inversion problem > usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd > usb: host: xhci-mem: allocate zeroed Scratchpad Buffer > xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton > usb: xhci: trace URB before giving it back instead of after > USB: host: xhci: use max-port define > USB: ehci-platform: fix companion-device leak > usb: r8a66597-hcd: select a different endpoint on timeout > usb: r8a66597-hcd: decrease timeout > > And look at the commits with the "Fixes:" tag in it, I do, I read every > one of them. See if writing a test for the majority of them would even > be possible... > > I don't mean to poo-poo the idea, but please realize that around 75% of > the kernel is hardware/arch support, so that means that 75% of the > changes/fixes deal with hardware things (yes, change is in direct > correlation to size of the codebase in the tree, strange but true). I would say that if it's for a specific hardware, then it's really up to the maintainer if there should be a test or not. As a lot of these is just to deal with some quirk or non standard that the hardware does. But are these regressions, or just some feature that's been broken on that hardware since its conception? That is, Thorsten this is more for you, how much real regressions are in hardware? A bug that's been there forever is not a regression. It's a feature ;-) A regression is something that use to work and now does not. Is that number still as high with hardware? Those probably could be where tests can be focused on. I'm worried more about infrastructure too. I would look at general functionality of say USB, to see if something can be written to test a device. Using the one change above that actually mentions "regression" would it be possible to test completion codes? (I have no idea, I only read the change log and I'm speaking out of my derrière) If we have a bunch of generic tests that can test hardware (general video tests, USB tests, network cards, etc) and people ran these on their own hardware, and it were to trigger a failure, then it would be easier for users to report these issues to the maintainers. And these would be easier to find and fix. No test should be written for a single specific hardware. It should be a general functionality that different hardware can execute. > > If only I had a subsystem that didn't have to deal with hardware, that > must be so easy to work with :) *smack*! ;-) -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 14:52 ` Mark Brown 2017-07-05 15:08 ` Carlos O'Donell 2017-07-09 13:46 ` Thorsten Leemhuis 2 siblings, 0 replies; 71+ messages in thread From: Mark Brown @ 2017-07-05 14:52 UTC (permalink / raw) To: Steven Rostedt Cc: Greg KH, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan [-- Attachment #1: Type: text/plain, Size: 638 bytes --] On Wed, Jul 05, 2017 at 10:33:35AM -0400, Steven Rostedt wrote: > That is, Thorsten this is more for you, how much real regressions are in > hardware? A bug that's been there forever is not a regression. It's a > feature ;-) A regression is something that use to work and now does > not. Is that number still as high with hardware? Those probably could > be where tests can be focused on. A relatively common case IME is things that were always bugs but depend on some external thing to become visible, like someone trying to use a device in a slightly different way, doing more detailed testing of some kind or some subsystem change. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:52 ` Mark Brown @ 2017-07-05 15:08 ` Carlos O'Donell [not found] ` <8c6843e8-73d9-a898-0366-0b72dfeb79a2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-07-09 13:46 ` Thorsten Leemhuis 2 siblings, 1 reply; 71+ messages in thread From: Carlos O'Donell @ 2017-07-05 15:08 UTC (permalink / raw) To: Steven Rostedt, Greg KH Cc: Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 07/05/2017 10:33 AM, Steven Rostedt wrote: > No test should be written for a single specific hardware. It should be a > general functionality that different hardware can execute. Why? We test all sorts of hardware in userspace and we see value in that testing. -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <8c6843e8-73d9-a898-0366-0b72dfeb79a2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <8c6843e8-73d9-a898-0366-0b72dfeb79a2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> @ 2017-07-05 16:10 ` Steven Rostedt [not found] ` <20170705121000.5430d7d0-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 16:10 UTC (permalink / raw) To: Carlos O'Donell Cc: Greg KH, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 5 Jul 2017 11:08:48 -0400 Carlos O'Donell <carlos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: > On 07/05/2017 10:33 AM, Steven Rostedt wrote: > > No test should be written for a single specific hardware. It should be a > > general functionality that different hardware can execute. > > Why? We test all sorts of hardware in userspace and we see value in that > testing. > One reason is for bit rot. I'm not totally against it. But I envision that if we have hundreds of tests for very specific pieces of hardware, it's value will diminish over time. Unless we can get a good infrastructure written where the hardware info is more of a data sheet then a single test itself. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705121000.5430d7d0-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705121000.5430d7d0-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-06 11:34 ` Laurent Pinchart 0 siblings, 0 replies; 71+ messages in thread From: Laurent Pinchart @ 2017-07-06 11:34 UTC (permalink / raw) To: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I Cc: Steven Rostedt, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan On Wednesday 05 Jul 2017 12:10:00 Steven Rostedt wrote: > On Wed, 5 Jul 2017 11:08:48 -0400 Carlos O'Donell wrote: > > On 07/05/2017 10:33 AM, Steven Rostedt wrote: > > > No test should be written for a single specific hardware. It should be a > > > general functionality that different hardware can execute. > > > > Why? We test all sorts of hardware in userspace and we see value in that > > testing. > > One reason is for bit rot. I'm not totally against it. But I envision > that if we have hundreds of tests for very specific pieces of hardware, > it's value will diminish over time. Unless we can get a good > infrastructure written where the hardware info is more of a data sheet > then a single test itself. That's all nice, but when the hardware is complex and not fully abstracted behind a kernel API, tests are bound to be hardware-specific. Of course, a bug or regression observed only with a specific device, but triggered through the usage of abstract APIs only, can lead to a test case written for that device but runnable with any device in the same category. In that case the test case should certainly be added to a test suite for the corresponding API/subsystem, not to an hidden test suite for a particular device. -- Regards, Laurent Pinchart ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:52 ` Mark Brown 2017-07-05 15:08 ` Carlos O'Donell @ 2017-07-09 13:46 ` Thorsten Leemhuis 2 siblings, 0 replies; 71+ messages in thread From: Thorsten Leemhuis @ 2017-07-09 13:46 UTC (permalink / raw) To: Steven Rostedt, Greg KH Cc: Carlos O'Donell, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 05.07.2017 16:33, Steven Rostedt wrote: > On Wed, 5 Jul 2017 16:06:07 +0200 > Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > [...] >> I don't mean to poo-poo the idea, but please realize that around 75% of >> the kernel is hardware/arch support, so that means that 75% of the >> changes/fixes deal with hardware things (yes, change is in direct >> correlation to size of the codebase in the tree, strange but true). > I would say that if it's for a specific hardware, then it's really up > to the maintainer if there should be a test or not. As a lot of these > is just to deal with some quirk or non standard that the hardware does. > But are these regressions, or just some feature that's been broken on > that hardware since its conception? > > That is, Thorsten this is more for you, how much real regressions are in > hardware? [...] >From this and other mails in this thread I got the impression some more data would be helpful -- for example a few percentage numbers on * how many of the regressions are in hardware-specific/driver code * how many regressions suddenly pop up due to a unrelated (and maybe even correct) change * for how many regressions does it make sense to write a selftest to catch similar issues beforehand in the future. I'll try to gather some of those numbers when doing regression tracking for 4.13 (sorry again that I had to skip 4.12), so be prepare yourself for a mail when you include a "Fixes:" tag in a commit ;-) Then there is some data to talk about on the summit or continue the discussion on this mailing list or LKML. BTW, Steven, you in this thread wrote "discuss if we want to consolidate the format of all the kselftests and have something that everyone (or most) developers agree on". I put that in my notes and try to make sure we do not forget about this. Or is this something you'll drive forward yourself? Ciao, Thorsten P.S.: Sorry, I'm a bit late with my reply here. My real job (which is not really about kernel work) and some other things required my attention in the past few days... ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 14:28 ` Carlos O'Donell 2017-07-05 14:33 ` Steven Rostedt @ 2017-07-05 14:33 ` Mark Brown [not found] ` <20170705143341.oees22k2snhtmkxo-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-05 15:16 ` Guenter Roeck 2017-07-05 16:54 ` Dan Williams 4 siblings, 1 reply; 71+ messages in thread From: Mark Brown @ 2017-07-05 14:33 UTC (permalink / raw) To: Greg KH Cc: Steven Rostedt, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan [-- Attachment #1: Type: text/plain, Size: 453 bytes --] On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: > I don't mean to poo-poo the idea, but please realize that around 75% of > the kernel is hardware/arch support, so that means that 75% of the > changes/fixes deal with hardware things (yes, change is in direct > correlation to size of the codebase in the tree, strange but true). Then add in all the fixes for concurrency/locking issues and so on that're hard to reliably reproduce as well... [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705143341.oees22k2snhtmkxo-GFdadSzt00ze9xe1eoZjHA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705143341.oees22k2snhtmkxo-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> @ 2017-07-05 14:36 ` Steven Rostedt [not found] ` <20170705103658.226099c6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 14:36 UTC (permalink / raw) To: Mark Brown Cc: Greg KH, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, 5 Jul 2017 15:33:41 +0100 Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: > > > I don't mean to poo-poo the idea, but please realize that around 75% of > > the kernel is hardware/arch support, so that means that 75% of the > > changes/fixes deal with hardware things (yes, change is in direct > > correlation to size of the codebase in the tree, strange but true). > > Then add in all the fixes for concurrency/locking issues and so on > that're hard to reliably reproduce as well... All tests should be run with lockdep enabled ;-) Which a surprising few developers appear to do :-p -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705103658.226099c6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705103658.226099c6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 14:50 ` James Bottomley [not found] ` <1499266228.3668.10.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 18:17 ` Daniel Vetter 1 sibling, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-05 14:50 UTC (permalink / raw) To: Steven Rostedt, Mark Brown Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote: > On Wed, 5 Jul 2017 15:33:41 +0100 > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > > > > > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: > > > > > > > > I don't mean to poo-poo the idea, but please realize that around > > > 75% of the kernel is hardware/arch support, so that means that > > > 75% of the changes/fixes deal with hardware things (yes, change > > > is in direct correlation to size of the codebase in the tree, > > > strange but true). > > > > Then add in all the fixes for concurrency/locking issues and so on > > that're hard to reliably reproduce as well... > > All tests should be run with lockdep enabled ;-) Which a surprising > few developers appear to do :-p Lockdep checks the locking hierarchies and makes assumptions about them which it then validates ... it doesn't tell you if the data you think you're protecting was accessed outside the lock, which is the usual source of concurrency problems. In other words lockdep is useful but it's not a panacea. James ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499266228.3668.10.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499266228.3668.10.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-05 14:56 ` Steven Rostedt [not found] ` <20170705105651.5da9c969-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 14:56 UTC (permalink / raw) To: James Bottomley Cc: Mark Brown, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 05 Jul 2017 07:50:28 -0700 James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote: > > On Wed, 5 Jul 2017 15:33:41 +0100 > > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > > > > > > > > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: > > > > > > > > > > > I don't mean to poo-poo the idea, but please realize that around > > > > 75% of the kernel is hardware/arch support, so that means that > > > > 75% of the changes/fixes deal with hardware things (yes, change > > > > is in direct correlation to size of the codebase in the tree, > > > > strange but true). > > > > > > Then add in all the fixes for concurrency/locking issues and so on > > > that're hard to reliably reproduce as well... > > > > All tests should be run with lockdep enabled ;-) Which a surprising > > few developers appear to do :-p > > Lockdep checks the locking hierarchies and makes assumptions about them > which it then validates ... it doesn't tell you if the data you think We should probably look at adding infrastructure that helps in that. RCU already has a lot of there to help know if data is being protected by RCU or not. Hmm, maybe we could add a __rcu like type that we can associate protected data with, where a config can associate access to a variable with a lock being held? > you're protecting was accessed outside the lock, which is the usual > source of concurrency problems. In other words lockdep is useful but > it's not a panacea. Still not an excuse to not have lockdep enabled during tests. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705105651.5da9c969-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705105651.5da9c969-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 15:09 ` James Bottomley [not found] ` <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-05 15:09 UTC (permalink / raw) To: Steven Rostedt Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Thorsten Leemhuis On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote: > On Wed, 05 Jul 2017 07:50:28 -0700 > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > > > On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote: > > > > > > On Wed, 5 Jul 2017 15:33:41 +0100 > > > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > > > > > > > > > > > > > > > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: > > > > > > > > > > > > > > > > > > > I don't mean to poo-poo the idea, but please realize that > > > > > around 75% of the kernel is hardware/arch support, so that > > > > > means that 75% of the changes/fixes deal with hardware things > > > > > (yes, change is in direct correlation to size of the codebase > > > > > in the tree, strange but true). > > > > > > > > Then add in all the fixes for concurrency/locking issues and so > > > > on that're hard to reliably reproduce as well... > > > > > > All tests should be run with lockdep enabled ;-) Which a > > > surprising few developers appear to do :-p > > > > Lockdep checks the locking hierarchies and makes assumptions about > > them which it then validates ... it doesn't tell you if the data > > you think > > We should probably look at adding infrastructure that helps in that. > RCU already has a lot of there to help know if data is being > protected by RCU or not. > > Hmm, maybe we could add a __rcu like type that we can associate > protected data with, where a config can associate access to a > variable with a lock being held? That's about 10x more complex than the releases/acquires/must_hold annotation, which we have fairly dismal coverage on. If you remember the hotplug annotations, which were a shining example: there's a limit of complexity before any annotation system simply becomes a make work tyranny. > > you're protecting was accessed outside the lock, which is the usual > > source of concurrency problems. In other words lockdep is useful > > but it's not a panacea. > > Still not an excuse to not have lockdep enabled during tests. OK, what makes you think lockdep isn't enabled? Since Kconfig is so complex, I usually use a distro config ... they have it enabled (or at least openSUSE does), so it's enabled for everything I do. James ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-05 15:20 ` Mark Brown [not found] ` <20170705152026.rkw73q2f6xmiju37-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-05 15:20 ` Steven Rostedt 2017-07-05 18:24 ` Daniel Vetter 2 siblings, 1 reply; 71+ messages in thread From: Mark Brown @ 2017-07-05 15:20 UTC (permalink / raw) To: James Bottomley Cc: Steven Rostedt, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Thorsten Leemhuis [-- Attachment #1: Type: text/plain, Size: 1003 bytes --] On Wed, Jul 05, 2017 at 08:09:49AM -0700, James Bottomley wrote: > On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote: > > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > you're protecting was accessed outside the lock, which is the usual > > > source of concurrency problems. In other words lockdep is useful > > > but it's not a panacea. > > Still not an excuse to not have lockdep enabled during tests. > OK, what makes you think lockdep isn't enabled? Since Kconfig is so > complex, I usually use a distro config ... they have it enabled (or at > least openSUSE does), so it's enabled for everything I do. Yeah, I see enough reports with it in embedded contexts to make me think people use it there. I know I tend to have it turned on most of the time. The concurrency stuff I'm thinking of here is more the things you're mentioning with just not taking locks at all when they are needed or concurrency with hardware. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705152026.rkw73q2f6xmiju37-GFdadSzt00ze9xe1eoZjHA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705152026.rkw73q2f6xmiju37-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> @ 2017-07-05 15:40 ` Geert Uytterhoeven 0 siblings, 0 replies; 71+ messages in thread From: Geert Uytterhoeven @ 2017-07-05 15:40 UTC (permalink / raw) To: Mark Brown Cc: James Bottomley, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Thorsten Leemhuis On Wed, Jul 5, 2017 at 5:20 PM, Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 08:09:49AM -0700, James Bottomley wrote: >> On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote: >> > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > >> > > you're protecting was accessed outside the lock, which is the usual >> > > source of concurrency problems. In other words lockdep is useful >> > > but it's not a panacea. > >> > Still not an excuse to not have lockdep enabled during tests. > >> OK, what makes you think lockdep isn't enabled? Since Kconfig is so >> complex, I usually use a distro config ... they have it enabled (or at >> least openSUSE does), so it's enabled for everything I do. > > Yeah, I see enough reports with it in embedded contexts to make me think > people use it there. I know I tend to have it turned on most of the > time. The concurrency stuff I'm thinking of here is more the things > you're mentioning with just not taking locks at all when they are needed > or concurrency with hardware. I try to have it enabled as much as possible. However, as it increases kernel size (huge static tables), hitting boot loader limitations on several boards, I cannot enable all debugging I would like to on all boards. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 15:20 ` Mark Brown @ 2017-07-05 15:20 ` Steven Rostedt [not found] ` <20170705112047.23ee09f6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 18:24 ` Daniel Vetter 2 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 15:20 UTC (permalink / raw) To: James Bottomley Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Thorsten Leemhuis On Wed, 05 Jul 2017 08:09:49 -0700 James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > you're protecting was accessed outside the lock, which is the usual > > > source of concurrency problems. In other words lockdep is useful > > > but it's not a panacea. > > > > Still not an excuse to not have lockdep enabled during tests. > > OK, what makes you think lockdep isn't enabled? Since Kconfig is so > complex, I usually use a distro config ... they have it enabled (or at > least openSUSE does), so it's enabled for everything I do. openSuSE has it enabled? I hope not for its production config, as lockdep has a huge performance penalty. I'm thinking you don't have it enabled. What config are you looking at? The actual config that does the testing of locks is CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705112047.23ee09f6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705112047.23ee09f6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 15:32 ` James Bottomley [not found] ` <1499268748.3668.20.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-05 15:32 UTC (permalink / raw) To: Steven Rostedt Cc: Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Thorsten Leemhuis On Wed, 2017-07-05 at 11:20 -0400, Steven Rostedt wrote: > On Wed, 05 Jul 2017 08:09:49 -0700 > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > > > > > > > > > > > > > you're protecting was accessed outside the lock, which is the > > > > usual source of concurrency problems. In other words lockdep > > > > is useful but it's not a panacea. > > > > > > Still not an excuse to not have lockdep enabled during tests. > > > > OK, what makes you think lockdep isn't enabled? Since Kconfig is > > so complex, I usually use a distro config ... they have it enabled > > (or at least openSUSE does), so it's enabled for everything I do. > > openSuSE has it enabled? I hope not for its production config, as > lockdep has a huge performance penalty. Then, surely, it's the last thing we want when tracking down race conditgions since it will alter timings dramatically. > I'm thinking you don't have it enabled. What config are you looking > at? The actual config that does the testing of locks is > CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in. This is what it has: jejb@jarvis:~/git/linux-build> grep LOCKDEP /boot/config-4.4.73-18.17-default CONFIG_LOCKDEP_SUPPORT=y James ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499268748.3668.20.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499268748.3668.20.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-05 15:43 ` Steven Rostedt 0 siblings, 0 replies; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 15:43 UTC (permalink / raw) To: James Bottomley Cc: Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Thorsten Leemhuis On Wed, 05 Jul 2017 08:32:28 -0700 James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > openSuSE has it enabled? I hope not for its production config, as > > lockdep has a huge performance penalty. > > Then, surely, it's the last thing we want when tracking down race > conditgions since it will alter timings dramatically. It's to be run when you want to make sure locking order is at least not an issue. And it's not about running when tracking down race conditions, its to be run when developing new code. > > > I'm thinking you don't have it enabled. What config are you looking > > at? The actual config that does the testing of locks is > > CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in. > > This is what it has: > > jejb@jarvis:~/git/linux-build> grep LOCKDEP /boot/config-4.4.73-18.17-default > CONFIG_LOCKDEP_SUPPORT=y That means your architecture supports it, it's not enabled. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 15:20 ` Mark Brown 2017-07-05 15:20 ` Steven Rostedt @ 2017-07-05 18:24 ` Daniel Vetter 2 siblings, 0 replies; 71+ messages in thread From: Daniel Vetter @ 2017-07-05 18:24 UTC (permalink / raw) To: James Bottomley Cc: Steven Rostedt, Carlos O'Donell, open list:ABI/API, Shuah Khan, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Thorsten Leemhuis On Wed, Jul 5, 2017 at 5:09 PM, James Bottomley <James.Bottomley-JuX6DAaQMKPCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: >> > > All tests should be run with lockdep enabled ;-) Which a >> > > surprising few developers appear to do :-p >> > >> > Lockdep checks the locking hierarchies and makes assumptions about >> > them which it then validates ... it doesn't tell you if the data >> > you think >> >> We should probably look at adding infrastructure that helps in that. >> RCU already has a lot of there to help know if data is being >> protected by RCU or not. >> >> Hmm, maybe we could add a __rcu like type that we can associate >> protected data with, where a config can associate access to a >> variable with a lock being held? > > That's about 10x more complex than the releases/acquires/must_hold > annotation, which we have fairly dismal coverage on. Yeah, I've never found those useful at all. What we're trying to do in drm code is liberally sprinkle lockdep_assert_held into accessor and helper functions (there's lots of nontrivial stuff where you need a little bit of computation around a pure access, so doesn't result in ugly code). That catches a lot of these, but of course not all. The problem with static annotations is that often the lock you need to hold isn't statically known, and annotating the entire callchain is a no-go as James points out. But maybe we could use such annotations plus a gcc plugin to auto-insert the right lockdep_assert_held every time you read/write into a given field? That's not going to cover locking rules where the locking rules change during the lifetime of an object, but I think even without that it would cover a _lot_ of cases. And if your static annotation would be allowed to chase pointers (well, just any C expression that takes the struct pointer as parameter would be sweet) you could even annotate fields where the protecting lock is in some parent struct. Another thing I'm really looking forward to (but it's somehow not moving fast) is the cross-release stuff. Too many times I've screamed at kernel backtraces stuck in wait_event, and lockdep could have directly told me what's wrong long before a stress test successfully hit that race. There's definitely a lot of room to prove more stuff in locking using tools. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705103658.226099c6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:50 ` James Bottomley @ 2017-07-05 18:17 ` Daniel Vetter 1 sibling, 0 replies; 71+ messages in thread From: Daniel Vetter @ 2017-07-05 18:17 UTC (permalink / raw) To: Steven Rostedt Cc: Mark Brown, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, open list:ABI/API On Wed, Jul 5, 2017 at 4:36 PM, Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> wrote: > On Wed, 5 Jul 2017 15:33:41 +0100 > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: >> On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote: >> > I don't mean to poo-poo the idea, but please realize that around 75% of >> > the kernel is hardware/arch support, so that means that 75% of the >> > changes/fixes deal with hardware things (yes, change is in direct >> > correlation to size of the codebase in the tree, strange but true). >> >> Then add in all the fixes for concurrency/locking issues and so on >> that're hard to reliably reproduce as well... > > All tests should be run with lockdep enabled ;-) Which a surprising > few developers appear to do :-p We're slowly working towards running the i915 testsuite with kasan enabled as the next level of evil. It's ... interesting, to say the least. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> ` (2 preceding siblings ...) 2017-07-05 14:33 ` Mark Brown @ 2017-07-05 15:16 ` Guenter Roeck [not found] ` <a462fb3b-a6d4-e969-b301-b404981de224-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-05 16:54 ` Dan Williams 4 siblings, 1 reply; 71+ messages in thread From: Guenter Roeck @ 2017-07-05 15:16 UTC (permalink / raw) To: Greg KH, Steven Rostedt Cc: Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On 07/05/2017 07:06 AM, Greg KH wrote: > On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote: >> Your "b" above is what I would like to push. But who's going to enforce >> this? With 10,000 changes per release, and a lot of them are fixes, the >> best we can do is the honor system. Start shaming people that don't >> have a regression test along with a Fixes tag (but we don't want people >> to fix bugs without adding that tag either). There is a fine line one >> must walk between getting people to change their approaches to bugs and >> regression tests, and pissing them off where they start doing the >> opposite of what would be best for the community. > > I would bet, for the huge majority of our fixes, they are fixes for > specific hardware, or workarounds for specific hardware issues. Now > writing tests for those is not an impossible task (look at what the i915 > developers have), but it is very very hard overall, especially if the > base infrastructure isn't there to do it. > > For specific examples, here's the shortlog for fixes that went into > drivers/usb/host/ for 4.12 after 4.12-rc1 came out. Do you know of a > way to write a test for these types of things? > usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk > usb: xhci: Fix USB 3.1 supported protocol parsing > usb: host: xhci-plat: propagate return value of platform_get_irq() > xhci: Fix command ring stop regression in 4.11 > xhci: remove GFP_DMA flag from allocation > USB: xhci: fix lock-inversion problem > usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd > usb: host: xhci-mem: allocate zeroed Scratchpad Buffer > xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton > usb: xhci: trace URB before giving it back instead of after > USB: host: xhci: use max-port define > USB: ehci-platform: fix companion-device leak > usb: r8a66597-hcd: select a different endpoint on timeout > usb: r8a66597-hcd: decrease timeout > > And look at the commits with the "Fixes:" tag in it, I do, I read every > one of them. See if writing a test for the majority of them would even > be possible... > > I don't mean to poo-poo the idea, but please realize that around 75% of > the kernel is hardware/arch support, so that means that 75% of the > changes/fixes deal with hardware things (yes, change is in direct > correlation to size of the codebase in the tree, strange but true). > The reproducers for several of the usb fixes I submitted recently took hours of stress test to reproduce the underlying problems. I have one more to fix which takes days to reproduce, if at all (I have seen that problem only two or three times during weeks of stress test). Due to the nature of the problems, reproducing them heavily depended on the underlying hardware. None of the reproducers can guarantee that the problem is fixed; they are intended to show the problem, not that it is fixed. This happens a lot with race conditions - in many cases it is impossible to prove that the problem is fixed; one can only prove that it still exists. Echoing what you said, I have no idea how it would even be possible to write unit tests to verify if the problems I fixed are really fixed. Several of the fixes I have submitted are based on single-instance error logs with no reproducer. Many others are compile time fixes or fix problems found with code inspection (manual or automatic). If we start shaming people for not providing unit tests, all we'll accomplish is that people will stop providing bug fixes. Guenter ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <a462fb3b-a6d4-e969-b301-b404981de224-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <a462fb3b-a6d4-e969-b301-b404981de224-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> @ 2017-07-05 15:27 ` Steven Rostedt [not found] ` <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:32 ` Greg KH 1 sibling, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 15:27 UTC (permalink / raw) To: Guenter Roeck Cc: Greg KH, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, 5 Jul 2017 08:16:33 -0700 Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > The reproducers for several of the usb fixes I submitted recently took hours of > stress test to reproduce the underlying problems. I have one more to fix which > takes days to reproduce, if at all (I have seen that problem only two or three > times during weeks of stress test). Due to the nature of the problems, reproducing > them heavily depended on the underlying hardware. None of the reproducers can > guarantee that the problem is fixed; they are intended to show the problem, > not that it is fixed. This happens a lot with race conditions - in many cases > it is impossible to prove that the problem is fixed; one can only prove that > it still exists. > > Echoing what you said, I have no idea how it would even be possible to write > unit tests to verify if the problems I fixed are really fixed. > > Several of the fixes I have submitted are based on single-instance error logs with > no reproducer. Many others are compile time fixes or fix problems found with code > inspection (manual or automatic). > > If we start shaming people for not providing unit tests, all we'll accomplish is > that people will stop providing bug fixes. I need to be clearer on this. What I meant was, if there's a bug where someone has a test that easily reproduces the bug, then if there's not a test added to selftests for said bug, then we should shame those into doing so. A bug that is found by inspection or hard to reproduce test cases are not applicable, as they don't have tests that can show a regression. And I'm betting that those bugs are NOT REGRESSIONS! Most likely are bugs that always existed, but because of the unpredictable hitting of the bug (as you said, it required hours of stress tests to reproduce), the bug was not previously hit during development. That's not a regression, that's a feature. Are we tracking regressions or just simply bugs? -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 15:36 ` James Bottomley [not found] ` <1499269015.3668.25.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 16:48 ` Guenter Roeck 2017-07-07 3:33 ` Fengguang Wu 2 siblings, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-05 15:36 UTC (permalink / raw) To: Steven Rostedt, Guenter Roeck Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 2017-07-05 at 11:27 -0400, Steven Rostedt wrote: > On Wed, 5 Jul 2017 08:16:33 -0700 > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > > > > > The reproducers for several of the usb fixes I submitted recently > > took hours of stress test to reproduce the underlying problems. I > > have one more to fix which takes days to reproduce, if at all (I > > have seen that problem only two or three times during weeks of > > stress test). Due to the nature of the problems, reproducing > > them heavily depended on the underlying hardware. None of the > > reproducers can guarantee that the problem is fixed; they are > > intended to show the problem, not that it is fixed. This happens a > > lot with race conditions - in many cases it is impossible to prove > > that the problem is fixed; one can only prove that it still exists. > > > > Echoing what you said, I have no idea how it would even be possible > > to write unit tests to verify if the problems I fixed are really > > fixed. > > > > Several of the fixes I have submitted are based on single-instance > > error logs with no reproducer. Many others are compile time fixes > > or fix problems found with code inspection (manual or automatic). > > > > If we start shaming people for not providing unit tests, all we'll > > accomplish is that people will stop providing bug fixes. > > I need to be clearer on this. What I meant was, if there's a bug > where someone has a test that easily reproduces the bug, then if > there's not a test added to selftests for said bug, then we should > shame those into doing so. > > A bug that is found by inspection or hard to reproduce test cases are > not applicable, as they don't have tests that can show a regression. > > And I'm betting that those bugs are NOT REGRESSIONS! Most likely are > bugs that always existed, but because of the unpredictable hitting of > the bug (as you said, it required hours of stress tests to > reproduce), the bug was not previously hit during development. That's > not a regression, that's a feature. > > Are we tracking regressions or just simply bugs? A lot of device driver regressions are bugs that previously existed in the code but which didn't manifest until something else happened. A huge number of locking and timing issues are like this. The irony is that a lot of them go from race always being won (so bug never noticed) to race being lost often enough to make something unusable. To a user that ends up being a kernel regression because it's a bug in the current kernel which they didn't see previously which makes it unusable for them. I've got to vote with my users here: that's a regression not a "feature". James ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499269015.3668.25.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499269015.3668.25.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-05 16:04 ` Steven Rostedt [not found] ` <20170705120459.41e81f7b-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 16:04 UTC (permalink / raw) To: James Bottomley Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, 05 Jul 2017 08:36:55 -0700 James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > Are we tracking regressions or just simply bugs? > > A lot of device driver regressions are bugs that previously existed in > the code but which didn't manifest until something else happened. A > huge number of locking and timing issues are like this. The irony is > that a lot of them go from race always being won (so bug never noticed) > to race being lost often enough to make something unusable. To a user > that ends up being a kernel regression because it's a bug in the > current kernel which they didn't see previously which makes it unusable > for them. > > I've got to vote with my users here: that's a regression not a > "feature". Let's take a step back. What exactly is the problem? The regressions that we want to track? Why are they not fixed? Is it because they are hard to reproduce? If so, how do we know they are a regression or just some hard to hit bug? If it's hard to hit, how do we know we fixed it? What exactly are the questions we want solved. Granted, I used this thread to push more use of kselftests, and I don't see any SCSI tests there at all! -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705120459.41e81f7b-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705120459.41e81f7b-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 16:58 ` James Bottomley [not found] ` <1499273908.3668.30.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-05 16:58 UTC (permalink / raw) To: Steven Rostedt Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan On Wed, 2017-07-05 at 12:04 -0400, Steven Rostedt wrote: > On Wed, 05 Jul 2017 08:36:55 -0700 > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > > > > > > > Are we tracking regressions or just simply bugs? > > > > A lot of device driver regressions are bugs that previously existed > > in the code but which didn't manifest until something else > > happened. A huge number of locking and timing issues are like > > this. The irony is that a lot of them go from race always being > > won (so bug never noticed) to race being lost often enough to make > > something unusable. To a user that ends up being a kernel > > regression because it's a bug in the current kernel which they > > didn't see previously which makes it unusable for them. > > > > I've got to vote with my users here: that's a regression not a > > "feature". > > Let's take a step back. What exactly is the problem? You mean what question was I answering? It was your "is your problem a regression?" one. > The regressions that we want to track? Why are they not fixed? Is it > because they are hard to reproduce? If so, how do we know they are a > regression or just some hard to hit bug? If it's hard to hit, how do > we know we fixed it? Usually for the exposed races we develop a theoretical model which tells us what the problem is and also the solution. > What exactly are the questions we want solved. In the context of this subthread? Tracking and fixing of regressions meaning behaviour that damages or destroys usability of version k+1 that wasn't present in version k. > Granted, I used this thread to push more use of kselftests, and I > don't see any SCSI tests there at all! It would be an interesting question for another thread to consider whether that's a problem or not. James ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499273908.3668.30.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499273908.3668.30.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-05 17:07 ` Steven Rostedt 0 siblings, 0 replies; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 17:07 UTC (permalink / raw) To: James Bottomley Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan On Wed, 05 Jul 2017 09:58:28 -0700 James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > On Wed, 2017-07-05 at 12:04 -0400, Steven Rostedt wrote: > > On Wed, 05 Jul 2017 08:36:55 -0700 > > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote: > > > > > > > > > > > > > Are we tracking regressions or just simply bugs? > > > > > > A lot of device driver regressions are bugs that previously existed > > > in the code but which didn't manifest until something else > > > happened. A huge number of locking and timing issues are like > > > this. The irony is that a lot of them go from race always being > > > won (so bug never noticed) to race being lost often enough to make > > > something unusable. To a user that ends up being a kernel > > > regression because it's a bug in the current kernel which they > > > didn't see previously which makes it unusable for them. > > > > > > I've got to vote with my users here: that's a regression not a > > > "feature". > > > > Let's take a step back. What exactly is the problem? > > You mean what question was I answering? It was your "is your problem a > regression?" one. No that's not what I meant. I mean that we are going off tangent to the original topic. > > > The regressions that we want to track? Why are they not fixed? Is it > > because they are hard to reproduce? If so, how do we know they are a > > regression or just some hard to hit bug? If it's hard to hit, how do > > we know we fixed it? > > Usually for the exposed races we develop a theoretical model which > tells us what the problem is and also the solution. I think the problem is that the regressions that are not being fixed happen to be where we have no model to create, as the problem may be too hard to hit, and it could just be a "works for me" issue. > > > What exactly are the questions we want solved. > > In the context of this subthread? Tracking and fixing of regressions > meaning behaviour that damages or destroys usability of version k+1 > that wasn't present in version k. Agreed with this part. And I believe this is also in the context of the entire thread. > > > Granted, I used this thread to push more use of kselftests, and I > > don't see any SCSI tests there at all! > > It would be an interesting question for another thread to consider > whether that's a problem or not. It's not a problem for me, but it begs the question of whether it would be useful or not. But I agree, that's for another thread. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:36 ` James Bottomley @ 2017-07-05 16:48 ` Guenter Roeck [not found] ` <c782a15a-4e73-7373-ca66-5b55e9406059-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-07 3:33 ` Fengguang Wu 2 siblings, 1 reply; 71+ messages in thread From: Guenter Roeck @ 2017-07-05 16:48 UTC (permalink / raw) To: Steven Rostedt Cc: Greg KH, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On 07/05/2017 08:27 AM, Steven Rostedt wrote: > On Wed, 5 Jul 2017 08:16:33 -0700 > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: [ ... ] >> >> If we start shaming people for not providing unit tests, all we'll accomplish is >> that people will stop providing bug fixes. > > I need to be clearer on this. What I meant was, if there's a bug > where someone has a test that easily reproduces the bug, then if > there's not a test added to selftests for said bug, then we should > shame those into doing so. > I don't think that public shaming of kernel developers is going to work any better than public shaming of children or teenagers. Maybe a friendlier approach would be more useful ? If a test to reproduce a problem exists, it might be more beneficial to suggest to the patch submitter that it would be great if that test would be submitted as unit test instead of shaming that person for not doing so. Acknowledging and praising kselftest submissions might help more than shaming for non-submissions. > A bug that is found by inspection or hard to reproduce test cases are > not applicable, as they don't have tests that can show a regression. > My concern would be that once the shaming starts, it won't stop. Guenter ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <c782a15a-4e73-7373-ca66-5b55e9406059-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <c782a15a-4e73-7373-ca66-5b55e9406059-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> @ 2017-07-05 16:58 ` Dan Williams 2017-07-05 17:02 ` Steven Rostedt 1 sibling, 0 replies; 71+ messages in thread From: Dan Williams @ 2017-07-05 16:58 UTC (permalink / raw) To: Guenter Roeck Cc: Steven Rostedt, ksummit, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, Linux API On Wed, Jul 5, 2017 at 9:48 AM, Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > On 07/05/2017 08:27 AM, Steven Rostedt wrote: >> >> On Wed, 5 Jul 2017 08:16:33 -0700 >> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > > [ ... ] >>> >>> >>> If we start shaming people for not providing unit tests, all we'll >>> accomplish is >>> that people will stop providing bug fixes. >> >> >> I need to be clearer on this. What I meant was, if there's a bug >> where someone has a test that easily reproduces the bug, then if >> there's not a test added to selftests for said bug, then we should >> shame those into doing so. >> > > I don't think that public shaming of kernel developers is going to work > any better than public shaming of children or teenagers. > > Maybe a friendlier approach would be more useful ? > > If a test to reproduce a problem exists, it might be more beneficial to > suggest > to the patch submitter that it would be great if that test would be > submitted > as unit test instead of shaming that person for not doing so. Acknowledging > and > praising kselftest submissions might help more than shaming for > non-submissions. > >> A bug that is found by inspection or hard to reproduce test cases are >> not applicable, as they don't have tests that can show a regression. >> > > My concern would be that once the shaming starts, it won't stop. Agreed, this shouldn't be a new burden for maintainers, this should be a contribution path for new kernel developers. Go beyond our standard "fix a bug" advice, which is a great advice, and also recommend "backstop a regression with a unit test". ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <c782a15a-4e73-7373-ca66-5b55e9406059-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-05 16:58 ` Dan Williams @ 2017-07-05 17:02 ` Steven Rostedt [not found] ` <20170705130200.7c653f61-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 17:02 UTC (permalink / raw) To: Guenter Roeck Cc: Greg KH, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, 5 Jul 2017 09:48:31 -0700 Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > On 07/05/2017 08:27 AM, Steven Rostedt wrote: > > On Wed, 5 Jul 2017 08:16:33 -0700 > > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > [ ... ] > >> > >> If we start shaming people for not providing unit tests, all we'll accomplish is > >> that people will stop providing bug fixes. > > > > I need to be clearer on this. What I meant was, if there's a bug > > where someone has a test that easily reproduces the bug, then if > > there's not a test added to selftests for said bug, then we should > > shame those into doing so. > > > > I don't think that public shaming of kernel developers is going to work > any better than public shaming of children or teenagers. > > Maybe a friendlier approach would be more useful ? I'm a friendly shamer ;-) > > If a test to reproduce a problem exists, it might be more beneficial to suggest > to the patch submitter that it would be great if that test would be submitted > as unit test instead of shaming that person for not doing so. Acknowledging and > praising kselftest submissions might help more than shaming for non-submissions. > > > A bug that is found by inspection or hard to reproduce test cases are > > not applicable, as they don't have tests that can show a regression. > > > > My concern would be that once the shaming starts, it won't stop. I think this is a communication issue. My word for "shaming" was to call out a developer for not submitting a test. It wasn't about making fun of them, or anything like that. I was only making a point about how to teach people that they need to be more aware of the testing infrastructure. Not about actually demeaning people. Lets take a hypothetical sample. Say someone posted a bug report with an associated reproducer for it. The developer then runs the reproducer sees the bug, makes a fix and sends it to Linus and stable. Now the developer forgets this and continues on their merry way. Along comes someone like myself and sees a reproducing test case for a bug, but sees no test added to kselftests. I would send an email along the lines of "Hi, I noticed that there was a reproducer for this bug you fixed. How come there was no test added to the kselftests to make sure it doesn't appear again?" There, I "shamed" them ;-) -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705130200.7c653f61-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705130200.7c653f61-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-06 9:28 ` Mark Brown [not found] ` <20170706092836.ifcnc2qqwufndhdl-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-31 16:54 ` Eric W. Biederman 1 sibling, 1 reply; 71+ messages in thread From: Mark Brown @ 2017-07-06 9:28 UTC (permalink / raw) To: Steven Rostedt Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA [-- Attachment #1: Type: text/plain, Size: 1191 bytes --] On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote: > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > > If a test to reproduce a problem exists, it might be more beneficial to suggest > > to the patch submitter that it would be great if that test would be submitted > > as unit test instead of shaming that person for not doing so. Acknowledging and > > praising kselftest submissions might help more than shaming for non-submissions. > > My concern would be that once the shaming starts, it won't stop. > I think this is a communication issue. My word for "shaming" was to > call out a developer for not submitting a test. It wasn't about making > fun of them, or anything like that. I was only making a point > about how to teach people that they need to be more aware of the > testing infrastructure. Not about actually demeaning people. I think before anything like that is viable we need to show a concerted and visible interest in actually running the tests we already have and paying attention to the results - if people can see that they're just checking a checkbox that will often result in low quality tests which can do more harm than good. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170706092836.ifcnc2qqwufndhdl-GFdadSzt00ze9xe1eoZjHA@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170706092836.ifcnc2qqwufndhdl-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> @ 2017-07-06 9:41 ` Daniel Vetter [not found] ` <CAKMK7uFH+Kz8Mdph=J_FCZ4LC3tzoOmwNJPpSO+snTz6p0Xz+w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-07-06 14:48 ` James Bottomley 1 sibling, 1 reply; 71+ messages in thread From: Daniel Vetter @ 2017-07-06 9:41 UTC (permalink / raw) To: Mark Brown Cc: Steven Rostedt, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, open list:ABI/API, Thorsten Leemhuis, Shuah Khan On Thu, Jul 6, 2017 at 11:28 AM, Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote: >> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > >> > If a test to reproduce a problem exists, it might be more beneficial to suggest >> > to the patch submitter that it would be great if that test would be submitted >> > as unit test instead of shaming that person for not doing so. Acknowledging and >> > praising kselftest submissions might help more than shaming for non-submissions. > >> > My concern would be that once the shaming starts, it won't stop. > >> I think this is a communication issue. My word for "shaming" was to >> call out a developer for not submitting a test. It wasn't about making >> fun of them, or anything like that. I was only making a point >> about how to teach people that they need to be more aware of the >> testing infrastructure. Not about actually demeaning people. > > I think before anything like that is viable we need to show a concerted > and visible interest in actually running the tests we already have and > paying attention to the results - if people can see that they're just > checking a checkbox that will often result in low quality tests which > can do more harm than good. +1. That pretty much means large-scale CI. The i915 test suite has suffered quite a bit over the past years because the CI infrastructure didn't keep up. Result is that running full CI kills pretty much every platform there is eventually, and it's really hard to get back to a state where the testsuite can be used to catch regressions again. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <CAKMK7uFH+Kz8Mdph=J_FCZ4LC3tzoOmwNJPpSO+snTz6p0Xz+w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <CAKMK7uFH+Kz8Mdph=J_FCZ4LC3tzoOmwNJPpSO+snTz6p0Xz+w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-07-06 14:53 ` Theodore Ts'o [not found] ` <20170706145346.6w2uzcf7xacbr3or-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Theodore Ts'o @ 2017-07-06 14:53 UTC (permalink / raw) To: Daniel Vetter Cc: Mark Brown, Steven Rostedt, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, open list:ABI/API, Thorsten Leemhuis, Shuah Khan On Thu, Jul 06, 2017 at 11:41:39AM +0200, Daniel Vetter wrote: > > +1. That pretty much means large-scale CI. The i915 test suite has > suffered quite a bit over the past years because the CI infrastructure > didn't keep up. Result is that running full CI kills pretty much every > platform there is eventually, and it's really hard to get back to a > state where the testsuite can be used to catch regressions again. I assume the i915 test suite requires real hardware and can't be run on VM's; is that correct? - Ted ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170706145346.6w2uzcf7xacbr3or-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170706145346.6w2uzcf7xacbr3or-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> @ 2017-07-06 21:28 ` Daniel Vetter 0 siblings, 0 replies; 71+ messages in thread From: Daniel Vetter @ 2017-07-06 21:28 UTC (permalink / raw) To: Theodore Ts'o Cc: Mark Brown, Steven Rostedt, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, open list:ABI/API, Thorsten Leemhuis, Shuah Khan On Thu, Jul 6, 2017 at 4:53 PM, Theodore Ts'o <tytso-3s7WtUTddSA@public.gmane.org> wrote: > On Thu, Jul 06, 2017 at 11:41:39AM +0200, Daniel Vetter wrote: >> +1. That pretty much means large-scale CI. The i915 test suite has >> suffered quite a bit over the past years because the CI infrastructure >> didn't keep up. Result is that running full CI kills pretty much every >> platform there is eventually, and it's really hard to get back to a >> state where the testsuite can be used to catch regressions again. > > I assume the i915 test suite requires real hardware and can't be run > on VM's; is that correct? Yes, that's another problem. If all bigger teams/subsystems would do what we'd do, but extended to all mailing lists, your patch series would get replies from a few hundred CI farms. Not sure that would scale ... And there's no way ever that one single entity will have hardware for everything. And if you only CI the mailing list for your own subsystem then every time a merge window happens your CI will be out of service (4.13 seems extremely bad, atm nothing survives an extended run on linux-next for us). -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170706092836.ifcnc2qqwufndhdl-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-06 9:41 ` Daniel Vetter @ 2017-07-06 14:48 ` James Bottomley [not found] ` <1499352485.2765.14.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: James Bottomley @ 2017-07-06 14:48 UTC (permalink / raw) To: Mark Brown, Steven Rostedt Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan [-- Attachment #1: Type: text/plain, Size: 2140 bytes --] On Thu, 2017-07-06 at 10:28 +0100, Mark Brown wrote: > On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote: > > > > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > > > > > > > > > If a test to reproduce a problem exists, it might be more > > > beneficial to suggest to the patch submitter that it would be > > > great if that test would be submitted as unit test instead of > > > shaming that person for not doing so. Acknowledging and > > > praising kselftest submissions might help more than shaming for > > > non-submissions. > > > > > > > > > My concern would be that once the shaming starts, it won't stop. > > > > > I think this is a communication issue. My word for "shaming" was to > > call out a developer for not submitting a test. It wasn't about > > making fun of them, or anything like that. I was only making a > > point about how to teach people that they need to be more aware of > > the testing infrastructure. Not about actually demeaning people. > > I think before anything like that is viable we need to show a > concerted and visible interest in actually running the tests we > already have and paying attention to the results - if people can see > that they're just checking a checkbox that will often result in low > quality tests which can do more harm than good. it depends what you mean by "we". I used to run a battery of tests over every SCSI commit. It was time consuming and slowed down the process, plus it was me who always got to diagnose failures. Nowadays I don't bother: I rely on 0day to run its usual tests plus a couple of extras I asked for it's a much more streamlined process (meaning less work for me) and everyone is happy. The corollary I take away from this is that the less intrusive the test infrastructure is (at least to my process) the happier I am. The 0day quantum leap for me was going from testing my tree and telling me of problems after I've added the patch to testing patches posted to the mailing list, which tells me of problems *before* the commit gets added to the tree. James [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <1499352485.2765.14.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <1499352485.2765.14.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> @ 2017-07-07 10:03 ` Mark Brown 0 siblings, 0 replies; 71+ messages in thread From: Mark Brown @ 2017-07-07 10:03 UTC (permalink / raw) To: James Bottomley Cc: Steven Rostedt, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan [-- Attachment #1: Type: text/plain, Size: 1388 bytes --] On Thu, Jul 06, 2017 at 07:48:05AM -0700, James Bottomley wrote: > On Thu, 2017-07-06 at 10:28 +0100, Mark Brown wrote: > > I think before anything like that is viable we need to show a > > concerted and visible interest in actually running the tests we > > already have and paying attention to the results - if people can see > > that they're just checking a checkbox that will often result in low > > quality tests which can do more harm than good. > it depends what you mean by "we". I used to run a battery of tests > over every SCSI commit. It was time consuming and slowed down the We as a community, I think something viable needs to be central services like kernelci that's automated and allows multiple people to be involved with the analysis. Hand running tests at scale just doesn't. > The corollary I take away from this is that the less intrusive the test > infrastructure is (at least to my process) the happier I am. The 0day > quantum leap for me was going from testing my tree and telling me of > problems after I've added the patch to testing patches posted to the > mailing list, which tells me of problems *before* the commit gets added > to the tree. I think we'd get a long way just by looking at what's ending up in -next - it's not as good as detecting things before they go in but it's workable if people keep on top of things. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705130200.7c653f61-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-06 9:28 ` Mark Brown @ 2017-07-31 16:54 ` Eric W. Biederman [not found] ` <87zibkzgve.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Eric W. Biederman @ 2017-07-31 16:54 UTC (permalink / raw) To: Steven Rostedt Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: > On Wed, 5 Jul 2017 09:48:31 -0700 > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: > >> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >> > On Wed, 5 Jul 2017 08:16:33 -0700 >> > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >> [ ... ] >> >> >> >> If we start shaming people for not providing unit tests, all we'll accomplish is >> >> that people will stop providing bug fixes. >> > >> > I need to be clearer on this. What I meant was, if there's a bug >> > where someone has a test that easily reproduces the bug, then if >> > there's not a test added to selftests for said bug, then we should >> > shame those into doing so. >> > >> >> I don't think that public shaming of kernel developers is going to work >> any better than public shaming of children or teenagers. >> >> Maybe a friendlier approach would be more useful ? > > I'm a friendly shamer ;-) > >> >> If a test to reproduce a problem exists, it might be more beneficial to suggest >> to the patch submitter that it would be great if that test would be submitted >> as unit test instead of shaming that person for not doing so. Acknowledging and >> praising kselftest submissions might help more than shaming for non-submissions. >> >> > A bug that is found by inspection or hard to reproduce test cases are >> > not applicable, as they don't have tests that can show a regression. >> > >> >> My concern would be that once the shaming starts, it won't stop. > > I think this is a communication issue. My word for "shaming" was to > call out a developer for not submitting a test. It wasn't about making > fun of them, or anything like that. I was only making a point > about how to teach people that they need to be more aware of the > testing infrastructure. Not about actually demeaning people. > > Lets take a hypothetical sample. Say someone posted a bug report with > an associated reproducer for it. The developer then runs the reproducer > sees the bug, makes a fix and sends it to Linus and stable. Now the > developer forgets this and continues on their merry way. Along comes > someone like myself and sees a reproducing test case for a bug, but > sees no test added to kselftests. I would send an email along the lines > of "Hi, I noticed that there was a reproducer for this bug you fixed. > How come there was no test added to the kselftests to make sure it > doesn't appear again?" There, I "shamed" them ;-) I just want to point out that kselftests are hard to build and run. As I was looking at another issue I found a bug in one of the tests. It had defined a constant wrong. I have a patch. It took me a week of poking at the kselftest code and trying one thing or another (between working on other things) before I could figure out which combination of things would let the test build and run. Until kselftests get easier to run I don't think they are something we want to push to hard. Eric ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <87zibkzgve.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <87zibkzgve.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> @ 2017-07-31 20:11 ` Steven Rostedt [not found] ` <20170731161123.4d1e80ac-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-08-02 16:53 ` Shuah Khan 1 sibling, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-31 20:11 UTC (permalink / raw) To: Eric W. Biederman Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Mon, 31 Jul 2017 11:54:45 -0500 ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org (Eric W. Biederman) wrote: > Until kselftests get easier to run I don't think they are something we > want to push to hard. Then perhaps we should push making them easier to run. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170731161123.4d1e80ac-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170731161123.4d1e80ac-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-31 20:12 ` Eric W. Biederman 0 siblings, 0 replies; 71+ messages in thread From: Eric W. Biederman @ 2017-07-31 20:12 UTC (permalink / raw) To: Steven Rostedt Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: > On Mon, 31 Jul 2017 11:54:45 -0500 > ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org (Eric W. Biederman) wrote: > >> Until kselftests get easier to run I don't think they are something we >> want to push to hard. > > Then perhaps we should push making them easier to run. Please. Eric ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <87zibkzgve.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 2017-07-31 20:11 ` Steven Rostedt @ 2017-08-02 16:53 ` Shuah Khan [not found] ` <fb399eba-91eb-21a5-e3f5-c3f919061c12-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Shuah Khan @ 2017-08-02 16:53 UTC (permalink / raw) To: Eric W. Biederman, Steven Rostedt Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Shuah Khan On 07/31/2017 10:54 AM, Eric W. Biederman wrote: > Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: > >> On Wed, 5 Jul 2017 09:48:31 -0700 >> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >> >>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>> [ ... ] >>>>> >>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>> that people will stop providing bug fixes. >>>> >>>> I need to be clearer on this. What I meant was, if there's a bug >>>> where someone has a test that easily reproduces the bug, then if >>>> there's not a test added to selftests for said bug, then we should >>>> shame those into doing so. >>>> >>> >>> I don't think that public shaming of kernel developers is going to work >>> any better than public shaming of children or teenagers. >>> >>> Maybe a friendlier approach would be more useful ? >> >> I'm a friendly shamer ;-) >> >>> >>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>> to the patch submitter that it would be great if that test would be submitted >>> as unit test instead of shaming that person for not doing so. Acknowledging and >>> praising kselftest submissions might help more than shaming for non-submissions. >>> >>>> A bug that is found by inspection or hard to reproduce test cases are >>>> not applicable, as they don't have tests that can show a regression. >>>> >>> >>> My concern would be that once the shaming starts, it won't stop. >> >> I think this is a communication issue. My word for "shaming" was to >> call out a developer for not submitting a test. It wasn't about making >> fun of them, or anything like that. I was only making a point >> about how to teach people that they need to be more aware of the >> testing infrastructure. Not about actually demeaning people. >> >> Lets take a hypothetical sample. Say someone posted a bug report with >> an associated reproducer for it. The developer then runs the reproducer >> sees the bug, makes a fix and sends it to Linus and stable. Now the >> developer forgets this and continues on their merry way. Along comes >> someone like myself and sees a reproducing test case for a bug, but >> sees no test added to kselftests. I would send an email along the lines >> of "Hi, I noticed that there was a reproducer for this bug you fixed. >> How come there was no test added to the kselftests to make sure it >> doesn't appear again?" There, I "shamed" them ;-) > > I just want to point out that kselftests are hard to build and run. > > As I was looking at another issue I found a bug in one of the tests. It > had defined a constant wrong. I have a patch. It took me a week of > poking at the kselftest code and trying one thing or another (between > working on other things) before I could figure out which combination of > things would let the test build and run. > > Until kselftests get easier to run I don't think they are something we > want to push to hard. > I would say it is easy to run ksefltests - "make kseflttest" from the main Makefile does this for you. You can also run individual tests: "make -C tools/testing/selftests/sync" for example to run sync tests. However, I think the main pain point at the moment is being able to sift through the output to make sense of it and being able to clearly identify run to run differences. At the moment, each test uses its own output format for results. As a result it is hard for users to understand the output and parse it. This problem is being addressed. There is active work in progress converting tests to TAP13 output format. I have been working several tests with help from others. Please give it a try on 4.13 latest or linux-kselftest next and suggest improvements. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <fb399eba-91eb-21a5-e3f5-c3f919061c12-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <fb399eba-91eb-21a5-e3f5-c3f919061c12-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> @ 2017-08-02 17:33 ` Eric W. Biederman [not found] ` <87lgn1nac8.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Eric W. Biederman @ 2017-08-02 17:33 UTC (permalink / raw) To: Shuah Khan Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: > On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >> >>> On Wed, 5 Jul 2017 09:48:31 -0700 >>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>> >>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>> [ ... ] >>>>>> >>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>> that people will stop providing bug fixes. >>>>> >>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>> where someone has a test that easily reproduces the bug, then if >>>>> there's not a test added to selftests for said bug, then we should >>>>> shame those into doing so. >>>>> >>>> >>>> I don't think that public shaming of kernel developers is going to work >>>> any better than public shaming of children or teenagers. >>>> >>>> Maybe a friendlier approach would be more useful ? >>> >>> I'm a friendly shamer ;-) >>> >>>> >>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>> to the patch submitter that it would be great if that test would be submitted >>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>> praising kselftest submissions might help more than shaming for non-submissions. >>>> >>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>> not applicable, as they don't have tests that can show a regression. >>>>> >>>> >>>> My concern would be that once the shaming starts, it won't stop. >>> >>> I think this is a communication issue. My word for "shaming" was to >>> call out a developer for not submitting a test. It wasn't about making >>> fun of them, or anything like that. I was only making a point >>> about how to teach people that they need to be more aware of the >>> testing infrastructure. Not about actually demeaning people. >>> >>> Lets take a hypothetical sample. Say someone posted a bug report with >>> an associated reproducer for it. The developer then runs the reproducer >>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>> developer forgets this and continues on their merry way. Along comes >>> someone like myself and sees a reproducing test case for a bug, but >>> sees no test added to kselftests. I would send an email along the lines >>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>> How come there was no test added to the kselftests to make sure it >>> doesn't appear again?" There, I "shamed" them ;-) >> >> I just want to point out that kselftests are hard to build and run. >> >> As I was looking at another issue I found a bug in one of the tests. It >> had defined a constant wrong. I have a patch. It took me a week of >> poking at the kselftest code and trying one thing or another (between >> working on other things) before I could figure out which combination of >> things would let the test build and run. >> >> Until kselftests get easier to run I don't think they are something we >> want to push to hard. >> > > I would say it is easy to run ksefltests - "make kseflttest" from the > main Makefile does this for you. You can also run individual tests: On 4.13-rc1 That doesn't work. $ make O=$PWD-build -j8 kselftests make[1]: Entering directory 'linux-build' make[1]: *** No rule to make target 'kselftests'. Stop. make[1]: Leaving directory 'linux-build' Makefile:145: recipe for target 'sub-make' failed make: *** [sub-make] Error 2 And why I have to use some esoteric command and not just the traditional "make path/to/test/output" to run an individual test is beyond me. Eric ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <87lgn1nac8.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <87lgn1nac8.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> @ 2017-08-02 17:46 ` Shuah Khan [not found] ` <61093c3f-6ad0-c033-a524-f2624f8d55ba-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Shuah Khan @ 2017-08-02 17:46 UTC (permalink / raw) To: Eric W. Biederman Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan On 08/02/2017 11:33 AM, Eric W. Biederman wrote: > Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: > >> On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >>> >>>> On Wed, 5 Jul 2017 09:48:31 -0700 >>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>> >>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>> [ ... ] >>>>>>> >>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>>> that people will stop providing bug fixes. >>>>>> >>>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>>> where someone has a test that easily reproduces the bug, then if >>>>>> there's not a test added to selftests for said bug, then we should >>>>>> shame those into doing so. >>>>>> >>>>> >>>>> I don't think that public shaming of kernel developers is going to work >>>>> any better than public shaming of children or teenagers. >>>>> >>>>> Maybe a friendlier approach would be more useful ? >>>> >>>> I'm a friendly shamer ;-) >>>> >>>>> >>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>>> to the patch submitter that it would be great if that test would be submitted >>>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>>> praising kselftest submissions might help more than shaming for non-submissions. >>>>> >>>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>>> not applicable, as they don't have tests that can show a regression. >>>>>> >>>>> >>>>> My concern would be that once the shaming starts, it won't stop. >>>> >>>> I think this is a communication issue. My word for "shaming" was to >>>> call out a developer for not submitting a test. It wasn't about making >>>> fun of them, or anything like that. I was only making a point >>>> about how to teach people that they need to be more aware of the >>>> testing infrastructure. Not about actually demeaning people. >>>> >>>> Lets take a hypothetical sample. Say someone posted a bug report with >>>> an associated reproducer for it. The developer then runs the reproducer >>>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>>> developer forgets this and continues on their merry way. Along comes >>>> someone like myself and sees a reproducing test case for a bug, but >>>> sees no test added to kselftests. I would send an email along the lines >>>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>>> How come there was no test added to the kselftests to make sure it >>>> doesn't appear again?" There, I "shamed" them ;-) >>> >>> I just want to point out that kselftests are hard to build and run. >>> >>> As I was looking at another issue I found a bug in one of the tests. It >>> had defined a constant wrong. I have a patch. It took me a week of >>> poking at the kselftest code and trying one thing or another (between >>> working on other things) before I could figure out which combination of >>> things would let the test build and run. >>> >>> Until kselftests get easier to run I don't think they are something we >>> want to push to hard. >>> >> >> I would say it is easy to run ksefltests - "make kseflttest" from the >> main Makefile does this for you. You can also run individual tests: > > On 4.13-rc1 That doesn't work. > > $ make O=$PWD-build -j8 kselftests > make[1]: Entering directory 'linux-build' > make[1]: *** No rule to make target 'kselftests'. Stop. > make[1]: Leaving directory 'linux-build' > Makefile:145: recipe for target 'sub-make' failed > make: *** [sub-make] Error 2 It is "make kselftest" > > And why I have to use some esoteric command and not just the > traditional "make path/to/test/output" to run an individual > test is beyond me. > make kselftest from top level Makefile is a way to run all the tests. As I mentioned in my previous email "You can also run individual tests: "make -C tools/testing/selftests/sync" for example to run sync tests. or you can also run: make -C tools/testing/selftests/ run_tests thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <61093c3f-6ad0-c033-a524-f2624f8d55ba-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <61093c3f-6ad0-c033-a524-f2624f8d55ba-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> @ 2017-08-02 17:58 ` Shuah Khan 2017-08-02 18:04 ` Eric W. Biederman 1 sibling, 0 replies; 71+ messages in thread From: Shuah Khan @ 2017-08-02 17:58 UTC (permalink / raw) To: Eric W. Biederman Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Shuah Khan On 08/02/2017 11:46 AM, Shuah Khan wrote: > On 08/02/2017 11:33 AM, Eric W. Biederman wrote: >> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: >> >>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >>>> >>>>> On Wed, 5 Jul 2017 09:48:31 -0700 >>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>> >>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>> [ ... ] >>>>>>>> >>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>>>> that people will stop providing bug fixes. >>>>>>> >>>>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>>>> where someone has a test that easily reproduces the bug, then if >>>>>>> there's not a test added to selftests for said bug, then we should >>>>>>> shame those into doing so. >>>>>>> >>>>>> >>>>>> I don't think that public shaming of kernel developers is going to work >>>>>> any better than public shaming of children or teenagers. >>>>>> >>>>>> Maybe a friendlier approach would be more useful ? >>>>> >>>>> I'm a friendly shamer ;-) >>>>> >>>>>> >>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>>>> to the patch submitter that it would be great if that test would be submitted >>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>>>> praising kselftest submissions might help more than shaming for non-submissions. >>>>>> >>>>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>>>> not applicable, as they don't have tests that can show a regression. >>>>>>> >>>>>> >>>>>> My concern would be that once the shaming starts, it won't stop. >>>>> >>>>> I think this is a communication issue. My word for "shaming" was to >>>>> call out a developer for not submitting a test. It wasn't about making >>>>> fun of them, or anything like that. I was only making a point >>>>> about how to teach people that they need to be more aware of the >>>>> testing infrastructure. Not about actually demeaning people. >>>>> >>>>> Lets take a hypothetical sample. Say someone posted a bug report with >>>>> an associated reproducer for it. The developer then runs the reproducer >>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>>>> developer forgets this and continues on their merry way. Along comes >>>>> someone like myself and sees a reproducing test case for a bug, but >>>>> sees no test added to kselftests. I would send an email along the lines >>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>>>> How come there was no test added to the kselftests to make sure it >>>>> doesn't appear again?" There, I "shamed" them ;-) >>>> >>>> I just want to point out that kselftests are hard to build and run. >>>> >>>> As I was looking at another issue I found a bug in one of the tests. It >>>> had defined a constant wrong. I have a patch. It took me a week of >>>> poking at the kselftest code and trying one thing or another (between >>>> working on other things) before I could figure out which combination of >>>> things would let the test build and run. >>>> >>>> Until kselftests get easier to run I don't think they are something we >>>> want to push to hard. >>>> >>> >>> I would say it is easy to run ksefltests - "make kseflttest" from the >>> main Makefile does this for you. You can also run individual tests: >> >> On 4.13-rc1 That doesn't work. >> >> $ make O=$PWD-build -j8 kselftests >> make[1]: Entering directory 'linux-build' >> make[1]: *** No rule to make target 'kselftests'. Stop. >> make[1]: Leaving directory 'linux-build' >> Makefile:145: recipe for target 'sub-make' failed >> make: *** [sub-make] Error 2 > > It is "make kselftest" > >> >> And why I have to use some esoteric command and not just the >> traditional "make path/to/test/output" to run an individual >> test is beyond me. >> > > make kselftest from top level Makefile is a way to run all the tests. > As I mentioned in my previous email > > "You can also run individual tests: > > "make -C tools/testing/selftests/sync" for example to run sync tests. > > or you can also run: > > make -C tools/testing/selftests/ run_tests > Please take a look at: Documentation/dev-tools/kselftest.rst for detailed use-cases for running selftests. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <61093c3f-6ad0-c033-a524-f2624f8d55ba-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-08-02 17:58 ` Shuah Khan @ 2017-08-02 18:04 ` Eric W. Biederman [not found] ` <87r2wtluc1.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Eric W. Biederman @ 2017-08-02 18:04 UTC (permalink / raw) To: Shuah Khan Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: > On 08/02/2017 11:33 AM, Eric W. Biederman wrote: >> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: >> >>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >>>> >>>>> On Wed, 5 Jul 2017 09:48:31 -0700 >>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>> >>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>> [ ... ] >>>>>>>> >>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>>>> that people will stop providing bug fixes. >>>>>>> >>>>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>>>> where someone has a test that easily reproduces the bug, then if >>>>>>> there's not a test added to selftests for said bug, then we should >>>>>>> shame those into doing so. >>>>>>> >>>>>> >>>>>> I don't think that public shaming of kernel developers is going to work >>>>>> any better than public shaming of children or teenagers. >>>>>> >>>>>> Maybe a friendlier approach would be more useful ? >>>>> >>>>> I'm a friendly shamer ;-) >>>>> >>>>>> >>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>>>> to the patch submitter that it would be great if that test would be submitted >>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>>>> praising kselftest submissions might help more than shaming for non-submissions. >>>>>> >>>>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>>>> not applicable, as they don't have tests that can show a regression. >>>>>>> >>>>>> >>>>>> My concern would be that once the shaming starts, it won't stop. >>>>> >>>>> I think this is a communication issue. My word for "shaming" was to >>>>> call out a developer for not submitting a test. It wasn't about making >>>>> fun of them, or anything like that. I was only making a point >>>>> about how to teach people that they need to be more aware of the >>>>> testing infrastructure. Not about actually demeaning people. >>>>> >>>>> Lets take a hypothetical sample. Say someone posted a bug report with >>>>> an associated reproducer for it. The developer then runs the reproducer >>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>>>> developer forgets this and continues on their merry way. Along comes >>>>> someone like myself and sees a reproducing test case for a bug, but >>>>> sees no test added to kselftests. I would send an email along the lines >>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>>>> How come there was no test added to the kselftests to make sure it >>>>> doesn't appear again?" There, I "shamed" them ;-) >>>> >>>> I just want to point out that kselftests are hard to build and run. >>>> >>>> As I was looking at another issue I found a bug in one of the tests. It >>>> had defined a constant wrong. I have a patch. It took me a week of >>>> poking at the kselftest code and trying one thing or another (between >>>> working on other things) before I could figure out which combination of >>>> things would let the test build and run. >>>> >>>> Until kselftests get easier to run I don't think they are something we >>>> want to push to hard. >>>> >>> >>> I would say it is easy to run ksefltests - "make kseflttest" from the >>> main Makefile does this for you. You can also run individual tests: >> >> On 4.13-rc1 That doesn't work. >> >> $ make O=$PWD-build -j8 kselftests >> make[1]: Entering directory 'linux-build' >> make[1]: *** No rule to make target 'kselftests'. Stop. >> make[1]: Leaving directory 'linux-build' >> Makefile:145: recipe for target 'sub-make' failed >> make: *** [sub-make] Error 2 > > It is "make kselftest" If I include the standard O= to keep my source tree pristine it still fails. Which is a practical issue. Especially because that "make kselftest" needs to be followed by I think "make mrproper" to get back to my normal development workflow. I don't remember exactly what the issue was but I could not get: tools/testing/selftests/x86/mpx-mini-test.c tools/testing/selftests/x86/protection_keys.c to build let alone run when I did "make kselftest" >> And why I have to use some esoteric command and not just the >> traditional "make path/to/test/output" to run an individual >> test is beyond me. >> > > make kselftest from top level Makefile is a way to run all the tests. > As I mentioned in my previous email > > "You can also run individual tests: > > "make -C tools/testing/selftests/sync" for example to run sync tests. > > or you can also run: > > make -C tools/testing/selftests/ run_tests As I said complicated. That is definitely not the ordinary way of building things in the kernel tree. Given that some of those other tests take a little while to run, running individual tests is actually quite important during development. Eric ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <87r2wtluc1.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <87r2wtluc1.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> @ 2017-08-02 18:23 ` Randy Dunlap 2017-08-02 18:42 ` Shuah Khan 1 sibling, 0 replies; 71+ messages in thread From: Randy Dunlap @ 2017-08-02 18:23 UTC (permalink / raw) To: Eric W. Biederman, Shuah Khan Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan On 08/02/2017 11:04 AM, Eric W. Biederman wrote: > Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: > >> On 08/02/2017 11:33 AM, Eric W. Biederman wrote: >>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: >>> >>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >>>>> >>>>>> On Wed, 5 Jul 2017 09:48:31 -0700 >>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>> >>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>>> [ ... ] >>>>>>>>> >>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>>>>> that people will stop providing bug fixes. >>>>>>>> >>>>>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>>>>> where someone has a test that easily reproduces the bug, then if >>>>>>>> there's not a test added to selftests for said bug, then we should >>>>>>>> shame those into doing so. >>>>>>>> >>>>>>> >>>>>>> I don't think that public shaming of kernel developers is going to work >>>>>>> any better than public shaming of children or teenagers. >>>>>>> >>>>>>> Maybe a friendlier approach would be more useful ? >>>>>> >>>>>> I'm a friendly shamer ;-) >>>>>> >>>>>>> >>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>>>>> to the patch submitter that it would be great if that test would be submitted >>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>>>>> praising kselftest submissions might help more than shaming for non-submissions. >>>>>>> >>>>>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>>>>> not applicable, as they don't have tests that can show a regression. >>>>>>>> >>>>>>> >>>>>>> My concern would be that once the shaming starts, it won't stop. >>>>>> >>>>>> I think this is a communication issue. My word for "shaming" was to >>>>>> call out a developer for not submitting a test. It wasn't about making >>>>>> fun of them, or anything like that. I was only making a point >>>>>> about how to teach people that they need to be more aware of the >>>>>> testing infrastructure. Not about actually demeaning people. >>>>>> >>>>>> Lets take a hypothetical sample. Say someone posted a bug report with >>>>>> an associated reproducer for it. The developer then runs the reproducer >>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>>>>> developer forgets this and continues on their merry way. Along comes >>>>>> someone like myself and sees a reproducing test case for a bug, but >>>>>> sees no test added to kselftests. I would send an email along the lines >>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>>>>> How come there was no test added to the kselftests to make sure it >>>>>> doesn't appear again?" There, I "shamed" them ;-) >>>>> >>>>> I just want to point out that kselftests are hard to build and run. >>>>> >>>>> As I was looking at another issue I found a bug in one of the tests. It >>>>> had defined a constant wrong. I have a patch. It took me a week of >>>>> poking at the kselftest code and trying one thing or another (between >>>>> working on other things) before I could figure out which combination of >>>>> things would let the test build and run. >>>>> >>>>> Until kselftests get easier to run I don't think they are something we >>>>> want to push to hard. >>>>> >>>> >>>> I would say it is easy to run ksefltests - "make kseflttest" from the >>>> main Makefile does this for you. You can also run individual tests: >>> >>> On 4.13-rc1 That doesn't work. >>> >>> $ make O=$PWD-build -j8 kselftests >>> make[1]: Entering directory 'linux-build' >>> make[1]: *** No rule to make target 'kselftests'. Stop. >>> make[1]: Leaving directory 'linux-build' >>> Makefile:145: recipe for target 'sub-make' failed >>> make: *** [sub-make] Error 2 >> >> It is "make kselftest" > > If I include the standard O= to keep my source tree pristine > it still fails. Which is a practical issue. Especially because > that "make kselftest" needs to be followed by I think "make mrproper" > to get back to my normal development workflow. I mentioned that problem/issue about 6 months ago. I too would like to use O=builddir, but it is ignored. > I don't remember exactly what the issue was but I could not get: > tools/testing/selftests/x86/mpx-mini-test.c > tools/testing/selftests/x86/protection_keys.c > > to build let alone run when I did "make kselftest" > >>> And why I have to use some esoteric command and not just the >>> traditional "make path/to/test/output" to run an individual >>> test is beyond me. >>> >> >> make kselftest from top level Makefile is a way to run all the tests. >> As I mentioned in my previous email >> >> "You can also run individual tests: >> >> "make -C tools/testing/selftests/sync" for example to run sync tests. >> >> or you can also run: >> >> make -C tools/testing/selftests/ run_tests > > As I said complicated. That is definitely not the ordinary way of > building things in the kernel tree. > > Given that some of those other tests take a little while to run, running > individual tests is actually quite important during development. -- ~Randy ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <87r2wtluc1.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 2017-08-02 18:23 ` Randy Dunlap @ 2017-08-02 18:42 ` Shuah Khan [not found] ` <92c31d6f-f98d-c764-eeec-bd0a2316d769-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Shuah Khan @ 2017-08-02 18:42 UTC (permalink / raw) To: Eric W. Biederman Cc: Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Shuah Khan On 08/02/2017 12:04 PM, Eric W. Biederman wrote: > Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: > >> On 08/02/2017 11:33 AM, Eric W. Biederman wrote: >>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes: >>> >>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote: >>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes: >>>>> >>>>>> On Wed, 5 Jul 2017 09:48:31 -0700 >>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>> >>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote: >>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700 >>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote: >>>>>>> [ ... ] >>>>>>>>> >>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is >>>>>>>>> that people will stop providing bug fixes. >>>>>>>> >>>>>>>> I need to be clearer on this. What I meant was, if there's a bug >>>>>>>> where someone has a test that easily reproduces the bug, then if >>>>>>>> there's not a test added to selftests for said bug, then we should >>>>>>>> shame those into doing so. >>>>>>>> >>>>>>> >>>>>>> I don't think that public shaming of kernel developers is going to work >>>>>>> any better than public shaming of children or teenagers. >>>>>>> >>>>>>> Maybe a friendlier approach would be more useful ? >>>>>> >>>>>> I'm a friendly shamer ;-) >>>>>> >>>>>>> >>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest >>>>>>> to the patch submitter that it would be great if that test would be submitted >>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and >>>>>>> praising kselftest submissions might help more than shaming for non-submissions. >>>>>>> >>>>>>>> A bug that is found by inspection or hard to reproduce test cases are >>>>>>>> not applicable, as they don't have tests that can show a regression. >>>>>>>> >>>>>>> >>>>>>> My concern would be that once the shaming starts, it won't stop. >>>>>> >>>>>> I think this is a communication issue. My word for "shaming" was to >>>>>> call out a developer for not submitting a test. It wasn't about making >>>>>> fun of them, or anything like that. I was only making a point >>>>>> about how to teach people that they need to be more aware of the >>>>>> testing infrastructure. Not about actually demeaning people. >>>>>> >>>>>> Lets take a hypothetical sample. Say someone posted a bug report with >>>>>> an associated reproducer for it. The developer then runs the reproducer >>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the >>>>>> developer forgets this and continues on their merry way. Along comes >>>>>> someone like myself and sees a reproducing test case for a bug, but >>>>>> sees no test added to kselftests. I would send an email along the lines >>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed. >>>>>> How come there was no test added to the kselftests to make sure it >>>>>> doesn't appear again?" There, I "shamed" them ;-) >>>>> >>>>> I just want to point out that kselftests are hard to build and run. >>>>> >>>>> As I was looking at another issue I found a bug in one of the tests. It >>>>> had defined a constant wrong. I have a patch. It took me a week of >>>>> poking at the kselftest code and trying one thing or another (between >>>>> working on other things) before I could figure out which combination of >>>>> things would let the test build and run. >>>>> >>>>> Until kselftests get easier to run I don't think they are something we >>>>> want to push to hard. >>>>> >>>> >>>> I would say it is easy to run ksefltests - "make kseflttest" from the >>>> main Makefile does this for you. You can also run individual tests: >>> >>> On 4.13-rc1 That doesn't work. >>> >>> $ make O=$PWD-build -j8 kselftests >>> make[1]: Entering directory 'linux-build' >>> make[1]: *** No rule to make target 'kselftests'. Stop. >>> make[1]: Leaving directory 'linux-build' >>> Makefile:145: recipe for target 'sub-make' failed >>> make: *** [sub-make] Error 2 There are multiple ways to run selftests at the moment. If your preferred use-case isn't supported, I don't see why it can't be added. Would you like to add support for this use-case? >> >> It is "make kselftest" > > If I include the standard O= to keep my source tree pristine > it still fails. Which is a practical issue. Especially because > that "make kselftest" needs to be followed by I think "make mrproper" > to get back to my normal development workflow. Hmm. not sure why you would need to run "make mrproper" after running "make kselftest" - I never have to. I would like to understand why though if you are seeing problems without running "make mrproper". > > I don't remember exactly what the issue was but I could not get: > tools/testing/selftests/x86/mpx-mini-test.c > tools/testing/selftests/x86/protection_keys.c > > to build let alone run when I did "make kselftest" > >>> And why I have to use some esoteric command and not just the >>> traditional "make path/to/test/output" to run an individual >>> test is beyond me. >>> >> >> make kselftest from top level Makefile is a way to run all the tests. >> As I mentioned in my previous email >> >> "You can also run individual tests: >> >> "make -C tools/testing/selftests/sync" for example to run sync tests. >> >> or you can also run: >> >> make -C tools/testing/selftests/ run_tests > > As I said complicated. That is definitely not the ordinary way of > building things in the kernel tree. > > Given that some of those other tests take a little while to run, running > individual tests is actually quite important during development. > > Eric > thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <92c31d6f-f98d-c764-eeec-bd0a2316d769-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <92c31d6f-f98d-c764-eeec-bd0a2316d769-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> @ 2017-08-03 3:03 ` Theodore Ts'o [not found] ` <20170803030327.gf7pl7foer3abdpe-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Theodore Ts'o @ 2017-08-03 3:03 UTC (permalink / raw) To: Shuah Khan Cc: Eric W. Biederman, Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote: > >>> $ make O=$PWD-build -j8 kselftests > >>> make[1]: Entering directory 'linux-build' > >>> make[1]: *** No rule to make target 'kselftests'. Stop. > >>> make[1]: Leaving directory 'linux-build' > >>> Makefile:145: recipe for target 'sub-make' failed > >>> make: *** [sub-make] Error 2 > > There are multiple ways to run selftests at the moment. If your > preferred use-case isn't supported, I don't see why it can't be > added. There are many of us who use separate object directories so we can build the same kernel tree with multiple configurations. So for example, my kernel sources will be on an SSD, but sometimes my /build directory will be on a HDD since storing the object files on an HDD don't really buy as much as keeping the sources on SSD. For example, so I might be doing builds via: make O=/build/ext4 -j8 and make O=/build/ext4-32 ARCH=i386 -j8 Or someone might have a separate config file for doing debugging which is different from their production kernel, which they could do by having a different .config file in /build/ext4-debug/.config. > >> It is "make kselftest" > > > > If I include the standard O= to keep my source tree pristine > > it still fails. Which is a practical issue. Especially because > > that "make kselftest" needs to be followed by I think "make mrproper" > > to get back to my normal development workflow. > > Hmm. not sure why you would need to run "make mrproper" after running > "make kselftest" - I never have to. I would like to understand why though > if you are seeing problems without running "make mrproper". The problem is that if you are doing builds in separate object directories, you must not have any build files in the source tree. Otherwise make gets terribly confused. If you have done a build in the source tree, then "make O=..." will because the Kernel makefile system has safety check which detects this case and complains: % make O=/build/linux-build make[1]: Entering directory '/build/linux-build' CHK include/config/kernel.release GEN ./Makefile CHK include/generated/uapi/linux/version.h CHK scripts/mod/devicetable-offsets.h Using /home/tytso/linux/linux as source for kernel /home/tytso/linux/linux is not clean, please run 'make mrproper' ... So this is why Eric said that he has to run "make mrproper" after doing "make kselftest". That's because doing any kind of build in the source tree breaks his (and my) normal kernel building system. Personally, I do all of my testing using xfstests (and gce-xfstests in particular). And so I don't have a lot of reason to spend time making kselftest work with the "make O=xxx" build paradigm. It's fair to say the fact that kselftest doesn't work with my kernel development workflow hasn't helped my motivation to try it out. But as I told the systemtap folks many years ago --- "you're not under any obligation to support my workflow; but if systemtap is hostile to my kernel development workflow, you also can't expect me to help support and grow the use of systemtap, either. It works both ways." The same I think applies to kselftest. Telling developers who complain that kselftest doesn't work well with their workflow to "send a patch" may not result in the reaction that you hope. Especially for something as similar as "make O=xxx", which I think is a pretty common development workflow. Anyway, I hope someone will care enough about kselftest to help support "make O=xxx"; I have to admit that someone isn't me, though. Sorry; I just don't have the time.... Cheers, - Ted ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170803030327.gf7pl7foer3abdpe-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org>]
* RE: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170803030327.gf7pl7foer3abdpe-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> @ 2017-08-03 17:42 ` Bird, Timothy [not found] ` <ECADFF3FD767C149AD96A924E7EA6EAF3AD01BAB-t8YLG9SB9XQEb75RT/aEbJZPSYnAH24X@public.gmane.org> 2017-08-03 18:51 ` Shuah Khan 1 sibling, 1 reply; 71+ messages in thread From: Bird, Timothy @ 2017-08-03 17:42 UTC (permalink / raw) To: Theodore Ts'o, Shuah Khan Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan > -----Original Message----- > From: Theodore Ts'o on Wednesday, August 02, 2017 8:03 PM > > On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote: > > >>> $ make O=$PWD-build -j8 kselftests > > >>> make[1]: Entering directory 'linux-build' > > >>> make[1]: *** No rule to make target 'kselftests'. Stop. > > >>> make[1]: Leaving directory 'linux-build' > > >>> Makefile:145: recipe for target 'sub-make' failed > > >>> make: *** [sub-make] Error 2 > > > > There are multiple ways to run selftests at the moment. If your > > preferred use-case isn't supported, I don't see why it can't be > > added. > > There are many of us who use separate object directories so we can > build the same kernel tree with multiple configurations. So for > example, my kernel sources will be on an SSD, but sometimes my /build > directory will be on a HDD since storing the object files on an HDD > don't really buy as much as keeping the sources on SSD. > > For example, so I might be doing builds via: > > make O=/build/ext4 -j8 > > and > > make O=/build/ext4-32 ARCH=i386 -j8 > > Or someone might have a separate config file for doing debugging which > is different from their production kernel, which they could do by > having a different .config file in /build/ext4-debug/.config. > > > >> It is "make kselftest" > > > > > > If I include the standard O= to keep my source tree pristine > > > it still fails. Which is a practical issue. Especially because > > > that "make kselftest" needs to be followed by I think "make mrproper" > > > to get back to my normal development workflow. > > > > Hmm. not sure why you would need to run "make mrproper" after running > > "make kselftest" - I never have to. I would like to understand why though > > if you are seeing problems without running "make mrproper". > > The problem is that if you are doing builds in separate object > directories, you must not have any build files in the source tree. > Otherwise make gets terribly confused. If you have done a build in > the source tree, then "make O=..." will because the Kernel makefile > system has safety check which detects this case and complains: > > % make O=/build/linux-build > make[1]: Entering directory '/build/linux-build' > CHK include/config/kernel.release > GEN ./Makefile > CHK include/generated/uapi/linux/version.h > CHK scripts/mod/devicetable-offsets.h > Using /home/tytso/linux/linux as source for kernel > /home/tytso/linux/linux is not clean, please run 'make mrproper' > ... > > So this is why Eric said that he has to run "make mrproper" after > doing "make kselftest". That's because doing any kind of build in the > source tree breaks his (and my) normal kernel building system. > > Personally, I do all of my testing using xfstests (and gce-xfstests in > particular). And so I don't have a lot of reason to spend time making > kselftest work with the "make O=xxx" build paradigm. > > It's fair to say the fact that kselftest doesn't work with my kernel > development workflow hasn't helped my motivation to try it out. But > as I told the systemtap folks many years ago --- "you're not under any > obligation to support my workflow; but if systemtap is hostile to my > kernel development workflow, you also can't expect me to help support > and grow the use of systemtap, either. It works both ways." > > The same I think applies to kselftest. Telling developers who > complain that kselftest doesn't work well with their workflow to "send > a patch" may not result in the reaction that you hope. Especially for > something as similar as "make O=xxx", which I think is a pretty common > development workflow. Anyway, I hope someone will care enough about > kselftest to help support "make O=xxx"; I have to admit that someone > isn't me, though. Sorry; I just don't have the time.... My preferred method of building for embedded boards also uses "make O=xxx" It's been on my to do list for a while to add support for that (KBUILD_OUTPUT) to kselftest, but it's one of those things I never got around to. Now that I know it's blocking other kernel developers, I'll prioritize it higher. Shuah - if no one else steps up to do this, I'll try to see if I can find time to do it in the next month or two (no promises). I think support for KBUILD_OUTPUT (or "make O=xxx") is pretty important. -- Tim ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <ECADFF3FD767C149AD96A924E7EA6EAF3AD01BAB-t8YLG9SB9XQEb75RT/aEbJZPSYnAH24X@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <ECADFF3FD767C149AD96A924E7EA6EAF3AD01BAB-t8YLG9SB9XQEb75RT/aEbJZPSYnAH24X@public.gmane.org> @ 2017-08-03 22:11 ` Shuah Khan 0 siblings, 0 replies; 71+ messages in thread From: Shuah Khan @ 2017-08-03 22:11 UTC (permalink / raw) To: Bird, Timothy, Theodore Ts'o Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan, Shuah Khan On 08/03/2017 11:42 AM, Bird, Timothy wrote: > > >> -----Original Message----- >> From: Theodore Ts'o on Wednesday, August 02, 2017 8:03 PM >> >> On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote: >>>>>> $ make O=$PWD-build -j8 kselftests >>>>>> make[1]: Entering directory 'linux-build' >>>>>> make[1]: *** No rule to make target 'kselftests'. Stop. >>>>>> make[1]: Leaving directory 'linux-build' >>>>>> Makefile:145: recipe for target 'sub-make' failed >>>>>> make: *** [sub-make] Error 2 >>> >>> There are multiple ways to run selftests at the moment. If your >>> preferred use-case isn't supported, I don't see why it can't be >>> added. >> >> There are many of us who use separate object directories so we can >> build the same kernel tree with multiple configurations. So for >> example, my kernel sources will be on an SSD, but sometimes my /build >> directory will be on a HDD since storing the object files on an HDD >> don't really buy as much as keeping the sources on SSD. >> >> For example, so I might be doing builds via: >> >> make O=/build/ext4 -j8 >> >> and >> >> make O=/build/ext4-32 ARCH=i386 -j8 >> >> Or someone might have a separate config file for doing debugging which >> is different from their production kernel, which they could do by >> having a different .config file in /build/ext4-debug/.config. >> >>>>> It is "make kselftest" >>>> >>>> If I include the standard O= to keep my source tree pristine >>>> it still fails. Which is a practical issue. Especially because >>>> that "make kselftest" needs to be followed by I think "make mrproper" >>>> to get back to my normal development workflow. >>> >>> Hmm. not sure why you would need to run "make mrproper" after running >>> "make kselftest" - I never have to. I would like to understand why though >>> if you are seeing problems without running "make mrproper". >> >> The problem is that if you are doing builds in separate object >> directories, you must not have any build files in the source tree. >> Otherwise make gets terribly confused. If you have done a build in >> the source tree, then "make O=..." will because the Kernel makefile >> system has safety check which detects this case and complains: >> >> % make O=/build/linux-build >> make[1]: Entering directory '/build/linux-build' >> CHK include/config/kernel.release >> GEN ./Makefile >> CHK include/generated/uapi/linux/version.h >> CHK scripts/mod/devicetable-offsets.h >> Using /home/tytso/linux/linux as source for kernel >> /home/tytso/linux/linux is not clean, please run 'make mrproper' >> ... >> >> So this is why Eric said that he has to run "make mrproper" after >> doing "make kselftest". That's because doing any kind of build in the >> source tree breaks his (and my) normal kernel building system. >> >> Personally, I do all of my testing using xfstests (and gce-xfstests in >> particular). And so I don't have a lot of reason to spend time making >> kselftest work with the "make O=xxx" build paradigm. >> >> It's fair to say the fact that kselftest doesn't work with my kernel >> development workflow hasn't helped my motivation to try it out. But >> as I told the systemtap folks many years ago --- "you're not under any >> obligation to support my workflow; but if systemtap is hostile to my >> kernel development workflow, you also can't expect me to help support >> and grow the use of systemtap, either. It works both ways." >> >> The same I think applies to kselftest. Telling developers who >> complain that kselftest doesn't work well with their workflow to "send >> a patch" may not result in the reaction that you hope. Especially for >> something as similar as "make O=xxx", which I think is a pretty common >> development workflow. Anyway, I hope someone will care enough about >> kselftest to help support "make O=xxx"; I have to admit that someone >> isn't me, though. Sorry; I just don't have the time.... > > My preferred method of building for embedded boards also uses "make O=xxx" > It's been on my to do list for a while to add support for that (KBUILD_OUTPUT) > to kselftest, but it's one of those things I never got around to. > > Now that I know it's blocking other kernel developers, I'll prioritize it higher. > > Shuah - if no one else steps up to do this, I'll try to see if I can find time to > do it in the next month or two (no promises). I think support for KBUILD_OUTPUT > (or "make O=xxx") is pretty important. > -- Tim > Tim, Thanks for the offer to help. I am going look into this and let you know if I need help. There has been some work done to support KBUILD_OUTPUT. I have to check and see if it can address this use-case. If it does, address it, great. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170803030327.gf7pl7foer3abdpe-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 2017-08-03 17:42 ` Bird, Timothy @ 2017-08-03 18:51 ` Shuah Khan [not found] ` <a313d401-d18b-ed9a-d82d-f6e12f606743-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Shuah Khan @ 2017-08-03 18:51 UTC (permalink / raw) To: Theodore Ts'o Cc: Eric W. Biederman, Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, Shuah Khan On 08/02/2017 09:03 PM, Theodore Ts'o wrote: > On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote: >>>>> $ make O=$PWD-build -j8 kselftests >>>>> make[1]: Entering directory 'linux-build' >>>>> make[1]: *** No rule to make target 'kselftests'. Stop. >>>>> make[1]: Leaving directory 'linux-build' >>>>> Makefile:145: recipe for target 'sub-make' failed >>>>> make: *** [sub-make] Error 2 >> >> There are multiple ways to run selftests at the moment. If your >> preferred use-case isn't supported, I don't see why it can't be >> added. > > There are many of us who use separate object directories so we can > build the same kernel tree with multiple configurations. So for > example, my kernel sources will be on an SSD, but sometimes my /build > directory will be on a HDD since storing the object files on an HDD > don't really buy as much as keeping the sources on SSD. > > For example, so I might be doing builds via: > > make O=/build/ext4 -j8 > > and > > make O=/build/ext4-32 ARCH=i386 -j8 > > Or someone might have a separate config file for doing debugging which > is different from their production kernel, which they could do by > having a different .config file in /build/ext4-debug/.config. > >>>> It is "make kselftest" >>> >>> If I include the standard O= to keep my source tree pristine >>> it still fails. Which is a practical issue. Especially because >>> that "make kselftest" needs to be followed by I think "make mrproper" >>> to get back to my normal development workflow. >> >> Hmm. not sure why you would need to run "make mrproper" after running >> "make kselftest" - I never have to. I would like to understand why though >> if you are seeing problems without running "make mrproper". > > The problem is that if you are doing builds in separate object > directories, you must not have any build files in the source tree. > Otherwise make gets terribly confused. If you have done a build in > the source tree, then "make O=..." will because the Kernel makefile > system has safety check which detects this case and complains: > > % make O=/build/linux-build > make[1]: Entering directory '/build/linux-build' > CHK include/config/kernel.release > GEN ./Makefile > CHK include/generated/uapi/linux/version.h > CHK scripts/mod/devicetable-offsets.h > Using /home/tytso/linux/linux as source for kernel > /home/tytso/linux/linux is not clean, please run 'make mrproper' > ... > > So this is why Eric said that he has to run "make mrproper" after > doing "make kselftest". That's because doing any kind of build in the > source tree breaks his (and my) normal kernel building system. Thanks for the detailed explanation. I have a better understanding of the workflow. > > Personally, I do all of my testing using xfstests (and gce-xfstests in > particular). And so I don't have a lot of reason to spend time making > kselftest work with the "make O=xxx" build paradigm. Now that I have a better understanding, I will take a look. Based on quick look, it might not be difficult to support it. > > It's fair to say the fact that kselftest doesn't work with my kernel > development workflow hasn't helped my motivation to try it out. But > as I told the systemtap folks many years ago --- "you're not under any > obligation to support my workflow; but if systemtap is hostile to my > kernel development workflow, you also can't expect me to help support > and grow the use of systemtap, either. It works both ways." > > The same I think applies to kselftest. Telling developers who > complain that kselftest doesn't work well with their workflow to "send > a patch" may not result in the reaction that you hope. Especially for > something as similar as "make O=xxx", which I think is a pretty common > development workflow. Anyway, I hope someone will care enough about > kselftest to help support "make O=xxx"; I have to admit that someone > isn't me, though. Sorry; I just don't have the time.... > It wasn't my intent to be hostile. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <a313d401-d18b-ed9a-d82d-f6e12f606743-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <a313d401-d18b-ed9a-d82d-f6e12f606743-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> @ 2017-08-04 1:15 ` Theodore Ts'o 0 siblings, 0 replies; 71+ messages in thread From: Theodore Ts'o @ 2017-08-04 1:15 UTC (permalink / raw) To: Shuah Khan Cc: Eric W. Biederman, Steven Rostedt, Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan On Thu, Aug 03, 2017 at 12:51:07PM -0600, Shuah Khan wrote: > > It wasn't my intent to be hostile. Sorry I wasn't clear; I wasn't saying you were being hostile. It's just that I am simultaneously trying to test multiple kernel cofigurations against a common source tree, and the reason why I use separate build directories is the alternative is each time I change configurations, around 2 GB of object files get rewritten. So a feature which requires that I give up a workflow such that I would be I constantly rebuild (and rewrite) gigabytes of object files is hostile to be my development workflow --- and so unless I hear testimonials about how it's amazingly useful, I'm highly unlikely to even experiment with it. Nothing personal.... it just doesn't fit with my development practices. Cheers, - Ted ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:36 ` James Bottomley 2017-07-05 16:48 ` Guenter Roeck @ 2017-07-07 3:33 ` Fengguang Wu [not found] ` <20170707033302.rgpq5knzx3qvvr2p-q6ZYBFIlbFFi0tQiZxhdj1DQ4js95KgL@public.gmane.org> 2 siblings, 1 reply; 71+ messages in thread From: Fengguang Wu @ 2017-07-07 3:33 UTC (permalink / raw) To: Steven Rostedt Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, Shuah Khan, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA On Wed, Jul 05, 2017 at 11:27:07AM -0400, Steven Rostedt wrote: [snip] >I need to be clearer on this. What I meant was, if there's a bug >where someone has a test that easily reproduces the bug, then if >there's not a test added to selftests for said bug, then we should >shame those into doing so. Besides shaming, there's one more option -- acknowledgement. When it's a test case or test tool that discovered the bug, we could acknowledge it by adding one line in the bug fixing patch. The exact forms can be discussed, but here are some examples to show the basic idea: Tool: lockdep Tool: ktest Tool: smatch Tool: trinity Tool: syzkaller Tool: xfstests/tests/ext4/025 Tool: scripts/coccinelle/locks/call_kern.cocci Tool: tools/testing/selftests/bpf/test_align.c Reports from test infrastructures like 0day could go further to help acknowledge the tool author or maintainer by showing such lines in its bug report email: You may consider adding these lines in the bug fixing patch: -----------------------[ cut here ]---------------------------------- Fixes: XXXXXXXXXX ("title of the buggy commit") Tool: tools/testing/selftests/bpf/test_align.c <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> Reported-by: 0day test robot <xiaolong.ye-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> -----------------------[ cut here ]---------------------------------- Regards, Fengguang ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170707033302.rgpq5knzx3qvvr2p-q6ZYBFIlbFFi0tQiZxhdj1DQ4js95KgL@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170707033302.rgpq5knzx3qvvr2p-q6ZYBFIlbFFi0tQiZxhdj1DQ4js95KgL@public.gmane.org> @ 2017-07-07 4:52 ` Frank Rowand 0 siblings, 0 replies; 71+ messages in thread From: Frank Rowand @ 2017-07-07 4:52 UTC (permalink / raw) To: Fengguang Wu, Steven Rostedt Cc: ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, Shuah Khan On 07/06/17 20:33, Fengguang Wu wrote: > On Wed, Jul 05, 2017 at 11:27:07AM -0400, Steven Rostedt wrote: > [snip] >> I need to be clearer on this. What I meant was, if there's a bug >> where someone has a test that easily reproduces the bug, then if >> there's not a test added to selftests for said bug, then we should >> shame those into doing so. > > Besides shaming, there's one more option -- acknowledgement. > > When it's a test case or test tool that discovered the bug, we could > acknowledge it by adding one line in the bug fixing patch. The exact > forms can be discussed, but here are some examples to show the basic > idea: > > Tool: lockdep > Tool: ktest > Tool: smatch > Tool: trinity > Tool: syzkaller > Tool: xfstests/tests/ext4/025 > Tool: scripts/coccinelle/locks/call_kern.cocci > Tool: tools/testing/selftests/bpf/test_align.c > > Reports from test infrastructures like 0day could go further to help > acknowledge the tool author or maintainer by showing such lines in its > bug report email: > > You may consider adding these lines in the bug fixing patch: > > -----------------------[ cut here ]---------------------------------- > Fixes: XXXXXXXXXX ("title of the buggy commit") > Tool: tools/testing/selftests/bpf/test_align.c <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org> > Reported-by: 0day test robot <xiaolong.ye-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> > -----------------------[ cut here ]---------------------------------- > > Regards, > Fengguang > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I@public.gmane.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss > That is a great idea! If a tool is shown to be catching a large number of bugs then I am more likely to add it to my test process. -Frank ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <a462fb3b-a6d4-e969-b301-b404981de224-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-05 15:27 ` Steven Rostedt @ 2017-07-05 15:32 ` Greg KH [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 1 sibling, 1 reply; 71+ messages in thread From: Greg KH @ 2017-07-05 15:32 UTC (permalink / raw) To: Guenter Roeck Cc: Steven Rostedt, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote: > If we start shaming people for not providing unit tests, all we'll accomplish is > that people will stop providing bug fixes. Yes, this is the key! Steven, just look at everything marked with a "Fixes:" or "stable@" tag from 4.12-rc1..4.12 and try to determine how you would write a test for the majority of them. Yes, for some subsystems this can work (look at xfstests as one great example for filesystems, same for the i915 tests), but for the majority of the kernel, at this point in time, it doesn't make sense. So take Carlos's advice, start small, do it for your subsystem if you don't touch hardware (easy peasy, right?), and let's see how it goes, and see if we have the infrastructure to do it even today. Right now, kselftests is finally getting a unified output format, which is great, it shows that people are starting to use and rely on it. What else will we need to make this more widely used, we don't know yet... thanks, greg k-h ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> @ 2017-07-05 15:36 ` Carlos O'Donell 2017-07-05 15:52 ` Steven Rostedt ` (2 subsequent siblings) 3 siblings, 0 replies; 71+ messages in thread From: Carlos O'Donell @ 2017-07-05 15:36 UTC (permalink / raw) To: Greg KH, Guenter Roeck Cc: Steven Rostedt, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On 07/05/2017 11:32 AM, Greg KH wrote: > So take Carlos's advice, start small, do it for your subsystem if you > don't touch hardware (easy peasy, right?), and let's see how it goes, > and see if we have the infrastructure to do it even today. Right now, > kselftests is finally getting a unified output format, which is great, > it shows that people are starting to use and rely on it. What else will > we need to make this more widely used, we don't know yet... +1 ;-) -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 15:36 ` Carlos O'Donell @ 2017-07-05 15:52 ` Steven Rostedt [not found] ` <20170705115219.02370220-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 18:29 ` Daniel Vetter 2017-07-06 22:24 ` Shuah Khan 3 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-05 15:52 UTC (permalink / raw) To: Greg KH Cc: Guenter Roeck, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, 5 Jul 2017 17:32:59 +0200 Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote: > > If we start shaming people for not providing unit tests, all we'll accomplish is > > that people will stop providing bug fixes. > > Yes, this is the key! And I mentioned this in my initial email. > > Steven, just look at everything marked with a "Fixes:" or "stable@" tag > from 4.12-rc1..4.12 and try to determine how you would write a test for > the majority of them. It only makes sense if there's a reproducible case. For cases where stress testing is required and you hope to hit the bug, well, that's never an easy answer, and this is not something that will fix it. > > Yes, for some subsystems this can work (look at xfstests as one great > example for filesystems, same for the i915 tests), but for the majority > of the kernel, at this point in time, it doesn't make sense. I already do. Actually, I have just fixed a bug that I need to add a selftest for. Yes, it is easier for non hardware, but for cases which has specs on hardware behavior, why can't we have tests to test if the hardware matches the spec? Everyone is focusing on that "shaming" comment and not looking at the rest of what I wrote. My main point is, there's a lot of reproducers in change logs or emails that are not in selftests. There's no excuse for that. Lets fix that issue, and not go into a bike shedding fight about the entire approach. > > So take Carlos's advice, start small, do it for your subsystem if you Yes, lets start small. What do you think about all reproducers getting into selftests? If it's not 100% reproducing, then it's up to the individual, but any test that can trigger a bug 100% should be added. I'd like to expand selftests to include configs too. If there's a config that triggers a bug, that should be added to a list of "configs" to be tested as well. > don't touch hardware (easy peasy, right?), and let's see how it goes, > and see if we have the infrastructure to do it even today. Right now, > kselftests is finally getting a unified output format, which is great, > it shows that people are starting to use and rely on it. What else will > we need to make this more widely used, we don't know yet... I've been using selftests for ftrace for some time. I have my own tests that I run (which do test any config that has failed me in the past), and I'm slowing getting those into the selftests directory as well. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705115219.02370220-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705115219.02370220-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-05 18:42 ` Greg KH 0 siblings, 0 replies; 71+ messages in thread From: Greg KH @ 2017-07-05 18:42 UTC (permalink / raw) To: Steven Rostedt Cc: Guenter Roeck, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On Wed, Jul 05, 2017 at 11:52:19AM -0400, Steven Rostedt wrote: > > So take Carlos's advice, start small, do it for your subsystem if you > > Yes, lets start small. What do you think about all reproducers getting > into selftests? If it's not 100% reproducing, then it's up to the > individual, but any test that can trigger a bug 100% should be added. That would be great. One could argue that we should be adding the "stack guard" testing apps to the selftest tree now, as a number of us have them floating around in their test directories at the moment. > I'd like to expand selftests to include configs too. If there's a > config that triggers a bug, that should be added to a list of "configs" > to be tested as well. So a test needs a specific configuration? We need a way to specify that in a generic fashion so that all tests don't have to duplicate that logic. Time to write a helper function to parse /proc/config.gz :) thanks, greg k-h ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 15:36 ` Carlos O'Donell 2017-07-05 15:52 ` Steven Rostedt @ 2017-07-05 18:29 ` Daniel Vetter 2017-07-06 22:24 ` Shuah Khan 3 siblings, 0 replies; 71+ messages in thread From: Daniel Vetter @ 2017-07-05 18:29 UTC (permalink / raw) To: Greg KH Cc: Guenter Roeck, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Carlos O'Donell, open list:ABI/API, Thorsten Leemhuis, Shuah Khan On Wed, Jul 5, 2017 at 5:32 PM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote: >> If we start shaming people for not providing unit tests, all we'll accomplish is >> that people will stop providing bug fixes. > > Yes, this is the key! > > Steven, just look at everything marked with a "Fixes:" or "stable@" tag > from 4.12-rc1..4.12 and try to determine how you would write a test for > the majority of them. > > Yes, for some subsystems this can work (look at xfstests as one great > example for filesystems, same for the i915 tests), but for the majority > of the kernel, at this point in time, it doesn't make sense. > > So take Carlos's advice, start small, do it for your subsystem if you > don't touch hardware (easy peasy, right?), and let's see how it goes, > and see if we have the infrastructure to do it even today. Right now, > kselftests is finally getting a unified output format, which is great, > it shows that people are starting to use and rely on it. What else will > we need to make this more widely used, we don't know yet... This is very hard work and takes a long time. Since 3 years I'm trying to establish the i915 test suite as an overall drm validation set. At least the generic parts like for the cross-driver kernel modeset interfaces, but also allowing other drivers to test their hw specific command submission. It's very slow going ... -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> ` (2 preceding siblings ...) 2017-07-05 18:29 ` Daniel Vetter @ 2017-07-06 22:24 ` Shuah Khan [not found] ` <a2fada39-76d4-e136-f2db-d8306d929902-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 3 siblings, 1 reply; 71+ messages in thread From: Shuah Khan @ 2017-07-06 22:24 UTC (permalink / raw) To: Greg KH, Guenter Roeck Cc: Steven Rostedt, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On 07/05/2017 09:32 AM, Greg KH wrote: > On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote: >> If we start shaming people for not providing unit tests, all we'll accomplish is >> that people will stop providing bug fixes. > > Yes, this is the key! > > Steven, just look at everything marked with a "Fixes:" or "stable@" tag > from 4.12-rc1..4.12 and try to determine how you would write a test for > the majority of them. > > Yes, for some subsystems this can work (look at xfstests as one great > example for filesystems, same for the i915 tests), but for the majority > of the kernel, at this point in time, it doesn't make sense. > > So take Carlos's advice, start small, do it for your subsystem if you > don't touch hardware (easy peasy, right?), and let's see how it goes, > and see if we have the infrastructure to do it even today. Right now, > kselftests is finally getting a unified output format, which is great, > it shows that people are starting to use and rely on it. What else will > we need to make this more widely used, we don't know yet... > Over the past couple of years, kselftests have seen improvements to run on ARM in kernel ci rings. TAP13 will definitely make it easier to find run to run differences. There is the effort to use ksefltests to test stable releases (4.4 LTS for example), which will help make the tests fail/skip gracefully when a feature isn't enabled/supported. The work so far is two fold: - enable them to run in test rings. - making them easy to use As per test development, we are constantly adding tests and I see new tests getting added for sub-systems that aren't hardware dependent. You will see lots of activity in mm, timers, seccomp, net, sys-calls to name a few. I am going to be looking for TAP13 format compliance for new tests starting 4.13. I am not sure how popular they are among developers and sub-system maintainers though. Maybe this is one area we can try to improve usage. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <a2fada39-76d4-e136-f2db-d8306d929902-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <a2fada39-76d4-e136-f2db-d8306d929902-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> @ 2017-07-06 22:32 ` Steven Rostedt [not found] ` <20170706183249.60b2aef9-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Steven Rostedt @ 2017-07-06 22:32 UTC (permalink / raw) To: Shuah Khan Cc: Greg KH, Guenter Roeck, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I On Thu, 6 Jul 2017 16:24:01 -0600 Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> wrote: > Over the past couple of years, kselftests have seen improvements to run > on ARM in kernel ci rings. TAP13 will definitely make it easier to find > run to run differences. There is the effort to use ksefltests to test > stable releases (4.4 LTS for example), which will help make the tests > fail/skip gracefully when a feature isn't enabled/supported. > > The work so far is two fold: > > - enable them to run in test rings. > - making them easy to use > > As per test development, we are constantly adding tests and I see new tests > getting added for sub-systems that aren't hardware dependent. You will see > lots of activity in mm, timers, seccomp, net, sys-calls to name a few. > > I am going to be looking for TAP13 format compliance for new tests starting > 4.13. > > I am not sure how popular they are among developers and sub-system maintainers > though. Maybe this is one area we can try to improve usage. Maybe this should be included in the MAINTAINERS SUMMIT as well. To consolidate the format of all the kselftests and have something that everyone (or most) developers agree on. -- Steve ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170706183249.60b2aef9-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170706183249.60b2aef9-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> @ 2017-07-06 22:40 ` Shuah Khan 0 siblings, 0 replies; 71+ messages in thread From: Shuah Khan @ 2017-07-06 22:40 UTC (permalink / raw) To: Steven Rostedt Cc: Greg KH, Guenter Roeck, Carlos O'Donell, linux-api-u79uwXL29TY76Z2rM5mHXA, Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan On 07/06/2017 04:32 PM, Steven Rostedt wrote: > On Thu, 6 Jul 2017 16:24:01 -0600 > Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> wrote: > > >> Over the past couple of years, kselftests have seen improvements to run >> on ARM in kernel ci rings. TAP13 will definitely make it easier to find >> run to run differences. There is the effort to use ksefltests to test >> stable releases (4.4 LTS for example), which will help make the tests >> fail/skip gracefully when a feature isn't enabled/supported. >> >> The work so far is two fold: >> >> - enable them to run in test rings. >> - making them easy to use >> >> As per test development, we are constantly adding tests and I see new tests >> getting added for sub-systems that aren't hardware dependent. You will see >> lots of activity in mm, timers, seccomp, net, sys-calls to name a few. >> >> I am going to be looking for TAP13 format compliance for new tests starting >> 4.13. >> >> I am not sure how popular they are among developers and sub-system maintainers >> though. Maybe this is one area we can try to improve usage. As a clarification, what I meant by "how popular they are among developers and sub-system maintainers" is that how often developers and sub-system maintainers run kselftests and are there any obstacles for running them. It would be good to get feedback on usage by us as in developers. > > Maybe this should be included in the MAINTAINERS SUMMIT as well. To > consolidate the format of all the kselftests and have something that > everyone (or most) developers agree on. thanks, -- Shuah ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> ` (3 preceding siblings ...) 2017-07-05 15:16 ` Guenter Roeck @ 2017-07-05 16:54 ` Dan Williams [not found] ` <CAPcyv4iOV2-hndx1rQmpPQF+myp=P8rmpf5JhXQXZxPhR6qoQw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 4 siblings, 1 reply; 71+ messages in thread From: Dan Williams @ 2017-07-05 16:54 UTC (permalink / raw) To: Greg KH Cc: Steven Rostedt, Carlos O'Donell, Linux API, Thorsten Leemhuis, ksummit, Shuah Khan On Wed, Jul 5, 2017 at 7:06 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote: >> Your "b" above is what I would like to push. But who's going to enforce >> this? With 10,000 changes per release, and a lot of them are fixes, the >> best we can do is the honor system. Start shaming people that don't >> have a regression test along with a Fixes tag (but we don't want people >> to fix bugs without adding that tag either). There is a fine line one >> must walk between getting people to change their approaches to bugs and >> regression tests, and pissing them off where they start doing the >> opposite of what would be best for the community. > > I would bet, for the huge majority of our fixes, they are fixes for > specific hardware, or workarounds for specific hardware issues. Now > writing tests for those is not an impossible task (look at what the i915 > developers have), but it is very very hard overall, especially if the > base infrastructure isn't there to do it. > > For specific examples, here's the shortlog for fixes that went into > drivers/usb/host/ for 4.12 after 4.12-rc1 came out. Do you know of a > way to write a test for these types of things? > usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk > usb: xhci: Fix USB 3.1 supported protocol parsing > usb: host: xhci-plat: propagate return value of platform_get_irq() > xhci: Fix command ring stop regression in 4.11 > xhci: remove GFP_DMA flag from allocation > USB: xhci: fix lock-inversion problem > usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd > usb: host: xhci-mem: allocate zeroed Scratchpad Buffer > xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton > usb: xhci: trace URB before giving it back instead of after > USB: host: xhci: use max-port define > USB: ehci-platform: fix companion-device leak > usb: r8a66597-hcd: select a different endpoint on timeout > usb: r8a66597-hcd: decrease timeout I wrote some test infrastructure to go after xhci TRB boundary conditions [1]. So, yes, some of these are possible to unit test, but of course not all. [1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2 ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <CAPcyv4iOV2-hndx1rQmpPQF+myp=P8rmpf5JhXQXZxPhR6qoQw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <CAPcyv4iOV2-hndx1rQmpPQF+myp=P8rmpf5JhXQXZxPhR6qoQw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2017-07-05 18:45 ` Greg KH [not found] ` <20170705184544.GB22044-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 71+ messages in thread From: Greg KH @ 2017-07-05 18:45 UTC (permalink / raw) To: Dan Williams Cc: Steven Rostedt, Carlos O'Donell, Linux API, Thorsten Leemhuis, ksummit, Shuah Khan On Wed, Jul 05, 2017 at 09:54:29AM -0700, Dan Williams wrote: > > I wrote some test infrastructure to go after xhci TRB boundary > conditions [1]. So, yes, some of these are possible to unit test, but > of course not all. > > [1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2 I forgot about that, what ever happened to it, any reason it never got merged? thanks, greg k-h ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <20170705184544.GB22044-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org>]
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705184544.GB22044-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> @ 2017-07-05 19:47 ` Dan Williams 0 siblings, 0 replies; 71+ messages in thread From: Dan Williams @ 2017-07-05 19:47 UTC (permalink / raw) To: Greg KH Cc: Steven Rostedt, Carlos O'Donell, Linux API, Thorsten Leemhuis, ksummit, Shuah Khan On Wed, Jul 5, 2017 at 11:45 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote: > On Wed, Jul 05, 2017 at 09:54:29AM -0700, Dan Williams wrote: >> >> I wrote some test infrastructure to go after xhci TRB boundary >> conditions [1]. So, yes, some of these are possible to unit test, but >> of course not all. >> >> [1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2 > > I forgot about that, what ever happened to it, any reason it never got > merged? Ran out of time before being consumed by NVDIMM stuff, but I did take some of the lessons learned over into tools/testing/nvdimm/. I haven't done the work to integrate that into kselftest, so far it's only exercised by the tests in the 'ndctl' [1] utility. [1]: https://github.com/pmem/ndctl ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:06 ` Greg KH @ 2017-07-05 14:06 ` Carlos O'Donell 1 sibling, 0 replies; 71+ messages in thread From: Carlos O'Donell @ 2017-07-05 14:06 UTC (permalink / raw) To: Steven Rostedt Cc: Thorsten Leemhuis, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I, Shuah Khan, linux-api-u79uwXL29TY76Z2rM5mHXA On 07/05/2017 09:27 AM, Steven Rostedt wrote: >> As a concrete action item, glibc core developers took a harder stance on >> (a) all user-visible bugs need a bug # (forces people to think about the > > Unfortunately, we don't have a good system for a "bug #". Most kernel > developers hate bugzilla, and I think that includes Linus ;-) Which > means, unless Linus builds us a new bug tracking system, there wont be > any mandate for it. Use the XMLRPC API to build a better interface for kernel developers? Our "fixed bugs" list is automatically culled via XMLRPC to generate our release announcement with "fixed bugs." The bug # mandate has had a few key effects. It allows non-developers to search for old similar regressions in an easier fashion than having to trawl the mailing list for incomprehensible (to them) discussions about semantics. The bugs are described and talked about in terms of user facing aspects, not internal implementation details. Regressed bugs can be reopened and discussed on the mailing list with links to the discussions and summaries of conclusions. All of this means we have a cleaner, clearer, description of the problem from the user side. This again needs priority from a group of people for whom time is precious, so you have to get buy in from them. I don't think (a) is needed, but the glibc community found it helpful. >> problem and file a coherent public bug about it) (b) all bugs needs a >> regression test if possible, (c) and if not possible we need to extend > > I would love all bug fixes to come with a test (when possible). We have lots of hardware-specific tests that are marked UNSUPPORTED if say you're not running on AVX512 enabled hardware. >> the testing framework to make it possible (we've started using kernel >> namespaces to create isolated test configurations). > > Well, we have a selftest directory that should include all of these. > And most people run them on either a test box or a VM. Improving the test infrastructure must also be a priority, otherwise you will grow to the limit of that infrastructure. >> This change in reviewer priorities has had a noticeable impact on developer >> priorities over the last 5 years. Timelines for this problem will be >> measured in years. > > Your "b" above is what I would like to push. But who's going to enforce > this? With 10,000 changes per release, and a lot of them are fixes, the > best we can do is the honor system. Start shaming people that don't > have a regression test along with a Fixes tag (but we don't want people > to fix bugs without adding that tag either). There is a fine line one > must walk between getting people to change their approaches to bugs and > regression tests, and pissing them off where they start doing the > opposite of what would be best for the community. I did say "hard problem" earlier didn't I? * Start with yourself. * For everyone you know well, and have met in person, be brutal and require them to submit regression tests with their bug fixes. These people are already committed to getting their fixes in and they will understand you are making an example of them. * For everyone you don't know well, be gentle, and begin reminding them you need a regression test, and if you feel generous try to write one yourself for them. Often the act of writing such a test will show you how hard it is, and what is missing from your infrastructure to make this easy, because if it was easy everyone would do it. YMMV. -- Cheers, Carlos. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking [not found] ` <4080ecc7-1aa8-2940-f230-1b79d656cdb4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-07-05 13:27 ` Steven Rostedt @ 2017-07-05 15:47 ` Mark Brown 1 sibling, 0 replies; 71+ messages in thread From: Mark Brown @ 2017-07-05 15:47 UTC (permalink / raw) To: Carlos O'Donell Cc: Steven Rostedt, Thorsten Leemhuis, linux-api-u79uwXL29TY76Z2rM5mHXA, Shuah Khan, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I [-- Attachment #1: Type: text/plain, Size: 1310 bytes --] On Wed, Jul 05, 2017 at 09:09:51AM -0400, Carlos O'Donell wrote: > This problem is a reflection of our own explicit or implicit priorities. > The priorities of developers and reviewers needs to change to make an > impact on the problem. This is a hard problem. Take a look at the trajectory for the build and boot testing for a concrete example of this - the failure rates go down over time but it's not a quick process. > As a concrete action item, glibc core developers took a harder stance on > (a) all user-visible bugs need a bug # (forces people to think about the > problem and file a coherent public bug about it) (b) all bugs needs a > regression test if possible, (c) and if not possible we need to extend > the testing framework to make it possible (we've started using kernel > namespaces to create isolated test configurations). One thing I'd really like to see here is an equivalent of the build and boot testing we currently have that exercises some of the testsuites on a regular basis so we can push on keeping them running cleanly. As well as just the intrisic value of the tests themselves I'd hope that a visible practical interest would help push more activity in this area. There's a couple of efforts I'm aware of in this area, hopefully one or both of them will start delivering. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 71+ messages in thread
end of thread, other threads:[~2017-08-04 1:15 UTC | newest] Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <576cea07-770a-4864-c3f5-0832ff211e94@leemhuis.info> [not found] ` <576cea07-770a-4864-c3f5-0832ff211e94-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> 2017-07-03 16:30 ` [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking Steven Rostedt [not found] ` <20170703123025.7479702e-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-03 18:50 ` Dan Williams 2017-07-04 19:03 ` Thorsten Leemhuis [not found] ` <ad94dc65-cc9c-f4f1-27c1-5a48603c7f59-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> 2017-07-05 12:45 ` Steven Rostedt [not found] ` <20170705084528.67499f8c-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 13:09 ` Carlos O'Donell [not found] ` <4080ecc7-1aa8-2940-f230-1b79d656cdb4-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-07-05 13:27 ` Steven Rostedt [not found] ` <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:06 ` Greg KH [not found] ` <20170705140607.GA30187-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 14:28 ` Carlos O'Donell 2017-07-05 14:33 ` Steven Rostedt [not found] ` <20170705103335.0cbd9984-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:52 ` Mark Brown 2017-07-05 15:08 ` Carlos O'Donell [not found] ` <8c6843e8-73d9-a898-0366-0b72dfeb79a2-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> 2017-07-05 16:10 ` Steven Rostedt [not found] ` <20170705121000.5430d7d0-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-06 11:34 ` Laurent Pinchart 2017-07-09 13:46 ` Thorsten Leemhuis 2017-07-05 14:33 ` Mark Brown [not found] ` <20170705143341.oees22k2snhtmkxo-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-05 14:36 ` Steven Rostedt [not found] ` <20170705103658.226099c6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 14:50 ` James Bottomley [not found] ` <1499266228.3668.10.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 14:56 ` Steven Rostedt [not found] ` <20170705105651.5da9c969-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:09 ` James Bottomley [not found] ` <1499267389.3668.16.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 15:20 ` Mark Brown [not found] ` <20170705152026.rkw73q2f6xmiju37-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-05 15:40 ` Geert Uytterhoeven 2017-07-05 15:20 ` Steven Rostedt [not found] ` <20170705112047.23ee09f6-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:32 ` James Bottomley [not found] ` <1499268748.3668.20.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 15:43 ` Steven Rostedt 2017-07-05 18:24 ` Daniel Vetter 2017-07-05 18:17 ` Daniel Vetter 2017-07-05 15:16 ` Guenter Roeck [not found] ` <a462fb3b-a6d4-e969-b301-b404981de224-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-05 15:27 ` Steven Rostedt [not found] ` <20170705112707.54d7f345-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 15:36 ` James Bottomley [not found] ` <1499269015.3668.25.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 16:04 ` Steven Rostedt [not found] ` <20170705120459.41e81f7b-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 16:58 ` James Bottomley [not found] ` <1499273908.3668.30.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-05 17:07 ` Steven Rostedt 2017-07-05 16:48 ` Guenter Roeck [not found] ` <c782a15a-4e73-7373-ca66-5b55e9406059-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> 2017-07-05 16:58 ` Dan Williams 2017-07-05 17:02 ` Steven Rostedt [not found] ` <20170705130200.7c653f61-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-06 9:28 ` Mark Brown [not found] ` <20170706092836.ifcnc2qqwufndhdl-GFdadSzt00ze9xe1eoZjHA@public.gmane.org> 2017-07-06 9:41 ` Daniel Vetter [not found] ` <CAKMK7uFH+Kz8Mdph=J_FCZ4LC3tzoOmwNJPpSO+snTz6p0Xz+w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-07-06 14:53 ` Theodore Ts'o [not found] ` <20170706145346.6w2uzcf7xacbr3or-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 2017-07-06 21:28 ` Daniel Vetter 2017-07-06 14:48 ` James Bottomley [not found] ` <1499352485.2765.14.camel-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> 2017-07-07 10:03 ` Mark Brown 2017-07-31 16:54 ` Eric W. Biederman [not found] ` <87zibkzgve.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 2017-07-31 20:11 ` Steven Rostedt [not found] ` <20170731161123.4d1e80ac-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-31 20:12 ` Eric W. Biederman 2017-08-02 16:53 ` Shuah Khan [not found] ` <fb399eba-91eb-21a5-e3f5-c3f919061c12-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-08-02 17:33 ` Eric W. Biederman [not found] ` <87lgn1nac8.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 2017-08-02 17:46 ` Shuah Khan [not found] ` <61093c3f-6ad0-c033-a524-f2624f8d55ba-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-08-02 17:58 ` Shuah Khan 2017-08-02 18:04 ` Eric W. Biederman [not found] ` <87r2wtluc1.fsf-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> 2017-08-02 18:23 ` Randy Dunlap 2017-08-02 18:42 ` Shuah Khan [not found] ` <92c31d6f-f98d-c764-eeec-bd0a2316d769-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-08-03 3:03 ` Theodore Ts'o [not found] ` <20170803030327.gf7pl7foer3abdpe-AKGzg7BKzIDYtjvyW6yDsg@public.gmane.org> 2017-08-03 17:42 ` Bird, Timothy [not found] ` <ECADFF3FD767C149AD96A924E7EA6EAF3AD01BAB-t8YLG9SB9XQEb75RT/aEbJZPSYnAH24X@public.gmane.org> 2017-08-03 22:11 ` Shuah Khan 2017-08-03 18:51 ` Shuah Khan [not found] ` <a313d401-d18b-ed9a-d82d-f6e12f606743-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-08-04 1:15 ` Theodore Ts'o 2017-07-07 3:33 ` Fengguang Wu [not found] ` <20170707033302.rgpq5knzx3qvvr2p-q6ZYBFIlbFFi0tQiZxhdj1DQ4js95KgL@public.gmane.org> 2017-07-07 4:52 ` Frank Rowand 2017-07-05 15:32 ` Greg KH [not found] ` <20170705153259.GA7265-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 15:36 ` Carlos O'Donell 2017-07-05 15:52 ` Steven Rostedt [not found] ` <20170705115219.02370220-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-05 18:42 ` Greg KH 2017-07-05 18:29 ` Daniel Vetter 2017-07-06 22:24 ` Shuah Khan [not found] ` <a2fada39-76d4-e136-f2db-d8306d929902-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> 2017-07-06 22:32 ` Steven Rostedt [not found] ` <20170706183249.60b2aef9-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> 2017-07-06 22:40 ` Shuah Khan 2017-07-05 16:54 ` Dan Williams [not found] ` <CAPcyv4iOV2-hndx1rQmpPQF+myp=P8rmpf5JhXQXZxPhR6qoQw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2017-07-05 18:45 ` Greg KH [not found] ` <20170705184544.GB22044-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> 2017-07-05 19:47 ` Dan Williams 2017-07-05 14:06 ` Carlos O'Donell 2017-07-05 15:47 ` Mark Brown
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).