Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Sun, 2 Jul 2017 19:51:43 +0200
Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote:

> Hi! Sorry, I know I'm late -- real life (travel, day job, ...) kept me
> away from spending time on Linux kernel regression work :-/
> 
> Maybe I'm taking it a bit to far for the new kid in town, but I think I
> want to propose two sessions. One for the maintainer summit, that deals
> with a the most critical issues relevant to regression tracking. And one
> technical session to deal with all the other stuff. Obviously we can
> move below mentioned topics from one to the other or talk about them at
> both if we want.
> 
> = [MAINTAINERS SUMMIT] Improve regression tracking =
> 
>  * Follow up from last year: What to do about bugzilla.kernel.org?
> Reporters still get stranded there.
>  * How to get subsystems maintainer involved more in regression tracking
> to better make sure that reported regressions are tracked and not
> forgotten accidentally.

We should push harder for all reproducer tests to be put into
selftests. I try to do that myself (although I admit, I forget to do it
myself here and there. But I'm pushing myself to be better)

>  * Frustrations with regression tracking aka. how to establish
> regression tracking properly to make sure it will never go away again.

By adding reproducing tests to selftests, we can easily see what
regressions are still there.

> 
> = [TECH TOPIC] Improve the kernels quality by getting more people
> involved in regression testing and reporting =

Again, this can be answered by placing more reproducers into selftests.

> 
>  * A short report from the outcome of the maintainer summit discussion;
> also pick up and topics here that where not properly discussed on the
> maintainer summit or were postponed to this session.
>  * How to get distros more involved in regression tracking; especially
> those that have a technical aware user base or normally ship up2date
> kernel images (and thus have an greater interest in avoiding
> regressions). I'm mainly thinking about Arch Linux, Debian, Fedora, and
> openSUSE Tumbleweed here; having Ubuntu in the boat would be good, too!
> (might be wise to talk about this on the maintainers summit as well, if
> the right people are there)
>  * How to make it more easy to (ideally automatically!) track the
> current status and the progress of each regression? Are there any tools
> that could make regression tracking easier for all of us while not
> introducing much overhead for maintainers?

What is selftests?  (Jeopardy answer for all of the above ;-)

> 
> = Details =
> 
> Below you'll find few more words about some points mentioned above;
> there are a few other topics as well we could discuss if we want. But
> first, a few general words on regression tracking from my point of view:
> 
>  * There are a lot of areas in regression tracking where things are far
> from good (read: in a bad state). That makes it easy to discuss current
> problems and their solutions for hours -- and at the same time forget
> that discussing itself doesn't get us much forward (the old bugzilla
> issue mentioned in this mail is a good example). We thus IMHO should
> focus on the most important issues and lay the groundwork to establish
> regression tracking properly again, then we move on to solve things that
> are harder to solve.
> 
>  * Regression tracking currently is quite boring and exhausting (read:
> high burn-out risk), as it involves quite a lot of manual work finding
> regressions and keeping track of their progress (and at the end of the
> day it does not feel like you achieved much). Some of that work can not
> be automated. But quite a bit can and that would help a great deal to
> establish regression tracking properly (currently I'm the only one doing
> it and some development cycles I simply don't find spare time for it).
> 
>    I currently don't see any existing solutions that fit well with our
> mail focused workflow and at the same time do not introduce much
> overhead for subsystem maintainers (which I assume is what everyone
> wants, as I fear solutions with much overhead won't fly at all). Ideas
> how to solve this tricky problem area are highly welcomed. It's
> something that can be discussed when the aforementioned points
> "establish regression tracking properly" and "make it more easy to
> manually or automatically track the current status of a regression" come up.
> 
> == What to do about bugzilla.kernel.org =
> 
> Discussed last year already; see https://lwn.net/Articles/705245/ for
> details. Situation didn't change much since then: the bugzilla instance
> was updated, but people still get stranded there as most subsystems
> ignore it. That afaics frustrates people and makes them stop testing or
> reporting bugs.
> 
> Discuss how to improve things. [my2cent] Maybe a short term solution
> like this could work: Serve a static page on bugzilla.kernel.org that
> tells people where regressions/bugs for certain subsystems can be
> reported, as it most of the time is some mailing list anyway. Such a
> page could get compiled from MAINTAINERS (there is the "B:" field now
> that points to bugzilla; if its not there point to a mailing lists; also
> explain get_maintainers.pl).
> 
>   Leave our bugzilla reachable via bugzilla.kernel.org/frontpage (or
> something like that) for those few subsystems that use it; that's afaics
> ACPI and PM (including Cpufreq, Cpuidle, Hibernation, Suspend, ...) and
> maybe PCI (not sure) -- or should we tell them to move to
> bugzilla.freedesktop.org (or somewhere else) to get rid of our bugzilla
> in the long etrm and make Konstantins life easier? Anyway: Make sure
> bugs for other subsystems can't get filed in bugzilla.kernel.org anymore
> to make sure they get lost there. [/my2cent]
> 
> == How to get subsystems maintainer more involved in regression tracking
> to […] ==
> 
> One reasons why I put this up is: It would help me a lot if people let
> regressions-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org (side note: might be wise to make a
> mailing-list that replaces this address) get told about regressions --
> simply CCing it on reports or answers to regressions reports is enough;
> forwarding/bouncing mails there (even without additional text) is fine,
> too.
> 
> The other reason I included it: This came up in last years discussion on
> this list and it seemed some people thought we can get the subsystems
> maintainers more involved; so I thought it might be wise to discuss it.
> Might also be a good idea to discuss here how to get distro kernel
> maintainer more involved if enough are around.
> 
> == How to establish regression tracking properly […] ==
> 
> This is a pretty vague topic on purpose. People seem to agree that
> regression tracking is important, but for years nobody did it (it
> stopped a little while after Rafael had to move on) and the little bit
> that I can do in my rare spare time won't help much (and I have no idea
> how long I can continue to find time for it).
> 
> == Make it easier to track the progress of regression ==
> 
> One of the main reasons that makes regression tracking hard currently:
> getting aware or regressions and tracking their progress is a lot of
> manual work. I plan one step that hopefully makes the job a little
> easier and at the same time might allow some automation in the long
> term: ask people to include a certain keyword in their regressions
> reports. Maybe something like "Linux-Regression" that doesn't get too
> much false positives when searching for it on lists and via Google
> (suggestions for a better tag welcome).
> 
> In addition, I plan to hand out some form of ID for each regressions I
> track and ask people to include it -- especially when they post patches
> that fix said regression or move the discussion to a new place (like
> "Corrects: Linux-Regression-d2afd"; again: suggestions welcome! Maybe I
> should just use a URL where people find details?).
> 
> That way I can notice more easy when a fix for a regression hits
> linux-next or master; I also get aware if a discussion moves from
> bugzilla to LKML or from one thread to another (fingers crossed).
> Obviously it depends on cooperation of those involved.
> 
> If this works out we could write a script or something that watches
> mailing lists, bug trackers and git trees for the tag in question. That
> script could file a database and automatically do some of the tracking job.
> 
> == get distros more involved ==
> 
> I assume at least Ben (Debian), Laura (Fedora), and Takashi (openSUSE)
> are around, so it might be a good idea to sit together and talk
> regression tracking in general and how we could get the distros kernel
> maintainers more involved. Even better would be to sit down before to
> maybe come up with some ideas/plans we could talk during this session.
> 
> One topic could be: How to make it easier for users of popular distros
> to get involved in testing. The "Kernel of the day" (KOTD) from
> SUSE/openSUSE was mentioned recently on this list already, but I got the
> impression that the existence of this repo is not well known; guess it's
> the same for my own Kernel Vanilla Repositories for Fedora (those
> contain packages with a quite recent mainline version; see
> https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories ) or the fact
> that Fedora rawhide ships a recent mainline snapshot all the time. But
> should distros also offer Linux-next somewhere? Or anything else? And
> should the distros send experienced users upstream when they found a
> regression? Or will subsystem maintainers send those users away because
> they assume those kernels are not vanilla?
> 
> 
> == Topics or vague ideas I left out on purpose ==
> 
> Here is a list of other things we could talk about, but I think better
> left for a later time:
> 
>  * Kerneloops (http://oops.kernel.org/): It was discussed last year on
> this list. I have no idea what the current status is. Is someone
> watching & analysing it? And poking the right people when needed? (I
> doubt it)
> 
>  * Regression tracking for stable kernels (many bugs only get noticed
> once a new mainline version got released; at that time it might still be
> easy to revert a certain patch in mainline and stable)
> 
>  * statistics: I didn't spend time to create statistics, like Rafael did
> in the past. They'd be nice to have, but for now I think my time is
> better spend elsewhere.
> 
>  * work towards growing the number of tester by making it easier for
> them (better documentation, easier configuration, bisection scripts, ...)
> 
>  * maybe document a few some procedures for those that are not regular
> kernel developers (like the "When users report bugs on the Fedora
> tracker that look like actual upstream bugs, what's the best way to have
> those reported?" thing that Laura mentioned earlier this month in the
> mail "Bug reporting feedback loop"
> 
>  * provide better services than only a plain text list of regression on
> a mailing list?
> 
>  * better documentation? for example explain the difference between bugs
> and regressions somewhere to make people understand why their bugs might
> get ignored, but as the same time know that we handle regressions more
> seriously.
> 
>  * Should the regression tracker nag subsystem maintainers (and
> reporters) more often if they are inactive? How do people for example
> feel about (Semi-)Automatic nagging mails for regressions where there is
> no progress?
> 
>  * Is the data and the format of the current reports show useful at all?
> If not: How to improve it?
> 
>  * regression tracking is a fair amount of work, and it's frustrating,
> and people burn out. How to avoid that? Can we maybe get regression
> tracking on solid ground by somehow building a healthy community around
> it (containing kernel developers, Distro maintainers and people that are
> willing to help in their spare time) that work on regressions
> testing/tracking and other QA stuff?
> 
>  * how to make the Linux kernel development so good that the mainstream
> distros stop their kernel forks and do what they do with Firefox: Ship
> the latest stable version (users get a new version with new features
> every few weeks) or a longterm branch (makes a big version jump about
> once a year; see Firefox ESR).

This wont ever happen (famous last words). Distros want "stable
kernels" with new features. That's not what stable is about.

> 
> Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked
> so closely into LWN.net articles about regression tracking and older
> discussions about it.

Anyway, I know that selftests are not the answer for everything, but
anything that has a way to reproduce a bug should be added to it. Sure,
it may depend on various hardware and/or file systems and different
configs, but if we have a central location to place all bug reproducing
tests (which we do have), then we should utilize it.

When it's in the kernel tree, it will be used much more often.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Dan Williams]

On Mon, Jul 3, 2017 at 9:30 AM, Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> wrote:
> On Sun, 2 Jul 2017 19:51:43 +0200
[..]
>>
>> Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked
>> so closely into LWN.net articles about regression tracking and older
>> discussions about it.
>
> Anyway, I know that selftests are not the answer for everything, but
> anything that has a way to reproduce a bug should be added to it. Sure,
> it may depend on various hardware and/or file systems and different
> configs, but if we have a central location to place all bug reproducing
> tests (which we do have), then we should utilize it.

I agree with Steven, and I would add that you don't necessarily need
specific hardware to write a test for a driver regression, see
examples in tools/testing/nvdimm. I also tend to think that
back-stopping regressions with new tests helps with the burn-out
problem of tracking regressions. Where building tools and tests is
potentially more fulfilling than just bug tracking.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Thorsten Leemhuis]

On 03.07.2017 18:30, Steven Rostedt wrote:
> On Sun, 2 Jul 2017 19:51:43 +0200
> Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote:
>>  * How to get subsystems maintainer involved more in regression tracking
>> to better make sure that reported regressions are tracked and not
>> forgotten accidentally.
> We should push harder for all reproducer tests to be put into
> selftests. I try to do that myself [...]
> [...]
> By adding reproducing tests to selftests, we can easily see what
> regressions are still there.
> [...]
> What is selftests?  (Jeopardy answer for all of the above ;-)

Sure, writing and running selftests is a good idea. But as you said
yourself in the later part of your mail: it won't help much in
situations where the kernel (or a selftest) needs to run on a certain
hardware or a specific (and maybe rare or complex) configuration. Sadly
a lot of the regressions in my recent reports were of this kind afaics :-/

In fact I got the impression that most of the regressions that might get
caught by selftests were directly handled by the subsystem maintainer
and never made it to me or my reports -- and thus I can't ask
maintainers to write selftests. *If* I got better aware of those
problems I (a) could make sure they are not forgotten and (b) sooner or
later could publicly state something like "hey, you had ten regressions
recently in your subsystem where writing a selftest might have been a
good idea, but you didn't even write one -- why?" (if we want something
like that).

> […]
>>  * how to make the Linux kernel development so good that the mainstream
>> distros stop their kernel forks and do what they do with Firefox: Ship
>> the latest stable version (users get a new version with new features
>> every few weeks) or a longterm branch (makes a big version jump about
>> once a year; see Firefox ESR).

Hehe, I maybe left the field "regression tracking" to much here and
wandered too far into QA territory.

> This wont ever happen (famous last words). Distros want "stable
> kernels" with new features. 

Ha, yes, it's a long shot (and maybe more a vague idea to work towards
to). And maybe Debian stable and RHEL will always use the model they use
today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and
some others are updating to the latest Linux kernel release every few
weeks already and it works fine for them. Maybe we can get Ubuntu and
others to follow sooner or later.

Sure, for some people a version jump to a major new kernel release will
sound crazy, but when Linus introduced the current development scheme  a
lot of people also said "that will never fly" -- that was 13 years ago
now and it works quite well. The situation was similar with Firefox as well.

> That's not what stable is about.

That afaics (disclaimer: English is not my mother tongue) depends on the
interpretation of the word, as it can mean "nothing changes" or "rock
solid/reliable" (even when two people have a "stable relationship" it
does not mean that nothing changes between them...).

>> Ugh, pretty long mail. Sorry about that. Maybe I shouldn't have looked
>> so closely into LWN.net articles about regression tracking and older
>> discussions about it.
> Anyway, I know that selftests are not the answer for everything, but
> anything that has a way to reproduce a bug should be added to it. Sure,
> it may depend on various hardware and/or file systems and different
> configs, but if we have a central location to place all bug reproducing
> tests (which we do have), then we should utilize it.
> When it's in the kernel tree, it will be used much more often.

+1

Ciao, Thorsten

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Tue, 4 Jul 2017 21:03:22 +0200
Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote:

> On 03.07.2017 18:30, Steven Rostedt wrote:
> > On Sun, 2 Jul 2017 19:51:43 +0200
> > Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org> wrote:  
> >>  * How to get subsystems maintainer involved more in regression tracking
> >> to better make sure that reported regressions are tracked and not
> >> forgotten accidentally.  
> > We should push harder for all reproducer tests to be put into
> > selftests. I try to do that myself [...]
> > [...]
> > By adding reproducing tests to selftests, we can easily see what
> > regressions are still there.
> > [...]
> > What is selftests?  (Jeopardy answer for all of the above ;-)  
> 
> Sure, writing and running selftests is a good idea. But as you said
> yourself in the later part of your mail: it won't help much in
> situations where the kernel (or a selftest) needs to run on a certain
> hardware or a specific (and maybe rare or complex) configuration. Sadly
> a lot of the regressions in my recent reports were of this kind afaics :-/
> 
> In fact I got the impression that most of the regressions that might get
> caught by selftests were directly handled by the subsystem maintainer
> and never made it to me or my reports -- and thus I can't ask
> maintainers to write selftests. *If* I got better aware of those
> problems I (a) could make sure they are not forgotten and (b) sooner or
> later could publicly state something like "hey, you had ten regressions
> recently in your subsystem where writing a selftest might have been a
> good idea, but you didn't even write one -- why?" (if we want something
> like that).

I'm betting there's a lot of reproducer code that never makes it into a
test. How do we solve that? Perhaps we need people looking at LKML for
any signs "I did this, and it caused a bug" or "Here's a test case
which can trigger the bug". Each of these instances should end up in
selftests, and I'm sure they are not.

We can't do much for special hardware, even though those tests should
still be in the selftests for those that have the hardware, but we can
do something about special configs. Perhaps selfttests should have a
"config test" section. I have that in my own tests, but I use ktest to
build them.


> 
> > […]  
> >>  * how to make the Linux kernel development so good that the mainstream
> >> distros stop their kernel forks and do what they do with Firefox: Ship
> >> the latest stable version (users get a new version with new features
> >> every few weeks) or a longterm branch (makes a big version jump about
> >> once a year; see Firefox ESR).  
> 
> Hehe, I maybe left the field "regression tracking" to much here and
> wandered too far into QA territory.
> 
> > This wont ever happen (famous last words). Distros want "stable
> > kernels" with new features.   
> 
> Ha, yes, it's a long shot (and maybe more a vague idea to work towards
> to). And maybe Debian stable and RHEL will always use the model they use
> today. But Fedora, rolling release distros (Tumbleweed, Arch, ...), and
> some others are updating to the latest Linux kernel release every few
> weeks already and it works fine for them. Maybe we can get Ubuntu and
> others to follow sooner or later.
> 
> Sure, for some people a version jump to a major new kernel release will
> sound crazy, but when Linus introduced the current development scheme  a
> lot of people also said "that will never fly" -- that was 13 years ago
> now and it works quite well. The situation was similar with Firefox as well.
> 
> > That's not what stable is about.  
> 
> That afaics (disclaimer: English is not my mother tongue) depends on the
> interpretation of the word, as it can mean "nothing changes" or "rock
> solid/reliable" (even when two people have a "stable relationship" it
> does not mean that nothing changes between them...).

Nothing to do with what language your mother tongue is ;-)

When the stable releases were created, there was some pretty strict
requirements for what should go into stable. Of course the requirements
have changed throughout the years. But there are big differences in
what Red Hat considers something "stable" and what the Linux stable
releases consider to be stable. That is where I meant that things wont
change.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Carlos O'Donell]

On 07/05/2017 08:45 AM, Steven Rostedt wrote:
> I'm betting there's a lot of reproducer code that never makes it into a
> test. How do we solve that? Perhaps we need people looking at LKML for
> any signs "I did this, and it caused a bug" or "Here's a test case
> which can trigger the bug". Each of these instances should end up in
> selftests, and I'm sure they are not.
> 
> We can't do much for special hardware, even though those tests should
> still be in the selftests for those that have the hardware, but we can
> do something about special configs. Perhaps selfttests should have a
> "config test" section. I have that in my own tests, but I use ktest to
> build them.

This problem is a reflection of our own explicit or implicit priorities.
The priorities of developers and reviewers needs to change to make an
impact on the problem. This is a hard problem.

As a concrete action item, glibc core developers took a harder stance on
(a) all user-visible bugs need a bug # (forces people to think about the
problem and file a coherent public bug about it) (b) all bugs needs a
regression test if possible, (c) and if not possible we need to extend
the testing framework to make it possible (we've started using kernel
namespaces to create isolated test configurations).

This change in reviewer priorities has had a noticeable impact on developer
priorities over the last 5 years. Timelines for this problem will be
measured in years.

-- 
Cheers,
Carlos.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 09:09:51 -0400
Carlos O'Donell <carlos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> This problem is a reflection of our own explicit or implicit priorities.
> The priorities of developers and reviewers needs to change to make an
> impact on the problem. This is a hard problem.

I 100% agree.

> 
> As a concrete action item, glibc core developers took a harder stance on
> (a) all user-visible bugs need a bug # (forces people to think about the

Unfortunately, we don't have a good system for a "bug #". Most kernel
developers hate bugzilla, and I think that includes Linus ;-) Which
means, unless Linus builds us a new bug tracking system, there wont be
any mandate for it.


> problem and file a coherent public bug about it) (b) all bugs needs a
> regression test if possible, (c) and if not possible we need to extend

I would love all bug fixes to come with a test (when possible).

> the testing framework to make it possible (we've started using kernel
> namespaces to create isolated test configurations).

Well, we have a selftest directory that should include all of these.
And most people run them on either a test box or a VM.

> 
> This change in reviewer priorities has had a noticeable impact on developer
> priorities over the last 5 years. Timelines for this problem will be
> measured in years.
> 

Your "b" above is what I would like to push. But who's going to enforce
this? With 10,000 changes per release, and a lot of them are fixes, the
best we can do is the honor system. Start shaming people that don't
have a regression test along with a Fixes tag (but we don't want people
to fix bugs without adding that tag either). There is a fine line one
must walk between getting people to change their approaches to bugs and
regression tests, and pissing them off where they start doing the
opposite of what would be best for the community.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Greg KH]

On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote:
> Your "b" above is what I would like to push. But who's going to enforce
> this? With 10,000 changes per release, and a lot of them are fixes, the
> best we can do is the honor system. Start shaming people that don't
> have a regression test along with a Fixes tag (but we don't want people
> to fix bugs without adding that tag either). There is a fine line one
> must walk between getting people to change their approaches to bugs and
> regression tests, and pissing them off where they start doing the
> opposite of what would be best for the community.

I would bet, for the huge majority of our fixes, they are fixes for
specific hardware, or workarounds for specific hardware issues.  Now
writing tests for those is not an impossible task (look at what the i915
developers have), but it is very very hard overall, especially if the
base infrastructure isn't there to do it.

For specific examples, here's the shortlog for fixes that went into
drivers/usb/host/ for 4.12 after 4.12-rc1 came out.  Do you know of a
way to write a test for these types of things?
	usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk
	usb: xhci: Fix USB 3.1 supported protocol parsing
	usb: host: xhci-plat: propagate return value of platform_get_irq()
	xhci: Fix command ring stop regression in 4.11
	xhci: remove GFP_DMA flag from allocation
	USB: xhci: fix lock-inversion problem
	usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd
	usb: host: xhci-mem: allocate zeroed Scratchpad Buffer
	xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton
	usb: xhci: trace URB before giving it back instead of after
	USB: host: xhci: use max-port define
	USB: ehci-platform: fix companion-device leak
	usb: r8a66597-hcd: select a different endpoint on timeout
	usb: r8a66597-hcd: decrease timeout

And look at the commits with the "Fixes:" tag in it, I do, I read every
one of them.  See if writing a test for the majority of them would even
be possible...

I don't mean to poo-poo the idea, but please realize that around 75% of
the kernel is hardware/arch support, so that means that 75% of the
changes/fixes deal with hardware things (yes, change is in direct
correlation to size of the codebase in the tree, strange but true).

If only I had a subsystem that didn't have to deal with hardware, that
must be so easy to work with :)

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Carlos O'Donell]

On 07/05/2017 10:06 AM, Greg KH wrote:
> I don't mean to poo-poo the idea, but please realize that around 75% of
> the kernel is hardware/arch support, so that means that 75% of the
> changes/fixes deal with hardware things (yes, change is in direct
> correlation to size of the codebase in the tree, strange but true).

We should distinguish between the reviewer reviewing the regression test
and running the regression test. As long as the submitter ran the 
regression test on their hardware, and it passed, the reviewer need only
review the test for logical consistency and correctness?

Lack of test infrastructure was a serious problem for us in glibc. We are
relying on namespaces for more complex network and filesystem testing.
Without namespaces we would have needed a much more complex setup that
might never have seen developer adoption. When I attended LPC 2016 I
prioritized listening in on namespaces discussions to make sure nothing
was changing that might break our testing framework.

This conversation is going to lead down the path of driver HAL or
emulation in order to provide regression testing for code above the actual
hardware, and that's another hard problem, but one need not go there.

Starting with real hardware tests can have benefit.

In glibc we test SSE, AVX, AVX512, TSX etc. but if you don't have the
extensions you get a bunch of UNSUPPORTED tests. While upstream kernel 
may have a more limited set of available hardware per-person, the
collective set of developers has hardware to cover all configurations,
and they should run the regression tests for hardware they care about
... and *must* do so if they submit a patch to fix a bug! :-)

-- 
Cheers,
Carlos.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 16:06:07 +0200
Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:

> On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote:
> > Your "b" above is what I would like to push. But who's going to enforce
> > this? With 10,000 changes per release, and a lot of them are fixes, the
> > best we can do is the honor system. Start shaming people that don't
> > have a regression test along with a Fixes tag (but we don't want people
> > to fix bugs without adding that tag either). There is a fine line one
> > must walk between getting people to change their approaches to bugs and
> > regression tests, and pissing them off where they start doing the
> > opposite of what would be best for the community.  
> 
> I would bet, for the huge majority of our fixes, they are fixes for
> specific hardware, or workarounds for specific hardware issues.  Now
> writing tests for those is not an impossible task (look at what the i915
> developers have), but it is very very hard overall, especially if the
> base infrastructure isn't there to do it.
> 
> For specific examples, here's the shortlog for fixes that went into
> drivers/usb/host/ for 4.12 after 4.12-rc1 came out.  Do you know of a
> way to write a test for these types of things?
> 	usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk
> 	usb: xhci: Fix USB 3.1 supported protocol parsing
> 	usb: host: xhci-plat: propagate return value of platform_get_irq()
> 	xhci: Fix command ring stop regression in 4.11
> 	xhci: remove GFP_DMA flag from allocation
> 	USB: xhci: fix lock-inversion problem
> 	usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd
> 	usb: host: xhci-mem: allocate zeroed Scratchpad Buffer
> 	xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton
> 	usb: xhci: trace URB before giving it back instead of after
> 	USB: host: xhci: use max-port define
> 	USB: ehci-platform: fix companion-device leak
> 	usb: r8a66597-hcd: select a different endpoint on timeout
> 	usb: r8a66597-hcd: decrease timeout
> 
> And look at the commits with the "Fixes:" tag in it, I do, I read every
> one of them.  See if writing a test for the majority of them would even
> be possible...
> 
> I don't mean to poo-poo the idea, but please realize that around 75% of
> the kernel is hardware/arch support, so that means that 75% of the
> changes/fixes deal with hardware things (yes, change is in direct
> correlation to size of the codebase in the tree, strange but true).

I would say that if it's for a specific hardware, then it's really up
to the maintainer if there should be a test or not. As a lot of these
is just to deal with some quirk or non standard that the hardware does.
But are these regressions, or just some feature that's been broken on
that hardware since its conception?

That is, Thorsten this is more for you, how much real regressions are in
hardware? A bug that's been there forever is not a regression. It's a
feature ;-)  A regression is something that use to work and now does
not. Is that number still as high with hardware? Those probably could
be where tests can be focused on.

I'm worried more about infrastructure too. I would look at general
functionality of say USB, to see if something can be written to test a
device. Using the one change above that actually mentions "regression"
would it be possible to test completion codes? (I have no idea, I only
read the change log and I'm speaking out of my derrière)

If we have a bunch of generic tests that can test hardware (general
video tests, USB tests, network cards, etc) and people ran these on
their own hardware, and it were to trigger a failure, then it would be
easier for users to report these issues to the maintainers. And these
would be easier to find and fix.

No test should be written for a single specific hardware. It should be a
general functionality that different hardware can execute.

> 
> If only I had a subsystem that didn't have to deal with hardware, that
> must be so easy to work with :)

*smack*!  ;-)

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On Wed, Jul 05, 2017 at 10:33:35AM -0400, Steven Rostedt wrote:

> That is, Thorsten this is more for you, how much real regressions are in
> hardware? A bug that's been there forever is not a regression. It's a
> feature ;-)  A regression is something that use to work and now does
> not. Is that number still as high with hardware? Those probably could
> be where tests can be focused on.

A relatively common case IME is things that were always bugs but depend
on some external thing to become visible, like someone trying to use a
device in a slightly different way, doing more detailed testing of some
kind or some subsystem change.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Carlos O'Donell]

On 07/05/2017 10:33 AM, Steven Rostedt wrote:
> No test should be written for a single specific hardware. It should be a
> general functionality that different hardware can execute.

Why? We test all sorts of hardware in userspace and we see value in that
testing.

-- 
Cheers,
Carlos.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 11:08:48 -0400
Carlos O'Donell <carlos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> On 07/05/2017 10:33 AM, Steven Rostedt wrote:
> > No test should be written for a single specific hardware. It should be a
> > general functionality that different hardware can execute.  
> 
> Why? We test all sorts of hardware in userspace and we see value in that
> testing.
> 

One reason is for bit rot. I'm not totally against it. But I envision
that if we have hundreds of tests for very specific pieces of hardware,
it's value will diminish over time. Unless we can get a good
infrastructure written where the hardware info is more of a data sheet
then a single test itself.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Laurent Pinchart]

On Wednesday 05 Jul 2017 12:10:00 Steven Rostedt wrote:
> On Wed, 5 Jul 2017 11:08:48 -0400 Carlos O'Donell wrote:
> > On 07/05/2017 10:33 AM, Steven Rostedt wrote:
> > > No test should be written for a single specific hardware. It should be a
> > > general functionality that different hardware can execute.
> > 
> > Why? We test all sorts of hardware in userspace and we see value in that
> > testing.
> 
> One reason is for bit rot. I'm not totally against it. But I envision
> that if we have hundreds of tests for very specific pieces of hardware,
> it's value will diminish over time. Unless we can get a good
> infrastructure written where the hardware info is more of a data sheet
> then a single test itself.

That's all nice, but when the hardware is complex and not fully abstracted 
behind a kernel API, tests are bound to be hardware-specific. Of course, a bug 
or regression observed only with a specific device, but triggered through the 
usage of abstract APIs only, can lead to a test case written for that device 
but runnable with any device in the same category. In that case the test case 
should certainly be added to a test suite for the corresponding API/subsystem, 
not to an hidden test suite for a particular device.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Thorsten Leemhuis]

On 05.07.2017 16:33, Steven Rostedt wrote:
> On Wed, 5 Jul 2017 16:06:07 +0200
> Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> [...]
>> I don't mean to poo-poo the idea, but please realize that around 75% of
>> the kernel is hardware/arch support, so that means that 75% of the
>> changes/fixes deal with hardware things (yes, change is in direct
>> correlation to size of the codebase in the tree, strange but true).
> I would say that if it's for a specific hardware, then it's really up
> to the maintainer if there should be a test or not. As a lot of these
> is just to deal with some quirk or non standard that the hardware does.
> But are these regressions, or just some feature that's been broken on
> that hardware since its conception?
> 
> That is, Thorsten this is more for you, how much real regressions are in
> hardware? [...]
>From this and other mails in this thread I got the impression some more
data would be helpful -- for example a few percentage numbers on
 * how many of the regressions are in hardware-specific/driver code
 * how many regressions suddenly pop up due to a unrelated (and maybe
even correct) change
 * for how many regressions does it make sense to write a selftest to
catch similar issues beforehand in the future.

I'll try to gather some of those numbers when doing regression tracking
for 4.13 (sorry again that I had to skip 4.12), so be prepare yourself
for a mail when you include a "Fixes:" tag in a commit ;-) Then there is
some data to talk about on the summit or continue the discussion on this
mailing list or LKML.

BTW, Steven, you in this thread wrote "discuss if we want to consolidate
the format of all the kselftests and have something that everyone (or
most) developers agree on". I put that in my notes and try to make sure
we do not forget about this. Or is this something you'll drive forward
yourself?

Ciao, Thorsten

P.S.: Sorry, I'm a bit late with my reply here. My real job (which is
not really about kernel work) and some other things required my
attention in the past few days...

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:

> I don't mean to poo-poo the idea, but please realize that around 75% of
> the kernel is hardware/arch support, so that means that 75% of the
> changes/fixes deal with hardware things (yes, change is in direct
> correlation to size of the codebase in the tree, strange but true).

Then add in all the fixes for concurrency/locking issues and so on
that're hard to reliably reproduce as well...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 15:33:41 +0100
Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:

> On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:
> 
> > I don't mean to poo-poo the idea, but please realize that around 75% of
> > the kernel is hardware/arch support, so that means that 75% of the
> > changes/fixes deal with hardware things (yes, change is in direct
> > correlation to size of the codebase in the tree, strange but true).  
> 
> Then add in all the fixes for concurrency/locking issues and so on
> that're hard to reliably reproduce as well...

All tests should be run with lockdep enabled ;-)  Which a surprising
few developers appear to do :-p

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote:
> On Wed, 5 Jul 2017 15:33:41 +0100
> Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> > 
> > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:
> > 
> > > 
> > > I don't mean to poo-poo the idea, but please realize that around
> > > 75% of the kernel is hardware/arch support, so that means that
> > > 75% of the changes/fixes deal with hardware things (yes, change
> > > is in direct correlation to size of the codebase in the tree,
> > > strange but true).  
> > 
> > Then add in all the fixes for concurrency/locking issues and so on
> > that're hard to reliably reproduce as well...
> 
> All tests should be run with lockdep enabled ;-)  Which a surprising
> few developers appear to do :-p

Lockdep checks the locking hierarchies and makes assumptions about them
which it then validates ... it doesn't tell you if the data you think
you're protecting was accessed outside the lock, which is the usual
source of concurrency problems.  In other words lockdep is useful but
it's not a panacea.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 05 Jul 2017 07:50:28 -0700
James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:

> On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote:
> > On Wed, 5 Jul 2017 15:33:41 +0100
> > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >   
> > > 
> > > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:
> > >   
> > > > 
> > > > I don't mean to poo-poo the idea, but please realize that around
> > > > 75% of the kernel is hardware/arch support, so that means that
> > > > 75% of the changes/fixes deal with hardware things (yes, change
> > > > is in direct correlation to size of the codebase in the tree,
> > > > strange but true).    
> > > 
> > > Then add in all the fixes for concurrency/locking issues and so on
> > > that're hard to reliably reproduce as well...  
> > 
> > All tests should be run with lockdep enabled ;-)  Which a surprising
> > few developers appear to do :-p  
> 
> Lockdep checks the locking hierarchies and makes assumptions about them
> which it then validates ... it doesn't tell you if the data you think

We should probably look at adding infrastructure that helps in that.
RCU already has a lot of there to help know if data is being protected
by RCU or not.

Hmm, maybe we could add a __rcu like type that we can associate
protected data with, where a config can associate access to a variable
with a lock being held?

> you're protecting was accessed outside the lock, which is the usual
> source of concurrency problems.  In other words lockdep is useful but
> it's not a panacea.

Still not an excuse to not have lockdep enabled during tests.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote:
> On Wed, 05 Jul 2017 07:50:28 -0700
> James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
> 
> > 
> > On Wed, 2017-07-05 at 10:36 -0400, Steven Rostedt wrote:
> > > 
> > > On Wed, 5 Jul 2017 15:33:41 +0100
> > > Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > >   
> > > > 
> > > > 
> > > > On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:
> > > >   
> > > > > 
> > > > > 
> > > > > I don't mean to poo-poo the idea, but please realize that
> > > > > around 75% of the kernel is hardware/arch support, so that
> > > > > means that 75% of the changes/fixes deal with hardware things
> > > > > (yes, change is in direct correlation to size of the codebase
> > > > > in the tree, strange but true).    
> > > > 
> > > > Then add in all the fixes for concurrency/locking issues and so
> > > > on that're hard to reliably reproduce as well...  
> > > 
> > > All tests should be run with lockdep enabled ;-)  Which a
> > > surprising few developers appear to do :-p  
> > 
> > Lockdep checks the locking hierarchies and makes assumptions about
> > them which it then validates ... it doesn't tell you if the data
> > you think
> 
> We should probably look at adding infrastructure that helps in that.
> RCU already has a lot of there to help know if data is being
> protected by RCU or not.
> 
> Hmm, maybe we could add a __rcu like type that we can associate
> protected data with, where a config can associate access to a
> variable with a lock being held?

That's about 10x more complex than the releases/acquires/must_hold
annotation, which we have fairly dismal coverage on.

If you remember the hotplug annotations, which were a shining example:
there's a limit of complexity before any annotation system simply
becomes a make work tyranny. 

> > you're protecting was accessed outside the lock, which is the usual
> > source of concurrency problems.  In other words lockdep is useful
> > but it's not a panacea.
> 
> Still not an excuse to not have lockdep enabled during tests.

OK, what makes you think lockdep isn't enabled?  Since Kconfig is so
complex, I usually use a distro config ... they have it enabled (or at
least openSUSE does), so it's enabled for everything I do.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1003 bytes --]

On Wed, Jul 05, 2017 at 08:09:49AM -0700, James Bottomley wrote:
> On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote:
> > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:

> > > you're protecting was accessed outside the lock, which is the usual
> > > source of concurrency problems.  In other words lockdep is useful
> > > but it's not a panacea.

> > Still not an excuse to not have lockdep enabled during tests.

> OK, what makes you think lockdep isn't enabled?  Since Kconfig is so
> complex, I usually use a distro config ... they have it enabled (or at
> least openSUSE does), so it's enabled for everything I do.

Yeah, I see enough reports with it in embedded contexts to make me think
people use it there.  I know I tend to have it turned on most of the
time.  The concurrency stuff I'm thinking of here is more the things
you're mentioning with just not taking locks at all when they are needed
or concurrency with hardware.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Geert Uytterhoeven]

On Wed, Jul 5, 2017 at 5:20 PM, Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Wed, Jul 05, 2017 at 08:09:49AM -0700, James Bottomley wrote:
>> On Wed, 2017-07-05 at 10:56 -0400, Steven Rostedt wrote:
>> > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
>
>> > > you're protecting was accessed outside the lock, which is the usual
>> > > source of concurrency problems.  In other words lockdep is useful
>> > > but it's not a panacea.
>
>> > Still not an excuse to not have lockdep enabled during tests.
>
>> OK, what makes you think lockdep isn't enabled?  Since Kconfig is so
>> complex, I usually use a distro config ... they have it enabled (or at
>> least openSUSE does), so it's enabled for everything I do.
>
> Yeah, I see enough reports with it in embedded contexts to make me think
> people use it there.  I know I tend to have it turned on most of the
> time.  The concurrency stuff I'm thinking of here is more the things
> you're mentioning with just not taking locks at all when they are needed
> or concurrency with hardware.

I try to have it enabled as much as possible.
However, as it increases kernel size (huge static tables), hitting boot
loader limitations on several boards, I cannot enable all debugging
I would like to on all boards.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 05 Jul 2017 08:09:49 -0700
James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:

 
> > > you're protecting was accessed outside the lock, which is the usual
> > > source of concurrency problems.  In other words lockdep is useful
> > > but it's not a panacea.  
> > 
> > Still not an excuse to not have lockdep enabled during tests.  
> 
> OK, what makes you think lockdep isn't enabled?  Since Kconfig is so
> complex, I usually use a distro config ... they have it enabled (or at
> least openSUSE does), so it's enabled for everything I do.

openSuSE has it enabled? I hope not for its production config, as
lockdep has a huge performance penalty.

I'm thinking you don't have it enabled. What config are you looking at?
The actual config that does the testing of locks is
CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

On Wed, 2017-07-05 at 11:20 -0400, Steven Rostedt wrote:
> On Wed, 05 Jul 2017 08:09:49 -0700
> James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
> 
>  
> > 
> > > 
> > > > 
> > > > you're protecting was accessed outside the lock, which is the
> > > > usual source of concurrency problems.  In other words lockdep
> > > > is useful but it's not a panacea.  
> > > 
> > > Still not an excuse to not have lockdep enabled during tests.  
> > 
> > OK, what makes you think lockdep isn't enabled?  Since Kconfig is
> > so complex, I usually use a distro config ... they have it enabled
> > (or at least openSUSE does), so it's enabled for everything I do.
> 
> openSuSE has it enabled? I hope not for its production config, as
> lockdep has a huge performance penalty.

Then, surely, it's the last thing we want when tracking down race
conditgions since it will alter timings dramatically.

> I'm thinking you don't have it enabled. What config are you looking
> at? The actual config that does the testing of locks is
> CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in.

This is what it has:

jejb@jarvis:~/git/linux-build> grep LOCKDEP /boot/config-4.4.73-18.17-default 
CONFIG_LOCKDEP_SUPPORT=y

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 05 Jul 2017 08:32:28 -0700
James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:


> > openSuSE has it enabled? I hope not for its production config, as
> > lockdep has a huge performance penalty.  
> 
> Then, surely, it's the last thing we want when tracking down race
> conditgions since it will alter timings dramatically.

It's to be run when you want to make sure locking order is at least not
an issue. And it's not about running when tracking down race
conditions, its to be run when developing new code.


> 
> > I'm thinking you don't have it enabled. What config are you looking
> > at? The actual config that does the testing of locks is
> > CONFIG_PROVE_LOCKING, which selects LOCKDEP to be compiled in.  
> 
> This is what it has:
> 
> jejb@jarvis:~/git/linux-build> grep LOCKDEP /boot/config-4.4.73-18.17-default 
> CONFIG_LOCKDEP_SUPPORT=y

That means your architecture supports it, it's not enabled.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Daniel Vetter]

On Wed, Jul 5, 2017 at 5:09 PM, James Bottomley
<James.Bottomley-JuX6DAaQMKPCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
>> > > All tests should be run with lockdep enabled ;-)  Which a
>> > > surprising few developers appear to do :-p
>> >
>> > Lockdep checks the locking hierarchies and makes assumptions about
>> > them which it then validates ... it doesn't tell you if the data
>> > you think
>>
>> We should probably look at adding infrastructure that helps in that.
>> RCU already has a lot of there to help know if data is being
>> protected by RCU or not.
>>
>> Hmm, maybe we could add a __rcu like type that we can associate
>> protected data with, where a config can associate access to a
>> variable with a lock being held?
>
> That's about 10x more complex than the releases/acquires/must_hold
> annotation, which we have fairly dismal coverage on.

Yeah, I've never found those useful at all. What we're trying to do in
drm code is liberally sprinkle lockdep_assert_held into accessor and
helper functions (there's lots of nontrivial stuff where you need a
little bit of computation around a pure access, so doesn't result in
ugly code). That catches a lot of these, but of course not all.

The problem with static annotations is that often the lock you need to
hold isn't statically known, and annotating the entire callchain is a
no-go as James points out. But maybe we could use such annotations
plus a gcc plugin to auto-insert the right lockdep_assert_held every
time you read/write into a given field?

That's not going to cover locking rules where the locking rules change
during the lifetime of an object, but I think even without that it
would cover a _lot_ of cases. And if your static annotation would be
allowed to chase pointers (well, just any C expression that takes the
struct pointer as parameter would be sweet) you could even annotate
fields where the protecting lock is in some parent struct.

Another thing I'm really looking forward to (but it's somehow not
moving fast) is the cross-release stuff. Too many times I've screamed
at kernel backtraces stuck in wait_event, and lockdep could have
directly told me what's wrong long before a stress test successfully
hit that race.

There's definitely a lot of room to prove more stuff in locking using tools.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Daniel Vetter]

On Wed, Jul 5, 2017 at 4:36 PM, Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> wrote:
> On Wed, 5 Jul 2017 15:33:41 +0100
> Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Wed, Jul 05, 2017 at 04:06:07PM +0200, Greg KH wrote:
>> > I don't mean to poo-poo the idea, but please realize that around 75% of
>> > the kernel is hardware/arch support, so that means that 75% of the
>> > changes/fixes deal with hardware things (yes, change is in direct
>> > correlation to size of the codebase in the tree, strange but true).
>>
>> Then add in all the fixes for concurrency/locking issues and so on
>> that're hard to reliably reproduce as well...
>
> All tests should be run with lockdep enabled ;-)  Which a surprising
> few developers appear to do :-p

We're slowly working towards running the i915 testsuite with kasan
enabled as the next level of evil. It's ... interesting, to say the
least.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Guenter Roeck]

On 07/05/2017 07:06 AM, Greg KH wrote:
> On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote:
>> Your "b" above is what I would like to push. But who's going to enforce
>> this? With 10,000 changes per release, and a lot of them are fixes, the
>> best we can do is the honor system. Start shaming people that don't
>> have a regression test along with a Fixes tag (but we don't want people
>> to fix bugs without adding that tag either). There is a fine line one
>> must walk between getting people to change their approaches to bugs and
>> regression tests, and pissing them off where they start doing the
>> opposite of what would be best for the community.
> 
> I would bet, for the huge majority of our fixes, they are fixes for
> specific hardware, or workarounds for specific hardware issues.  Now
> writing tests for those is not an impossible task (look at what the i915
> developers have), but it is very very hard overall, especially if the
> base infrastructure isn't there to do it.
> 
> For specific examples, here's the shortlog for fixes that went into
> drivers/usb/host/ for 4.12 after 4.12-rc1 came out.  Do you know of a
> way to write a test for these types of things?
> 	usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk
> 	usb: xhci: Fix USB 3.1 supported protocol parsing
> 	usb: host: xhci-plat: propagate return value of platform_get_irq()
> 	xhci: Fix command ring stop regression in 4.11
> 	xhci: remove GFP_DMA flag from allocation
> 	USB: xhci: fix lock-inversion problem
> 	usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd
> 	usb: host: xhci-mem: allocate zeroed Scratchpad Buffer
> 	xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton
> 	usb: xhci: trace URB before giving it back instead of after
> 	USB: host: xhci: use max-port define
> 	USB: ehci-platform: fix companion-device leak
> 	usb: r8a66597-hcd: select a different endpoint on timeout
> 	usb: r8a66597-hcd: decrease timeout
> 
> And look at the commits with the "Fixes:" tag in it, I do, I read every
> one of them.  See if writing a test for the majority of them would even
> be possible...
> 
> I don't mean to poo-poo the idea, but please realize that around 75% of
> the kernel is hardware/arch support, so that means that 75% of the
> changes/fixes deal with hardware things (yes, change is in direct
> correlation to size of the codebase in the tree, strange but true).
> 

The reproducers for several of the usb fixes I submitted recently took hours of
stress test to reproduce the underlying problems. I have one more to fix which
takes days to reproduce, if at all (I have seen that problem only two or three
times during weeks of stress test). Due to the nature of the problems, reproducing
them heavily depended on the underlying hardware. None of the reproducers can
guarantee that the problem is fixed; they are intended to show the problem,
not that it is fixed. This happens a lot with race conditions - in many cases
it is impossible to prove that the problem is fixed; one can only prove that
it still exists.

Echoing what you said, I have no idea how it would even be possible to write
unit tests to verify if the problems I fixed are really fixed.

Several of the fixes I have submitted are based on single-instance error logs with
no reproducer. Many others are compile time fixes or fix problems found with code
inspection (manual or automatic).

If we start shaming people for not providing unit tests, all we'll accomplish is
that people will stop providing bug fixes.

Guenter

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 08:16:33 -0700
Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:

> The reproducers for several of the usb fixes I submitted recently took hours of
> stress test to reproduce the underlying problems. I have one more to fix which
> takes days to reproduce, if at all (I have seen that problem only two or three
> times during weeks of stress test). Due to the nature of the problems, reproducing
> them heavily depended on the underlying hardware. None of the reproducers can
> guarantee that the problem is fixed; they are intended to show the problem,
> not that it is fixed. This happens a lot with race conditions - in many cases
> it is impossible to prove that the problem is fixed; one can only prove that
> it still exists.
> 
> Echoing what you said, I have no idea how it would even be possible to write
> unit tests to verify if the problems I fixed are really fixed.
> 
> Several of the fixes I have submitted are based on single-instance error logs with
> no reproducer. Many others are compile time fixes or fix problems found with code
> inspection (manual or automatic).
> 
> If we start shaming people for not providing unit tests, all we'll accomplish is
> that people will stop providing bug fixes.

I need to be clearer on this. What I meant was, if there's a bug
where someone has a test that easily reproduces the bug, then if
there's not a test added to selftests for said bug, then we should
shame those into doing so.

A bug that is found by inspection or hard to reproduce test cases are
not applicable, as they don't have tests that can show a regression.

And I'm betting that those bugs are NOT REGRESSIONS! Most likely are
bugs that always existed, but because of the unpredictable hitting of
the bug (as you said, it required hours of stress tests to reproduce),
the bug was not previously hit during development. That's not a
regression, that's a feature.

Are we tracking regressions or just simply bugs?

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

On Wed, 2017-07-05 at 11:27 -0400, Steven Rostedt wrote:
> On Wed, 5 Jul 2017 08:16:33 -0700
> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
> 
> > 
> > The reproducers for several of the usb fixes I submitted recently
> > took hours of stress test to reproduce the underlying problems. I
> > have one more to fix which takes days to reproduce, if at all (I
> > have seen that problem only two or three times during weeks of
> > stress test). Due to the nature of the problems, reproducing
> > them heavily depended on the underlying hardware. None of the
> > reproducers can guarantee that the problem is fixed; they are
> > intended to show the problem, not that it is fixed. This happens a
> > lot with race conditions - in many cases it is impossible to prove
> > that the problem is fixed; one can only prove that it still exists.
> > 
> > Echoing what you said, I have no idea how it would even be possible
> > to write unit tests to verify if the problems I fixed are really
> > fixed.
> > 
> > Several of the fixes I have submitted are based on single-instance
> > error logs with no reproducer. Many others are compile time fixes
> > or fix problems found with code inspection (manual or automatic).
> > 
> > If we start shaming people for not providing unit tests, all we'll
> > accomplish is that people will stop providing bug fixes.
> 
> I need to be clearer on this. What I meant was, if there's a bug
> where someone has a test that easily reproduces the bug, then if
> there's not a test added to selftests for said bug, then we should
> shame those into doing so.
> 
> A bug that is found by inspection or hard to reproduce test cases are
> not applicable, as they don't have tests that can show a regression.
> 
> And I'm betting that those bugs are NOT REGRESSIONS! Most likely are
> bugs that always existed, but because of the unpredictable hitting of
> the bug (as you said, it required hours of stress tests to
> reproduce), the bug was not previously hit during development. That's
> not a regression, that's a feature.
> 
> Are we tracking regressions or just simply bugs?

A lot of device driver regressions are bugs that previously existed in
the code but which didn't manifest until something else happened.  A
huge number of locking and timing issues are like this.  The irony is
that a lot of them go from race always being won (so bug never noticed)
to race being lost often enough to make something unusable.  To a user
that ends up being a kernel regression because it's a bug in the
current kernel which they didn't see previously which makes it unusable
for them.

I've got to vote with my users here: that's a regression not a
"feature".

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 05 Jul 2017 08:36:55 -0700
James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:

> > Are we tracking regressions or just simply bugs?  
> 
> A lot of device driver regressions are bugs that previously existed in
> the code but which didn't manifest until something else happened.  A
> huge number of locking and timing issues are like this.  The irony is
> that a lot of them go from race always being won (so bug never noticed)
> to race being lost often enough to make something unusable.  To a user
> that ends up being a kernel regression because it's a bug in the
> current kernel which they didn't see previously which makes it unusable
> for them.
> 
> I've got to vote with my users here: that's a regression not a
> "feature".

Let's take a step back. What exactly is the problem?

The regressions that we want to track? Why are they not fixed? Is it
because they are hard to reproduce? If so, how do we know they are a
regression or just some hard to hit bug? If it's hard to hit, how do we
know we fixed it?

What exactly are the questions we want solved.

Granted, I used this thread to push more use of kselftests, and I don't
see any SCSI tests there at all!

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

On Wed, 2017-07-05 at 12:04 -0400, Steven Rostedt wrote:
> On Wed, 05 Jul 2017 08:36:55 -0700
> James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
> 
> > 
> > > 
> > > Are we tracking regressions or just simply bugs?  
> > 
> > A lot of device driver regressions are bugs that previously existed
> > in the code but which didn't manifest until something else
> > happened.  A huge number of locking and timing issues are like
> > this.  The irony is that a lot of them go from race always being
> > won (so bug never noticed) to race being lost often enough to make
> > something unusable.  To a user that ends up being a kernel
> > regression because it's a bug in the current kernel which they
> > didn't see previously which makes it unusable for them.
> > 
> > I've got to vote with my users here: that's a regression not a
> > "feature".
> 
> Let's take a step back. What exactly is the problem?

You mean what question was I answering?  It was your "is your problem a
regression?" one.

> The regressions that we want to track? Why are they not fixed? Is it
> because they are hard to reproduce? If so, how do we know they are a
> regression or just some hard to hit bug? If it's hard to hit, how do
> we know we fixed it?

Usually for the exposed races we develop a theoretical model which
tells us what the problem is and also the solution.

> What exactly are the questions we want solved.

In the context of this subthread?  Tracking and fixing of regressions
meaning behaviour that damages or destroys usability of version k+1
that wasn't present in version k.

> Granted, I used this thread to push more use of kselftests, and I
> don't see any SCSI tests there at all!

It would be an interesting question for another thread to consider
whether that's a problem or not.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 05 Jul 2017 09:58:28 -0700
James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:

> On Wed, 2017-07-05 at 12:04 -0400, Steven Rostedt wrote:
> > On Wed, 05 Jul 2017 08:36:55 -0700
> > James Bottomley <James.Bottomley-d9PhHud1JfjCXq6kfMZ53/egYHeGw8Jk@public.gmane.org> wrote:
> >   
> > >   
> > > > 
> > > > Are we tracking regressions or just simply bugs?    
> > > 
> > > A lot of device driver regressions are bugs that previously existed
> > > in the code but which didn't manifest until something else
> > > happened.  A huge number of locking and timing issues are like
> > > this.  The irony is that a lot of them go from race always being
> > > won (so bug never noticed) to race being lost often enough to make
> > > something unusable.  To a user that ends up being a kernel
> > > regression because it's a bug in the current kernel which they
> > > didn't see previously which makes it unusable for them.
> > > 
> > > I've got to vote with my users here: that's a regression not a
> > > "feature".  
> > 
> > Let's take a step back. What exactly is the problem?  
> 
> You mean what question was I answering?  It was your "is your problem a
> regression?" one.

No that's not what I meant. I mean that we are going off tangent to the
original topic.

> 
> > The regressions that we want to track? Why are they not fixed? Is it
> > because they are hard to reproduce? If so, how do we know they are a
> > regression or just some hard to hit bug? If it's hard to hit, how do
> > we know we fixed it?  
> 
> Usually for the exposed races we develop a theoretical model which
> tells us what the problem is and also the solution.

I think the problem is that the regressions that are not being fixed
happen to be where we have no model to create, as the problem may be
too hard to hit, and it could just be a "works for me" issue.

> 
> > What exactly are the questions we want solved.  
> 
> In the context of this subthread?  Tracking and fixing of regressions
> meaning behaviour that damages or destroys usability of version k+1
> that wasn't present in version k.

Agreed with this part. And I believe this is also in the context of the
entire thread.

> 
> > Granted, I used this thread to push more use of kselftests, and I
> > don't see any SCSI tests there at all!  
> 
> It would be an interesting question for another thread to consider
> whether that's a problem or not.

It's not a problem for me, but it begs the question of whether it would
be useful or not. But I agree, that's for another thread.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Guenter Roeck]

On 07/05/2017 08:27 AM, Steven Rostedt wrote:
> On Wed, 5 Jul 2017 08:16:33 -0700
> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
[ ... ]
>>
>> If we start shaming people for not providing unit tests, all we'll accomplish is
>> that people will stop providing bug fixes.
> 
> I need to be clearer on this. What I meant was, if there's a bug
> where someone has a test that easily reproduces the bug, then if
> there's not a test added to selftests for said bug, then we should
> shame those into doing so.
> 

I don't think that public shaming of kernel developers is going to work
any better than public shaming of children or teenagers.

Maybe a friendlier approach would be more useful ?

If a test to reproduce a problem exists, it might be more beneficial to suggest
to the patch submitter that it would be great if that test would be submitted
as unit test instead of shaming that person for not doing so. Acknowledging and
praising kselftest submissions might help more than shaming for non-submissions.

> A bug that is found by inspection or hard to reproduce test cases are
> not applicable, as they don't have tests that can show a regression.
> 

My concern would be that once the shaming starts, it won't stop.

Guenter

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Dan Williams]

On Wed, Jul 5, 2017 at 9:48 AM, Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>
>> On Wed, 5 Jul 2017 08:16:33 -0700
>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>
> [ ... ]
>>>
>>>
>>> If we start shaming people for not providing unit tests, all we'll
>>> accomplish is
>>> that people will stop providing bug fixes.
>>
>>
>> I need to be clearer on this. What I meant was, if there's a bug
>> where someone has a test that easily reproduces the bug, then if
>> there's not a test added to selftests for said bug, then we should
>> shame those into doing so.
>>
>
> I don't think that public shaming of kernel developers is going to work
> any better than public shaming of children or teenagers.
>
> Maybe a friendlier approach would be more useful ?
>
> If a test to reproduce a problem exists, it might be more beneficial to
> suggest
> to the patch submitter that it would be great if that test would be
> submitted
> as unit test instead of shaming that person for not doing so. Acknowledging
> and
> praising kselftest submissions might help more than shaming for
> non-submissions.
>
>> A bug that is found by inspection or hard to reproduce test cases are
>> not applicable, as they don't have tests that can show a regression.
>>
>
> My concern would be that once the shaming starts, it won't stop.

Agreed, this shouldn't be a new burden for maintainers, this should be
a contribution path for new kernel developers. Go beyond our standard
"fix a bug" advice, which is a great advice, and also recommend
"backstop a regression with a unit test".

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 09:48:31 -0700
Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:

> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
> > On Wed, 5 Jul 2017 08:16:33 -0700
> > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
> [ ... ]
> >>
> >> If we start shaming people for not providing unit tests, all we'll accomplish is
> >> that people will stop providing bug fixes.  
> > 
> > I need to be clearer on this. What I meant was, if there's a bug
> > where someone has a test that easily reproduces the bug, then if
> > there's not a test added to selftests for said bug, then we should
> > shame those into doing so.
> >   
> 
> I don't think that public shaming of kernel developers is going to work
> any better than public shaming of children or teenagers.
> 
> Maybe a friendlier approach would be more useful ?

I'm a friendly shamer ;-)

> 
> If a test to reproduce a problem exists, it might be more beneficial to suggest
> to the patch submitter that it would be great if that test would be submitted
> as unit test instead of shaming that person for not doing so. Acknowledging and
> praising kselftest submissions might help more than shaming for non-submissions.
> 
> > A bug that is found by inspection or hard to reproduce test cases are
> > not applicable, as they don't have tests that can show a regression.
> >   
> 
> My concern would be that once the shaming starts, it won't stop.

I think this is a communication issue. My word for "shaming" was to
call out a developer for not submitting a test. It wasn't about making
fun of them, or anything like that. I was only making a point
about how to teach people that they need to be more aware of the
testing infrastructure. Not about actually demeaning people.

Lets take a hypothetical sample. Say someone posted a bug report with
an associated reproducer for it. The developer then runs the reproducer
sees the bug, makes a fix and sends it to Linus and stable. Now the
developer forgets this and continues on their merry way. Along comes
someone like myself and sees a reproducing test case for a bug, but
sees no test added to kselftests. I would send an email along the lines
of "Hi, I noticed that there was a reproducer for this bug you fixed.
How come there was no test added to the kselftests to make sure it
doesn't appear again?" There, I "shamed" them ;-)

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote:
> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:

> > If a test to reproduce a problem exists, it might be more beneficial to suggest
> > to the patch submitter that it would be great if that test would be submitted
> > as unit test instead of shaming that person for not doing so. Acknowledging and
> > praising kselftest submissions might help more than shaming for non-submissions.

> > My concern would be that once the shaming starts, it won't stop.

> I think this is a communication issue. My word for "shaming" was to
> call out a developer for not submitting a test. It wasn't about making
> fun of them, or anything like that. I was only making a point
> about how to teach people that they need to be more aware of the
> testing infrastructure. Not about actually demeaning people.

I think before anything like that is viable we need to show a concerted
and visible interest in actually running the tests we already have and
paying attention to the results - if people can see that they're just
checking a checkbox that will often result in low quality tests which
can do more harm than good.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Daniel Vetter]

On Thu, Jul 6, 2017 at 11:28 AM, Mark Brown <broonie-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote:
>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>
>> > If a test to reproduce a problem exists, it might be more beneficial to suggest
>> > to the patch submitter that it would be great if that test would be submitted
>> > as unit test instead of shaming that person for not doing so. Acknowledging and
>> > praising kselftest submissions might help more than shaming for non-submissions.
>
>> > My concern would be that once the shaming starts, it won't stop.
>
>> I think this is a communication issue. My word for "shaming" was to
>> call out a developer for not submitting a test. It wasn't about making
>> fun of them, or anything like that. I was only making a point
>> about how to teach people that they need to be more aware of the
>> testing infrastructure. Not about actually demeaning people.
>
> I think before anything like that is viable we need to show a concerted
> and visible interest in actually running the tests we already have and
> paying attention to the results - if people can see that they're just
> checking a checkbox that will often result in low quality tests which
> can do more harm than good.

+1. That pretty much means large-scale CI. The i915 test suite has
suffered quite a bit over the past years because the CI infrastructure
didn't keep up. Result is that running full CI kills pretty much every
platform there is eventually, and it's really hard to get back to a
state where the testsuite can be used to catch regressions again.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Theodore Ts'o]

On Thu, Jul 06, 2017 at 11:41:39AM +0200, Daniel Vetter wrote:
> 
> +1. That pretty much means large-scale CI. The i915 test suite has
> suffered quite a bit over the past years because the CI infrastructure
> didn't keep up. Result is that running full CI kills pretty much every
> platform there is eventually, and it's really hard to get back to a
> state where the testsuite can be used to catch regressions again.

I assume the i915 test suite requires real hardware and can't be run
on VM's; is that correct?

						- Ted

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Daniel Vetter]

On Thu, Jul 6, 2017 at 4:53 PM, Theodore Ts'o <tytso-3s7WtUTddSA@public.gmane.org> wrote:
> On Thu, Jul 06, 2017 at 11:41:39AM +0200, Daniel Vetter wrote:
>> +1. That pretty much means large-scale CI. The i915 test suite has
>> suffered quite a bit over the past years because the CI infrastructure
>> didn't keep up. Result is that running full CI kills pretty much every
>> platform there is eventually, and it's really hard to get back to a
>> state where the testsuite can be used to catch regressions again.
>
> I assume the i915 test suite requires real hardware and can't be run
> on VM's; is that correct?

Yes, that's another problem. If all bigger teams/subsystems would do
what we'd do, but extended to all mailing lists, your patch series
would get replies from a few hundred CI farms. Not sure that would
scale ... And there's no way ever that one single entity will have
hardware for everything.

And if you only CI the mailing list for your own subsystem then every
time a merge window happens your CI will be out of service (4.13 seems
extremely bad, atm nothing survives an extended run on linux-next for
us).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[James Bottomley]

[-- Attachment #1: Type: text/plain, Size: 2140 bytes --]

On Thu, 2017-07-06 at 10:28 +0100, Mark Brown wrote:
> On Wed, Jul 05, 2017 at 01:02:00PM -0400, Steven Rostedt wrote:
> > 
> > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
> 
> > 
> > > 
> > > If a test to reproduce a problem exists, it might be more
> > > beneficial to suggest to the patch submitter that it would be
> > > great if that test would be submitted as unit test instead of
> > > shaming that person for not doing so. Acknowledging and
> > > praising kselftest submissions might help more than shaming for
> > > non-submissions.
> 
> > 
> > > 
> > > My concern would be that once the shaming starts, it won't stop.
> 
> > 
> > I think this is a communication issue. My word for "shaming" was to
> > call out a developer for not submitting a test. It wasn't about
> > making fun of them, or anything like that. I was only making a
> > point about how to teach people that they need to be more aware of
> > the testing infrastructure. Not about actually demeaning people.
> 
> I think before anything like that is viable we need to show a
> concerted and visible interest in actually running the tests we
> already have and paying attention to the results - if people can see
> that they're just checking a checkbox that will often result in low
> quality tests which can do more harm than good.

it depends what you mean by "we".  I used to run a battery of tests
over every SCSI commit.  It was time consuming and slowed down the
process, plus it was me who always got to diagnose failures.  Nowadays
I don't bother: I rely on 0day to run its usual tests plus a couple of
extras I asked for it's a much more streamlined process (meaning less
work for me) and everyone is happy.

The corollary I take away from this is that the less intrusive the test
infrastructure is (at least to my process) the happier I am.  The 0day
quantum leap for me was going from testing my tree and telling me of
problems after I've added the patch to testing patches posted to the
mailing list, which tells me of problems *before* the commit gets added
to the tree.

James

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Thu, Jul 06, 2017 at 07:48:05AM -0700, James Bottomley wrote:
> On Thu, 2017-07-06 at 10:28 +0100, Mark Brown wrote:

> > I think before anything like that is viable we need to show a
> > concerted and visible interest in actually running the tests we
> > already have and paying attention to the results - if people can see
> > that they're just checking a checkbox that will often result in low
> > quality tests which can do more harm than good.

> it depends what you mean by "we".  I used to run a battery of tests
> over every SCSI commit.  It was time consuming and slowed down the

We as a community, I think something viable needs to be central services
like kernelci that's automated and allows multiple people to be involved
with the analysis.  Hand running tests at scale just doesn't.

> The corollary I take away from this is that the less intrusive the test
> infrastructure is (at least to my process) the happier I am.  The 0day
> quantum leap for me was going from testing my tree and telling me of
> problems after I've added the patch to testing patches posted to the
> mailing list, which tells me of problems *before* the commit gets added
> to the tree.

I think we'd get a long way just by looking at what's ending up in -next
- it's not as good as detecting things before they go in but it's
workable if people keep on top of things.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Eric W. Biederman]

Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:

> On Wed, 5 Jul 2017 09:48:31 -0700
> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>
>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>> > On Wed, 5 Jul 2017 08:16:33 -0700
>> > Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>> [ ... ]
>> >>
>> >> If we start shaming people for not providing unit tests, all we'll accomplish is
>> >> that people will stop providing bug fixes.  
>> > 
>> > I need to be clearer on this. What I meant was, if there's a bug
>> > where someone has a test that easily reproduces the bug, then if
>> > there's not a test added to selftests for said bug, then we should
>> > shame those into doing so.
>> >   
>> 
>> I don't think that public shaming of kernel developers is going to work
>> any better than public shaming of children or teenagers.
>> 
>> Maybe a friendlier approach would be more useful ?
>
> I'm a friendly shamer ;-)
>
>> 
>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>> to the patch submitter that it would be great if that test would be submitted
>> as unit test instead of shaming that person for not doing so. Acknowledging and
>> praising kselftest submissions might help more than shaming for non-submissions.
>> 
>> > A bug that is found by inspection or hard to reproduce test cases are
>> > not applicable, as they don't have tests that can show a regression.
>> >   
>> 
>> My concern would be that once the shaming starts, it won't stop.
>
> I think this is a communication issue. My word for "shaming" was to
> call out a developer for not submitting a test. It wasn't about making
> fun of them, or anything like that. I was only making a point
> about how to teach people that they need to be more aware of the
> testing infrastructure. Not about actually demeaning people.
>
> Lets take a hypothetical sample. Say someone posted a bug report with
> an associated reproducer for it. The developer then runs the reproducer
> sees the bug, makes a fix and sends it to Linus and stable. Now the
> developer forgets this and continues on their merry way. Along comes
> someone like myself and sees a reproducing test case for a bug, but
> sees no test added to kselftests. I would send an email along the lines
> of "Hi, I noticed that there was a reproducer for this bug you fixed.
> How come there was no test added to the kselftests to make sure it
> doesn't appear again?" There, I "shamed" them ;-)

I just want to point out that kselftests are hard to build and run.

As I was looking at another issue I found a bug in one of the tests.  It
had defined a constant wrong.  I have a patch.  It took me a week of
poking at the kselftest code and trying one thing or another (between
working on other things) before I could figure out which combination of
things would let the test build and run.

Until kselftests get easier to run I don't think they are something we
want to push to hard.

Eric

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Mon, 31 Jul 2017 11:54:45 -0500
ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org (Eric W. Biederman) wrote:

> Until kselftests get easier to run I don't think they are something we
> want to push to hard.

Then perhaps we should push making them easier to run.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Eric W. Biederman]

Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:

> On Mon, 31 Jul 2017 11:54:45 -0500
> ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org (Eric W. Biederman) wrote:
>
>> Until kselftests get easier to run I don't think they are something we
>> want to push to hard.
>
> Then perhaps we should push making them easier to run.

Please.

Eric

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
> 
>> On Wed, 5 Jul 2017 09:48:31 -0700
>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>
>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>> [ ... ]
>>>>>
>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>> that people will stop providing bug fixes.  
>>>>
>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>> where someone has a test that easily reproduces the bug, then if
>>>> there's not a test added to selftests for said bug, then we should
>>>> shame those into doing so.
>>>>   
>>>
>>> I don't think that public shaming of kernel developers is going to work
>>> any better than public shaming of children or teenagers.
>>>
>>> Maybe a friendlier approach would be more useful ?
>>
>> I'm a friendly shamer ;-)
>>
>>>
>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>> to the patch submitter that it would be great if that test would be submitted
>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>
>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>> not applicable, as they don't have tests that can show a regression.
>>>>   
>>>
>>> My concern would be that once the shaming starts, it won't stop.
>>
>> I think this is a communication issue. My word for "shaming" was to
>> call out a developer for not submitting a test. It wasn't about making
>> fun of them, or anything like that. I was only making a point
>> about how to teach people that they need to be more aware of the
>> testing infrastructure. Not about actually demeaning people.
>>
>> Lets take a hypothetical sample. Say someone posted a bug report with
>> an associated reproducer for it. The developer then runs the reproducer
>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>> developer forgets this and continues on their merry way. Along comes
>> someone like myself and sees a reproducing test case for a bug, but
>> sees no test added to kselftests. I would send an email along the lines
>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>> How come there was no test added to the kselftests to make sure it
>> doesn't appear again?" There, I "shamed" them ;-)
> 
> I just want to point out that kselftests are hard to build and run.
> 
> As I was looking at another issue I found a bug in one of the tests.  It
> had defined a constant wrong.  I have a patch.  It took me a week of
> poking at the kselftest code and trying one thing or another (between
> working on other things) before I could figure out which combination of
> things would let the test build and run.
> 
> Until kselftests get easier to run I don't think they are something we
> want to push to hard.
> 

I would say it is easy to run ksefltests - "make kseflttest" from the
main Makefile does this for you. You can also run individual tests:

"make -C tools/testing/selftests/sync" for example to run sync tests.

However, I think the main pain point at the moment is being able to sift
through the output to make sense of it and being able to clearly identify
run to run differences.

At the moment, each test uses its own output format for results. As a result
it is hard for users to understand the output and parse it. This problem
is being addressed. There is active work in progress converting tests to TAP13
output format. I have been working several tests with help from others.

Please give it a try on 4.13 latest or linux-kselftest next and suggest
improvements.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Eric W. Biederman]

Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:

> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>> 
>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>
>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>> [ ... ]
>>>>>>
>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>> that people will stop providing bug fixes.  
>>>>>
>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>> where someone has a test that easily reproduces the bug, then if
>>>>> there's not a test added to selftests for said bug, then we should
>>>>> shame those into doing so.
>>>>>   
>>>>
>>>> I don't think that public shaming of kernel developers is going to work
>>>> any better than public shaming of children or teenagers.
>>>>
>>>> Maybe a friendlier approach would be more useful ?
>>>
>>> I'm a friendly shamer ;-)
>>>
>>>>
>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>> to the patch submitter that it would be great if that test would be submitted
>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>
>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>   
>>>>
>>>> My concern would be that once the shaming starts, it won't stop.
>>>
>>> I think this is a communication issue. My word for "shaming" was to
>>> call out a developer for not submitting a test. It wasn't about making
>>> fun of them, or anything like that. I was only making a point
>>> about how to teach people that they need to be more aware of the
>>> testing infrastructure. Not about actually demeaning people.
>>>
>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>> an associated reproducer for it. The developer then runs the reproducer
>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>> developer forgets this and continues on their merry way. Along comes
>>> someone like myself and sees a reproducing test case for a bug, but
>>> sees no test added to kselftests. I would send an email along the lines
>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>> How come there was no test added to the kselftests to make sure it
>>> doesn't appear again?" There, I "shamed" them ;-)
>> 
>> I just want to point out that kselftests are hard to build and run.
>> 
>> As I was looking at another issue I found a bug in one of the tests.  It
>> had defined a constant wrong.  I have a patch.  It took me a week of
>> poking at the kselftest code and trying one thing or another (between
>> working on other things) before I could figure out which combination of
>> things would let the test build and run.
>> 
>> Until kselftests get easier to run I don't think they are something we
>> want to push to hard.
>> 
>
> I would say it is easy to run ksefltests - "make kseflttest" from the
> main Makefile does this for you. You can also run individual tests:

On 4.13-rc1  That doesn't work.

$ make O=$PWD-build -j8 kselftests
make[1]: Entering directory 'linux-build'
make[1]: *** No rule to make target 'kselftests'.  Stop.
make[1]: Leaving directory 'linux-build'
Makefile:145: recipe for target 'sub-make' failed
make: *** [sub-make] Error 2

And why I have to use some esoteric command and not just the
traditional "make path/to/test/output" to run an individual
test is beyond me.

Eric

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 08/02/2017 11:33 AM, Eric W. Biederman wrote:
> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
> 
>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>>>
>>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>>
>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>>> [ ... ]
>>>>>>>
>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>>> that people will stop providing bug fixes.  
>>>>>>
>>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>>> where someone has a test that easily reproduces the bug, then if
>>>>>> there's not a test added to selftests for said bug, then we should
>>>>>> shame those into doing so.
>>>>>>   
>>>>>
>>>>> I don't think that public shaming of kernel developers is going to work
>>>>> any better than public shaming of children or teenagers.
>>>>>
>>>>> Maybe a friendlier approach would be more useful ?
>>>>
>>>> I'm a friendly shamer ;-)
>>>>
>>>>>
>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>>> to the patch submitter that it would be great if that test would be submitted
>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>>
>>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>>   
>>>>>
>>>>> My concern would be that once the shaming starts, it won't stop.
>>>>
>>>> I think this is a communication issue. My word for "shaming" was to
>>>> call out a developer for not submitting a test. It wasn't about making
>>>> fun of them, or anything like that. I was only making a point
>>>> about how to teach people that they need to be more aware of the
>>>> testing infrastructure. Not about actually demeaning people.
>>>>
>>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>>> an associated reproducer for it. The developer then runs the reproducer
>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>>> developer forgets this and continues on their merry way. Along comes
>>>> someone like myself and sees a reproducing test case for a bug, but
>>>> sees no test added to kselftests. I would send an email along the lines
>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>>> How come there was no test added to the kselftests to make sure it
>>>> doesn't appear again?" There, I "shamed" them ;-)
>>>
>>> I just want to point out that kselftests are hard to build and run.
>>>
>>> As I was looking at another issue I found a bug in one of the tests.  It
>>> had defined a constant wrong.  I have a patch.  It took me a week of
>>> poking at the kselftest code and trying one thing or another (between
>>> working on other things) before I could figure out which combination of
>>> things would let the test build and run.
>>>
>>> Until kselftests get easier to run I don't think they are something we
>>> want to push to hard.
>>>
>>
>> I would say it is easy to run ksefltests - "make kseflttest" from the
>> main Makefile does this for you. You can also run individual tests:
> 
> On 4.13-rc1  That doesn't work.
> 
> $ make O=$PWD-build -j8 kselftests
> make[1]: Entering directory 'linux-build'
> make[1]: *** No rule to make target 'kselftests'.  Stop.
> make[1]: Leaving directory 'linux-build'
> Makefile:145: recipe for target 'sub-make' failed
> make: *** [sub-make] Error 2

It is "make kselftest"

> 
> And why I have to use some esoteric command and not just the
> traditional "make path/to/test/output" to run an individual
> test is beyond me.
> 

make kselftest from top level Makefile is a way to run all the tests.
As I mentioned in my previous email 

"You can also run individual tests:

"make -C tools/testing/selftests/sync" for example to run sync tests.

or you can also run:

make -C tools/testing/selftests/ run_tests

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 08/02/2017 11:46 AM, Shuah Khan wrote:
> On 08/02/2017 11:33 AM, Eric W. Biederman wrote:
>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
>>
>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>>>>
>>>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>>>
>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>>>> [ ... ]
>>>>>>>>
>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>>>> that people will stop providing bug fixes.  
>>>>>>>
>>>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>>>> where someone has a test that easily reproduces the bug, then if
>>>>>>> there's not a test added to selftests for said bug, then we should
>>>>>>> shame those into doing so.
>>>>>>>   
>>>>>>
>>>>>> I don't think that public shaming of kernel developers is going to work
>>>>>> any better than public shaming of children or teenagers.
>>>>>>
>>>>>> Maybe a friendlier approach would be more useful ?
>>>>>
>>>>> I'm a friendly shamer ;-)
>>>>>
>>>>>>
>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>>>> to the patch submitter that it would be great if that test would be submitted
>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>>>
>>>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>>>   
>>>>>>
>>>>>> My concern would be that once the shaming starts, it won't stop.
>>>>>
>>>>> I think this is a communication issue. My word for "shaming" was to
>>>>> call out a developer for not submitting a test. It wasn't about making
>>>>> fun of them, or anything like that. I was only making a point
>>>>> about how to teach people that they need to be more aware of the
>>>>> testing infrastructure. Not about actually demeaning people.
>>>>>
>>>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>>>> an associated reproducer for it. The developer then runs the reproducer
>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>>>> developer forgets this and continues on their merry way. Along comes
>>>>> someone like myself and sees a reproducing test case for a bug, but
>>>>> sees no test added to kselftests. I would send an email along the lines
>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>>>> How come there was no test added to the kselftests to make sure it
>>>>> doesn't appear again?" There, I "shamed" them ;-)
>>>>
>>>> I just want to point out that kselftests are hard to build and run.
>>>>
>>>> As I was looking at another issue I found a bug in one of the tests.  It
>>>> had defined a constant wrong.  I have a patch.  It took me a week of
>>>> poking at the kselftest code and trying one thing or another (between
>>>> working on other things) before I could figure out which combination of
>>>> things would let the test build and run.
>>>>
>>>> Until kselftests get easier to run I don't think they are something we
>>>> want to push to hard.
>>>>
>>>
>>> I would say it is easy to run ksefltests - "make kseflttest" from the
>>> main Makefile does this for you. You can also run individual tests:
>>
>> On 4.13-rc1  That doesn't work.
>>
>> $ make O=$PWD-build -j8 kselftests
>> make[1]: Entering directory 'linux-build'
>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>> make[1]: Leaving directory 'linux-build'
>> Makefile:145: recipe for target 'sub-make' failed
>> make: *** [sub-make] Error 2
> 
> It is "make kselftest"
> 
>>
>> And why I have to use some esoteric command and not just the
>> traditional "make path/to/test/output" to run an individual
>> test is beyond me.
>>
> 
> make kselftest from top level Makefile is a way to run all the tests.
> As I mentioned in my previous email 
> 
> "You can also run individual tests:
> 
> "make -C tools/testing/selftests/sync" for example to run sync tests.
> 
> or you can also run:
> 
> make -C tools/testing/selftests/ run_tests
> 


Please take a look at:

Documentation/dev-tools/kselftest.rst

for detailed use-cases for running selftests.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Eric W. Biederman]

Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:

> On 08/02/2017 11:33 AM, Eric W. Biederman wrote:
>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
>> 
>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>>>>
>>>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>>>
>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>>>> [ ... ]
>>>>>>>>
>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>>>> that people will stop providing bug fixes.  
>>>>>>>
>>>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>>>> where someone has a test that easily reproduces the bug, then if
>>>>>>> there's not a test added to selftests for said bug, then we should
>>>>>>> shame those into doing so.
>>>>>>>   
>>>>>>
>>>>>> I don't think that public shaming of kernel developers is going to work
>>>>>> any better than public shaming of children or teenagers.
>>>>>>
>>>>>> Maybe a friendlier approach would be more useful ?
>>>>>
>>>>> I'm a friendly shamer ;-)
>>>>>
>>>>>>
>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>>>> to the patch submitter that it would be great if that test would be submitted
>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>>>
>>>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>>>   
>>>>>>
>>>>>> My concern would be that once the shaming starts, it won't stop.
>>>>>
>>>>> I think this is a communication issue. My word for "shaming" was to
>>>>> call out a developer for not submitting a test. It wasn't about making
>>>>> fun of them, or anything like that. I was only making a point
>>>>> about how to teach people that they need to be more aware of the
>>>>> testing infrastructure. Not about actually demeaning people.
>>>>>
>>>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>>>> an associated reproducer for it. The developer then runs the reproducer
>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>>>> developer forgets this and continues on their merry way. Along comes
>>>>> someone like myself and sees a reproducing test case for a bug, but
>>>>> sees no test added to kselftests. I would send an email along the lines
>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>>>> How come there was no test added to the kselftests to make sure it
>>>>> doesn't appear again?" There, I "shamed" them ;-)
>>>>
>>>> I just want to point out that kselftests are hard to build and run.
>>>>
>>>> As I was looking at another issue I found a bug in one of the tests.  It
>>>> had defined a constant wrong.  I have a patch.  It took me a week of
>>>> poking at the kselftest code and trying one thing or another (between
>>>> working on other things) before I could figure out which combination of
>>>> things would let the test build and run.
>>>>
>>>> Until kselftests get easier to run I don't think they are something we
>>>> want to push to hard.
>>>>
>>>
>>> I would say it is easy to run ksefltests - "make kseflttest" from the
>>> main Makefile does this for you. You can also run individual tests:
>> 
>> On 4.13-rc1  That doesn't work.
>> 
>> $ make O=$PWD-build -j8 kselftests
>> make[1]: Entering directory 'linux-build'
>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>> make[1]: Leaving directory 'linux-build'
>> Makefile:145: recipe for target 'sub-make' failed
>> make: *** [sub-make] Error 2
>
> It is "make kselftest"

If I include the standard O= to keep my source tree pristine
it still fails.  Which is a practical issue.  Especially because
that "make kselftest" needs to be followed by I think "make mrproper"
to get back to my normal development workflow.

I don't remember exactly what the issue was but I could not get:
tools/testing/selftests/x86/mpx-mini-test.c
tools/testing/selftests/x86/protection_keys.c

to build let alone run when I did "make kselftest"

>> And why I have to use some esoteric command and not just the
>> traditional "make path/to/test/output" to run an individual
>> test is beyond me.
>> 
>
> make kselftest from top level Makefile is a way to run all the tests.
> As I mentioned in my previous email 
>
> "You can also run individual tests:
>
> "make -C tools/testing/selftests/sync" for example to run sync tests.
>
> or you can also run:
>
> make -C tools/testing/selftests/ run_tests

As I said complicated.  That is definitely not the ordinary way of
building things in the kernel tree.

Given that some of those other tests take a little while to run, running
individual tests is actually quite important during development.

Eric

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Randy Dunlap]

On 08/02/2017 11:04 AM, Eric W. Biederman wrote:
> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
> 
>> On 08/02/2017 11:33 AM, Eric W. Biederman wrote:
>>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
>>>
>>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>>>>>
>>>>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>>>>
>>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>>>>> [ ... ]
>>>>>>>>>
>>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>>>>> that people will stop providing bug fixes.  
>>>>>>>>
>>>>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>>>>> where someone has a test that easily reproduces the bug, then if
>>>>>>>> there's not a test added to selftests for said bug, then we should
>>>>>>>> shame those into doing so.
>>>>>>>>   
>>>>>>>
>>>>>>> I don't think that public shaming of kernel developers is going to work
>>>>>>> any better than public shaming of children or teenagers.
>>>>>>>
>>>>>>> Maybe a friendlier approach would be more useful ?
>>>>>>
>>>>>> I'm a friendly shamer ;-)
>>>>>>
>>>>>>>
>>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>>>>> to the patch submitter that it would be great if that test would be submitted
>>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>>>>
>>>>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>>>>   
>>>>>>>
>>>>>>> My concern would be that once the shaming starts, it won't stop.
>>>>>>
>>>>>> I think this is a communication issue. My word for "shaming" was to
>>>>>> call out a developer for not submitting a test. It wasn't about making
>>>>>> fun of them, or anything like that. I was only making a point
>>>>>> about how to teach people that they need to be more aware of the
>>>>>> testing infrastructure. Not about actually demeaning people.
>>>>>>
>>>>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>>>>> an associated reproducer for it. The developer then runs the reproducer
>>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>>>>> developer forgets this and continues on their merry way. Along comes
>>>>>> someone like myself and sees a reproducing test case for a bug, but
>>>>>> sees no test added to kselftests. I would send an email along the lines
>>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>>>>> How come there was no test added to the kselftests to make sure it
>>>>>> doesn't appear again?" There, I "shamed" them ;-)
>>>>>
>>>>> I just want to point out that kselftests are hard to build and run.
>>>>>
>>>>> As I was looking at another issue I found a bug in one of the tests.  It
>>>>> had defined a constant wrong.  I have a patch.  It took me a week of
>>>>> poking at the kselftest code and trying one thing or another (between
>>>>> working on other things) before I could figure out which combination of
>>>>> things would let the test build and run.
>>>>>
>>>>> Until kselftests get easier to run I don't think they are something we
>>>>> want to push to hard.
>>>>>
>>>>
>>>> I would say it is easy to run ksefltests - "make kseflttest" from the
>>>> main Makefile does this for you. You can also run individual tests:
>>>
>>> On 4.13-rc1  That doesn't work.
>>>
>>> $ make O=$PWD-build -j8 kselftests
>>> make[1]: Entering directory 'linux-build'
>>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>>> make[1]: Leaving directory 'linux-build'
>>> Makefile:145: recipe for target 'sub-make' failed
>>> make: *** [sub-make] Error 2
>>
>> It is "make kselftest"
> 
> If I include the standard O= to keep my source tree pristine
> it still fails.  Which is a practical issue.  Especially because
> that "make kselftest" needs to be followed by I think "make mrproper"
> to get back to my normal development workflow.

I mentioned that problem/issue about 6 months ago. I too would like to
use O=builddir, but it is ignored.

> I don't remember exactly what the issue was but I could not get:
> tools/testing/selftests/x86/mpx-mini-test.c
> tools/testing/selftests/x86/protection_keys.c
> 
> to build let alone run when I did "make kselftest"
> 
>>> And why I have to use some esoteric command and not just the
>>> traditional "make path/to/test/output" to run an individual
>>> test is beyond me.
>>>
>>
>> make kselftest from top level Makefile is a way to run all the tests.
>> As I mentioned in my previous email 
>>
>> "You can also run individual tests:
>>
>> "make -C tools/testing/selftests/sync" for example to run sync tests.
>>
>> or you can also run:
>>
>> make -C tools/testing/selftests/ run_tests
> 
> As I said complicated.  That is definitely not the ordinary way of
> building things in the kernel tree.
> 
> Given that some of those other tests take a little while to run, running
> individual tests is actually quite important during development.


-- 
~Randy

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 08/02/2017 12:04 PM, Eric W. Biederman wrote:
> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
> 
>> On 08/02/2017 11:33 AM, Eric W. Biederman wrote:
>>> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> writes:
>>>
>>>> On 07/31/2017 10:54 AM, Eric W. Biederman wrote:
>>>>> Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> writes:
>>>>>
>>>>>> On Wed, 5 Jul 2017 09:48:31 -0700
>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:
>>>>>>
>>>>>>> On 07/05/2017 08:27 AM, Steven Rostedt wrote:
>>>>>>>> On Wed, 5 Jul 2017 08:16:33 -0700
>>>>>>>> Guenter Roeck <linux-0h96xk9xTtrk1uMJSBkQmQ@public.gmane.org> wrote:  
>>>>>>> [ ... ]
>>>>>>>>>
>>>>>>>>> If we start shaming people for not providing unit tests, all we'll accomplish is
>>>>>>>>> that people will stop providing bug fixes.  
>>>>>>>>
>>>>>>>> I need to be clearer on this. What I meant was, if there's a bug
>>>>>>>> where someone has a test that easily reproduces the bug, then if
>>>>>>>> there's not a test added to selftests for said bug, then we should
>>>>>>>> shame those into doing so.
>>>>>>>>   
>>>>>>>
>>>>>>> I don't think that public shaming of kernel developers is going to work
>>>>>>> any better than public shaming of children or teenagers.
>>>>>>>
>>>>>>> Maybe a friendlier approach would be more useful ?
>>>>>>
>>>>>> I'm a friendly shamer ;-)
>>>>>>
>>>>>>>
>>>>>>> If a test to reproduce a problem exists, it might be more beneficial to suggest
>>>>>>> to the patch submitter that it would be great if that test would be submitted
>>>>>>> as unit test instead of shaming that person for not doing so. Acknowledging and
>>>>>>> praising kselftest submissions might help more than shaming for non-submissions.
>>>>>>>
>>>>>>>> A bug that is found by inspection or hard to reproduce test cases are
>>>>>>>> not applicable, as they don't have tests that can show a regression.
>>>>>>>>   
>>>>>>>
>>>>>>> My concern would be that once the shaming starts, it won't stop.
>>>>>>
>>>>>> I think this is a communication issue. My word for "shaming" was to
>>>>>> call out a developer for not submitting a test. It wasn't about making
>>>>>> fun of them, or anything like that. I was only making a point
>>>>>> about how to teach people that they need to be more aware of the
>>>>>> testing infrastructure. Not about actually demeaning people.
>>>>>>
>>>>>> Lets take a hypothetical sample. Say someone posted a bug report with
>>>>>> an associated reproducer for it. The developer then runs the reproducer
>>>>>> sees the bug, makes a fix and sends it to Linus and stable. Now the
>>>>>> developer forgets this and continues on their merry way. Along comes
>>>>>> someone like myself and sees a reproducing test case for a bug, but
>>>>>> sees no test added to kselftests. I would send an email along the lines
>>>>>> of "Hi, I noticed that there was a reproducer for this bug you fixed.
>>>>>> How come there was no test added to the kselftests to make sure it
>>>>>> doesn't appear again?" There, I "shamed" them ;-)
>>>>>
>>>>> I just want to point out that kselftests are hard to build and run.
>>>>>
>>>>> As I was looking at another issue I found a bug in one of the tests.  It
>>>>> had defined a constant wrong.  I have a patch.  It took me a week of
>>>>> poking at the kselftest code and trying one thing or another (between
>>>>> working on other things) before I could figure out which combination of
>>>>> things would let the test build and run.
>>>>>
>>>>> Until kselftests get easier to run I don't think they are something we
>>>>> want to push to hard.
>>>>>
>>>>
>>>> I would say it is easy to run ksefltests - "make kseflttest" from the
>>>> main Makefile does this for you. You can also run individual tests:
>>>
>>> On 4.13-rc1  That doesn't work.
>>>
>>> $ make O=$PWD-build -j8 kselftests
>>> make[1]: Entering directory 'linux-build'
>>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>>> make[1]: Leaving directory 'linux-build'
>>> Makefile:145: recipe for target 'sub-make' failed
>>> make: *** [sub-make] Error 2

There are multiple ways to run selftests at the moment. If your
preferred use-case isn't supported, I don't see why it can't be
added.

Would you like to add support for this use-case?

>>
>> It is "make kselftest"
> 
> If I include the standard O= to keep my source tree pristine
> it still fails.  Which is a practical issue.  Especially because
> that "make kselftest" needs to be followed by I think "make mrproper"
> to get back to my normal development workflow.

Hmm. not sure why you would need to run "make mrproper" after running
"make kselftest" - I never have to. I would like to understand why though
if you are seeing problems without running "make mrproper".

> 
> I don't remember exactly what the issue was but I could not get:
> tools/testing/selftests/x86/mpx-mini-test.c
> tools/testing/selftests/x86/protection_keys.c
> 
> to build let alone run when I did "make kselftest"
> 
>>> And why I have to use some esoteric command and not just the
>>> traditional "make path/to/test/output" to run an individual
>>> test is beyond me.
>>>
>>
>> make kselftest from top level Makefile is a way to run all the tests.
>> As I mentioned in my previous email 
>>
>> "You can also run individual tests:
>>
>> "make -C tools/testing/selftests/sync" for example to run sync tests.
>>
>> or you can also run:
>>
>> make -C tools/testing/selftests/ run_tests
> 
> As I said complicated.  That is definitely not the ordinary way of
> building things in the kernel tree.
> 
> Given that some of those other tests take a little while to run, running
> individual tests is actually quite important during development.
> 
> Eric
> 

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Theodore Ts'o]

On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote:
> >>> $ make O=$PWD-build -j8 kselftests
> >>> make[1]: Entering directory 'linux-build'
> >>> make[1]: *** No rule to make target 'kselftests'.  Stop.
> >>> make[1]: Leaving directory 'linux-build'
> >>> Makefile:145: recipe for target 'sub-make' failed
> >>> make: *** [sub-make] Error 2
> 
> There are multiple ways to run selftests at the moment. If your
> preferred use-case isn't supported, I don't see why it can't be
> added.

There are many of us who use separate object directories so we can
build the same kernel tree with multiple configurations.  So for
example, my kernel sources will be on an SSD, but sometimes my /build
directory will be on a HDD since storing the object files on an HDD
don't really buy as much as keeping the sources on SSD.

For example, so I might be doing builds via: 

make O=/build/ext4 -j8

   and

make O=/build/ext4-32 ARCH=i386 -j8

Or someone might have a separate config file for doing debugging which
is different from their production kernel, which they could do by
having a different .config file in /build/ext4-debug/.config.

> >> It is "make kselftest"
> > 
> > If I include the standard O= to keep my source tree pristine
> > it still fails.  Which is a practical issue.  Especially because
> > that "make kselftest" needs to be followed by I think "make mrproper"
> > to get back to my normal development workflow.
> 
> Hmm. not sure why you would need to run "make mrproper" after running
> "make kselftest" - I never have to. I would like to understand why though
> if you are seeing problems without running "make mrproper".

The problem is that if you are doing builds in separate object
directories, you must not have any build files in the source tree.
Otherwise make gets terribly confused.  If you have done a build in
the source tree, then "make O=..." will because the Kernel makefile
system has safety check which detects this case and complains:

% make O=/build/linux-build
make[1]: Entering directory '/build/linux-build'
  CHK     include/config/kernel.release
  GEN     ./Makefile
  CHK     include/generated/uapi/linux/version.h
  CHK     scripts/mod/devicetable-offsets.h
  Using /home/tytso/linux/linux as source for kernel
  /home/tytso/linux/linux is not clean, please run 'make mrproper'
  ...

So this is why Eric said that he has to run "make mrproper" after
doing "make kselftest".  That's because doing any kind of build in the
source tree breaks his (and my) normal kernel building system.

Personally, I do all of my testing using xfstests (and gce-xfstests in
particular).  And so I don't have a lot of reason to spend time making
kselftest work with the "make O=xxx" build paradigm.

It's fair to say the fact that kselftest doesn't work with my kernel
development workflow hasn't helped my motivation to try it out.  But
as I told the systemtap folks many years ago --- "you're not under any
obligation to support my workflow; but if systemtap is hostile to my
kernel development workflow, you also can't expect me to help support
and grow the use of systemtap, either.  It works both ways."

The same I think applies to kselftest.  Telling developers who
complain that kselftest doesn't work well with their workflow to "send
a patch" may not result in the reaction that you hope.  Especially for
something as similar as "make O=xxx", which I think is a pretty common
development workflow.  Anyway, I hope someone will care enough about
kselftest to help support "make O=xxx"; I have to admit that someone
isn't me, though.  Sorry; I just don't have the time....

Cheers,

						- Ted

RE: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Bird, Timothy]

> -----Original Message-----
> From: Theodore Ts'o on Wednesday, August 02, 2017 8:03 PM
> 
> On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote:
> > >>> $ make O=$PWD-build -j8 kselftests
> > >>> make[1]: Entering directory 'linux-build'
> > >>> make[1]: *** No rule to make target 'kselftests'.  Stop.
> > >>> make[1]: Leaving directory 'linux-build'
> > >>> Makefile:145: recipe for target 'sub-make' failed
> > >>> make: *** [sub-make] Error 2
> >
> > There are multiple ways to run selftests at the moment. If your
> > preferred use-case isn't supported, I don't see why it can't be
> > added.
> 
> There are many of us who use separate object directories so we can
> build the same kernel tree with multiple configurations.  So for
> example, my kernel sources will be on an SSD, but sometimes my /build
> directory will be on a HDD since storing the object files on an HDD
> don't really buy as much as keeping the sources on SSD.
> 
> For example, so I might be doing builds via:
> 
> make O=/build/ext4 -j8
> 
>    and
> 
> make O=/build/ext4-32 ARCH=i386 -j8
> 
> Or someone might have a separate config file for doing debugging which
> is different from their production kernel, which they could do by
> having a different .config file in /build/ext4-debug/.config.
> 
> > >> It is "make kselftest"
> > >
> > > If I include the standard O= to keep my source tree pristine
> > > it still fails.  Which is a practical issue.  Especially because
> > > that "make kselftest" needs to be followed by I think "make mrproper"
> > > to get back to my normal development workflow.
> >
> > Hmm. not sure why you would need to run "make mrproper" after running
> > "make kselftest" - I never have to. I would like to understand why though
> > if you are seeing problems without running "make mrproper".
> 
> The problem is that if you are doing builds in separate object
> directories, you must not have any build files in the source tree.
> Otherwise make gets terribly confused.  If you have done a build in
> the source tree, then "make O=..." will because the Kernel makefile
> system has safety check which detects this case and complains:
> 
> % make O=/build/linux-build
> make[1]: Entering directory '/build/linux-build'
>   CHK     include/config/kernel.release
>   GEN     ./Makefile
>   CHK     include/generated/uapi/linux/version.h
>   CHK     scripts/mod/devicetable-offsets.h
>   Using /home/tytso/linux/linux as source for kernel
>   /home/tytso/linux/linux is not clean, please run 'make mrproper'
>   ...
> 
> So this is why Eric said that he has to run "make mrproper" after
> doing "make kselftest".  That's because doing any kind of build in the
> source tree breaks his (and my) normal kernel building system.
> 
> Personally, I do all of my testing using xfstests (and gce-xfstests in
> particular).  And so I don't have a lot of reason to spend time making
> kselftest work with the "make O=xxx" build paradigm.
> 
> It's fair to say the fact that kselftest doesn't work with my kernel
> development workflow hasn't helped my motivation to try it out.  But
> as I told the systemtap folks many years ago --- "you're not under any
> obligation to support my workflow; but if systemtap is hostile to my
> kernel development workflow, you also can't expect me to help support
> and grow the use of systemtap, either.  It works both ways."
> 
> The same I think applies to kselftest.  Telling developers who
> complain that kselftest doesn't work well with their workflow to "send
> a patch" may not result in the reaction that you hope.  Especially for
> something as similar as "make O=xxx", which I think is a pretty common
> development workflow.  Anyway, I hope someone will care enough about
> kselftest to help support "make O=xxx"; I have to admit that someone
> isn't me, though.  Sorry; I just don't have the time....

My preferred method of building for embedded boards also uses "make O=xxx"
It's been on my to do list for a while to add support for that (KBUILD_OUTPUT)
to kselftest, but it's one of those things I never got around to.

Now that I know it's blocking other kernel developers, I'll prioritize it higher.

Shuah - if no one else steps up to do this, I'll try to see if I can find time to
do it in the next month or two (no promises).  I think support for KBUILD_OUTPUT
(or "make O=xxx") is pretty important. 
   -- Tim 

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 08/03/2017 11:42 AM, Bird, Timothy wrote:
> 
> 
>> -----Original Message-----
>> From: Theodore Ts'o on Wednesday, August 02, 2017 8:03 PM
>>
>> On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote:
>>>>>> $ make O=$PWD-build -j8 kselftests
>>>>>> make[1]: Entering directory 'linux-build'
>>>>>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>>>>>> make[1]: Leaving directory 'linux-build'
>>>>>> Makefile:145: recipe for target 'sub-make' failed
>>>>>> make: *** [sub-make] Error 2
>>>
>>> There are multiple ways to run selftests at the moment. If your
>>> preferred use-case isn't supported, I don't see why it can't be
>>> added.
>>
>> There are many of us who use separate object directories so we can
>> build the same kernel tree with multiple configurations.  So for
>> example, my kernel sources will be on an SSD, but sometimes my /build
>> directory will be on a HDD since storing the object files on an HDD
>> don't really buy as much as keeping the sources on SSD.
>>
>> For example, so I might be doing builds via:
>>
>> make O=/build/ext4 -j8
>>
>>    and
>>
>> make O=/build/ext4-32 ARCH=i386 -j8
>>
>> Or someone might have a separate config file for doing debugging which
>> is different from their production kernel, which they could do by
>> having a different .config file in /build/ext4-debug/.config.
>>
>>>>> It is "make kselftest"
>>>>
>>>> If I include the standard O= to keep my source tree pristine
>>>> it still fails.  Which is a practical issue.  Especially because
>>>> that "make kselftest" needs to be followed by I think "make mrproper"
>>>> to get back to my normal development workflow.
>>>
>>> Hmm. not sure why you would need to run "make mrproper" after running
>>> "make kselftest" - I never have to. I would like to understand why though
>>> if you are seeing problems without running "make mrproper".
>>
>> The problem is that if you are doing builds in separate object
>> directories, you must not have any build files in the source tree.
>> Otherwise make gets terribly confused.  If you have done a build in
>> the source tree, then "make O=..." will because the Kernel makefile
>> system has safety check which detects this case and complains:
>>
>> % make O=/build/linux-build
>> make[1]: Entering directory '/build/linux-build'
>>   CHK     include/config/kernel.release
>>   GEN     ./Makefile
>>   CHK     include/generated/uapi/linux/version.h
>>   CHK     scripts/mod/devicetable-offsets.h
>>   Using /home/tytso/linux/linux as source for kernel
>>   /home/tytso/linux/linux is not clean, please run 'make mrproper'
>>   ...
>>
>> So this is why Eric said that he has to run "make mrproper" after
>> doing "make kselftest".  That's because doing any kind of build in the
>> source tree breaks his (and my) normal kernel building system.
>>
>> Personally, I do all of my testing using xfstests (and gce-xfstests in
>> particular).  And so I don't have a lot of reason to spend time making
>> kselftest work with the "make O=xxx" build paradigm.
>>
>> It's fair to say the fact that kselftest doesn't work with my kernel
>> development workflow hasn't helped my motivation to try it out.  But
>> as I told the systemtap folks many years ago --- "you're not under any
>> obligation to support my workflow; but if systemtap is hostile to my
>> kernel development workflow, you also can't expect me to help support
>> and grow the use of systemtap, either.  It works both ways."
>>
>> The same I think applies to kselftest.  Telling developers who
>> complain that kselftest doesn't work well with their workflow to "send
>> a patch" may not result in the reaction that you hope.  Especially for
>> something as similar as "make O=xxx", which I think is a pretty common
>> development workflow.  Anyway, I hope someone will care enough about
>> kselftest to help support "make O=xxx"; I have to admit that someone
>> isn't me, though.  Sorry; I just don't have the time....
> 
> My preferred method of building for embedded boards also uses "make O=xxx"
> It's been on my to do list for a while to add support for that (KBUILD_OUTPUT)
> to kselftest, but it's one of those things I never got around to.
> 
> Now that I know it's blocking other kernel developers, I'll prioritize it higher.
> 
> Shuah - if no one else steps up to do this, I'll try to see if I can find time to
> do it in the next month or two (no promises).  I think support for KBUILD_OUTPUT
> (or "make O=xxx") is pretty important. 
>    -- Tim 
> 

Tim,

Thanks for the offer to help. I am going look into this and let you know if I need
help. There has been some work done to support KBUILD_OUTPUT. I have to check and
see if it can address this use-case. If it does, address it, great.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 08/02/2017 09:03 PM, Theodore Ts'o wrote:
> On Wed, Aug 02, 2017 at 12:42:04PM -0600, Shuah Khan wrote:
>>>>> $ make O=$PWD-build -j8 kselftests
>>>>> make[1]: Entering directory 'linux-build'
>>>>> make[1]: *** No rule to make target 'kselftests'.  Stop.
>>>>> make[1]: Leaving directory 'linux-build'
>>>>> Makefile:145: recipe for target 'sub-make' failed
>>>>> make: *** [sub-make] Error 2
>>
>> There are multiple ways to run selftests at the moment. If your
>> preferred use-case isn't supported, I don't see why it can't be
>> added.
> 
> There are many of us who use separate object directories so we can
> build the same kernel tree with multiple configurations.  So for
> example, my kernel sources will be on an SSD, but sometimes my /build
> directory will be on a HDD since storing the object files on an HDD
> don't really buy as much as keeping the sources on SSD.
> 
> For example, so I might be doing builds via: 
> 
> make O=/build/ext4 -j8
> 
>    and
> 
> make O=/build/ext4-32 ARCH=i386 -j8
> 
> Or someone might have a separate config file for doing debugging which
> is different from their production kernel, which they could do by
> having a different .config file in /build/ext4-debug/.config.
> 
>>>> It is "make kselftest"
>>>
>>> If I include the standard O= to keep my source tree pristine
>>> it still fails.  Which is a practical issue.  Especially because
>>> that "make kselftest" needs to be followed by I think "make mrproper"
>>> to get back to my normal development workflow.
>>
>> Hmm. not sure why you would need to run "make mrproper" after running
>> "make kselftest" - I never have to. I would like to understand why though
>> if you are seeing problems without running "make mrproper".
> 
> The problem is that if you are doing builds in separate object
> directories, you must not have any build files in the source tree.
> Otherwise make gets terribly confused.  If you have done a build in
> the source tree, then "make O=..." will because the Kernel makefile
> system has safety check which detects this case and complains:
> 
> % make O=/build/linux-build
> make[1]: Entering directory '/build/linux-build'
>   CHK     include/config/kernel.release
>   GEN     ./Makefile
>   CHK     include/generated/uapi/linux/version.h
>   CHK     scripts/mod/devicetable-offsets.h
>   Using /home/tytso/linux/linux as source for kernel
>   /home/tytso/linux/linux is not clean, please run 'make mrproper'
>   ...
> 
> So this is why Eric said that he has to run "make mrproper" after
> doing "make kselftest".  That's because doing any kind of build in the
> source tree breaks his (and my) normal kernel building system.

Thanks for the detailed explanation. I have a better understanding
of the workflow.

> 
> Personally, I do all of my testing using xfstests (and gce-xfstests in
> particular).  And so I don't have a lot of reason to spend time making
> kselftest work with the "make O=xxx" build paradigm.
Now that I have a better understanding, I will take a look. Based on
quick look, it might not be difficult to support it.

> 
> It's fair to say the fact that kselftest doesn't work with my kernel
> development workflow hasn't helped my motivation to try it out.  But
> as I told the systemtap folks many years ago --- "you're not under any
> obligation to support my workflow; but if systemtap is hostile to my
> kernel development workflow, you also can't expect me to help support
> and grow the use of systemtap, either.  It works both ways."
> 
> The same I think applies to kselftest.  Telling developers who
> complain that kselftest doesn't work well with their workflow to "send
> a patch" may not result in the reaction that you hope.  Especially for
> something as similar as "make O=xxx", which I think is a pretty common
> development workflow.  Anyway, I hope someone will care enough about
> kselftest to help support "make O=xxx"; I have to admit that someone
> isn't me, though.  Sorry; I just don't have the time....
> 

It wasn't my intent to be hostile.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Theodore Ts'o]

On Thu, Aug 03, 2017 at 12:51:07PM -0600, Shuah Khan wrote:
> 
> It wasn't my intent to be hostile.

Sorry I wasn't clear; I wasn't saying you were being hostile.  It's
just that I am simultaneously trying to test multiple kernel
cofigurations against a common source tree, and the reason why I use
separate build directories is the alternative is each time I change
configurations, around 2 GB of object files get rewritten.  So a
feature which requires that I give up a workflow such that I would be
I constantly rebuild (and rewrite) gigabytes of object files is
hostile to be my development workflow --- and so unless I hear
testimonials about how it's amazingly useful, I'm highly unlikely to
even experiment with it.

Nothing personal.... it just doesn't fit with my development practices.

Cheers,

					- Ted

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Fengguang Wu]

On Wed, Jul 05, 2017 at 11:27:07AM -0400, Steven Rostedt wrote:
[snip]
>I need to be clearer on this. What I meant was, if there's a bug
>where someone has a test that easily reproduces the bug, then if
>there's not a test added to selftests for said bug, then we should
>shame those into doing so.

Besides shaming, there's one more option -- acknowledgement.

When it's a test case or test tool that discovered the bug, we could
acknowledge it by adding one line in the bug fixing patch. The exact
forms can be discussed, but here are some examples to show the basic
idea:

Tool: lockdep
Tool: ktest
Tool: smatch
Tool: trinity
Tool: syzkaller
Tool: xfstests/tests/ext4/025
Tool: scripts/coccinelle/locks/call_kern.cocci
Tool: tools/testing/selftests/bpf/test_align.c

Reports from test infrastructures like 0day could go further to help
acknowledge the tool author or maintainer by showing such lines in its
bug report email:

You may consider adding these lines in the bug fixing patch:

-----------------------[ cut here ]----------------------------------
Fixes: XXXXXXXXXX ("title of the buggy commit")
Tool: tools/testing/selftests/bpf/test_align.c <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
Reported-by: 0day test robot <xiaolong.ye-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
-----------------------[ cut here ]----------------------------------

Regards,
Fengguang

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Frank Rowand]

On 07/06/17 20:33, Fengguang Wu wrote:
> On Wed, Jul 05, 2017 at 11:27:07AM -0400, Steven Rostedt wrote:
> [snip]
>> I need to be clearer on this. What I meant was, if there's a bug
>> where someone has a test that easily reproduces the bug, then if
>> there's not a test added to selftests for said bug, then we should
>> shame those into doing so.
> 
> Besides shaming, there's one more option -- acknowledgement.
> 
> When it's a test case or test tool that discovered the bug, we could
> acknowledge it by adding one line in the bug fixing patch. The exact
> forms can be discussed, but here are some examples to show the basic
> idea:
> 
> Tool: lockdep
> Tool: ktest
> Tool: smatch
> Tool: trinity
> Tool: syzkaller
> Tool: xfstests/tests/ext4/025
> Tool: scripts/coccinelle/locks/call_kern.cocci
> Tool: tools/testing/selftests/bpf/test_align.c
> 
> Reports from test infrastructures like 0day could go further to help
> acknowledge the tool author or maintainer by showing such lines in its
> bug report email:
> 
> You may consider adding these lines in the bug fixing patch:
> 
> -----------------------[ cut here ]----------------------------------
> Fixes: XXXXXXXXXX ("title of the buggy commit")
> Tool: tools/testing/selftests/bpf/test_align.c <davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org>
> Reported-by: 0day test robot <xiaolong.ye-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> -----------------------[ cut here ]----------------------------------
> 
> Regards,
> Fengguang
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 

That is a great idea!  If a tool is shown to be catching a large
number of bugs then I am more likely to add it to my test process.

-Frank

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Greg KH]

On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote:
> If we start shaming people for not providing unit tests, all we'll accomplish is
> that people will stop providing bug fixes.

Yes, this is the key!

Steven, just look at everything marked with a "Fixes:" or "stable@" tag
from 4.12-rc1..4.12 and try to determine how you would write a test for
the majority of them.

Yes, for some subsystems this can work (look at xfstests as one great
example for filesystems, same for the i915 tests), but for the majority
of the kernel, at this point in time, it doesn't make sense.

So take Carlos's advice, start small, do it for your subsystem if you
don't touch hardware (easy peasy, right?), and let's see how it goes,
and see if we have the infrastructure to do it even today.  Right now,
kselftests is finally getting a unified output format, which is great,
it shows that people are starting to use and rely on it.  What else will
we need to make this more widely used, we don't know yet...

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Carlos O'Donell]

On 07/05/2017 11:32 AM, Greg KH wrote:
> So take Carlos's advice, start small, do it for your subsystem if you
> don't touch hardware (easy peasy, right?), and let's see how it goes,
> and see if we have the infrastructure to do it even today.  Right now,
> kselftests is finally getting a unified output format, which is great,
> it shows that people are starting to use and rely on it.  What else will
> we need to make this more widely used, we don't know yet...

+1 ;-)

-- 
Cheers,
Carlos.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Wed, 5 Jul 2017 17:32:59 +0200
Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:

> On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote:
> > If we start shaming people for not providing unit tests, all we'll accomplish is
> > that people will stop providing bug fixes.  
> 
> Yes, this is the key!

And I mentioned this in my initial email.

> 
> Steven, just look at everything marked with a "Fixes:" or "stable@" tag
> from 4.12-rc1..4.12 and try to determine how you would write a test for
> the majority of them.

It only makes sense if there's a reproducible case. For cases where
stress testing is required and you hope to hit the bug, well, that's
never an easy answer, and this is not something that will fix it.

> 
> Yes, for some subsystems this can work (look at xfstests as one great
> example for filesystems, same for the i915 tests), but for the majority
> of the kernel, at this point in time, it doesn't make sense.

I already do. Actually, I have just fixed a bug that I need to add a
selftest for. Yes, it is easier for non hardware, but for cases which
has specs on hardware behavior, why can't we have tests to test if the
hardware matches the spec?

Everyone is focusing on that "shaming" comment and not looking at the
rest of what I wrote. My main point is, there's a lot of reproducers in
change logs or emails that are not in selftests. There's no excuse for
that. Lets fix that issue, and not go into a bike shedding fight about
the entire approach.

> 
> So take Carlos's advice, start small, do it for your subsystem if you

Yes, lets start small. What do you think about all reproducers getting
into selftests? If it's not 100% reproducing, then it's up to the
individual, but any test that can trigger a bug 100% should be added.

I'd like to expand selftests to include configs too. If there's a
config that triggers a bug, that should be added to a list of "configs"
to be tested as well.

> don't touch hardware (easy peasy, right?), and let's see how it goes,
> and see if we have the infrastructure to do it even today.  Right now,
> kselftests is finally getting a unified output format, which is great,
> it shows that people are starting to use and rely on it.  What else will
> we need to make this more widely used, we don't know yet...

I've been using selftests for ftrace for some time. I have my own tests
that I run (which do test any config that has failed me in the past),
and I'm slowing getting those into the selftests directory as well.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Greg KH]

On Wed, Jul 05, 2017 at 11:52:19AM -0400, Steven Rostedt wrote:
> > So take Carlos's advice, start small, do it for your subsystem if you
> 
> Yes, lets start small. What do you think about all reproducers getting
> into selftests? If it's not 100% reproducing, then it's up to the
> individual, but any test that can trigger a bug 100% should be added.

That would be great.  One could argue that we should be adding the
"stack guard" testing apps to the selftest tree now, as a number of us
have them floating around in their test directories at the moment.

> I'd like to expand selftests to include configs too. If there's a
> config that triggers a bug, that should be added to a list of "configs"
> to be tested as well.

So a test needs a specific configuration?  We need a way to specify that
in a generic fashion so that all tests don't have to duplicate that
logic.  Time to write a helper function to parse /proc/config.gz :)

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Daniel Vetter]

On Wed, Jul 5, 2017 at 5:32 PM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote:
>> If we start shaming people for not providing unit tests, all we'll accomplish is
>> that people will stop providing bug fixes.
>
> Yes, this is the key!
>
> Steven, just look at everything marked with a "Fixes:" or "stable@" tag
> from 4.12-rc1..4.12 and try to determine how you would write a test for
> the majority of them.
>
> Yes, for some subsystems this can work (look at xfstests as one great
> example for filesystems, same for the i915 tests), but for the majority
> of the kernel, at this point in time, it doesn't make sense.
>
> So take Carlos's advice, start small, do it for your subsystem if you
> don't touch hardware (easy peasy, right?), and let's see how it goes,
> and see if we have the infrastructure to do it even today.  Right now,
> kselftests is finally getting a unified output format, which is great,
> it shows that people are starting to use and rely on it.  What else will
> we need to make this more widely used, we don't know yet...

This is very hard work and takes a long time. Since 3 years I'm trying
to establish the i915 test suite as an overall drm validation set. At
least the generic parts like for the cross-driver kernel modeset
interfaces, but also allowing other drivers to test their hw specific
command submission. It's very slow going ...
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 07/05/2017 09:32 AM, Greg KH wrote:
> On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote:
>> If we start shaming people for not providing unit tests, all we'll accomplish is
>> that people will stop providing bug fixes.
> 
> Yes, this is the key!
> 
> Steven, just look at everything marked with a "Fixes:" or "stable@" tag
> from 4.12-rc1..4.12 and try to determine how you would write a test for
> the majority of them.
> 
> Yes, for some subsystems this can work (look at xfstests as one great
> example for filesystems, same for the i915 tests), but for the majority
> of the kernel, at this point in time, it doesn't make sense.
> 
> So take Carlos's advice, start small, do it for your subsystem if you
> don't touch hardware (easy peasy, right?), and let's see how it goes,
> and see if we have the infrastructure to do it even today.  Right now,
> kselftests is finally getting a unified output format, which is great,
> it shows that people are starting to use and rely on it.  What else will
> we need to make this more widely used, we don't know yet...
> 

Over the past couple of years, kselftests have seen improvements to run
on ARM in kernel ci rings. TAP13 will definitely make it easier to find
run to run differences. There is the effort to use ksefltests to test
stable releases (4.4 LTS for example), which will help make the tests
fail/skip gracefully when a feature isn't enabled/supported.

The work so far is two fold:

- enable them to run in test rings.
- making them easy to use

As per test development, we are constantly adding tests and I see new tests
getting added for sub-systems that aren't hardware dependent. You will see
lots of activity in mm, timers, seccomp, net, sys-calls to name a few.

I am going to be looking for TAP13 format compliance for new tests starting
4.13.

I am not sure how popular they are among developers and sub-system maintainers
though. Maybe this is one area we can try to improve usage.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Steven Rostedt]

On Thu, 6 Jul 2017 16:24:01 -0600
Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> wrote:


> Over the past couple of years, kselftests have seen improvements to run
> on ARM in kernel ci rings. TAP13 will definitely make it easier to find
> run to run differences. There is the effort to use ksefltests to test
> stable releases (4.4 LTS for example), which will help make the tests
> fail/skip gracefully when a feature isn't enabled/supported.
> 
> The work so far is two fold:
> 
> - enable them to run in test rings.
> - making them easy to use
> 
> As per test development, we are constantly adding tests and I see new tests
> getting added for sub-systems that aren't hardware dependent. You will see
> lots of activity in mm, timers, seccomp, net, sys-calls to name a few.
> 
> I am going to be looking for TAP13 format compliance for new tests starting
> 4.13.
> 
> I am not sure how popular they are among developers and sub-system maintainers
> though. Maybe this is one area we can try to improve usage.

Maybe this should be included in the MAINTAINERS SUMMIT as well. To
consolidate the format of all the kselftests and have something that
everyone (or most) developers agree on.

-- Steve

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan]

On 07/06/2017 04:32 PM, Steven Rostedt wrote:
> On Thu, 6 Jul 2017 16:24:01 -0600
> Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org> wrote:
> 
> 
>> Over the past couple of years, kselftests have seen improvements to run
>> on ARM in kernel ci rings. TAP13 will definitely make it easier to find
>> run to run differences. There is the effort to use ksefltests to test
>> stable releases (4.4 LTS for example), which will help make the tests
>> fail/skip gracefully when a feature isn't enabled/supported.
>>
>> The work so far is two fold:
>>
>> - enable them to run in test rings.
>> - making them easy to use
>>
>> As per test development, we are constantly adding tests and I see new tests
>> getting added for sub-systems that aren't hardware dependent. You will see
>> lots of activity in mm, timers, seccomp, net, sys-calls to name a few.
>>
>> I am going to be looking for TAP13 format compliance for new tests starting
>> 4.13.
>>
>> I am not sure how popular they are among developers and sub-system maintainers
>> though. Maybe this is one area we can try to improve usage.

As a clarification, what I meant by "how popular they are among developers and
sub-system maintainers" is that how often developers and sub-system maintainers
run kselftests and are there any obstacles for running them.

It would be good to get feedback on usage by us as in developers.

> 
> Maybe this should be included in the MAINTAINERS SUMMIT as well. To
> consolidate the format of all the kselftests and have something that
> everyone (or most) developers agree on.


thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Dan Williams]

On Wed, Jul 5, 2017 at 7:06 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Jul 05, 2017 at 09:27:57AM -0400, Steven Rostedt wrote:
>> Your "b" above is what I would like to push. But who's going to enforce
>> this? With 10,000 changes per release, and a lot of them are fixes, the
>> best we can do is the honor system. Start shaming people that don't
>> have a regression test along with a Fixes tag (but we don't want people
>> to fix bugs without adding that tag either). There is a fine line one
>> must walk between getting people to change their approaches to bugs and
>> regression tests, and pissing them off where they start doing the
>> opposite of what would be best for the community.
>
> I would bet, for the huge majority of our fixes, they are fixes for
> specific hardware, or workarounds for specific hardware issues.  Now
> writing tests for those is not an impossible task (look at what the i915
> developers have), but it is very very hard overall, especially if the
> base infrastructure isn't there to do it.
>
> For specific examples, here's the shortlog for fixes that went into
> drivers/usb/host/ for 4.12 after 4.12-rc1 came out.  Do you know of a
> way to write a test for these types of things?
>         usb: xhci: ASMedia ASM1042A chipset need shorts TX quirk
>         usb: xhci: Fix USB 3.1 supported protocol parsing
>         usb: host: xhci-plat: propagate return value of platform_get_irq()
>         xhci: Fix command ring stop regression in 4.11
>         xhci: remove GFP_DMA flag from allocation
>         USB: xhci: fix lock-inversion problem
>         usb: host: xhci-ring: don't need to clear interrupt pending for MSI enabled hcd
>         usb: host: xhci-mem: allocate zeroed Scratchpad Buffer
>         xhci: apply PME_STUCK_QUIRK and MISSING_CAS quirk for Denverton
>         usb: xhci: trace URB before giving it back instead of after
>         USB: host: xhci: use max-port define
>         USB: ehci-platform: fix companion-device leak
>         usb: r8a66597-hcd: select a different endpoint on timeout
>         usb: r8a66597-hcd: decrease timeout

I wrote some test infrastructure to go after xhci TRB boundary
conditions [1]. So, yes, some of these are possible to unit test, but
of course not all.

[1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Greg KH]

On Wed, Jul 05, 2017 at 09:54:29AM -0700, Dan Williams wrote:
> 
> I wrote some test infrastructure to go after xhci TRB boundary
> conditions [1]. So, yes, some of these are possible to unit test, but
> of course not all.
> 
> [1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2

I forgot about that, what ever happened to it, any reason it never got
merged?

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Dan Williams]

On Wed, Jul 5, 2017 at 11:45 AM, Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Jul 05, 2017 at 09:54:29AM -0700, Dan Williams wrote:
>>
>> I wrote some test infrastructure to go after xhci TRB boundary
>> conditions [1]. So, yes, some of these are possible to unit test, but
>> of course not all.
>>
>> [1]: http://marc.info/?l=linux-usb&m=140872785411304&w=2
>
> I forgot about that, what ever happened to it, any reason it never got
> merged?

Ran out of time before being consumed by NVDIMM stuff, but I did take
some of the lessons learned over into tools/testing/nvdimm/. I haven't
done the work to integrate that into kselftest, so far it's only
exercised by the tests in the 'ndctl' [1] utility.

[1]: https://github.com/pmem/ndctl

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Carlos O'Donell]

On 07/05/2017 09:27 AM, Steven Rostedt wrote:
>> As a concrete action item, glibc core developers took a harder stance on
>> (a) all user-visible bugs need a bug # (forces people to think about the
> 
> Unfortunately, we don't have a good system for a "bug #". Most kernel
> developers hate bugzilla, and I think that includes Linus ;-) Which
> means, unless Linus builds us a new bug tracking system, there wont be
> any mandate for it.

Use the XMLRPC API to build a better interface for kernel developers?

Our "fixed bugs" list is automatically culled via XMLRPC to generate our
release announcement with "fixed bugs."

The bug # mandate has had a few key effects. It allows non-developers to
search for old similar regressions in an easier fashion than having to
trawl the mailing list for incomprehensible (to them) discussions about
semantics. The bugs are described and talked about in terms of user
facing aspects, not internal implementation details. Regressed bugs can
be reopened and discussed on the mailing list with links to the discussions
and summaries of conclusions.

All of this means we have a cleaner, clearer, description of the problem
from the user side. This again needs priority from a group of people for
whom time is precious, so you have to get buy in from them.

I don't think (a) is needed, but the glibc community found it helpful.
 
>> problem and file a coherent public bug about it) (b) all bugs needs a
>> regression test if possible, (c) and if not possible we need to extend
> 
> I would love all bug fixes to come with a test (when possible).

We have lots of hardware-specific tests that are marked UNSUPPORTED if
say you're not running on AVX512 enabled hardware.

>> the testing framework to make it possible (we've started using kernel
>> namespaces to create isolated test configurations).
> 
> Well, we have a selftest directory that should include all of these.
> And most people run them on either a test box or a VM.
Improving the test infrastructure must also be a priority, otherwise you
will grow to the limit of that infrastructure.

>> This change in reviewer priorities has had a noticeable impact on developer
>> priorities over the last 5 years. Timelines for this problem will be
>> measured in years.
> 
> Your "b" above is what I would like to push. But who's going to enforce
> this? With 10,000 changes per release, and a lot of them are fixes, the
> best we can do is the honor system. Start shaming people that don't
> have a regression test along with a Fixes tag (but we don't want people
> to fix bugs without adding that tag either). There is a fine line one
> must walk between getting people to change their approaches to bugs and
> regression tests, and pissing them off where they start doing the
> opposite of what would be best for the community.

I did say "hard problem" earlier didn't I?

* Start with yourself.

* For everyone you know well, and have met in person, be brutal and
  require them to submit regression tests with their bug fixes. These
  people are already committed to getting their fixes in and they will
  understand you are making an example of them.

* For everyone you don't know well, be gentle, and begin reminding
  them you need a regression test, and if you feel generous try to write
  one yourself for them. Often the act of writing such a test will show
  you how hard it is, and what is missing from your infrastructure to make
  this easy, because if it was easy everyone would do it.

YMMV.

-- 
Cheers,
Carlos.

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

On Wed, Jul 05, 2017 at 09:09:51AM -0400, Carlos O'Donell wrote:

> This problem is a reflection of our own explicit or implicit priorities.
> The priorities of developers and reviewers needs to change to make an
> impact on the problem. This is a hard problem.

Take a look at the trajectory for the build and boot testing for a
concrete example of this - the failure rates go down over time but it's
not a quick process.

> As a concrete action item, glibc core developers took a harder stance on
> (a) all user-visible bugs need a bug # (forces people to think about the
> problem and file a coherent public bug about it) (b) all bugs needs a
> regression test if possible, (c) and if not possible we need to extend
> the testing framework to make it possible (we've started using kernel
> namespaces to create isolated test configurations).

One thing I'd really like to see here is an equivalent of the build and
boot testing we currently have that exercises some of the testsuites on
a regular basis so we can push on keeping them running cleanly.  As well
as just the intrisic value of the tests themselves I'd hope that a
visible practical interest would help push more activity in this area.

There's a couple of efforts I'm aware of in this area, hopefully one or
both of them will start delivering.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]