* [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful @ 2018-09-05 10:13 James Bottomley 2018-09-05 11:37 ` Mark Brown ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: James Bottomley @ 2018-09-05 10:13 UTC (permalink / raw) To: ksummit-discuss I'm seeing a lot of wasted effort by our customers on kernel bugs and trying to engage the distribution to fix them. As a caveat, I'm working in the cloud, so the distributions in question are usually community ones not enterprise ones. However, we do have a fair few customers on LTS kernels from Distributions. Mostly they find a cloud performance regression, they try to engage the distro, spend ages working on it or submitting bugs and usually end up with an unsatisfactory result. By the time they call my team in, we've likely only got a week to fix the issue. However, step one is always confirming whether upstream works (95% of the time it does) and then finding the fix by bisection (usually assisted by knowledge of where the bug is). To do the bisection we usually have to build a kernel package with our guesses and get them to try it, so it can be a bit slow. Once we have the backport, we send it to stable and notify the distribution to include it in their next kernel release. Here's the rub: community distributions (even LTS ones) don't have the resources even to triage cloud bugs in environments they likely can't reproduce, so we really need to develop assistive tools for customers to perform bisections to identify what caused the bug or (in the 95% case) what fixed it. Having a bugzilla and using it as first line of support implies a service expectation (usually coming from Enterprise) that simply isn't met, so distributions need to fix this at the point of interaction: bugzilla. The first suggestion is that kernel builds are pretty much automated and we try to make every commit buildable, so could we automate the machinery that allows a customer to do bisection simply by installing a kernel package? (we here, obviously means the distro, but going from git bisect to kernel package would be the useful link). Second suggestion is that the bugzillas need to say much more strongly that the reporter really needs to confirm the fix in upstream and do the bisection themselves (and ideally request the backport to stable themselves). James ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 10:13 [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful James Bottomley @ 2018-09-05 11:37 ` Mark Brown 2018-09-05 15:03 ` Paul E. McKenney 2018-09-05 13:16 ` Takashi Iwai 2018-09-05 17:41 ` Laura Abbott 2 siblings, 1 reply; 33+ messages in thread From: Mark Brown @ 2018-09-05 11:37 UTC (permalink / raw) To: James Bottomley; +Cc: ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 799 bytes --] On Wed, Sep 05, 2018 at 11:13:52AM +0100, James Bottomley wrote: > The first suggestion is that kernel builds are pretty much automated > and we try to make every commit buildable, so could we automate the > machinery that allows a customer to do bisection simply by installing a > kernel package? (we here, obviously means the distro, but going from > git bisect to kernel package would be the useful link). Improving bisectability would obviously help with other testing efforts too - we have existing users, Guillaume Tucker implemented automated bisection support in KernelCI which is incredibly useful providing one can actually bisect. Right now it works pretty well a lot of the time but there are cases where it gets messy, especially when you add boot issues onto the buildability ones. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 11:37 ` Mark Brown @ 2018-09-05 15:03 ` Paul E. McKenney 2018-09-05 15:50 ` Steven Rostedt 0 siblings, 1 reply; 33+ messages in thread From: Paul E. McKenney @ 2018-09-05 15:03 UTC (permalink / raw) To: Mark Brown; +Cc: James Bottomley, ksummit-discuss On Wed, Sep 05, 2018 at 12:37:15PM +0100, Mark Brown wrote: > On Wed, Sep 05, 2018 at 11:13:52AM +0100, James Bottomley wrote: > > > The first suggestion is that kernel builds are pretty much automated > > and we try to make every commit buildable, so could we automate the > > machinery that allows a customer to do bisection simply by installing a > > kernel package? (we here, obviously means the distro, but going from > > git bisect to kernel package would be the useful link). > > Improving bisectability would obviously help with other testing efforts > too - we have existing users, Guillaume Tucker implemented automated > bisection support in KernelCI which is incredibly useful providing one > can actually bisect. Right now it works pretty well a lot of the time > but there are cases where it gets messy, especially when you add boot > issues onto the buildability ones. I am one of those strange people who rebase in order to improve bisectability. But one reason I can do that is that I have relatively few patches, and it gets harder the more patches I am carrying. I suppose that someone (not me!) could rebase -stable to make it more bisectable, but that sounds difficult, painful, and error-prone. Could added tooling make bisection work better? Sounds valuable, but non-trivial. In some of my bisection efforts, I have had to apply fix patches to fix various unrelated bugs. I suppose that this could be automated, for example by tracking which fix-patches are needed at which potential bisection points, though this sounds like a large effort. Of course, automated backporting of patches would make it easier, and much else easier as well. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 15:03 ` Paul E. McKenney @ 2018-09-05 15:50 ` Steven Rostedt 2018-09-05 16:20 ` Paul E. McKenney 0 siblings, 1 reply; 33+ messages in thread From: Steven Rostedt @ 2018-09-05 15:50 UTC (permalink / raw) To: Paul E. McKenney; +Cc: James Bottomley, ksummit-discuss On Wed, 5 Sep 2018 08:03:15 -0700 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > I am one of those strange people who rebase in order to improve > bisectability. But one reason I can do that is that I have relatively > few patches, and it gets harder the more patches I am carrying. I suppose > that someone (not me!) could rebase -stable to make it more bisectable, How would rebasing it make stable more bisectable? Once you rebase, you don't have a tree that use to work? Although I guess you may find the commit that caused the problem better. But rebasing creates a lot of other issues, I would not recommend rebasing stable, as that would totally break the RT stable tree work flow. -- Steve > but that sounds difficult, painful, and error-prone. Could added tooling > make bisection work better? Sounds valuable, but non-trivial. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 15:50 ` Steven Rostedt @ 2018-09-05 16:20 ` Paul E. McKenney 2018-09-05 16:45 ` James Bottomley 0 siblings, 1 reply; 33+ messages in thread From: Paul E. McKenney @ 2018-09-05 16:20 UTC (permalink / raw) To: Steven Rostedt; +Cc: James Bottomley, ksummit-discuss On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote: > On Wed, 5 Sep 2018 08:03:15 -0700 > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > > > I am one of those strange people who rebase in order to improve > > bisectability. But one reason I can do that is that I have relatively > > few patches, and it gets harder the more patches I am carrying. I suppose > > that someone (not me!) could rebase -stable to make it more bisectable, > > How would rebasing it make stable more bisectable? Once you rebase, you > don't have a tree that use to work? Although I guess you may find the > commit that caused the problem better. But rebasing creates a lot of > other issues, I would not recommend rebasing stable, as that would > totally break the RT stable tree work flow. Instead of leaving the buggy commit and the span where the bug exists, you rebase the fix into the original buggy fix. And I bet that rebasing -stable would cause no end of broken glass in a great many projects. ;-) Thanx, Paul ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 16:20 ` Paul E. McKenney @ 2018-09-05 16:45 ` James Bottomley 2018-09-05 17:00 ` Paul E. McKenney 2018-09-05 19:25 ` Jiri Kosina 0 siblings, 2 replies; 33+ messages in thread From: James Bottomley @ 2018-09-05 16:45 UTC (permalink / raw) To: paulmck, Steven Rostedt; +Cc: ksummit-discuss On Wed, 2018-09-05 at 09:20 -0700, Paul E. McKenney wrote: > On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote: > > On Wed, 5 Sep 2018 08:03:15 -0700 > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > > > > > I am one of those strange people who rebase in order to improve > > > bisectability. But one reason I can do that is that I have > > > relatively > > > few patches, and it gets harder the more patches I am > > > carrying. I suppose > > > that someone (not me!) could rebase -stable to make it more > > > bisectable, > > > > How would rebasing it make stable more bisectable? Once you rebase, > > you don't have a tree that use to work? Although I guess you may > > find the commit that caused the problem better. But rebasing > > creates a lot of other issues, I would not recommend rebasing > > stable, as that would totally break the RT stable tree work flow. > > Instead of leaving the buggy commit and the span where the bug > exists, you rebase the fix into the original buggy fix. We do this in SCSI as well, but only if the tree hasn't yet been submitted to Linus. The technical term is folding. It's obviously better to fix buggy commits that haven't gone upstream because it improves bisectability. > And I bet that rebasing -stable would cause no end of broken glass in > a great many projects. ;-) If others rely on your tree, rebasing is harder and must be done more carefully and with co-ordination, but it's not impossible assuming you have a problem big enough. Again, it's an expediency based trade off. James ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 16:45 ` James Bottomley @ 2018-09-05 17:00 ` Paul E. McKenney 2018-09-05 19:25 ` Jiri Kosina 1 sibling, 0 replies; 33+ messages in thread From: Paul E. McKenney @ 2018-09-05 17:00 UTC (permalink / raw) To: James Bottomley; +Cc: ksummit-discuss On Wed, Sep 05, 2018 at 05:45:14PM +0100, James Bottomley wrote: > On Wed, 2018-09-05 at 09:20 -0700, Paul E. McKenney wrote: > > On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote: > > > On Wed, 5 Sep 2018 08:03:15 -0700 > > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > > > > > > > I am one of those strange people who rebase in order to improve > > > > bisectability. But one reason I can do that is that I have > > > > relatively > > > > few patches, and it gets harder the more patches I am > > > > carrying. I suppose > > > > that someone (not me!) could rebase -stable to make it more > > > > bisectable, > > > > > > How would rebasing it make stable more bisectable? Once you rebase, > > > you don't have a tree that use to work? Although I guess you may > > > find the commit that caused the problem better. But rebasing > > > creates a lot of other issues, I would not recommend rebasing > > > stable, as that would totally break the RT stable tree work flow. > > > > Instead of leaving the buggy commit and the span where the bug > > exists, you rebase the fix into the original buggy fix. > > We do this in SCSI as well, but only if the tree hasn't yet been > submitted to Linus. The technical term is folding. It's obviously > better to fix buggy commits that haven't gone upstream because it > improves bisectability. So I am not the only one doing this. ;-) > > And I bet that rebasing -stable would cause no end of broken glass in > > a great many projects. ;-) > > If others rely on your tree, rebasing is harder and must be done more > carefully and with co-ordination, but it's not impossible assuming you > have a problem big enough. Again, it's an expediency based trade off. I use date-coded branches to keep the old commits around. I delete them periodically, but keep them around for at least six months, on the theory that if you are developing against a two-year old -rcu branch, you have bigger problems. And you should have instead developed against a formal release anyway. Thanx, Paul ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 16:45 ` James Bottomley 2018-09-05 17:00 ` Paul E. McKenney @ 2018-09-05 19:25 ` Jiri Kosina 2018-09-05 19:40 ` James Bottomley 1 sibling, 1 reply; 33+ messages in thread From: Jiri Kosina @ 2018-09-05 19:25 UTC (permalink / raw) To: James Bottomley; +Cc: ksummit-discuss On Wed, 5 Sep 2018, James Bottomley wrote: > We do this in SCSI as well, but only if the tree hasn't yet been > submitted to Linus. The technical term is folding. It's obviously > better to fix buggy commits that haven't gone upstream because it > improves bisectability. We are drifting away a bit here, but now that you have mentioned it, let me add a datapoint to this -- it's actually causing issues to our workflow, as we have scsi.git as one of the upstreams [1], and when you rebase, it blows up our git workflow and we have to fixup things manually. So if you are aware of your tree having downstreams, and care about not breaking them and want to be nice to them, you shouldn't rebase that tree [2]. [1] there are some funny technical details, but in basic principle it's exactly like that [2] https://yarchive.net/comp/linux/git_rebase.html Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 19:25 ` Jiri Kosina @ 2018-09-05 19:40 ` James Bottomley 2018-09-06 19:54 ` Jiri Kosina 0 siblings, 1 reply; 33+ messages in thread From: James Bottomley @ 2018-09-05 19:40 UTC (permalink / raw) To: Jiri Kosina; +Cc: ksummit-discuss On Wed, 2018-09-05 at 21:25 +0200, Jiri Kosina wrote: > On Wed, 5 Sep 2018, James Bottomley wrote: > > > We do this in SCSI as well, but only if the tree hasn't yet been > > submitted to Linus. The technical term is folding. It's > > obviously better to fix buggy commits that haven't gone upstream > > because it improves bisectability. > > We are drifting away a bit here, but now that you have mentioned it, > let me add a datapoint to this -- it's actually causing issues to > our workflow, as we have scsi.git as one of the upstreams [1], and > when you rebase, it blows up our git workflow and we have to fixup > things manually. Describe the issues and we'll try to come up with a fix, but on the whole you should regard the scsi trees somewhat similarly to linux- next: it's our proposal for a patch set but we may update it. > So if you are aware of your tree having downstreams, and care about > not breaking them and want to be nice to them, you shouldn't rebase > that tree [2]. Well, I wasn't aware of this one, but I'm sure we can come up with a functional workflow once I understand what's happening with the downstream tree. James > [1] there are some funny technical details, but in basic principle > it's exactly like that > > [2] https://yarchive.net/comp/linux/git_rebase.html > > Thanks, > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 19:40 ` James Bottomley @ 2018-09-06 19:54 ` Jiri Kosina 2018-09-18 13:43 ` Martin K. Petersen 0 siblings, 1 reply; 33+ messages in thread From: Jiri Kosina @ 2018-09-06 19:54 UTC (permalink / raw) To: James Bottomley; +Cc: ksummit-discuss On Wed, 5 Sep 2018, James Bottomley wrote: > > We are drifting away a bit here, but now that you have mentioned it, > > let me add a datapoint to this -- it's actually causing issues to > > our workflow, as we have scsi.git as one of the upstreams [1], and > > when you rebase, it blows up our git workflow and we have to fixup > > things manually. > > Describe the issues and we'll try to come up with a fix, but on the > whole you should regard the scsi trees somewhat similarly to linux- > next: it's our proposal for a patch set but we may update it. We order patches in our trees in the same git-topological-ordering as they are upstream. It has a lot of benefits, most importantly: it doesn't introduce artificial conflicts that don't exist in reality. In order to achieve that, we of course need 1:1 mapping between our patches and upstream commits. Rebases destroy that mapping. And in some areas (scsi is one, but not the only one), we basically had no other choice than considering maintainer's tree to be already "upstream enough", without waiting for Linus' tree merge. > > So if you are aware of your tree having downstreams, and care about > > not breaking them and want to be nice to them, you shouldn't rebase > > that tree [2]. > > Well, I wasn't aware of this one, but I'm sure we can come up with a > functional workflow once I understand what's happening with the > downstream tree. Thanks, -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-06 19:54 ` Jiri Kosina @ 2018-09-18 13:43 ` Martin K. Petersen 2018-09-18 14:12 ` Geert Uytterhoeven ` (2 more replies) 0 siblings, 3 replies; 33+ messages in thread From: Martin K. Petersen @ 2018-09-18 13:43 UTC (permalink / raw) To: Jiri Kosina; +Cc: James Bottomley, ksummit-discuss Hi Jiri, Sorry I'm a little late to the game here. Been out on vacation. > We order patches in our trees in the same git-topological-ordering as they > are upstream. It has a lot of benefits, most importantly: it doesn't > introduce artificial conflicts that don't exist in reality. > > In order to achieve that, we of course need 1:1 mapping between our > patches and upstream commits. Rebases destroy that mapping. > > And in some areas (scsi is one, but not the only one), we basically had no > other choice than considering maintainer's tree to be already "upstream > enough", without waiting for Linus' tree merge. When I discussed this with Johannes a little while ago, I suggested you guys used git patch-id to track patches instead of commit ids. That's how we track patches applied across many different trees internally. Works much better than using the upstream sha. I would like to understand your "upstream enough" requirement. Why do you need a tree that's stable before Linus pulls the changes? Note that I am generally only rebasing as a last resort and typically only very early in the rc cycle. It usually happens when I need to drop a patch series that turned out to be unfixable in its current state. And before everyone screams because I'm not supposed to be pushing stuff that breaks, please realize that it is impossible to test all the different types of hardware I have to merge drivers for. There is no regression test suite or lab setup with anything resembling comprehensive coverage. I test changes to the SCSI core code and do some rudimentary testing on SAS and FC on x86_64. But that's really the best I can do. Even though most patches posted to linux-scsi get picked up by 0day, more often than not they only get x86_64 build coverage. Whereas 0day build failures on arm, mips, sparc32, whatever typically only get reported after patches have been simmering in linux-next for a while. Depends how busy 0day is. Also, actual driver failures on platforms not officially supported and tested by the controller vendor are only found after the fact. And most of the time it's not a matter of reverting a single patch but effectively dropping all of the patches in the series until they can be reworked. Sometimes a workaround takes a week or two to deliver, and people don't appreciate not being able to boot their systems in the meantime. So that's why I generally drop the series instead. I would love for every patch sent to linux-scsi to be bug free and instantly build tested by 0day on every architecture. And I would love for hardware vendors to be more cognizant about architectures they don't commercially support. But reality is that things break frequently when I merge big, complex driver update patch series. As a result, the preference has been to have the flexibility to amend or drop patches early in every cycle. It hasn't really been a problem because there have been no downstream users of SCSI at all. I only recently found out about your use case. I'm pretty flexible about how to address this, there are a couple of ways to go about it. 1. I could just always revert instead of dropping the patches. The downside is that we end up with a pretty messy history because, as I pointed out above, it's usually a matter of dropping tens of patches at a time and not reverting a single offending commit. In addition, having a messy history makes it harder on distro kernel people to track driver updates. 2. The other option is that I set up a scsi-staging tree where stuff can simmer for a bit and hopefully get some 0day coverage before getting shuffled over to scsi-queue. However, I do question how much actual real hardware coverage we'll get by having a SCSI tree that people would explicitly have to pull to test. As opposed to linux-next which at least gets some coverage by test farms and users. 3. The third option is that we pick a number like rc4 and I will promise not to rebase after that. We can see how that works out. I'm open to ideas. The most important thing to me is that 0day and linux-next are indispensable tools in my workflow given that I have no way to personally test most of the code I merge. So it is imperative that I have the ability to push code early because that's where I get the bulk of my (non-SCSI-core) build and test coverage. -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 13:43 ` Martin K. Petersen @ 2018-09-18 14:12 ` Geert Uytterhoeven 2018-09-18 15:01 ` Martin K. Petersen 2018-09-18 16:12 ` Mark Brown 2018-09-18 20:37 ` Takashi Iwai 2 siblings, 1 reply; 33+ messages in thread From: Geert Uytterhoeven @ 2018-09-18 14:12 UTC (permalink / raw) To: Martin K. Petersen; +Cc: James Bottomley, ksummit-discuss On Tue, Sep 18, 2018 at 3:43 PM Martin K. Petersen <martin.petersen@oracle.com> wrote: > > We order patches in our trees in the same git-topological-ordering as they > > are upstream. It has a lot of benefits, most importantly: it doesn't > > introduce artificial conflicts that don't exist in reality. > > > > In order to achieve that, we of course need 1:1 mapping between our > > patches and upstream commits. Rebases destroy that mapping. > > > > And in some areas (scsi is one, but not the only one), we basically had no > > other choice than considering maintainer's tree to be already "upstream > > enough", without waiting for Linus' tree merge. > > When I discussed this with Johannes a little while ago, I suggested you > guys used git patch-id to track patches instead of commit ids. That's > how we track patches applied across many different trees internally. > Works much better than using the upstream sha. > > I would like to understand your "upstream enough" requirement. Why do > you need a tree that's stable before Linus pulls the changes? > > Note that I am generally only rebasing as a last resort and typically > only very early in the rc cycle. It usually happens when I need to drop > a patch series that turned out to be unfixable in its current state. > > And before everyone screams because I'm not supposed to be pushing stuff > that breaks, please realize that it is impossible to test all the > different types of hardware I have to merge drivers for. There is no > regression test suite or lab setup with anything resembling > comprehensive coverage. I test changes to the SCSI core code and do some > rudimentary testing on SAS and FC on x86_64. But that's really the best > I can do. > > Even though most patches posted to linux-scsi get picked up by 0day, > more often than not they only get x86_64 build coverage. Whereas 0day > build failures on arm, mips, sparc32, whatever typically only get > reported after patches have been simmering in linux-next for a > while. Depends how busy 0day is. > > Also, actual driver failures on platforms not officially supported and > tested by the controller vendor are only found after the fact. And most > of the time it's not a matter of reverting a single patch but > effectively dropping all of the patches in the series until they can be > reworked. Sometimes a workaround takes a week or two to deliver, and > people don't appreciate not being able to boot their systems in the > meantime. So that's why I generally drop the series instead. > > I would love for every patch sent to linux-scsi to be bug free and > instantly build tested by 0day on every architecture. And I would love > for hardware vendors to be more cognizant about architectures they don't > commercially support. But reality is that things break frequently when > I merge big, complex driver update patch series. > > As a result, the preference has been to have the flexibility to amend or > drop patches early in every cycle. It hasn't really been a problem > because there have been no downstream users of SCSI at all. I only > recently found out about your use case. > > I'm pretty flexible about how to address this, there are a couple of > ways to go about it. > > 1. I could just always revert instead of dropping the patches. The > downside is that we end up with a pretty messy history because, as I > pointed out above, it's usually a matter of dropping tens of patches > at a time and not reverting a single offending commit. In addition, > having a messy history makes it harder on distro kernel people to > track driver updates. > > 2. The other option is that I set up a scsi-staging tree where stuff can > simmer for a bit and hopefully get some 0day coverage before getting > shuffled over to scsi-queue. However, I do question how much actual > real hardware coverage we'll get by having a SCSI tree that people > would explicitly have to pull to test. As opposed to linux-next which > at least gets some coverage by test farms and users. > > 3. The third option is that we pick a number like rc4 and I will promise > not to rebase after that. We can see how that works out. > > I'm open to ideas. The most important thing to me is that 0day and > linux-next are indispensable tools in my workflow given that I have no > way to personally test most of the code I merge. So it is imperative > that I have the ability to push code early because that's where I get > the bulk of my (non-SCSI-core) build and test coverage. You could start using topic branches (for core, and per driver/vendor), and recreate your for-next branch as a merge of all topic branches on a daily basis. If one patch series turns out to be bad, at least the commit IDs in the other topic branches will remain stable. I believe Mark Brown used to have such a system (except for dropping patches, he never rebases his topic branches), unless he got bashed by Linus for his cephalopodic merges having more branches than an octopus(?). Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 14:12 ` Geert Uytterhoeven @ 2018-09-18 15:01 ` Martin K. Petersen 2018-09-18 15:27 ` Christoph Hellwig 0 siblings, 1 reply; 33+ messages in thread From: Martin K. Petersen @ 2018-09-18 15:01 UTC (permalink / raw) To: Geert Uytterhoeven; +Cc: James Bottomley, ksummit-discuss Geert, > You could start using topic branches (for core, and per driver/vendor), > and recreate your for-next branch as a merge of all topic branches on a > daily basis. If one patch series turns out to be bad, at least the commit > IDs in the other topic branches will remain stable. Yeah. I thought about doing what Jens does, splitting core and drivers into separate branches. However, compared to block I have so few SCSI core updates that I am not convinced it's worth it. Core changes only amount to a couple percent of the total number of commits every release. For topic branches to be truly effective I'd need a separate branch per driver. Not impossible, but it seems like a pretty big hammer... -- Martin K. Petersen Oracle Linux Engineering ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 15:01 ` Martin K. Petersen @ 2018-09-18 15:27 ` Christoph Hellwig 2018-09-18 15:34 ` Jens Axboe 2018-09-18 17:08 ` Mark Brown 0 siblings, 2 replies; 33+ messages in thread From: Christoph Hellwig @ 2018-09-18 15:27 UTC (permalink / raw) To: Martin K. Petersen; +Cc: James Bottomley, ksummit-discuss On Tue, Sep 18, 2018 at 11:01:50AM -0400, Martin K. Petersen wrote: > Yeah. I thought about doing what Jens does, splitting core and drivers > into separate branches. However, compared to block I have so few SCSI > core updates that I am not convinced it's worth it. Core changes only > amount to a couple percent of the total number of commits every release. I did that for the SCSI tree a long time ago, and it did not work out at all as the few core changes are usually required by driver changes. Note that I've not really seen the split recently in the block tree either, probably for the same reason. ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 15:27 ` Christoph Hellwig @ 2018-09-18 15:34 ` Jens Axboe 2018-09-18 17:08 ` Mark Brown 1 sibling, 0 replies; 33+ messages in thread From: Jens Axboe @ 2018-09-18 15:34 UTC (permalink / raw) To: Christoph Hellwig, Martin K. Petersen; +Cc: James Bottomley, ksummit-discuss On 9/18/18 9:27 AM, Christoph Hellwig wrote: > On Tue, Sep 18, 2018 at 11:01:50AM -0400, Martin K. Petersen wrote: >> Yeah. I thought about doing what Jens does, splitting core and drivers >> into separate branches. However, compared to block I have so few SCSI >> core updates that I am not convinced it's worth it. Core changes only >> amount to a couple percent of the total number of commits every release. > > I did that for the SCSI tree a long time ago, and it did not work out > at all as the few core changes are usually required by driver changes. > > Note that I've not really seen the split recently in the block tree > either, probably for the same reason. Yeah exactly, I basically stopped doing that as the most active driver was NVMe, and for quite a while that also helped drive core changes. Since they went hand in hand, there wasn't much point in doing the split branches anymore, in fact it was harmful. I'd love to start doing it again, and I may do that, since we've now got less feature additions that end up depending on core changes. -- Jens Axboe ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 15:27 ` Christoph Hellwig 2018-09-18 15:34 ` Jens Axboe @ 2018-09-18 17:08 ` Mark Brown 1 sibling, 0 replies; 33+ messages in thread From: Mark Brown @ 2018-09-18 17:08 UTC (permalink / raw) To: Christoph Hellwig; +Cc: James Bottomley, ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 658 bytes --] On Tue, Sep 18, 2018 at 08:27:11AM -0700, Christoph Hellwig wrote: > I did that for the SCSI tree a long time ago, and it did not work out > at all as the few core changes are usually required by driver changes. > Note that I've not really seen the split recently in the block tree > either, probably for the same reason. Most of my topic branches end up being per-driver ones, with embedded stuff that often winds up being useful for cross tree merges that weren't anticipated. The core changes that affect or are used by drivers either get the relevant driver changes stacked onto the same branch or merged into the driver branches depending on taste. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 13:43 ` Martin K. Petersen 2018-09-18 14:12 ` Geert Uytterhoeven @ 2018-09-18 16:12 ` Mark Brown 2018-09-18 20:20 ` Takashi Iwai 2018-09-18 20:37 ` Takashi Iwai 2 siblings, 1 reply; 33+ messages in thread From: Mark Brown @ 2018-09-18 16:12 UTC (permalink / raw) To: Martin K. Petersen; +Cc: James Bottomley, ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 781 bytes --] On Tue, Sep 18, 2018 at 09:43:08AM -0400, Martin K. Petersen wrote: > 1. I could just always revert instead of dropping the patches. The > downside is that we end up with a pretty messy history because, as I > pointed out above, it's usually a matter of dropping tens of patches > at a time and not reverting a single offending commit. In addition, > having a messy history makes it harder on distro kernel people to > track driver updates. I used to deal with this by using topic branches heavily and making my -next be an automated merge of those branches, if something went badly I could just throw away the branch. I stopped for a while because Linus didn't like the number of branches I was creating, though it wasn't a problem with the approach in general. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 16:12 ` Mark Brown @ 2018-09-18 20:20 ` Takashi Iwai 2018-09-19 0:08 ` Mark Brown 0 siblings, 1 reply; 33+ messages in thread From: Takashi Iwai @ 2018-09-18 20:20 UTC (permalink / raw) To: Mark Brown; +Cc: James Bottomley, ksummit-discuss On Tue, 18 Sep 2018 18:12:01 +0200, Mark Brown wrote: > > On Tue, Sep 18, 2018 at 09:43:08AM -0400, Martin K. Petersen wrote: > > > 1. I could just always revert instead of dropping the patches. The > > downside is that we end up with a pretty messy history because, as I > > pointed out above, it's usually a matter of dropping tens of patches > > at a time and not reverting a single offending commit. In addition, > > having a messy history makes it harder on distro kernel people to > > track driver updates. > > I used to deal with this by using topic branches heavily and making my > -next be an automated merge of those branches, if something went badly I > could just throw away the branch. I stopped for a while because Linus > didn't like the number of branches I was creating, though it wasn't a > problem with the approach in general. I think that the topic branch approach would work well if you merge topic branches back to the main branch more often. That is, each topic branch lives only for a relatively short time (e.g. a few weeks), and not merging the whole branches in a shot at the end of the development cycle. Basically the merge to the main branch "fixates" the developments of the given topic, hence other people can start working on the commits safely. Meanwhile the not-yet-merged branches can be still thrown away if they are really bad. Takashi ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 20:20 ` Takashi Iwai @ 2018-09-19 0:08 ` Mark Brown 0 siblings, 0 replies; 33+ messages in thread From: Mark Brown @ 2018-09-19 0:08 UTC (permalink / raw) To: Takashi Iwai; +Cc: James Bottomley, ksummit-discuss [-- Attachment #1: Type: text/plain, Size: 1301 bytes --] On Tue, Sep 18, 2018 at 10:20:17PM +0200, Takashi Iwai wrote: > Mark Brown wrote: > > I used to deal with this by using topic branches heavily and making my > > -next be an automated merge of those branches, if something went badly I > > could just throw away the branch. I stopped for a while because Linus > > didn't like the number of branches I was creating, though it wasn't a > > problem with the approach in general. > I think that the topic branch approach would work well if you merge > topic branches back to the main branch more often. That is, each > topic branch lives only for a relatively short time (e.g. a few > weeks), and not merging the whole branches in a shot at the end of the > development cycle. That approach tends to lead to more merges (as you end up with more of the small topic branches) and if you don't have any commits actually on the trunk then the long series of merges is also seen as an antipattern by some. > Basically the merge to the main branch "fixates" the developments of > the given topic, hence other people can start working on the commits > safely. Meanwhile the not-yet-merged branches can be still thrown > away if they are really bad. For less widely used stuff it can be unfortunately difficult to get the confidence that things are done :( [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 13:43 ` Martin K. Petersen 2018-09-18 14:12 ` Geert Uytterhoeven 2018-09-18 16:12 ` Mark Brown @ 2018-09-18 20:37 ` Takashi Iwai 2018-09-19 6:16 ` Geert Uytterhoeven 2 siblings, 1 reply; 33+ messages in thread From: Takashi Iwai @ 2018-09-18 20:37 UTC (permalink / raw) To: Martin K. Petersen; +Cc: James Bottomley, ksummit-discuss On Tue, 18 Sep 2018 15:43:08 +0200, Martin K. Petersen wrote: > > > Hi Jiri, > > Sorry I'm a little late to the game here. Been out on vacation. > > > We order patches in our trees in the same git-topological-ordering as they > > are upstream. It has a lot of benefits, most importantly: it doesn't > > introduce artificial conflicts that don't exist in reality. > > > > In order to achieve that, we of course need 1:1 mapping between our > > patches and upstream commits. Rebases destroy that mapping. > > > > And in some areas (scsi is one, but not the only one), we basically had no > > other choice than considering maintainer's tree to be already "upstream > > enough", without waiting for Linus' tree merge. > > When I discussed this with Johannes a little while ago, I suggested you > guys used git patch-id to track patches instead of commit ids. That's > how we track patches applied across many different trees internally. > Works much better than using the upstream sha. What is git patch id? Is there any internal patch tag? > I would like to understand your "upstream enough" requirement. Why do > you need a tree that's stable before Linus pulls the changes? A few reasons: - We'd like to sort also the multiple would-be-upstream patches in the git topo order, which allows us to keep the patches closer to the original shape. - Once when such patches are merged to upstream, they are moved to the upstream section, again ordered. For moving them, of course it'd be easy if the commit ids are kept. And here the worst case is that the old commit ID disappears after the rebase. In that sense, at least, it'd be helpful if you can preserve the old branches via tag or another named branch in the repo; then the commits are not pruned, and we can still track the order of patches. > Note that I am generally only rebasing as a last resort and typically > only very early in the rc cycle. It usually happens when I need to drop > a patch series that turned out to be unfixable in its current state. ... and we shouldn't take such a patch series always as "would-be-upstream" thing, either. So it sounds like a sort of lack of communication to me; you might be interested in that a certain patch series are already pushed to distro kernels, and we are definitely interested in whether the patches are merged or dropped. Hmm... Takashi > And before everyone screams because I'm not supposed to be pushing stuff > that breaks, please realize that it is impossible to test all the > different types of hardware I have to merge drivers for. There is no > regression test suite or lab setup with anything resembling > comprehensive coverage. I test changes to the SCSI core code and do some > rudimentary testing on SAS and FC on x86_64. But that's really the best > I can do. > > Even though most patches posted to linux-scsi get picked up by 0day, > more often than not they only get x86_64 build coverage. Whereas 0day > build failures on arm, mips, sparc32, whatever typically only get > reported after patches have been simmering in linux-next for a > while. Depends how busy 0day is. > > Also, actual driver failures on platforms not officially supported and > tested by the controller vendor are only found after the fact. And most > of the time it's not a matter of reverting a single patch but > effectively dropping all of the patches in the series until they can be > reworked. Sometimes a workaround takes a week or two to deliver, and > people don't appreciate not being able to boot their systems in the > meantime. So that's why I generally drop the series instead. > > I would love for every patch sent to linux-scsi to be bug free and > instantly build tested by 0day on every architecture. And I would love > for hardware vendors to be more cognizant about architectures they don't > commercially support. But reality is that things break frequently when > I merge big, complex driver update patch series. > > As a result, the preference has been to have the flexibility to amend or > drop patches early in every cycle. It hasn't really been a problem > because there have been no downstream users of SCSI at all. I only > recently found out about your use case. > > I'm pretty flexible about how to address this, there are a couple of > ways to go about it. > > 1. I could just always revert instead of dropping the patches. The > downside is that we end up with a pretty messy history because, as I > pointed out above, it's usually a matter of dropping tens of patches > at a time and not reverting a single offending commit. In addition, > having a messy history makes it harder on distro kernel people to > track driver updates. > > 2. The other option is that I set up a scsi-staging tree where stuff can > simmer for a bit and hopefully get some 0day coverage before getting > shuffled over to scsi-queue. However, I do question how much actual > real hardware coverage we'll get by having a SCSI tree that people > would explicitly have to pull to test. As opposed to linux-next which > at least gets some coverage by test farms and users. > > 3. The third option is that we pick a number like rc4 and I will promise > not to rebase after that. We can see how that works out. > > I'm open to ideas. The most important thing to me is that 0day and > linux-next are indispensable tools in my workflow given that I have no > way to personally test most of the code I merge. So it is imperative > that I have the ability to push code early because that's where I get > the bulk of my (non-SCSI-core) build and test coverage. > > -- > Martin K. Petersen Oracle Linux Engineering > _______________________________________________ > Ksummit-discuss mailing list > Ksummit-discuss@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss > ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-18 20:37 ` Takashi Iwai @ 2018-09-19 6:16 ` Geert Uytterhoeven 2018-09-19 6:31 ` Takashi Iwai 0 siblings, 1 reply; 33+ messages in thread From: Geert Uytterhoeven @ 2018-09-19 6:16 UTC (permalink / raw) To: Takashi Iwai; +Cc: James Bottomley, ksummit-discuss Hi Iwai-san, On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote: > On Tue, 18 Sep 2018 15:43:08 +0200, > Martin K. Petersen wrote: > > Sorry I'm a little late to the game here. Been out on vacation. > > > > > We order patches in our trees in the same git-topological-ordering as they > > > are upstream. It has a lot of benefits, most importantly: it doesn't > > > introduce artificial conflicts that don't exist in reality. > > > > > > In order to achieve that, we of course need 1:1 mapping between our > > > patches and upstream commits. Rebases destroy that mapping. > > > > > > And in some areas (scsi is one, but not the only one), we basically had no > > > other choice than considering maintainer's tree to be already "upstream > > > enough", without waiting for Linus' tree merge. > > > > When I discussed this with Johannes a little while ago, I suggested you > > guys used git patch-id to track patches instead of commit ids. That's > > how we track patches applied across many different trees internally. > > Works much better than using the upstream sha. > > What is git patch id? Is there any internal patch tag? It's the output of "git patch-id", which is a hash of the patch, without including its dependencies. This allows to match identical commits in different branches, and is what "git cherry" and "git rebase" use to detect already applied patches. Example (from v4.18.y): $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 Author: Robert Munteanu <rombert@apache.org> Date: Tue Jun 19 11:20:40 2018 +0300 HID: redragon: fix num lock and caps lock LEDs commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream. The redragon asura keyboard registers two input devices. The initial commit 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura Keyboard") considered this an error and prevented one of the devices from registering. However, once this is done the num lock and caps lock leds no longer toggle on and off, although the key functionality is not affected. This commit removes the code that prevents the input device registration and restores the num lock and caps lock LEDs. Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura Keyboard") Signed-off-by: Robert Munteanu <rombert@apache.org> Signed-off-by: Jiri Kosina <jkosina@suse.cz> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id 9d4546a92b459de7249748c0a38b50c989f13bd1 e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id 9d4546a92b459de7249748c0a38b50c989f13bd1 dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e So both commits are identical, as they have the same patch ID. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-19 6:16 ` Geert Uytterhoeven @ 2018-09-19 6:31 ` Takashi Iwai 2018-09-19 9:23 ` Jan Kara 0 siblings, 1 reply; 33+ messages in thread From: Takashi Iwai @ 2018-09-19 6:31 UTC (permalink / raw) To: Geert Uytterhoeven; +Cc: James Bottomley, ksummit-discuss On Wed, 19 Sep 2018 08:16:17 +0200, Geert Uytterhoeven wrote: > > Hi Iwai-san, > > On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote: > > On Tue, 18 Sep 2018 15:43:08 +0200, > > Martin K. Petersen wrote: > > > Sorry I'm a little late to the game here. Been out on vacation. > > > > > > > We order patches in our trees in the same git-topological-ordering as they > > > > are upstream. It has a lot of benefits, most importantly: it doesn't > > > > introduce artificial conflicts that don't exist in reality. > > > > > > > > In order to achieve that, we of course need 1:1 mapping between our > > > > patches and upstream commits. Rebases destroy that mapping. > > > > > > > > And in some areas (scsi is one, but not the only one), we basically had no > > > > other choice than considering maintainer's tree to be already "upstream > > > > enough", without waiting for Linus' tree merge. > > > > > > When I discussed this with Johannes a little while ago, I suggested you > > > guys used git patch-id to track patches instead of commit ids. That's > > > how we track patches applied across many different trees internally. > > > Works much better than using the upstream sha. > > > > What is git patch id? Is there any internal patch tag? > > It's the output of "git patch-id", which is a hash of the patch, without > including its dependencies. This allows to match identical commits in > different branches, and is what "git cherry" and "git rebase" use to detect > already applied patches. > > Example (from v4.18.y): > > $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > Author: Robert Munteanu <rombert@apache.org> > Date: Tue Jun 19 11:20:40 2018 +0300 > > HID: redragon: fix num lock and caps lock LEDs > > commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream. > > The redragon asura keyboard registers two input devices. The initial commit > 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura > Keyboard") > considered this an error and prevented one of the devices from registering. > However, once this is done the num lock and caps lock leds no > longer toggle on > and off, although the key functionality is not affected. > > This commit removes the code that prevents the input device > registration and restores the num lock and caps lock LEDs. > > Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for > Redragon Asura Keyboard") > Signed-off-by: Robert Munteanu <rombert@apache.org> > Signed-off-by: Jiri Kosina <jkosina@suse.cz> > Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > > $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id > 9d4546a92b459de7249748c0a38b50c989f13bd1 > e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id > 9d4546a92b459de7249748c0a38b50c989f13bd1 > dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e > > So both commits are identical, as they have the same patch ID. Thanks for the explanation! This would help for identifying the rebased patch, yes. But it still doesn't help for sorting as is. You cannot get the topology information from the patch id, so you don't know the patch order, either. (You can deduce the actual commit from the patch id, but it's too heavy if you have 20,000 patches :) Takashi ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-19 6:31 ` Takashi Iwai @ 2018-09-19 9:23 ` Jan Kara 2018-09-19 9:27 ` Takashi Iwai 0 siblings, 1 reply; 33+ messages in thread From: Jan Kara @ 2018-09-19 9:23 UTC (permalink / raw) To: Takashi Iwai; +Cc: James Bottomley, ksummit-discuss On Wed 19-09-18 08:31:04, Takashi Iwai wrote: > On Wed, 19 Sep 2018 08:16:17 +0200, > Geert Uytterhoeven wrote: > > > > Hi Iwai-san, > > > > On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote: > > > On Tue, 18 Sep 2018 15:43:08 +0200, > > > Martin K. Petersen wrote: > > > > Sorry I'm a little late to the game here. Been out on vacation. > > > > > > > > > We order patches in our trees in the same git-topological-ordering as they > > > > > are upstream. It has a lot of benefits, most importantly: it doesn't > > > > > introduce artificial conflicts that don't exist in reality. > > > > > > > > > > In order to achieve that, we of course need 1:1 mapping between our > > > > > patches and upstream commits. Rebases destroy that mapping. > > > > > > > > > > And in some areas (scsi is one, but not the only one), we basically had no > > > > > other choice than considering maintainer's tree to be already "upstream > > > > > enough", without waiting for Linus' tree merge. > > > > > > > > When I discussed this with Johannes a little while ago, I suggested you > > > > guys used git patch-id to track patches instead of commit ids. That's > > > > how we track patches applied across many different trees internally. > > > > Works much better than using the upstream sha. > > > > > > What is git patch id? Is there any internal patch tag? > > > > It's the output of "git patch-id", which is a hash of the patch, without > > including its dependencies. This allows to match identical commits in > > different branches, and is what "git cherry" and "git rebase" use to detect > > already applied patches. > > > > Example (from v4.18.y): > > > > $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > Author: Robert Munteanu <rombert@apache.org> > > Date: Tue Jun 19 11:20:40 2018 +0300 > > > > HID: redragon: fix num lock and caps lock LEDs > > > > commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream. > > > > The redragon asura keyboard registers two input devices. The initial commit > > 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura > > Keyboard") > > considered this an error and prevented one of the devices from registering. > > However, once this is done the num lock and caps lock leds no > > longer toggle on > > and off, although the key functionality is not affected. > > > > This commit removes the code that prevents the input device > > registration and restores the num lock and caps lock LEDs. > > > > Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for > > Redragon Asura Keyboard") > > Signed-off-by: Robert Munteanu <rombert@apache.org> > > Signed-off-by: Jiri Kosina <jkosina@suse.cz> > > Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > > > > $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id > > 9d4546a92b459de7249748c0a38b50c989f13bd1 > > e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id > > 9d4546a92b459de7249748c0a38b50c989f13bd1 > > dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e > > > > So both commits are identical, as they have the same patch ID. > > Thanks for the explanation! > > This would help for identifying the rebased patch, yes. But it still > doesn't help for sorting as is. You cannot get the topology > information from the patch id, so you don't know the patch order, > either. (You can deduce the actual commit from the patch id, but it's > too heavy if you have 20,000 patches :) Actually we could use the patch id to identify new commit id of a rebased patch and infer proper ordering once we have the new commit id. So I think using patch id (additionally to commit id) would really help us and solve 95% of rebases of e.g. SCSI tree by our automation without human interaction. Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-19 9:23 ` Jan Kara @ 2018-09-19 9:27 ` Takashi Iwai 0 siblings, 0 replies; 33+ messages in thread From: Takashi Iwai @ 2018-09-19 9:27 UTC (permalink / raw) To: Jan Kara; +Cc: James Bottomley, ksummit-discuss On Wed, 19 Sep 2018 11:23:43 +0200, Jan Kara wrote: > > On Wed 19-09-18 08:31:04, Takashi Iwai wrote: > > On Wed, 19 Sep 2018 08:16:17 +0200, > > Geert Uytterhoeven wrote: > > > > > > Hi Iwai-san, > > > > > > On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote: > > > > On Tue, 18 Sep 2018 15:43:08 +0200, > > > > Martin K. Petersen wrote: > > > > > Sorry I'm a little late to the game here. Been out on vacation. > > > > > > > > > > > We order patches in our trees in the same git-topological-ordering as they > > > > > > are upstream. It has a lot of benefits, most importantly: it doesn't > > > > > > introduce artificial conflicts that don't exist in reality. > > > > > > > > > > > > In order to achieve that, we of course need 1:1 mapping between our > > > > > > patches and upstream commits. Rebases destroy that mapping. > > > > > > > > > > > > And in some areas (scsi is one, but not the only one), we basically had no > > > > > > other choice than considering maintainer's tree to be already "upstream > > > > > > enough", without waiting for Linus' tree merge. > > > > > > > > > > When I discussed this with Johannes a little while ago, I suggested you > > > > > guys used git patch-id to track patches instead of commit ids. That's > > > > > how we track patches applied across many different trees internally. > > > > > Works much better than using the upstream sha. > > > > > > > > What is git patch id? Is there any internal patch tag? > > > > > > It's the output of "git patch-id", which is a hash of the patch, without > > > including its dependencies. This allows to match identical commits in > > > different branches, and is what "git cherry" and "git rebase" use to detect > > > already applied patches. > > > > > > Example (from v4.18.y): > > > > > > $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > > commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > > Author: Robert Munteanu <rombert@apache.org> > > > Date: Tue Jun 19 11:20:40 2018 +0300 > > > > > > HID: redragon: fix num lock and caps lock LEDs > > > > > > commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream. > > > > > > The redragon asura keyboard registers two input devices. The initial commit > > > 85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura > > > Keyboard") > > > considered this an error and prevented one of the devices from registering. > > > However, once this is done the num lock and caps lock leds no > > > longer toggle on > > > and off, although the key functionality is not affected. > > > > > > This commit removes the code that prevents the input device > > > registration and restores the num lock and caps lock LEDs. > > > > > > Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for > > > Redragon Asura Keyboard") > > > Signed-off-by: Robert Munteanu <rombert@apache.org> > > > Signed-off-by: Jiri Kosina <jkosina@suse.cz> > > > Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > > > > > > $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id > > > 9d4546a92b459de7249748c0a38b50c989f13bd1 > > > e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 > > > $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id > > > 9d4546a92b459de7249748c0a38b50c989f13bd1 > > > dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e > > > > > > So both commits are identical, as they have the same patch ID. > > > > Thanks for the explanation! > > > > This would help for identifying the rebased patch, yes. But it still > > doesn't help for sorting as is. You cannot get the topology > > information from the patch id, so you don't know the patch order, > > either. (You can deduce the actual commit from the patch id, but it's > > too heavy if you have 20,000 patches :) > > Actually we could use the patch id to identify new commit id of a rebased > patch and infer proper ordering once we have the new commit id. So I think > using patch id (additionally to commit id) would really help us and solve > 95% of rebases of e.g. SCSI tree by our automation without human > interaction. Yes, we can use the patch id in addition to the commit id as complementary information for tracking the upstream. I chatted with people here now, and it seems feasible. Thanks for the nice suggestion! Takashi ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 10:13 [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful James Bottomley 2018-09-05 11:37 ` Mark Brown @ 2018-09-05 13:16 ` Takashi Iwai 2018-09-05 13:20 ` Jiri Kosina 2018-09-05 13:39 ` Konstantin Ryabitsev 2018-09-05 17:41 ` Laura Abbott 2 siblings, 2 replies; 33+ messages in thread From: Takashi Iwai @ 2018-09-05 13:16 UTC (permalink / raw) To: James Bottomley; +Cc: ksummit-discuss On Wed, 05 Sep 2018 12:13:52 +0200, James Bottomley wrote: > > The first suggestion is that kernel builds are pretty much automated > and we try to make every commit buildable, so could we automate the > machinery that allows a customer to do bisection simply by installing a > kernel package? (we here, obviously means the distro, but going from > git bisect to kernel package would be the useful link). Oh yeah, that's a thing I've been dreaming for years! :) A part of problems is the build power: it still takes an hour or longer (depending on arch) to build a full distro kernel package, for example, on OBS. That's too painful for bisection. So some semi-automatic way to reduce the config would be required for usable bisection. I guess automatic make localmodconfig, prep the repo and kick off. Also some garbage collections for stale repos, etc. > Second suggestion is that the bugzillas need to say much more strongly > that the reporter really needs to confirm the fix in upstream and do > the bisection themselves (and ideally request the backport to stable > themselves). OK, distros definitely need to try hard not to annoy upstream devs. In the case of SUSE Kernel, we usually ask testing the latest (more-or-less) vanilla kernel at first. If it's an upstream problem, then it's often tossed to the upstream. If it's already addressed in the upstream kernel, we take the responsibility for backports. Asking bisection by reporter is usually the last resort. It'd be helpful if we get any suggestion to improve the process. thanks, Takashi ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 13:16 ` Takashi Iwai @ 2018-09-05 13:20 ` Jiri Kosina 2018-09-05 13:39 ` Konstantin Ryabitsev 1 sibling, 0 replies; 33+ messages in thread From: Jiri Kosina @ 2018-09-05 13:20 UTC (permalink / raw) To: Takashi Iwai; +Cc: James Bottomley, ksummit-discuss On Wed, 5 Sep 2018, Takashi Iwai wrote: > Asking bisection by reporter is usually the last resort. > > It'd be helpful if we get any suggestion to improve the process. IIRC, years ago someone started on "web-based bisection tool", but that was never finished. The idea was that the user would just click "Good" and "Bad" buttons, and would be getting kernel RPMs to test. If this would have been picked up and finished (GSoC perhaps?), it might help a lot with this aspect. -- Jiri Kosina SUSE Labs ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 13:16 ` Takashi Iwai 2018-09-05 13:20 ` Jiri Kosina @ 2018-09-05 13:39 ` Konstantin Ryabitsev 2018-09-05 15:16 ` Sasha Levin 2018-09-05 16:44 ` Laura Abbott 1 sibling, 2 replies; 33+ messages in thread From: Konstantin Ryabitsev @ 2018-09-05 13:39 UTC (permalink / raw) To: Takashi Iwai; +Cc: James Bottomley, ksummit-discuss On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote: > > Second suggestion is that the bugzillas need to say much more strongly > > that the reporter really needs to confirm the fix in upstream and do > > the bisection themselves (and ideally request the backport to stable > > themselves). > > OK, distros definitely need to try hard not to annoy upstream devs. > > In the case of SUSE Kernel, we usually ask testing the latest > (more-or-less) vanilla kernel at first. If it's an upstream problem, > then it's often tossed to the upstream. If it's already addressed in > the upstream kernel, we take the responsibility for backports. Asking > bisection by reporter is usually the last resort. > > It'd be helpful if we get any suggestion to improve the process. It would be awesome to have a "bisect@home" type of thing with a similar idea like seti@home and folding@home. Have a central queue where developers can submit upstream commits and testcases, and a swarm of volunteer drones would grab and bisect-build them until the bug-introducing commit is identified and reported back. I'll totally host the hell out of this. -K ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 13:39 ` Konstantin Ryabitsev @ 2018-09-05 15:16 ` Sasha Levin 2018-09-05 16:44 ` Laura Abbott 1 sibling, 0 replies; 33+ messages in thread From: Sasha Levin @ 2018-09-05 15:16 UTC (permalink / raw) To: Takashi Iwai, James Bottomley, ksummit-discuss On Wed, Sep 05, 2018 at 09:39:16AM -0400, Konstantin Ryabitsev wrote: >On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote: >> > Second suggestion is that the bugzillas need to say much more strongly >> > that the reporter really needs to confirm the fix in upstream and do >> > the bisection themselves (and ideally request the backport to stable >> > themselves). >> >> OK, distros definitely need to try hard not to annoy upstream devs. >> >> In the case of SUSE Kernel, we usually ask testing the latest >> (more-or-less) vanilla kernel at first. If it's an upstream problem, >> then it's often tossed to the upstream. If it's already addressed in >> the upstream kernel, we take the responsibility for backports. Asking >> bisection by reporter is usually the last resort. >> >> It'd be helpful if we get any suggestion to improve the process. > >It would be awesome to have a "bisect@home" type of thing with a similar >idea like seti@home and folding@home. Have a central queue where >developers can submit upstream commits and testcases, and a swarm of >volunteer drones would grab and bisect-build them until the >bug-introducing commit is identified and reported back. > >I'll totally host the hell out of this. I don't think KernelCI are too far on this front. They're already working on auto bisection, so if there's an ability to add (simple) testcases you'll be all set. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 13:39 ` Konstantin Ryabitsev 2018-09-05 15:16 ` Sasha Levin @ 2018-09-05 16:44 ` Laura Abbott 2018-09-05 20:15 ` Konstantin Ryabitsev 1 sibling, 1 reply; 33+ messages in thread From: Laura Abbott @ 2018-09-05 16:44 UTC (permalink / raw) To: Takashi Iwai, James Bottomley, ksummit-discuss On 09/05/2018 06:39 AM, Konstantin Ryabitsev wrote: > On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote: >>> Second suggestion is that the bugzillas need to say much more strongly >>> that the reporter really needs to confirm the fix in upstream and do >>> the bisection themselves (and ideally request the backport to stable >>> themselves). >> >> OK, distros definitely need to try hard not to annoy upstream devs. >> >> In the case of SUSE Kernel, we usually ask testing the latest >> (more-or-less) vanilla kernel at first. If it's an upstream problem, >> then it's often tossed to the upstream. If it's already addressed in >> the upstream kernel, we take the responsibility for backports. Asking >> bisection by reporter is usually the last resort. >> >> It'd be helpful if we get any suggestion to improve the process. > > It would be awesome to have a "bisect@home" type of thing with a similar > idea like seti@home and folding@home. Have a central queue where > developers can submit upstream commits and testcases, and a swarm of > volunteer drones would grab and bisect-build them until the > bug-introducing commit is identified and reported back. > > I'll totally host the hell out of this. > I played around with bisect scripts for Fedora a while ago but mostly abandoned them due to lack of interest and fragility. I was attempting to use the full Fedora patch set and spec file but making that work for arbitrary kernel commits was a pain. Developers usually have no problem building and bisecting kernels, it's non-kernel developers who often struggle with bisection. One idea that I haven't followed up on was to extend the existing targets for building distro packages to just build the source side of things and then take advantage of existing environments (e.g. COPR) to build the package binaries. I'd love a web interface that would handle some of this automatically but, again, lack of resources and knowledge of web frameworks. Thanks, Laura ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 16:44 ` Laura Abbott @ 2018-09-05 20:15 ` Konstantin Ryabitsev 2018-09-05 20:36 ` Takashi Iwai 2018-09-07 20:24 ` Mauro Carvalho Chehab 0 siblings, 2 replies; 33+ messages in thread From: Konstantin Ryabitsev @ 2018-09-05 20:15 UTC (permalink / raw) To: Laura Abbott; +Cc: James Bottomley, ksummit-discuss On Wed, Sep 05, 2018 at 09:44:23AM -0700, Laura Abbott wrote: >>It would be awesome to have a "bisect@home" type of thing with a >>similar >>idea like seti@home and folding@home. Have a central queue where >>developers can submit upstream commits and testcases, and a swarm of >>volunteer drones would grab and bisect-build them until the >>bug-introducing commit is identified and reported back. >> >>I'll totally host the hell out of this. >> >Developers usually have no problem building and bisecting kernels, >it's non-kernel developers who often struggle with bisection. >One idea that I haven't followed up on was to extend the existing >targets for building distro packages to just build the source >side of things and then take advantage of existing environments >(e.g. COPR) to build the package binaries. I'd love a web interface >that would handle some of this automatically but, again, lack of >resources and knowledge of web frameworks. I'm excited that kernelci.org is coming on board as a full-fledged Linux Foundation project, since I'm hoping that their charter would include funding this kind of development. [1] I was already chatting with Kevin about some of the cool things we could do to make various CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting as a service" to my list of suggestions for the glorious kernel CI tool of the future. :) -K [1] I don't bring this up too often, but kernel.org's official organizational purpose is "to distribute the Linux kernel and other Open Source software to the public without charge" -- so drifting too far away from such activity could potentially jeopardize its legal standing as far as non-profit filing paperwork is concerned. This is why we can organize funding for things like public-inbox and lore.kernel.org, but not other things that wouldn't be overall related to "distributing the Linux kernel free of charge." ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 20:15 ` Konstantin Ryabitsev @ 2018-09-05 20:36 ` Takashi Iwai 2018-09-07 20:24 ` Mauro Carvalho Chehab 1 sibling, 0 replies; 33+ messages in thread From: Takashi Iwai @ 2018-09-05 20:36 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: James Bottomley, ksummit-discuss On Wed, 05 Sep 2018 22:15:33 +0200, Konstantin Ryabitsev wrote: > > On Wed, Sep 05, 2018 at 09:44:23AM -0700, Laura Abbott wrote: > >> It would be awesome to have a "bisect@home" type of thing with a > >> similar > >>idea like seti@home and folding@home. Have a central queue where > >>developers can submit upstream commits and testcases, and a swarm of > >>volunteer drones would grab and bisect-build them until the > >>bug-introducing commit is identified and reported back. > >> > >>I'll totally host the hell out of this. > >> > >Developers usually have no problem building and bisecting kernels, > >it's non-kernel developers who often struggle with bisection. > >One idea that I haven't followed up on was to extend the existing > >targets for building distro packages to just build the source > >side of things and then take advantage of existing environments > >(e.g. COPR) to build the package binaries. I'd love a web interface > >that would handle some of this automatically but, again, lack of > >resources and knowledge of web frameworks. > > I'm excited that kernelci.org is coming on board as a full-fledged > Linux Foundation project, since I'm hoping that their charter would > include funding this kind of development. [1] I was already chatting > with Kevin about some of the cool things we could do to make various > CI/fuzzing/bug-reporting tools more streamlined, so I'll add > "bisecting as a service" to my list of suggestions for the glorious > kernel CI tool of the future. :) That sounds promising, thanks! Takashi ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 20:15 ` Konstantin Ryabitsev 2018-09-05 20:36 ` Takashi Iwai @ 2018-09-07 20:24 ` Mauro Carvalho Chehab 1 sibling, 0 replies; 33+ messages in thread From: Mauro Carvalho Chehab @ 2018-09-07 20:24 UTC (permalink / raw) To: Konstantin Ryabitsev; +Cc: James Bottomley, ksummit-discuss Em Wed, 5 Sep 2018 16:15:33 -0400 Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu: > I'm excited that kernelci.org is coming on board as a full-fledged Linux > Foundation project, since I'm hoping that their charter would include > funding this kind of development. [1] I was already chatting with Kevin > about some of the cool things we could do to make various > CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting > as a service" to my list of suggestions for the glorious kernel CI tool > of the future. :) Kernel-ci seems interesting. Yet, 1/3 of the Kernel patches are drivers. At least for media stuff, IMO, CI is too far away from our needs, as one would need some real hardware to do some testing stuff (like for example, some webcams and some ways to check if the webcam support is actually working). To make it worse, there are a wide variety of different hardware with different features and their quirk needs. I suspect that similar troubles would affect sound devices as well. It could be worth to have some discussions with CI people with regards to how it can better cope with driver's subsystem needs. Thanks, Mauro ^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful 2018-09-05 10:13 [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful James Bottomley 2018-09-05 11:37 ` Mark Brown 2018-09-05 13:16 ` Takashi Iwai @ 2018-09-05 17:41 ` Laura Abbott 2 siblings, 0 replies; 33+ messages in thread From: Laura Abbott @ 2018-09-05 17:41 UTC (permalink / raw) To: James Bottomley, ksummit-discuss On 09/05/2018 03:13 AM, James Bottomley wrote: > I'm seeing a lot of wasted effort by our customers on kernel bugs and > trying to engage the distribution to fix them. As a caveat, I'm > working in the cloud, so the distributions in question are usually > community ones not enterprise ones. However, we do have a fair few > customers on LTS kernels from Distributions. > > Mostly they find a cloud performance regression, they try to engage the > distro, spend ages working on it or submitting bugs and usually end up > with an unsatisfactory result. By the time they call my team in, we've > likely only got a week to fix the issue. However, step one is always > confirming whether upstream works (95% of the time it does) and then > finding the fix by bisection (usually assisted by knowledge of where > the bug is). To do the bisection we usually have to build a kernel > package with our guesses and get them to try it, so it can be a bit > slow. Once we have the backport, we send it to stable and notify the > distribution to include it in their next kernel release. > > Here's the rub: community distributions (even LTS ones) don't have the > resources even to triage cloud bugs in environments they likely can't > reproduce, so we really need to develop assistive tools for customers > to perform bisections to identify what caused the bug or (in the 95% > case) what fixed it. Having a bugzilla and using it as first line of > support implies a service expectation (usually coming from Enterprise) > that simply isn't met, so distributions need to fix this at the point > of interaction: bugzilla. > > The first suggestion is that kernel builds are pretty much automated > and we try to make every commit buildable, so could we automate the > machinery that allows a customer to do bisection simply by installing a > kernel package? (we here, obviously means the distro, but going from > git bisect to kernel package would be the useful link). > As mentioned further down, I did try having scripts to do this but it was ultimately fairly fragile. I think a better approach is leveraging the existing in tree support for building a distro package and using that with regular git bisect. > Second suggestion is that the bugzillas need to say much more strongly > that the reporter really needs to confirm the fix in upstream and do > the bisection themselves (and ideally request the backport to stable > themselves). > At least in Fedora this is something we hit fairly frequently. We do strongly encourage people to report bugs and bisect. This runs into a number of problems: - Bisecting on local machines is slow and people often don't want to give up their machine resources. - If people give me a test case I can reproduce, I'm usually okay to run a bisect myself but it's pretty rare to get a test case. - People are hesitant to run bisections and build kernels. There's a lot of steps involved. We try and point people to wiki pages with instructions but many times we end up having to go back and forth explaining how to do the setup. - People are hesitant to report bugs to the upstream. I end up having to explain where to report bugs or run get_maintainer.pl for people otherwise they just file it against kernel.org bugzilla or just e-mail LKML. I started a skeleton of a web project to make a web interface for get_maintainer.pl but it never got very far. Tooling can certainly help with a lot of this but some of this may also just be more documentation and needing to guide people. Thanks, Laura ^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2018-09-19 9:27 UTC | newest] Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-09-05 10:13 [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful James Bottomley 2018-09-05 11:37 ` Mark Brown 2018-09-05 15:03 ` Paul E. McKenney 2018-09-05 15:50 ` Steven Rostedt 2018-09-05 16:20 ` Paul E. McKenney 2018-09-05 16:45 ` James Bottomley 2018-09-05 17:00 ` Paul E. McKenney 2018-09-05 19:25 ` Jiri Kosina 2018-09-05 19:40 ` James Bottomley 2018-09-06 19:54 ` Jiri Kosina 2018-09-18 13:43 ` Martin K. Petersen 2018-09-18 14:12 ` Geert Uytterhoeven 2018-09-18 15:01 ` Martin K. Petersen 2018-09-18 15:27 ` Christoph Hellwig 2018-09-18 15:34 ` Jens Axboe 2018-09-18 17:08 ` Mark Brown 2018-09-18 16:12 ` Mark Brown 2018-09-18 20:20 ` Takashi Iwai 2018-09-19 0:08 ` Mark Brown 2018-09-18 20:37 ` Takashi Iwai 2018-09-19 6:16 ` Geert Uytterhoeven 2018-09-19 6:31 ` Takashi Iwai 2018-09-19 9:23 ` Jan Kara 2018-09-19 9:27 ` Takashi Iwai 2018-09-05 13:16 ` Takashi Iwai 2018-09-05 13:20 ` Jiri Kosina 2018-09-05 13:39 ` Konstantin Ryabitsev 2018-09-05 15:16 ` Sasha Levin 2018-09-05 16:44 ` Laura Abbott 2018-09-05 20:15 ` Konstantin Ryabitsev 2018-09-05 20:36 ` Takashi Iwai 2018-09-07 20:24 ` Mauro Carvalho Chehab 2018-09-05 17:41 ` Laura Abbott
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.