* [LSF/MM TOPIC] FS, MM, and stable trees @ 2019-02-12 17:00 Sasha Levin 2019-02-12 21:32 ` Steve French 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2019-02-12 17:00 UTC (permalink / raw) To: lsf-pc; +Cc: linux-fsdevel, linux-mm, linux-kernel Hi all, I'd like to propose a discussion about the workflow of the stable trees when it comes to fs/ and mm/. In the past year we had some friction with regards to the policies and the procedures around picking patches for stable tree, and I feel it would be very useful to establish better flow with the folks who might be attending LSF/MM. I feel that fs/ and mm/ are in very different places with regards to which patches go in -stable, what tests are expected, and the timeline of patches from the point they are proposed on a mailing list to the point they are released in a stable tree. Therefore, I'd like to propose two different sessions on this (one for fs/ and one for mm/), as a common session might be less conductive to agreeing on a path forward as the starting point for both subsystems are somewhat different. We can go through the existing processes, automation, and testing mechanisms we employ when building stable trees, and see how we can improve these to address the concerns of fs/ and mm/ folks. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-12 17:00 [LSF/MM TOPIC] FS, MM, and stable trees Sasha Levin @ 2019-02-12 21:32 ` Steve French 2019-02-13 7:20 ` Amir Goldstein 0 siblings, 1 reply; 19+ messages in thread From: Steve French @ 2019-02-12 21:32 UTC (permalink / raw) To: Sasha Levin; +Cc: lsf-pc, linux-fsdevel, linux-mm, LKML Makes sense - e.g. I would like to have a process to make automation of the xfstests for proposed patches for stable for cifs.ko easier and part of the process (as we already do for cifs/smb3 related checkins to for-next ie linux next before sending to mainline for cifs.ko). Each filesystem has a different set of xfstests (and perhaps other mechanisms) to run so might be very specific to each file system, but would be helpful to discuss On Tue, Feb 12, 2019 at 9:32 AM Sasha Levin <sashal@kernel.org> wrote: > > Hi all, > > I'd like to propose a discussion about the workflow of the stable trees > when it comes to fs/ and mm/. In the past year we had some friction with > regards to the policies and the procedures around picking patches for > stable tree, and I feel it would be very useful to establish better flow > with the folks who might be attending LSF/MM. > > I feel that fs/ and mm/ are in very different places with regards to > which patches go in -stable, what tests are expected, and the timeline > of patches from the point they are proposed on a mailing list to the > point they are released in a stable tree. Therefore, I'd like to propose > two different sessions on this (one for fs/ and one for mm/), as a > common session might be less conductive to agreeing on a path forward as > the starting point for both subsystems are somewhat different. > > We can go through the existing processes, automation, and testing > mechanisms we employ when building stable trees, and see how we can > improve these to address the concerns of fs/ and mm/ folks. > > -- > Thanks, > Sasha -- Thanks, Steve ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-12 21:32 ` Steve French @ 2019-02-13 7:20 ` Amir Goldstein 2019-02-13 7:37 ` Greg KH 0 siblings, 1 reply; 19+ messages in thread From: Amir Goldstein @ 2019-02-13 7:20 UTC (permalink / raw) To: Steve French, Sasha Levin Cc: lsf-pc, linux-fsdevel, linux-mm, LKML, Greg KH, Luis R. Rodriguez On Tue, Feb 12, 2019 at 11:56 PM Steve French <smfrench@gmail.com> wrote: > > Makes sense - e.g. I would like to have a process to make automation > of the xfstests for proposed patches for stable for cifs.ko easier and > part of the process (as we already do for cifs/smb3 related checkins > to for-next ie linux next before sending to mainline for cifs.ko). > Each filesystem has a different set of xfstests (and perhaps other > mechanisms) to run so might be very specific to each file system, but > would be helpful to discuss > Agreed. Perhaps it is just a matter of communicating the stable tree workflow. I currently only see notice emails from Greg about patches being queued for stable. I never saw an email from you or Greg saying, the branch "stable-xxx" is in review. Please run your tests. I have seen reports from LTP about stable kernels, so I know it is being run regularly and I recently saw the set of xfstests configurations that Sasha and Luis posted. Is there any publicly available information about which tests are being run on stable candidate branches? Thanks, Amir. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 7:20 ` Amir Goldstein @ 2019-02-13 7:37 ` Greg KH 2019-02-13 9:01 ` Amir Goldstein 0 siblings, 1 reply; 19+ messages in thread From: Greg KH @ 2019-02-13 7:37 UTC (permalink / raw) To: Amir Goldstein Cc: Steve French, Sasha Levin, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 09:20:00AM +0200, Amir Goldstein wrote: > I never saw an email from you or Greg saying, the branch "stable-xxx" is > in review. Please run your tests. That is what my "Subject: [PATCH 4.9 000/137] 4.9.156-stable review" type emails are supposed to kick off. They are sent both to the stable mailing list and lkml. This message already starts the testing systems going for a number of different groups out there, do you want to be added to the cc: list so you get them directly? thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 7:37 ` Greg KH @ 2019-02-13 9:01 ` Amir Goldstein 2019-02-13 9:18 ` Greg KH 0 siblings, 1 reply; 19+ messages in thread From: Amir Goldstein @ 2019-02-13 9:01 UTC (permalink / raw) To: Greg KH Cc: Steve French, Sasha Levin, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 9:37 AM Greg KH <gregkh@linuxfoundation.org> wrote: > > On Wed, Feb 13, 2019 at 09:20:00AM +0200, Amir Goldstein wrote: > > I never saw an email from you or Greg saying, the branch "stable-xxx" is > > in review. Please run your tests. > > That is what my "Subject: [PATCH 4.9 000/137] 4.9.156-stable review" > type emails are supposed to kick off. They are sent both to the stable > mailing list and lkml. > > This message already starts the testing systems going for a number of > different groups out there, do you want to be added to the cc: list so > you get them directly? > No thanks, I'll fix my email filters ;-) I think the main difference between these review announcements and true CI is what kind of guaranty you get for a release candidate from NOT getting a test failure response, which is one of the main reasons that where holding back xfs stable fixes for so long. Best effort testing in timely manner is good, but a good way to improve confidence in stable kernel releases is a publicly available list of tests that the release went through. Do you have any such list of tests that you *know* are being run, that you (or Sasha) run yourself or that you actively wait on an ACK from a group before a release? Thanks, Amir. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 9:01 ` Amir Goldstein @ 2019-02-13 9:18 ` Greg KH 2019-02-13 19:25 ` Sasha Levin 0 siblings, 1 reply; 19+ messages in thread From: Greg KH @ 2019-02-13 9:18 UTC (permalink / raw) To: Amir Goldstein Cc: Steve French, Sasha Levin, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote: > I think the main difference between these review announcements > and true CI is what kind of guaranty you get for a release candidate > from NOT getting a test failure response, which is one of the main > reasons that where holding back xfs stable fixes for so long. That's not true, I know to wait for some responses before doing a release of these kernels. > Best effort testing in timely manner is good, but a good way to > improve confidence in stable kernel releases is a publicly > available list of tests that the release went through. We have that, you aren't noticing them... > Do you have any such list of tests that you *know* are being run, > that you (or Sasha) run yourself or that you actively wait on an > ACK from a group before a release? Yes, look at the responses to those messages from Guenter, Shuah, Jon, kernel.ci, Red Hat testing, the Linaro testing teams, and a few other testers that come and go over time. Those list out all of the tests that are being run, and the results of those tests. I also get a number of private responses from different build systems from companies that don't want to post in public, which is fine, I understand the issues involved with that. I would argue that the stable releases are better tested than Linus's releases for that reason alone :) thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 9:18 ` Greg KH @ 2019-02-13 19:25 ` Sasha Levin 2019-02-13 19:52 ` Greg KH 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2019-02-13 19:25 UTC (permalink / raw) To: Greg KH Cc: Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote: >On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote: >> Best effort testing in timely manner is good, but a good way to >> improve confidence in stable kernel releases is a publicly >> available list of tests that the release went through. > >We have that, you aren't noticing them... This is one of the biggest things I want to address: there is a disconnect between the stable kernel testing story and the tests the fs/ and mm/ folks expect to see here. On one had, the stable kernel folks see these kernels go through entire suites of testing by multiple individuals and organizations, receiving way more coverage than any of Linus's releases. On the other hand, things like LTP and selftests tend to barely scratch the surface of our mm/ and fs/ code, and the maintainers of these subsystems do not see LTP-like suites as something that adds significant value and ignore them. Instead, they have a (convoluted) set of testing they do with different tools and configurations that qualifies their code as being "tested". So really, it sounds like a low hanging fruit: we don't really need to write much more testing code code nor do we have to refactor existing test suites. We just need to make sure the right tests are running on stable kernels. I really want to clarify what each subsystem sees as "sufficient" (and have that documented somewhere). -- Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 19:25 ` Sasha Levin @ 2019-02-13 19:52 ` Greg KH 2019-02-13 20:14 ` James Bottomley 2019-03-20 3:46 ` Jon Masters 0 siblings, 2 replies; 19+ messages in thread From: Greg KH @ 2019-02-13 19:52 UTC (permalink / raw) To: Sasha Levin Cc: Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote: > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote: > > > Best effort testing in timely manner is good, but a good way to > > > improve confidence in stable kernel releases is a publicly > > > available list of tests that the release went through. > > > > We have that, you aren't noticing them... > > This is one of the biggest things I want to address: there is a > disconnect between the stable kernel testing story and the tests the fs/ > and mm/ folks expect to see here. > > On one had, the stable kernel folks see these kernels go through entire > suites of testing by multiple individuals and organizations, receiving > way more coverage than any of Linus's releases. > > On the other hand, things like LTP and selftests tend to barely scratch > the surface of our mm/ and fs/ code, and the maintainers of these > subsystems do not see LTP-like suites as something that adds significant > value and ignore them. Instead, they have a (convoluted) set of testing > they do with different tools and configurations that qualifies their > code as being "tested". > > So really, it sounds like a low hanging fruit: we don't really need to > write much more testing code code nor do we have to refactor existing > test suites. We just need to make sure the right tests are running on > stable kernels. I really want to clarify what each subsystem sees as > "sufficient" (and have that documented somewhere). kernel.ci and 0-day and Linaro are starting to add the fs and mm tests to their test suites to address these issues (I think 0-day already has many of them). So this is happening, but not quite obvious. I know I keep asking Linaro about this :( Anyway, just having a list of what tests each subsystem things is "good to run" would be great to have somewhere. Ideally in the kernel tree itself, as that's what kselftests are for :) thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 19:52 ` Greg KH @ 2019-02-13 20:14 ` James Bottomley 2019-02-15 1:50 ` Sasha Levin 2019-03-20 3:46 ` Jon Masters 1 sibling, 1 reply; 19+ messages in thread From: James Bottomley @ 2019-02-13 20:14 UTC (permalink / raw) To: Greg KH, Sasha Levin Cc: Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote: > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: > > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote: > > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote: > > > > Best effort testing in timely manner is good, but a good way to > > > > improve confidence in stable kernel releases is a publicly > > > > available list of tests that the release went through. > > > > > > We have that, you aren't noticing them... > > > > This is one of the biggest things I want to address: there is a > > disconnect between the stable kernel testing story and the tests > > the fs/ and mm/ folks expect to see here. > > > > On one had, the stable kernel folks see these kernels go through > > entire suites of testing by multiple individuals and organizations, > > receiving way more coverage than any of Linus's releases. > > > > On the other hand, things like LTP and selftests tend to barely > > scratch the surface of our mm/ and fs/ code, and the maintainers of > > these subsystems do not see LTP-like suites as something that adds > > significant value and ignore them. Instead, they have a > > (convoluted) set of testing they do with different tools and > > configurations that qualifies their code as being "tested". > > > > So really, it sounds like a low hanging fruit: we don't really need > > to write much more testing code code nor do we have to refactor > > existing test suites. We just need to make sure the right tests are > > running on stable kernels. I really want to clarify what each > > subsystem sees as "sufficient" (and have that documented > > somewhere). > > kernel.ci and 0-day and Linaro are starting to add the fs and mm > tests to their test suites to address these issues (I think 0-day > already has many of them). So this is happening, but not quite > obvious. I know I keep asking Linaro about this :( 0day has xfstests at least, but it's opt-in only (you have to request that it be run on your trees). When I did it for the SCSI tree, I had to email Fenguangg directly, there wasn't any other way of getting it. James ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 20:14 ` James Bottomley @ 2019-02-15 1:50 ` Sasha Levin 2019-02-15 2:48 ` James Bottomley 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2019-02-15 1:50 UTC (permalink / raw) To: James Bottomley Cc: Greg KH, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote: >On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote: >> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: >> > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote: >> > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote: >> > > > Best effort testing in timely manner is good, but a good way to >> > > > improve confidence in stable kernel releases is a publicly >> > > > available list of tests that the release went through. >> > > >> > > We have that, you aren't noticing them... >> > >> > This is one of the biggest things I want to address: there is a >> > disconnect between the stable kernel testing story and the tests >> > the fs/ and mm/ folks expect to see here. >> > >> > On one had, the stable kernel folks see these kernels go through >> > entire suites of testing by multiple individuals and organizations, >> > receiving way more coverage than any of Linus's releases. >> > >> > On the other hand, things like LTP and selftests tend to barely >> > scratch the surface of our mm/ and fs/ code, and the maintainers of >> > these subsystems do not see LTP-like suites as something that adds >> > significant value and ignore them. Instead, they have a >> > (convoluted) set of testing they do with different tools and >> > configurations that qualifies their code as being "tested". >> > >> > So really, it sounds like a low hanging fruit: we don't really need >> > to write much more testing code code nor do we have to refactor >> > existing test suites. We just need to make sure the right tests are >> > running on stable kernels. I really want to clarify what each >> > subsystem sees as "sufficient" (and have that documented >> > somewhere). >> >> kernel.ci and 0-day and Linaro are starting to add the fs and mm >> tests to their test suites to address these issues (I think 0-day >> already has many of them). So this is happening, but not quite >> obvious. I know I keep asking Linaro about this :( > >0day has xfstests at least, but it's opt-in only (you have to request >that it be run on your trees). When I did it for the SCSI tree, I had >to email Fenguangg directly, there wasn't any other way of getting it. It's very tricky to do even if someone would just run it. I worked with the xfs folks for quite a while to gather the various configs they want to use, and to establish the baseline for a few of the stable trees (some tests are know to fail, etc). So just running xfstests "blindly" doesn't add much value beyond ltp I think. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-15 1:50 ` Sasha Levin @ 2019-02-15 2:48 ` James Bottomley 2019-02-16 18:28 ` Theodore Y. Ts'o 0 siblings, 1 reply; 19+ messages in thread From: James Bottomley @ 2019-02-15 2:48 UTC (permalink / raw) To: Sasha Levin Cc: Greg KH, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Thu, 2019-02-14 at 20:50 -0500, Sasha Levin wrote: > On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote: > > On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote: > > > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: > > > > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote: > > > > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein > > > > > wrote: > > > > > > Best effort testing in timely manner is good, but a good > > > > > > way to improve confidence in stable kernel releases is a > > > > > > publicly available list of tests that the release went > > > > > > through. > > > > > > > > > > We have that, you aren't noticing them... > > > > > > > > This is one of the biggest things I want to address: there is a > > > > disconnect between the stable kernel testing story and the > > > > tests the fs/ and mm/ folks expect to see here. > > > > > > > > On one had, the stable kernel folks see these kernels go > > > > through entire suites of testing by multiple individuals and > > > > organizations, receiving way more coverage than any of Linus's > > > > releases. > > > > > > > > On the other hand, things like LTP and selftests tend to barely > > > > scratch the surface of our mm/ and fs/ code, and the > > > > maintainers of these subsystems do not see LTP-like suites as > > > > something that adds significant value and ignore them. Instead, > > > > they have a (convoluted) set of testing they do with different > > > > tools and configurations that qualifies their code as being > > > > "tested". > > > > > > > > So really, it sounds like a low hanging fruit: we don't really > > > > need to write much more testing code code nor do we have to > > > > refactor existing test suites. We just need to make sure the > > > > right tests are running on stable kernels. I really want to > > > > clarify what each subsystem sees as "sufficient" (and have that > > > > documented somewhere). > > > > > > kernel.ci and 0-day and Linaro are starting to add the fs and mm > > > tests to their test suites to address these issues (I think 0-day > > > already has many of them). So this is happening, but not quite > > > obvious. I know I keep asking Linaro about this :( > > > > 0day has xfstests at least, but it's opt-in only (you have to > > request that it be run on your trees). When I did it for the SCSI > > tree, I had to email Fenguangg directly, there wasn't any other way > > of getting it. > > It's very tricky to do even if someone would just run it. It is? It's a test suite, so you just run it and it exercises standard and growing set of regression tests. > I worked with the xfs folks for quite a while to gather the various > configs they want to use, and to establish the baseline for a few of > the stable trees (some tests are know to fail, etc). The only real config issue is per-fs non-standard tests (features specific to a given filesystem). I just want it to exercise the storage underneath, so the SCSI tree is configured for the default set on xfs. > So just running xfstests "blindly" doesn't add much value beyond ltp > I think. Well, we differ on the value of running regression tests, then. The whole point of a test infrastructure is that it's simple to run 'make check' in autoconf parlance. xfstests does provide a useful baseline set of regression tests. However, since my goal is primarily to detect problems in the storage path rather than the filesystem, the utility is exercising that path, although I fully appreciate that filesystem regression tests aren't going to catch every SCSI issue, they do provide some level of assurance against bugs. Hopefully we can switch over to blktests when it's ready, but in the meantime xfstests is way better than nothing. James ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-15 2:48 ` James Bottomley @ 2019-02-16 18:28 ` Theodore Y. Ts'o 2019-02-21 15:34 ` Luis Chamberlain 0 siblings, 1 reply; 19+ messages in thread From: Theodore Y. Ts'o @ 2019-02-16 18:28 UTC (permalink / raw) To: James Bottomley Cc: Sasha Levin, Greg KH, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Thu, Feb 14, 2019 at 06:48:22PM -0800, James Bottomley wrote: > Well, we differ on the value of running regression tests, then. The > whole point of a test infrastructure is that it's simple to run 'make > check' in autoconf parlance. xfstests does provide a useful baseline > set of regression tests. However, since my goal is primarily to detect > problems in the storage path rather than the filesystem, the utility is > exercising that path, although I fully appreciate that filesystem > regression tests aren't going to catch every SCSI issue, they do > provide some level of assurance against bugs. > > Hopefully we can switch over to blktests when it's ready, but in the > meantime xfstests is way better than nothing. blktests isn't yet comprehensive, but I think there's value in running blktests as well as xfstests. I've been integrating blktests into {kvm,gce}-xfstets because if the problem is caused to some regression introduced in the block layer, I'm not wasting time trying to figure out if it's caused by the block layer or not. It won't catch everything, but at least it has some value... The block/*, loop/* and scsi/* tests in blktests do seem to be in pretty good shape. The nvme, nvmeof, and srp tests are *definitely* not as mature. - Ted ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-16 18:28 ` Theodore Y. Ts'o @ 2019-02-21 15:34 ` Luis Chamberlain 2019-02-21 18:52 ` Theodore Y. Ts'o 0 siblings, 1 reply; 19+ messages in thread From: Luis Chamberlain @ 2019-02-21 15:34 UTC (permalink / raw) To: Theodore Y. Ts'o, James Bottomley, Sasha Levin, Greg KH, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote: > The block/*, loop/* and scsi/* tests in blktests do seem to be in > pretty good shape. The nvme, nvmeof, and srp tests are *definitely* > not as mature. Can you say more about this later part. What would you like to see more of for nvme tests for instance? It sounds like a productive session would include tracking our: a) sour spots b) who's already working on these c) gather volutneers for these sour spots Luis ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-21 15:34 ` Luis Chamberlain @ 2019-02-21 18:52 ` Theodore Y. Ts'o 0 siblings, 0 replies; 19+ messages in thread From: Theodore Y. Ts'o @ 2019-02-21 18:52 UTC (permalink / raw) To: Luis Chamberlain Cc: James Bottomley, Sasha Levin, Greg KH, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML On Thu, Feb 21, 2019 at 07:34:15AM -0800, Luis Chamberlain wrote: > On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote: > > The block/*, loop/* and scsi/* tests in blktests do seem to be in > > pretty good shape. The nvme, nvmeof, and srp tests are *definitely* > > not as mature. > > Can you say more about this later part. What would you like to see more > of for nvme tests for instance? > > It sounds like a productive session would include tracking our: > > a) sour spots > b) who's already working on these > c) gather volutneers for these sour spots I complained on another LSF/MM topic thread, but there are a lot of failures where it's not clear whether it's because I guessed incorrectly about which version of nvme-cli I should be using (debian stable and head of nvme-cli both are apparently wrong answers), or kernel bugs or kernel misconfiguration issues on my side. Current nvme/* failures that I'm still seeing are attached below. - Ted nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [failed] runtime ... 100.265s something found in dmesg: [ 1857.188083] run blktests nvme/012 at 2019-02-12 01:11:33 [ 1857.437322] nvmet: adding nsid 1 to subsystem blktests-subsystem-1 [ 1857.456187] nvmet: creating controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:78dc695a-2c99-4841-968d-c2c16a49a02a. [ 1857.458162] nvme nvme0: ANA group 1: optimized. [ 1857.458257] nvme nvme0: creating 2 I/O queues. [ 1857.460893] nvme nvme0: new ctrl: "blktests-subsystem-1" [ 1857.720666] ============================================ [ 1857.726308] WARNING: possible recursive locking detected [ 1857.731784] 5.0.0-rc3-xfstests-00014-g1236f7d60242 #843 Not tainted ... (See '/results/nodev/nvme/012.dmesg' for the entire message) nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [failed] runtime ... 32.634s --- tests/nvme/013.out 2019-02-11 18:57:39.000000000 -0500 +++ /results/nodev/nvme/013.out.bad 2019-02-12 01:13:46.708757206 -0500 @@ -1,5 +1,9 @@ Running nvme/013 91fdba0d-f87b-4c25-b80f-db7be1418b9e uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=329326592, buflen=4096 +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=467435520, buflen=4096 +fio exited with status 0 +4;fio-3.2;verify;0;5;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;192672;6182;1546;31166;4;9044;63.332763;57.979218;482;29948;10268.332290;1421.459893;1.000000%=4145;5.000000%=9109;10.000000%=9502;20.000000%=9764;30.000000%=10027;40.000000%=10289;50.000000%=10420;60.000000%=10551;70.000000%=10682;80.000000%=10682;90.000000%=10944;95.000000%=11206;99.000000%=13172;99.500000%=16318;99.900000%=24510;99.950000%=27394;99.990000%=29229;0%=0;0%=0;0%=0;507;30005;10331.973087;1421.131712;6040;8232;100.000000%;6189.741935;267.091495;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;3.991657%;6.484839%;91142;0;1024;0.1%;0.1%;0.1%;0.1%;100.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.20%;0.34%;0.25%;0.19%;27.86%;70.89%;0.23%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n1;0;0;0;0;0;0;0;0.00% ... (Run 'diff -u tests/nvme/013.out /results/nodev/nvme/013.out.bad' to see the entire diff) nvme/015 (unit test for NVMe flush for file backed ns) [failed] runtime ... 8.914s --- tests/nvme/015.out 2019-02-11 18:57:39.000000000 -0500 +++ /results/nodev/nvme/015.out.bad 2019-02-12 01:14:05.429328259 -0500 @@ -1,6 +1,6 @@ Running nvme/015 91fdba0d-f87b-4c25-b80f-db7be1418b9e uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e -NVMe Flush: success +NVME IO command error:INTERNAL: The command was not completed successfully due to an internal error(6006) NQN:blktests-subsystem-1 disconnected 1 controller(s) Test complete nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) runtime ... nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [failed] runtime ... 23.576s --- tests/nvme/016.out 2019-02-11 18:57:39.000000000 -0500 +++ /results/nodev/nvme/016.out.bad 2019-02-12 01:14:29.173378854 -0500 @@ -1,11 +1,11 @@ Running nvme/016 -Discovery Log Number of Records 1, Generation counter 1 +Discovery Log Number of Records 1, Generation counter 5 =====Discovery Log Entry 0====== trtype: loop adrfam: pci ... (Run 'diff -u tests/nvme/016.out /results/nodev/nvme/016.out.bad' to see the entire diff) nvme/017 (create/delete many file-ns and test discovery) runtime ... nvme/017 (create/delete many file-ns and test discovery) [failed] runtime ... 23.592s --- tests/nvme/017.out 2019-02-11 18:57:39.000000000 -0500 +++ /results/nodev/nvme/017.out.bad 2019-02-12 01:14:52.880762691 -0500 @@ -1,11 +1,11 @@ Running nvme/017 -Discovery Log Number of Records 1, Generation counter 1 +Discovery Log Number of Records 1, Generation counter 2 =====Discovery Log Entry 0====== trtype: loop adrfam: pci ... (Run 'diff -u tests/nvme/017.out /results/nodev/nvme/017.out.bad' to see the entire diff) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-02-13 19:52 ` Greg KH 2019-02-13 20:14 ` James Bottomley @ 2019-03-20 3:46 ` Jon Masters 2019-03-20 5:06 ` Greg KH 1 sibling, 1 reply; 19+ messages in thread From: Jon Masters @ 2019-03-20 3:46 UTC (permalink / raw) To: Greg KH, Sasha Levin Cc: Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On 2/13/19 2:52 PM, Greg KH wrote: > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: >> So really, it sounds like a low hanging fruit: we don't really need to >> write much more testing code code nor do we have to refactor existing >> test suites. We just need to make sure the right tests are running on >> stable kernels. I really want to clarify what each subsystem sees as >> "sufficient" (and have that documented somewhere). > > kernel.ci and 0-day and Linaro are starting to add the fs and mm tests > to their test suites to address these issues (I think 0-day already has > many of them). So this is happening, but not quite obvious. I know I > keep asking Linaro about this :( We're working on investments for LDCG[0] in 2019 that include kernel CI changes for server use cases. Please keep us informed of what you folks ultimately want to see, and I'll pass on to the steering committee too. Ultimately I've been pushing for a kernel 0-day project for Arm. That's probably going to require a lot of duplicated effort since the original 0-day project isn't open, but creating an open one could help everyone. Jon. [0] Linaro DataCenter Group (formerly "LEG") ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-03-20 3:46 ` Jon Masters @ 2019-03-20 5:06 ` Greg KH 2019-03-20 6:14 ` Jon Masters 0 siblings, 1 reply; 19+ messages in thread From: Greg KH @ 2019-03-20 5:06 UTC (permalink / raw) To: Jon Masters Cc: Sasha Levin, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote: > On 2/13/19 2:52 PM, Greg KH wrote: > > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: > > >> So really, it sounds like a low hanging fruit: we don't really need to > >> write much more testing code code nor do we have to refactor existing > >> test suites. We just need to make sure the right tests are running on > >> stable kernels. I really want to clarify what each subsystem sees as > >> "sufficient" (and have that documented somewhere). > > > > kernel.ci and 0-day and Linaro are starting to add the fs and mm tests > > to their test suites to address these issues (I think 0-day already has > > many of them). So this is happening, but not quite obvious. I know I > > keep asking Linaro about this :( > > We're working on investments for LDCG[0] in 2019 that include kernel CI > changes for server use cases. Please keep us informed of what you folks > ultimately want to see, and I'll pass on to the steering committee too. > > Ultimately I've been pushing for a kernel 0-day project for Arm. That's > probably going to require a lot of duplicated effort since the original > 0-day project isn't open, but creating an open one could help everyone. Why are you trying to duplicate it on your own? That's what kernel.ci should be doing, please join in and invest in that instead. It's an open source project with its own governance and needs sponsors, why waste time and money doing it all on your own? thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-03-20 5:06 ` Greg KH @ 2019-03-20 6:14 ` Jon Masters 2019-03-20 6:28 ` Greg KH 0 siblings, 1 reply; 19+ messages in thread From: Jon Masters @ 2019-03-20 6:14 UTC (permalink / raw) To: Greg KH Cc: Sasha Levin, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On 3/20/19 1:06 AM, Greg KH wrote: > On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote: >> On 2/13/19 2:52 PM, Greg KH wrote: >>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: >> >>>> So really, it sounds like a low hanging fruit: we don't really need to >>>> write much more testing code code nor do we have to refactor existing >>>> test suites. We just need to make sure the right tests are running on >>>> stable kernels. I really want to clarify what each subsystem sees as >>>> "sufficient" (and have that documented somewhere). >>> >>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests >>> to their test suites to address these issues (I think 0-day already has >>> many of them). So this is happening, but not quite obvious. I know I >>> keep asking Linaro about this :( >> >> We're working on investments for LDCG[0] in 2019 that include kernel CI >> changes for server use cases. Please keep us informed of what you folks >> ultimately want to see, and I'll pass on to the steering committee too. >> >> Ultimately I've been pushing for a kernel 0-day project for Arm. That's >> probably going to require a lot of duplicated effort since the original >> 0-day project isn't open, but creating an open one could help everyone. > > Why are you trying to duplicate it on your own? That's what kernel.ci > should be doing, please join in and invest in that instead. It's an > open source project with its own governance and needs sponsors, why > waste time and money doing it all on your own? To clarify, I'm pushing for investment in kernel.ci to achieve that goal that it could provide the same 0-day capability for Arm and others. It'll ultimately result in duplicated effort vs if 0-day were open. Jon. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-03-20 6:14 ` Jon Masters @ 2019-03-20 6:28 ` Greg KH 2019-03-20 6:32 ` Jon Masters 0 siblings, 1 reply; 19+ messages in thread From: Greg KH @ 2019-03-20 6:28 UTC (permalink / raw) To: Jon Masters Cc: Sasha Levin, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On Wed, Mar 20, 2019 at 02:14:09AM -0400, Jon Masters wrote: > On 3/20/19 1:06 AM, Greg KH wrote: > > On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote: > >> On 2/13/19 2:52 PM, Greg KH wrote: > >>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: > >> > >>>> So really, it sounds like a low hanging fruit: we don't really need to > >>>> write much more testing code code nor do we have to refactor existing > >>>> test suites. We just need to make sure the right tests are running on > >>>> stable kernels. I really want to clarify what each subsystem sees as > >>>> "sufficient" (and have that documented somewhere). > >>> > >>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests > >>> to their test suites to address these issues (I think 0-day already has > >>> many of them). So this is happening, but not quite obvious. I know I > >>> keep asking Linaro about this :( > >> > >> We're working on investments for LDCG[0] in 2019 that include kernel CI > >> changes for server use cases. Please keep us informed of what you folks > >> ultimately want to see, and I'll pass on to the steering committee too. > >> > >> Ultimately I've been pushing for a kernel 0-day project for Arm. That's > >> probably going to require a lot of duplicated effort since the original > >> 0-day project isn't open, but creating an open one could help everyone. > > > > Why are you trying to duplicate it on your own? That's what kernel.ci > > should be doing, please join in and invest in that instead. It's an > > open source project with its own governance and needs sponsors, why > > waste time and money doing it all on your own? > > To clarify, I'm pushing for investment in kernel.ci to achieve that goal > that it could provide the same 0-day capability for Arm and others. Great, that's what I was trying to suggest :) > It'll ultimately result in duplicated effort vs if 0-day were open. "Half" of 0-day is open, but it's that other half that is still needed... thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [LSF/MM TOPIC] FS, MM, and stable trees 2019-03-20 6:28 ` Greg KH @ 2019-03-20 6:32 ` Jon Masters 0 siblings, 0 replies; 19+ messages in thread From: Jon Masters @ 2019-03-20 6:32 UTC (permalink / raw) To: Greg KH Cc: Sasha Levin, Amir Goldstein, Steve French, lsf-pc, linux-fsdevel, linux-mm, LKML, Luis R. Rodriguez On 3/20/19 2:28 AM, Greg KH wrote: > On Wed, Mar 20, 2019 at 02:14:09AM -0400, Jon Masters wrote: >> On 3/20/19 1:06 AM, Greg KH wrote: >>> On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote: >>>> On 2/13/19 2:52 PM, Greg KH wrote: >>>>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote: >>>> >>>>>> So really, it sounds like a low hanging fruit: we don't really need to >>>>>> write much more testing code code nor do we have to refactor existing >>>>>> test suites. We just need to make sure the right tests are running on >>>>>> stable kernels. I really want to clarify what each subsystem sees as >>>>>> "sufficient" (and have that documented somewhere). >>>>> >>>>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests >>>>> to their test suites to address these issues (I think 0-day already has >>>>> many of them). So this is happening, but not quite obvious. I know I >>>>> keep asking Linaro about this :( >>>> >>>> We're working on investments for LDCG[0] in 2019 that include kernel CI >>>> changes for server use cases. Please keep us informed of what you folks >>>> ultimately want to see, and I'll pass on to the steering committee too. >>>> >>>> Ultimately I've been pushing for a kernel 0-day project for Arm. That's >>>> probably going to require a lot of duplicated effort since the original >>>> 0-day project isn't open, but creating an open one could help everyone. >>> >>> Why are you trying to duplicate it on your own? That's what kernel.ci >>> should be doing, please join in and invest in that instead. It's an >>> open source project with its own governance and needs sponsors, why >>> waste time and money doing it all on your own? >> >> To clarify, I'm pushing for investment in kernel.ci to achieve that goal >> that it could provide the same 0-day capability for Arm and others. > > Great, that's what I was trying to suggest :) > >> It'll ultimately result in duplicated effort vs if 0-day were open. > > "Half" of 0-day is open, but it's that other half that is still > needed... ;) I'm hoping this might also help that to happen... Best, Jon. ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2019-03-20 6:32 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-02-12 17:00 [LSF/MM TOPIC] FS, MM, and stable trees Sasha Levin 2019-02-12 21:32 ` Steve French 2019-02-13 7:20 ` Amir Goldstein 2019-02-13 7:37 ` Greg KH 2019-02-13 9:01 ` Amir Goldstein 2019-02-13 9:18 ` Greg KH 2019-02-13 19:25 ` Sasha Levin 2019-02-13 19:52 ` Greg KH 2019-02-13 20:14 ` James Bottomley 2019-02-15 1:50 ` Sasha Levin 2019-02-15 2:48 ` James Bottomley 2019-02-16 18:28 ` Theodore Y. Ts'o 2019-02-21 15:34 ` Luis Chamberlain 2019-02-21 18:52 ` Theodore Y. Ts'o 2019-03-20 3:46 ` Jon Masters 2019-03-20 5:06 ` Greg KH 2019-03-20 6:14 ` Jon Masters 2019-03-20 6:28 ` Greg KH 2019-03-20 6:32 ` Jon Masters
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).