[LSF/MM TOPIC] FS, MM, and stable trees

[LSF/MM TOPIC] FS, MM, and stable trees

[Sasha Levin]

Hi all,

I'd like to propose a discussion about the workflow of the stable trees
when it comes to fs/ and mm/. In the past year we had some friction with
regards to the policies and the procedures around picking patches for
stable tree, and I feel it would be very useful to establish better flow
with the folks who might be attending LSF/MM.

I feel that fs/ and mm/ are in very different places with regards to
which patches go in -stable, what tests are expected, and the timeline
of patches from the point they are proposed on a mailing list to the
point they are released in a stable tree. Therefore, I'd like to propose
two different sessions on this (one for fs/ and one for mm/), as a
common session might be less conductive to agreeing on a path forward as
the starting point for both subsystems are somewhat different.

We can go through the existing processes, automation, and testing
mechanisms we employ when building stable trees, and see how we can
improve these to address the concerns of fs/ and mm/ folks.

--
Thanks,
Sasha

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Steve French]

Makes sense - e.g. I would like to have a process to make automation
of the xfstests for proposed patches for stable for cifs.ko easier and
part of the process (as we already do for cifs/smb3 related checkins
to for-next ie linux next before sending to mainline for cifs.ko).
Each filesystem has a different set of xfstests (and perhaps other
mechanisms) to run so might be very specific to each file system, but
would be helpful to discuss

On Tue, Feb 12, 2019 at 9:32 AM Sasha Levin <sashal@kernel.org> wrote:
>
> Hi all,
>
> I'd like to propose a discussion about the workflow of the stable trees
> when it comes to fs/ and mm/. In the past year we had some friction with
> regards to the policies and the procedures around picking patches for
> stable tree, and I feel it would be very useful to establish better flow
> with the folks who might be attending LSF/MM.
>
> I feel that fs/ and mm/ are in very different places with regards to
> which patches go in -stable, what tests are expected, and the timeline
> of patches from the point they are proposed on a mailing list to the
> point they are released in a stable tree. Therefore, I'd like to propose
> two different sessions on this (one for fs/ and one for mm/), as a
> common session might be less conductive to agreeing on a path forward as
> the starting point for both subsystems are somewhat different.
>
> We can go through the existing processes, automation, and testing
> mechanisms we employ when building stable trees, and see how we can
> improve these to address the concerns of fs/ and mm/ folks.
>
> --
> Thanks,
> Sasha



-- 
Thanks,

Steve

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Amir Goldstein]

On Tue, Feb 12, 2019 at 11:56 PM Steve French <smfrench@gmail.com> wrote:
>
> Makes sense - e.g. I would like to have a process to make automation
> of the xfstests for proposed patches for stable for cifs.ko easier and
> part of the process (as we already do for cifs/smb3 related checkins
> to for-next ie linux next before sending to mainline for cifs.ko).
> Each filesystem has a different set of xfstests (and perhaps other
> mechanisms) to run so might be very specific to each file system, but
> would be helpful to discuss
>

Agreed.

Perhaps it is just a matter of communicating the stable tree workflow.
I currently only see notice emails from Greg about patches being queued
for stable.

I never saw an email from you or Greg saying, the branch "stable-xxx" is
in review. Please run your tests.

I have seen reports from LTP about stable kernels, so I know it is
being run regularly and I recently saw the set of xfstests configurations
that Sasha and Luis posted.

Is there any publicly available information about which tests are being run
on stable candidate branches?

Thanks,
Amir.

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Greg KH]

On Wed, Feb 13, 2019 at 09:20:00AM +0200, Amir Goldstein wrote:
> I never saw an email from you or Greg saying, the branch "stable-xxx" is
> in review. Please run your tests.

That is what my "Subject: [PATCH 4.9 000/137] 4.9.156-stable review"
type emails are supposed to kick off.  They are sent both to the stable
mailing list and lkml.

This message already starts the testing systems going for a number of
different groups out there, do you want to be added to the cc: list so
you get them directly?

thanks,

greg k-h

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Amir Goldstein]

On Wed, Feb 13, 2019 at 9:37 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Wed, Feb 13, 2019 at 09:20:00AM +0200, Amir Goldstein wrote:
> > I never saw an email from you or Greg saying, the branch "stable-xxx" is
> > in review. Please run your tests.
>
> That is what my "Subject: [PATCH 4.9 000/137] 4.9.156-stable review"
> type emails are supposed to kick off.  They are sent both to the stable
> mailing list and lkml.
>
> This message already starts the testing systems going for a number of
> different groups out there, do you want to be added to the cc: list so
> you get them directly?
>

No thanks, I'll fix my email filters ;-)

I think the main difference between these review announcements
and true CI is what kind of guaranty you get for a release candidate
from NOT getting a test failure response, which is one of the main
reasons that where holding back xfs stable fixes for so long.

Best effort testing in timely manner is good, but a good way to
improve confidence in stable kernel releases is a publicly
available list of tests that the release went through.

Do you have any such list of tests that you *know* are being run,
that you (or Sasha) run yourself or that you actively wait on an
ACK from a group before a release?

Thanks,
Amir.

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Greg KH]

On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
> I think the main difference between these review announcements
> and true CI is what kind of guaranty you get for a release candidate
> from NOT getting a test failure response, which is one of the main
> reasons that where holding back xfs stable fixes for so long.

That's not true, I know to wait for some responses before doing a
release of these kernels.

> Best effort testing in timely manner is good, but a good way to
> improve confidence in stable kernel releases is a publicly
> available list of tests that the release went through.

We have that, you aren't noticing them...

> Do you have any such list of tests that you *know* are being run,
> that you (or Sasha) run yourself or that you actively wait on an
> ACK from a group before a release?

Yes, look at the responses to those messages from Guenter, Shuah, Jon,
kernel.ci, Red Hat testing, the Linaro testing teams, and a few other
testers that come and go over time.  Those list out all of the tests
that are being run, and the results of those tests.

I also get a number of private responses from different build systems
from companies that don't want to post in public, which is fine, I
understand the issues involved with that.

I would argue that the stable releases are better tested than Linus's
releases for that reason alone :)

thanks,

greg k-h

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Sasha Levin]

On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
>On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
>> Best effort testing in timely manner is good, but a good way to
>> improve confidence in stable kernel releases is a publicly
>> available list of tests that the release went through.
>
>We have that, you aren't noticing them...

This is one of the biggest things I want to address: there is a
disconnect between the stable kernel testing story and the tests the fs/
and mm/ folks expect to see here.

On one had, the stable kernel folks see these kernels go through entire
suites of testing by multiple individuals and organizations, receiving
way more coverage than any of Linus's releases.

On the other hand, things like LTP and selftests tend to barely scratch
the surface of our mm/ and fs/ code, and the maintainers of these
subsystems do not see LTP-like suites as something that adds significant
value and ignore them. Instead, they have a (convoluted) set of testing
they do with different tools and configurations that qualifies their
code as being "tested".

So really, it sounds like a low hanging fruit: we don't really need to
write much more testing code code nor do we have to refactor existing
test suites. We just need to make sure the right tests are running on
stable kernels. I really want to clarify what each subsystem sees as
"sufficient" (and have that documented somewhere).

--
Thanks,
Sasha

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Greg KH]

On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
> > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
> > > Best effort testing in timely manner is good, but a good way to
> > > improve confidence in stable kernel releases is a publicly
> > > available list of tests that the release went through.
> > 
> > We have that, you aren't noticing them...
> 
> This is one of the biggest things I want to address: there is a
> disconnect between the stable kernel testing story and the tests the fs/
> and mm/ folks expect to see here.
> 
> On one had, the stable kernel folks see these kernels go through entire
> suites of testing by multiple individuals and organizations, receiving
> way more coverage than any of Linus's releases.
> 
> On the other hand, things like LTP and selftests tend to barely scratch
> the surface of our mm/ and fs/ code, and the maintainers of these
> subsystems do not see LTP-like suites as something that adds significant
> value and ignore them. Instead, they have a (convoluted) set of testing
> they do with different tools and configurations that qualifies their
> code as being "tested".
> 
> So really, it sounds like a low hanging fruit: we don't really need to
> write much more testing code code nor do we have to refactor existing
> test suites. We just need to make sure the right tests are running on
> stable kernels. I really want to clarify what each subsystem sees as
> "sufficient" (and have that documented somewhere).

kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
to their test suites to address these issues (I think 0-day already has
many of them).  So this is happening, but not quite obvious.  I know I
keep asking Linaro about this :(

Anyway, just having a list of what tests each subsystem things is "good
to run" would be great to have somewhere.  Ideally in the kernel tree
itself, as that's what kselftests are for :)

thanks,

greg k-h

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[James Bottomley]

On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote:
> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
> > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
> > > > Best effort testing in timely manner is good, but a good way to
> > > > improve confidence in stable kernel releases is a publicly
> > > > available list of tests that the release went through.
> > > 
> > > We have that, you aren't noticing them...
> > 
> > This is one of the biggest things I want to address: there is a
> > disconnect between the stable kernel testing story and the tests
> > the fs/ and mm/ folks expect to see here.
> > 
> > On one had, the stable kernel folks see these kernels go through
> > entire suites of testing by multiple individuals and organizations,
> > receiving way more coverage than any of Linus's releases.
> > 
> > On the other hand, things like LTP and selftests tend to barely
> > scratch the surface of our mm/ and fs/ code, and the maintainers of
> > these subsystems do not see LTP-like suites as something that adds
> > significant value and ignore them. Instead, they have a
> > (convoluted) set of testing they do with different tools and
> > configurations that qualifies their code as being "tested".
> > 
> > So really, it sounds like a low hanging fruit: we don't really need
> > to write much more testing code code nor do we have to refactor
> > existing test suites. We just need to make sure the right tests are
> > running on stable kernels. I really want to clarify what each
> > subsystem sees as "sufficient" (and have that documented
> > somewhere).
> 
> kernel.ci and 0-day and Linaro are starting to add the fs and mm
> tests to their test suites to address these issues (I think 0-day
> already has many of them).  So this is happening, but not quite
> obvious.  I know I keep asking Linaro about this :(

0day has xfstests at least, but it's opt-in only (you have to request
that it be run on your trees).  When I did it for the SCSI tree, I had
to email Fenguangg directly, there wasn't any other way of getting it.

James

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Sasha Levin]

On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote:
>On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote:
>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
>> > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
>> > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein wrote:
>> > > > Best effort testing in timely manner is good, but a good way to
>> > > > improve confidence in stable kernel releases is a publicly
>> > > > available list of tests that the release went through.
>> > >
>> > > We have that, you aren't noticing them...
>> >
>> > This is one of the biggest things I want to address: there is a
>> > disconnect between the stable kernel testing story and the tests
>> > the fs/ and mm/ folks expect to see here.
>> >
>> > On one had, the stable kernel folks see these kernels go through
>> > entire suites of testing by multiple individuals and organizations,
>> > receiving way more coverage than any of Linus's releases.
>> >
>> > On the other hand, things like LTP and selftests tend to barely
>> > scratch the surface of our mm/ and fs/ code, and the maintainers of
>> > these subsystems do not see LTP-like suites as something that adds
>> > significant value and ignore them. Instead, they have a
>> > (convoluted) set of testing they do with different tools and
>> > configurations that qualifies their code as being "tested".
>> >
>> > So really, it sounds like a low hanging fruit: we don't really need
>> > to write much more testing code code nor do we have to refactor
>> > existing test suites. We just need to make sure the right tests are
>> > running on stable kernels. I really want to clarify what each
>> > subsystem sees as "sufficient" (and have that documented
>> > somewhere).
>>
>> kernel.ci and 0-day and Linaro are starting to add the fs and mm
>> tests to their test suites to address these issues (I think 0-day
>> already has many of them).  So this is happening, but not quite
>> obvious.  I know I keep asking Linaro about this :(
>
>0day has xfstests at least, but it's opt-in only (you have to request
>that it be run on your trees).  When I did it for the SCSI tree, I had
>to email Fenguangg directly, there wasn't any other way of getting it.

It's very tricky to do even if someone would just run it. I worked with
the xfs folks for quite a while to gather the various configs they want
to use, and to establish the baseline for a few of the stable trees
(some tests are know to fail, etc).

So just running xfstests "blindly" doesn't add much value beyond ltp I
think.

--
Thanks,
Sasha

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[James Bottomley]

On Thu, 2019-02-14 at 20:50 -0500, Sasha Levin wrote:
> On Wed, Feb 13, 2019 at 12:14:35PM -0800, James Bottomley wrote:
> > On Wed, 2019-02-13 at 20:52 +0100, Greg KH wrote:
> > > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> > > > On Wed, Feb 13, 2019 at 10:18:03AM +0100, Greg KH wrote:
> > > > > On Wed, Feb 13, 2019 at 11:01:25AM +0200, Amir Goldstein
> > > > > wrote:
> > > > > > Best effort testing in timely manner is good, but a good
> > > > > > way to improve confidence in stable kernel releases is a
> > > > > > publicly available list of tests that the release went
> > > > > > through.
> > > > > 
> > > > > We have that, you aren't noticing them...
> > > > 
> > > > This is one of the biggest things I want to address: there is a
> > > > disconnect between the stable kernel testing story and the
> > > > tests the fs/ and mm/ folks expect to see here.
> > > > 
> > > > On one had, the stable kernel folks see these kernels go
> > > > through entire suites of testing by multiple individuals and
> > > > organizations, receiving way more coverage than any of Linus's
> > > > releases.
> > > > 
> > > > On the other hand, things like LTP and selftests tend to barely
> > > > scratch the surface of our mm/ and fs/ code, and the
> > > > maintainers of these subsystems do not see LTP-like suites as
> > > > something that adds significant value and ignore them. Instead,
> > > > they have a (convoluted) set of testing they do with different
> > > > tools and configurations that qualifies their code as being
> > > > "tested".
> > > > 
> > > > So really, it sounds like a low hanging fruit: we don't really
> > > > need to write much more testing code code nor do we have to
> > > > refactor existing test suites. We just need to make sure the
> > > > right tests are running on stable kernels. I really want to
> > > > clarify what each subsystem sees as "sufficient" (and have that
> > > > documented somewhere).
> > > 
> > > kernel.ci and 0-day and Linaro are starting to add the fs and mm
> > > tests to their test suites to address these issues (I think 0-day
> > > already has many of them).  So this is happening, but not quite
> > > obvious.  I know I keep asking Linaro about this :(
> > 
> > 0day has xfstests at least, but it's opt-in only (you have to
> > request that it be run on your trees).  When I did it for the SCSI
> > tree, I had to email Fenguangg directly, there wasn't any other way
> > of getting it.
> 
> It's very tricky to do even if someone would just run it.

It is?  It's a test suite, so you just run it and it exercises standard
and growing set of regression tests.

>  I worked with the xfs folks for quite a while to gather the various
> configs they want to use, and to establish the baseline for a few of
> the stable trees (some tests are know to fail, etc).

The only real config issue is per-fs non-standard tests (features
specific to a given filesystem).  I just want it to exercise the
storage underneath, so the SCSI tree is configured for the default set
on xfs.

> So just running xfstests "blindly" doesn't add much value beyond ltp
> I think.

Well, we differ on the value of running regression tests, then.  The
whole point of a test infrastructure is that it's simple to run 'make
check' in autoconf parlance.  xfstests does provide a useful baseline
set of regression tests.  However, since my goal is primarily to detect
problems in the storage path rather than the filesystem, the utility is
exercising that path, although I fully appreciate that filesystem
regression tests aren't going to catch every SCSI issue, they do
provide some level of assurance against bugs.

Hopefully we can switch over to blktests when it's ready, but in the
meantime xfstests is way better than nothing.

James

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Theodore Y. Ts'o]

On Thu, Feb 14, 2019 at 06:48:22PM -0800, James Bottomley wrote:
> Well, we differ on the value of running regression tests, then.  The
> whole point of a test infrastructure is that it's simple to run 'make
> check' in autoconf parlance.  xfstests does provide a useful baseline
> set of regression tests.  However, since my goal is primarily to detect
> problems in the storage path rather than the filesystem, the utility is
> exercising that path, although I fully appreciate that filesystem
> regression tests aren't going to catch every SCSI issue, they do
> provide some level of assurance against bugs.
> 
> Hopefully we can switch over to blktests when it's ready, but in the
> meantime xfstests is way better than nothing.

blktests isn't yet comprehensive, but I think there's value in running
blktests as well as xfstests.  I've been integrating blktests into
{kvm,gce}-xfstets because if the problem is caused to some regression
introduced in the block layer, I'm not wasting time trying to figure
out if it's caused by the block layer or not.  It won't catch
everything, but at least it has some value...

The block/*, loop/* and scsi/* tests in blktests do seem to be in
pretty good shape.  The nvme, nvmeof, and srp tests are *definitely*
not as mature.

				- Ted

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Luis Chamberlain]

On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote:
> The block/*, loop/* and scsi/* tests in blktests do seem to be in
> pretty good shape.  The nvme, nvmeof, and srp tests are *definitely*
> not as mature.

Can you say more about this later part. What would you like to see more
of for nvme tests for instance?

It sounds like a productive session would include tracking our:

  a) sour spots
  b) who's already working on these
  c) gather volutneers for these sour spots

 Luis

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Theodore Y. Ts'o]

On Thu, Feb 21, 2019 at 07:34:15AM -0800, Luis Chamberlain wrote:
> On Sat, Feb 16, 2019 at 01:28:35PM -0500, Theodore Y. Ts'o wrote:
> > The block/*, loop/* and scsi/* tests in blktests do seem to be in
> > pretty good shape.  The nvme, nvmeof, and srp tests are *definitely*
> > not as mature.
> 
> Can you say more about this later part. What would you like to see more
> of for nvme tests for instance?
> 
> It sounds like a productive session would include tracking our:
> 
>   a) sour spots
>   b) who's already working on these
>   c) gather volutneers for these sour spots

I complained on another LSF/MM topic thread, but there are a lot of
failures where it's not clear whether it's because I guessed
incorrectly about which version of nvme-cli I should be using (debian
stable and head of nvme-cli both are apparently wrong answers), or
kernel bugs or kernel misconfiguration issues on my side.

Current nvme/* failures that I'm still seeing are attached below.

	       		     	       - Ted

nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [failed]
    runtime  ...  100.265s
    something found in dmesg:
    [ 1857.188083] run blktests nvme/012 at 2019-02-12 01:11:33
    [ 1857.437322] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
    [ 1857.456187] nvmet: creating controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:78dc695a-2c99-4841-968d-c2c16a49a02a.
    [ 1857.458162] nvme nvme0: ANA group 1: optimized.
    [ 1857.458257] nvme nvme0: creating 2 I/O queues.
    [ 1857.460893] nvme nvme0: new ctrl: "blktests-subsystem-1"
    
    [ 1857.720666] ============================================
    [ 1857.726308] WARNING: possible recursive locking detected
    [ 1857.731784] 5.0.0-rc3-xfstests-00014-g1236f7d60242 #843 Not tainted
    ...
    (See '/results/nodev/nvme/012.dmesg' for the entire message)
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed ns) [failed]
    runtime  ...  32.634s
    --- tests/nvme/013.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/013.out.bad	2019-02-12 01:13:46.708757206 -0500
    @@ -1,5 +1,9 @@
     Running nvme/013
     91fdba0d-f87b-4c25-b80f-db7be1418b9e
     uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
    +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=329326592, buflen=4096
    +fio: io_u error on file /mnt/blktests///verify.0.0: Input/output error: write offset=467435520, buflen=4096
    +fio exited with status 0
    +4;fio-3.2;verify;0;5;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;192672;6182;1546;31166;4;9044;63.332763;57.979218;482;29948;10268.332290;1421.459893;1.000000%=4145;5.000000%=9109;10.000000%=9502;20.000000%=9764;30.000000%=10027;40.000000%=10289;50.000000%=10420;60.000000%=10551;70.000000%=10682;80.000000%=10682;90.000000%=10944;95.000000%=11206;99.000000%=13172;99.500000%=16318;99.900000%=24510;99.950000%=27394;99.990000%=29229;0%=0;0%=0;0%=0;507;30005;10331.973087;1421.131712;6040;8232;100.000000%;6189.741935;267.091495;0;0;0;0;0;0;0.000000;0.000000;0;0;0.000000;0.000000;1.000000%=0;5.000000%=0;10.000000%=0;20.000000%=0;30.000000%=0;40.000000%=0;50.000000%=0;60.000000%=0;70.000000%=0;80.000000%=0;90.000000%=0;95.000000%=0;99.000000%=0;99.500000%=0;99.900000%=0;99.950000%=0;99.990000%=0;0%=0;0%=0;0%=0;0;0;0.000000;0.000000;0;0;0.000000%;0.000000;0.000000;3.991657%;6.484839%;91142;0;1024;0.1%;0.1%;0.1%;0.1%;100.0%;0.0%;0.0%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.01%;0.20%;0.34%;0.25%;0.19%;27.86%;70.89%;0.23%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;0.00%;nvme0n1;0;0;0;0;0;0;0;0.00%
    ...
    (Run 'diff -u tests/nvme/013.out /results/nodev/nvme/013.out.bad' to see the entire diff)
nvme/015 (unit test for NVMe flush for file backed ns)       [failed]
    runtime  ...  8.914s
    --- tests/nvme/015.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/015.out.bad	2019-02-12 01:14:05.429328259 -0500
    @@ -1,6 +1,6 @@
     Running nvme/015
     91fdba0d-f87b-4c25-b80f-db7be1418b9e
     uuid.91fdba0d-f87b-4c25-b80f-db7be1418b9e
    -NVMe Flush: success
    +NVME IO command error:INTERNAL: The command was not completed successfully due to an internal error(6006)
     NQN:blktests-subsystem-1 disconnected 1 controller(s)
     Test complete
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery)
    runtime  ...
nvme/016 (create/delete many NVMeOF block device-backed ns and test discovery) [failed]
    runtime  ...  23.576s
    --- tests/nvme/016.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/016.out.bad	2019-02-12 01:14:29.173378854 -0500
    @@ -1,11 +1,11 @@
     Running nvme/016
     
    -Discovery Log Number of Records 1, Generation counter 1
    +Discovery Log Number of Records 1, Generation counter 5
     =====Discovery Log Entry 0======
     trtype:  loop
     adrfam:  pci
    ...
    (Run 'diff -u tests/nvme/016.out /results/nodev/nvme/016.out.bad' to see the entire diff)
nvme/017 (create/delete many file-ns and test discovery)    
    runtime  ...
nvme/017 (create/delete many file-ns and test discovery)     [failed]
    runtime  ...  23.592s
    --- tests/nvme/017.out	2019-02-11 18:57:39.000000000 -0500
    +++ /results/nodev/nvme/017.out.bad	2019-02-12 01:14:52.880762691 -0500
    @@ -1,11 +1,11 @@
     Running nvme/017
     
    -Discovery Log Number of Records 1, Generation counter 1
    +Discovery Log Number of Records 1, Generation counter 2
     =====Discovery Log Entry 0======
     trtype:  loop
     adrfam:  pci
    ...
    (Run 'diff -u tests/nvme/017.out /results/nodev/nvme/017.out.bad' to see the entire diff)

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Jon Masters]

On 2/13/19 2:52 PM, Greg KH wrote:
> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:

>> So really, it sounds like a low hanging fruit: we don't really need to
>> write much more testing code code nor do we have to refactor existing
>> test suites. We just need to make sure the right tests are running on
>> stable kernels. I really want to clarify what each subsystem sees as
>> "sufficient" (and have that documented somewhere).
> 
> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
> to their test suites to address these issues (I think 0-day already has
> many of them).  So this is happening, but not quite obvious.  I know I
> keep asking Linaro about this :(

We're working on investments for LDCG[0] in 2019 that include kernel CI
changes for server use cases. Please keep us informed of what you folks
ultimately want to see, and I'll pass on to the steering committee too.

Ultimately I've been pushing for a kernel 0-day project for Arm. That's
probably going to require a lot of duplicated effort since the original
0-day project isn't open, but creating an open one could help everyone.

Jon.

[0] Linaro DataCenter Group (formerly "LEG")

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Greg KH]

On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote:
> On 2/13/19 2:52 PM, Greg KH wrote:
> > On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> 
> >> So really, it sounds like a low hanging fruit: we don't really need to
> >> write much more testing code code nor do we have to refactor existing
> >> test suites. We just need to make sure the right tests are running on
> >> stable kernels. I really want to clarify what each subsystem sees as
> >> "sufficient" (and have that documented somewhere).
> > 
> > kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
> > to their test suites to address these issues (I think 0-day already has
> > many of them).  So this is happening, but not quite obvious.  I know I
> > keep asking Linaro about this :(
> 
> We're working on investments for LDCG[0] in 2019 that include kernel CI
> changes for server use cases. Please keep us informed of what you folks
> ultimately want to see, and I'll pass on to the steering committee too.
> 
> Ultimately I've been pushing for a kernel 0-day project for Arm. That's
> probably going to require a lot of duplicated effort since the original
> 0-day project isn't open, but creating an open one could help everyone.

Why are you trying to duplicate it on your own?  That's what kernel.ci
should be doing, please join in and invest in that instead.  It's an
open source project with its own governance and needs sponsors, why
waste time and money doing it all on your own?

thanks,

greg k-h

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Jon Masters]

On 3/20/19 1:06 AM, Greg KH wrote:
> On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote:
>> On 2/13/19 2:52 PM, Greg KH wrote:
>>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
>>
>>>> So really, it sounds like a low hanging fruit: we don't really need to
>>>> write much more testing code code nor do we have to refactor existing
>>>> test suites. We just need to make sure the right tests are running on
>>>> stable kernels. I really want to clarify what each subsystem sees as
>>>> "sufficient" (and have that documented somewhere).
>>>
>>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
>>> to their test suites to address these issues (I think 0-day already has
>>> many of them).  So this is happening, but not quite obvious.  I know I
>>> keep asking Linaro about this :(
>>
>> We're working on investments for LDCG[0] in 2019 that include kernel CI
>> changes for server use cases. Please keep us informed of what you folks
>> ultimately want to see, and I'll pass on to the steering committee too.
>>
>> Ultimately I've been pushing for a kernel 0-day project for Arm. That's
>> probably going to require a lot of duplicated effort since the original
>> 0-day project isn't open, but creating an open one could help everyone.
> 
> Why are you trying to duplicate it on your own?  That's what kernel.ci
> should be doing, please join in and invest in that instead.  It's an
> open source project with its own governance and needs sponsors, why
> waste time and money doing it all on your own?

To clarify, I'm pushing for investment in kernel.ci to achieve that goal
that it could provide the same 0-day capability for Arm and others.
It'll ultimately result in duplicated effort vs if 0-day were open.

Jon.

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Greg KH]

On Wed, Mar 20, 2019 at 02:14:09AM -0400, Jon Masters wrote:
> On 3/20/19 1:06 AM, Greg KH wrote:
> > On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote:
> >> On 2/13/19 2:52 PM, Greg KH wrote:
> >>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
> >>
> >>>> So really, it sounds like a low hanging fruit: we don't really need to
> >>>> write much more testing code code nor do we have to refactor existing
> >>>> test suites. We just need to make sure the right tests are running on
> >>>> stable kernels. I really want to clarify what each subsystem sees as
> >>>> "sufficient" (and have that documented somewhere).
> >>>
> >>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
> >>> to their test suites to address these issues (I think 0-day already has
> >>> many of them).  So this is happening, but not quite obvious.  I know I
> >>> keep asking Linaro about this :(
> >>
> >> We're working on investments for LDCG[0] in 2019 that include kernel CI
> >> changes for server use cases. Please keep us informed of what you folks
> >> ultimately want to see, and I'll pass on to the steering committee too.
> >>
> >> Ultimately I've been pushing for a kernel 0-day project for Arm. That's
> >> probably going to require a lot of duplicated effort since the original
> >> 0-day project isn't open, but creating an open one could help everyone.
> > 
> > Why are you trying to duplicate it on your own?  That's what kernel.ci
> > should be doing, please join in and invest in that instead.  It's an
> > open source project with its own governance and needs sponsors, why
> > waste time and money doing it all on your own?
> 
> To clarify, I'm pushing for investment in kernel.ci to achieve that goal
> that it could provide the same 0-day capability for Arm and others.

Great, that's what I was trying to suggest :)

> It'll ultimately result in duplicated effort vs if 0-day were open.

"Half" of 0-day is open, but it's that other half that is still
needed...

thanks,

greg k-h

Re: [LSF/MM TOPIC] FS, MM, and stable trees

[Jon Masters]

On 3/20/19 2:28 AM, Greg KH wrote:
> On Wed, Mar 20, 2019 at 02:14:09AM -0400, Jon Masters wrote:
>> On 3/20/19 1:06 AM, Greg KH wrote:
>>> On Tue, Mar 19, 2019 at 11:46:09PM -0400, Jon Masters wrote:
>>>> On 2/13/19 2:52 PM, Greg KH wrote:
>>>>> On Wed, Feb 13, 2019 at 02:25:12PM -0500, Sasha Levin wrote:
>>>>
>>>>>> So really, it sounds like a low hanging fruit: we don't really need to
>>>>>> write much more testing code code nor do we have to refactor existing
>>>>>> test suites. We just need to make sure the right tests are running on
>>>>>> stable kernels. I really want to clarify what each subsystem sees as
>>>>>> "sufficient" (and have that documented somewhere).
>>>>>
>>>>> kernel.ci and 0-day and Linaro are starting to add the fs and mm tests
>>>>> to their test suites to address these issues (I think 0-day already has
>>>>> many of them).  So this is happening, but not quite obvious.  I know I
>>>>> keep asking Linaro about this :(
>>>>
>>>> We're working on investments for LDCG[0] in 2019 that include kernel CI
>>>> changes for server use cases. Please keep us informed of what you folks
>>>> ultimately want to see, and I'll pass on to the steering committee too.
>>>>
>>>> Ultimately I've been pushing for a kernel 0-day project for Arm. That's
>>>> probably going to require a lot of duplicated effort since the original
>>>> 0-day project isn't open, but creating an open one could help everyone.
>>>
>>> Why are you trying to duplicate it on your own?  That's what kernel.ci
>>> should be doing, please join in and invest in that instead.  It's an
>>> open source project with its own governance and needs sponsors, why
>>> waste time and money doing it all on your own?
>>
>> To clarify, I'm pushing for investment in kernel.ci to achieve that goal
>> that it could provide the same 0-day capability for Arm and others.
> 
> Great, that's what I was trying to suggest :)
> 
>> It'll ultimately result in duplicated effort vs if 0-day were open.
> 
> "Half" of 0-day is open, but it's that other half that is still
> needed...

;) I'm hoping this might also help that to happen...

Best,

Jon.