[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

Hi,

Since discussion of the storage stack and device driver at the
LSFMM 2017 (https://lwn.net/Articles/717699/), Omar Sandoval introduced
a new framework "blktests" dedicated for Linux Kernel Block layer
testing that is maintained by Shinichiro Kawasaki :-

https://lwn.net/Articles/722785/
https://github.com/osandov/blktests

As Linux Kernel Block layer is central to the various file systems and
underlying low-level device drivers it is important to have a centralized
testing framework and make sure it grows with the latest block layer
changes which are being added based on the different device features from
different device types (e.g. NVMe devices with Zoned Namespace support).

Since then blktests has grown and became go-to framework where we have
integrated different stand-alone test suites like SRP-tests, NVMFTESTS,
NVMe Multipath tests, zone block device tests, into one central
framework, which has made an overall block layer testing and development
much easier than having to configure and execute different test cases
for each kernel release for different subsystems such as FS, NVMe, Zone
Block devices, etc).

Here is the list of the existing test categories:-

block : 32
dm    : 1
loop  : 9
nbd   : 4
nvme  : 49
scsi  : 6
srp   : 15
ublk  : 6
zbd   : 10
----------------------------------------------------------------
9 Categories     :147 Tests

This project has gathered much attention and storage stack community is
actively participating and adding new test cases with different
categories to the framework.

Since addition of this framework we are consistently finding bugs
proactively with the help of blktests testcases.

For storage track, I would like to propose a session dedicated to
blktests. It is a great opportunity for the storage developers to gather
and have a discussion about:-

1. Current status of the blktests framework.
2. Any new/missing features that we want to add in the blktests.
3. Any new kernel features that could be used to make testing easier?
4. DM/MD Testcases.
5. Potentially adding VM support in the blktests.

E.g. Implementing new features in the null_blk.c in order to have device
independent complete test coverage. (e.g. adding discard command for
null_blk or any other specific REQ_OP). Discussion about having any new
tracepoint events in the block layer.

Any new test cases/categories which are lacking in the blktests
framework.

Required attendees :-

Shinichiro Kawasaki
Damien Le Moal
Daniel Wanger
Hannes Reinecke

-ck

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Bart Van Assche]

On 1/8/24 22:30, Chaitanya Kulkarni wrote:
> 5. Potentially adding VM support in the blktests.

What is "VM support"? I'm running blktests in a VM all the time since
this test suite was introduced.

Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Chaitanya Kulkarni]

On 1/9/2024 1:31 PM, Bart Van Assche wrote:
> On 1/8/24 22:30, Chaitanya Kulkarni wrote:
>> 5. Potentially adding VM support in the blktests.
> 
> What is "VM support"? I'm running blktests in a VM all the time since
> this test suite was introduced.
> 
> Bart.

An ability to start, stop, suspend, issue commands to vm, recent
patchset async shutdown on linux-nvme list is one of the example why we
may need this feature, see [1].

-ck

[1]

https://lists.infradead.org/pipermail/linux-nvme/2024-Januar/044135.html

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Bart Van Assche]

On 1/9/24 14:01, Chaitanya Kulkarni wrote:
> On 1/9/2024 1:31 PM, Bart Van Assche wrote:
>> On 1/8/24 22:30, Chaitanya Kulkarni wrote:
>>> 5. Potentially adding VM support in the blktests.
>>
>> What is "VM support"? I'm running blktests in a VM all the time since
>> this test suite was introduced.
>>
>> Bart.
> 
> An ability to start, stop, suspend, issue commands to vm, recent
> patchset async shutdown on linux-nvme list is one of the example why we
> may need this feature, see [1].
> 
> -ck
> 
> [1]
> 
> https://lists.infradead.org/pipermail/linux-nvme/2024-Januar/044135.html
If I try to open that link, the following appears: "Not Found - The requested
URL was not found on this server." Anyway, I think I know which patch series
you are referring to. There may be better ways to trigger shutdown calls than
by stopping a VM.

Thanks,

Bart.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Tue, Jan 09, 2024 at 06:30:46AM +0000, Chaitanya Kulkarni wrote:
> For storage track, I would like to propose a session dedicated to
> blktests. It is a great opportunity for the storage developers to gather
> and have a discussion about:-
> 
> 1. Current status of the blktests framework.
> 2. Any new/missing features that we want to add in the blktests.
> 3. Any new kernel features that could be used to make testing easier?
> 4. DM/MD Testcases.
> 5. Potentially adding VM support in the blktests.

I am interested in such a session.

> Required attendees :-
> 
> Shinichiro Kawasaki
> Damien Le Moal
> Daniel Wanger
> Hannes Reinecke

If I get an invite I'll be there.

Thanks,
Daniel

Re: Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Wed, Jan 17, 2024 at 09:50:50AM +0100, Daniel Wagner wrote:
> On Tue, Jan 09, 2024 at 06:30:46AM +0000, Chaitanya Kulkarni wrote:
> > For storage track, I would like to propose a session dedicated to
> > blktests. It is a great opportunity for the storage developers to gather
> > and have a discussion about:-
> > 
> > 1. Current status of the blktests framework.
> > 2. Any new/missing features that we want to add in the blktests.
> > 3. Any new kernel features that could be used to make testing easier?
> > 4. DM/MD Testcases.
> > 5. Potentially adding VM support in the blktests.
> 
> I am interested in such a session.

One discussion point I'd like to add is

  - running blktest against real hardare/target

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Shinichiro Kawasaki]

On Jan 23, 2024 / 16:07, Daniel Wagner wrote:
> On Wed, Jan 17, 2024 at 09:50:50AM +0100, Daniel Wagner wrote:
> > On Tue, Jan 09, 2024 at 06:30:46AM +0000, Chaitanya Kulkarni wrote:
> > > For storage track, I would like to propose a session dedicated to
> > > blktests. It is a great opportunity for the storage developers to gather
> > > and have a discussion about:-
> > > 
> > > 1. Current status of the blktests framework.
> > > 2. Any new/missing features that we want to add in the blktests.
> > > 3. Any new kernel features that could be used to make testing easier?
> > > 4. DM/MD Testcases.
> > > 5. Potentially adding VM support in the blktests.
> > 
> > I am interested in such a session.

Thanks Chaitanya, I'm interested in them too. I can share my view on the current
status of blktests.

> 
> One discussion point I'd like to add is
> 
>   - running blktest against real hardare/target

Agreed. I guess this maybe meant for real RDMA hardware, which was discussed in
a couple of GitHub pull requests [1][2].

  [1] https://github.com/osandov/blktests/pull/86
  [2] https://github.com/osandov/blktests/pull/127

Another topic I suggest is,

 - Automated blktests runs and reports

Recently I learned that CKI project runs blktests regularly against Linus master
branch and linux-block for-next branch, then makes the run results visible on
the net [3][4]. Now I'm trying to understand detailed conditions of the test
runs. If we can discuss and clarify what kind of run conditions will help
storage kernel developers, it will be a good input to CKI project, hopefully.

  [3] https://datawarehouse.cki-project.org/kcidb/tests?tree_filter=mainline.kernel.org&kernel_version_filter=&arch_filter=x86_64&test_filter=blktests&host_filter=&testresult_filter=
  [4] https://datawarehouse.cki-project.org/kcidb/tests?tree_filter=block&kernel_version_filter=&arch_filter=x86_64&test_filter=blktests&host_filter=&testresult_filter=

Re: Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Luis Chamberlain]

On Tue, Jan 23, 2024 at 04:07:48PM +0100, Daniel Wagner wrote:
> On Wed, Jan 17, 2024 at 09:50:50AM +0100, Daniel Wagner wrote:
> > On Tue, Jan 09, 2024 at 06:30:46AM +0000, Chaitanya Kulkarni wrote:
> > > For storage track, I would like to propose a session dedicated to
> > > blktests. It is a great opportunity for the storage developers to gather
> > > and have a discussion about:-
> > > 
> > > 1. Current status of the blktests framework.
> > > 2. Any new/missing features that we want to add in the blktests.
> > > 3. Any new kernel features that could be used to make testing easier?
> > > 4. DM/MD Testcases.
> > > 5. Potentially adding VM support in the blktests.
> > 
> > I am interested in such a session.
> 
> One discussion point I'd like to add is
> 
>   - running blktest against real hardare/target

We've resolved this in fstests with canonicalizing device symlinks, and
through kdevops its possible to even use PCIe passthrough onto a guest
using dynamic kconfig (ie, specific to the host).

It should be possible to do that in blktests too, but the dynamic
kconfig thing is outside of scope, but this is a long winded way of
suggestin that if we extend blktests to add a canonon-similar device
function, then since kdevops supports blktests you get that pcie
passthrough for free too.

  Luis

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Wed, Feb 21, 2024 at 10:32:05AM -0800, Luis Chamberlain wrote:
> > One discussion point I'd like to add is
> > 
> >   - running blktest against real hardare/target
> 
> We've resolved this in fstests with canonicalizing device symlinks, and
> through kdevops its possible to even use PCIe passthrough onto a guest
> using dynamic kconfig (ie, specific to the host).
> 
> It should be possible to do that in blktests too, but the dynamic
> kconfig thing is outside of scope, but this is a long winded way of
> suggestin that if we extend blktests to add a canonon-similar device
> function, then since kdevops supports blktests you get that pcie
> passthrough for free too.

I should have been more precise here, I was trying to say supporting
real fabrics targets. blktests already has some logic for PCI targets
with $TEST_DEV but I haven't really looked into this part yet.

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Luis Chamberlain]

On Thu, Feb 22, 2024 at 10:31:53AM +0100, Daniel Wagner wrote:
> On Wed, Feb 21, 2024 at 10:32:05AM -0800, Luis Chamberlain wrote:
> > > One discussion point I'd like to add is
> > > 
> > >   - running blktest against real hardare/target
> > 
> > We've resolved this in fstests with canonicalizing device symlinks, and
> > through kdevops its possible to even use PCIe passthrough onto a guest
> > using dynamic kconfig (ie, specific to the host).
> > 
> > It should be possible to do that in blktests too, but the dynamic
> > kconfig thing is outside of scope, but this is a long winded way of
> > suggestin that if we extend blktests to add a canonon-similar device
> > function, then since kdevops supports blktests you get that pcie
> > passthrough for free too.
> 
> I should have been more precise here, I was trying to say supporting
> real fabrics targets. blktests already has some logic for PCI targets
> with $TEST_DEV but I haven't really looked into this part yet.

Do fabric targets have a symlink which remains static?

  Luis

Re: [LSF/MM/BPF ATTEND][LSF/MM/BPF TOPIC] : blktests: status, expansion plan for the storage stack test framework

[Daniel Wagner]

On Thu, Feb 22, 2024 at 07:54:18AM -0800, Luis Chamberlain wrote:
> > I should have been more precise here, I was trying to say supporting
> > real fabrics targets. blktests already has some logic for PCI targets
> > with $TEST_DEV but I haven't really looked into this part yet.
> 
> Do fabric targets have a symlink which remains static?

A pretty typical nvme fabric test is:

setup phase target side:
 - create backing device (file/block)
 - create loopback device
 - create nvme subsystem

setup phase host side:
 - discover
 - connect to the target

test phase
 do something like reading/writing from '/dev/nvmeX'
 or 'nvme id-ctrl /dev/nvmeX', etc.

cleanup phase host side:
 - disconnect from the target

cleanup phase target side:
 - remove nvme subsystem
 - remove loopback device
 - remove backing device

I'd like to make the setup and cleanup target side more flexible. The
host side will not be affected at all by exchanging the current soft
target side (aka nvmet) with something else. This means it's not about
any device links in /dev.

Hope this makes it a bit clearer.