can't run nvme-mp blktests

can't run nvme-mp blktests

[Luis Chamberlain]

I do have CONFIG_NVME_MULTIPATH=y but I also have:

cat /etc/modprobe.d/nvme.conf 
options nvme_core multipath=N

And yet I always end up booting with:

cat /sys/module/nvme_core/parameters/multipath 
Y

So trying to run:

nvme_trtype=rdma ./check nvmeof-mp

I end up with the warning:

nvmeof-mp/***                                                [not run]
    CONFIG_NVME_MULTIPATH has been set in .config and multipathing has been enabled in the nvme_core kernel module

Are there times where one cannot disable multipath? I'm not using
any nvme drive at boot, but I do use one for a random data parition.

Any tips?

  Luis

Re: can't run nvme-mp blktests

[Yi Zhang]

On Wed, Apr 13, 2022 at 8:24 AM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> I do have CONFIG_NVME_MULTIPATH=y but I also have:
>
> cat /etc/modprobe.d/nvme.conf
> options nvme_core multipath=N
>
> And yet I always end up booting with:
>
> cat /sys/module/nvme_core/parameters/multipath
> Y
>
> So trying to run:
>
> nvme_trtype=rdma ./check nvmeof-mp
>
> I end up with the warning:
>
> nvmeof-mp/***                                                [not run]
>     CONFIG_NVME_MULTIPATH has been set in .config and multipathing has been enabled in the nvme_core kernel module
>
> Are there times where one cannot disable multipath? I'm not using
> any nvme drive at boot, but I do use one for a random data parition.

So the multipath is not updated with N, pls try manually removing the
nvme_core module and retest.
Or just reboot can also help update the parameter.

BTW, to run blktests nvmeof-mp, the correct way is:
# ./check nvmeof-mp
# use_siw=1 ./check nvmeof-mp

>
> Any tips?
>
>   Luis
>


-- 
Best Regards,
  Yi Zhang

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Wed, Apr 13, 2022 at 11:02:47AM +0800, Yi Zhang wrote:
> On Wed, Apr 13, 2022 at 8:24 AM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > I do have CONFIG_NVME_MULTIPATH=y but I also have:
> >
> > cat /etc/modprobe.d/nvme.conf
> > options nvme_core multipath=N
> >
> > And yet I always end up booting with:
> >
> > cat /sys/module/nvme_core/parameters/multipath
> > Y
> >
> > So trying to run:
> >
> > nvme_trtype=rdma ./check nvmeof-mp
> >
> > I end up with the warning:
> >
> > nvmeof-mp/***                                                [not run]
> >     CONFIG_NVME_MULTIPATH has been set in .config and multipathing has been enabled in the nvme_core kernel module
> >
> > Are there times where one cannot disable multipath? I'm not using
> > any nvme drive at boot, but I do use one for a random data parition.
> 
> So the multipath is not updated with N, pls try manually removing the
> nvme_core module and retest.

I can't remove the module as a filesystem is mounted on one of the nvme
drives used at boot. It is not an OS partition, ie, it is just /data/.
The blktests is intended for some other nvme drives.

> Or just reboot can also help update the parameter.

Reboot does not help, that is my point.

> BTW, to run blktests nvmeof-mp, the correct way is:
> # ./check nvmeof-mp
> # use_siw=1 ./check nvmeof-mp

That does not let me do anything different as CONFIG_NVME_MULTIPATH is
still found to be set and the module parameter for nvme_core for
multipath is still "Y".

  Luis

Re: can't run nvme-mp blktests

[Christoph Hellwig]

On Tue, Apr 12, 2022 at 05:24:04PM -0700, Luis Chamberlain wrote:
> I do have CONFIG_NVME_MULTIPATH=y but I also have:

I'd suggest to ignore broken tests that require a deprecated option
that will eventually be removed.

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Tue, Apr 12, 2022 at 09:53:12PM -0700, Christoph Hellwig wrote:
> On Tue, Apr 12, 2022 at 05:24:04PM -0700, Luis Chamberlain wrote:
> > I do have CONFIG_NVME_MULTIPATH=y but I also have:
> 
> I'd suggest to ignore broken tests that require a deprecated option
> that will eventually be removed.

CONFIG_NVME_MULTIPATH will eventually be nuked?

  Luis

Re: can't run nvme-mp blktests

[Christoph Hellwig]

On Wed, Apr 13, 2022 at 10:38:50AM -0700, Luis Chamberlain wrote:
> On Tue, Apr 12, 2022 at 09:53:12PM -0700, Christoph Hellwig wrote:
> > On Tue, Apr 12, 2022 at 05:24:04PM -0700, Luis Chamberlain wrote:
> > > I do have CONFIG_NVME_MULTIPATH=y but I also have:
> > 
> > I'd suggest to ignore broken tests that require a deprecated option
> > that will eventually be removed.
> 
> CONFIG_NVME_MULTIPATH will eventually be nuked?

You'll only get a single namespace without it once the phaseout is
complete, yes.  If you want nvme multipathing you'll need it, which
is the story since day one.

If Bart wants to test dm-multipath I think his earlier srp_tests are
the much better choice.

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Wed, Apr 13, 2022 at 10:41:12AM -0700, Christoph Hellwig wrote:
> On Wed, Apr 13, 2022 at 10:38:50AM -0700, Luis Chamberlain wrote:
> > On Tue, Apr 12, 2022 at 09:53:12PM -0700, Christoph Hellwig wrote:
> > > On Tue, Apr 12, 2022 at 05:24:04PM -0700, Luis Chamberlain wrote:
> > > > I do have CONFIG_NVME_MULTIPATH=y but I also have:
> > > 
> > > I'd suggest to ignore broken tests that require a deprecated option
> > > that will eventually be removed.
> > 
> > CONFIG_NVME_MULTIPATH will eventually be nuked?
> 
> You'll only get a single namespace without it once the phaseout is
> complete, yes.

OK so CONFIG_NVME_MULTIPATH will be nuked, but the module parameter
will remain, and if the module parameter is used with multipath=N you'll
end up with a single namespace once the phase out of
CONFIG_NVME_MULTIPATH is complete.

> If you want nvme multipathing you'll need it, which
> is the story since day one.

Sounds good.

> If Bart wants to test dm-multipath I think his earlier srp_tests are
> the much better choice.

OK thanks for the heads up.

  Luis

Re: can't run nvme-mp blktests

[Christoph Hellwig]

On Wed, Apr 13, 2022 at 11:12:26AM -0700, Luis Chamberlain wrote:
> OK so CONFIG_NVME_MULTIPATH will be nuked, but the module parameter
> will remain, and if the module parameter is used with multipath=N you'll
> end up with a single namespace once the phase out of
> CONFIG_NVME_MULTIPATH is complete.

The other way around.  The plan is that CONFIG_NVME_MULTIPATH will
stay around forever - for deeply embedded devices that only have
a soldered on or sd-card nvme device with a single path there is no
need to carry the multipathing code around.  But the code to support
multiple controllers in a subsystem without CONFIG_NVME_MULTIPATH
will go away.

Re: can't run nvme-mp blktests

[Bart Van Assche]

On 4/13/22 10:41, Christoph Hellwig wrote:
> On Wed, Apr 13, 2022 at 10:38:50AM -0700, Luis Chamberlain wrote:
>> On Tue, Apr 12, 2022 at 09:53:12PM -0700, Christoph Hellwig wrote:
>>> On Tue, Apr 12, 2022 at 05:24:04PM -0700, Luis Chamberlain wrote:
>>>> I do have CONFIG_NVME_MULTIPATH=y but I also have:
>>>
>>> I'd suggest to ignore broken tests that require a deprecated option
>>> that will eventually be removed.
>>
>> CONFIG_NVME_MULTIPATH will eventually be nuked?
> 
> You'll only get a single namespace without it once the phaseout is
> complete, yes.  If you want nvme multipathing you'll need it, which
> is the story since day one.
> 
> If Bart wants to test dm-multipath I think his earlier srp_tests are
> the much better choice.

I'm not sure whether the nvme-mp tests test any code that is not yet tested by
any nvme or srp test. How about removing these tests since these tests make
it harder than necessary to run blktests?

Thanks,

Bart.

Re: can't run nvme-mp blktests

[Christoph Hellwig]

On Wed, Apr 13, 2022 at 03:46:16PM -0700, Bart Van Assche wrote:
> I'm not sure whether the nvme-mp tests test any code that is not yet tested by
> any nvme or srp test. How about removing these tests since these tests make
> it harder than necessary to run blktests?

I haven't looked at the details recently, but if these tests still are
basically a copy and paste of the srp mpath tests I'm all for removing
them!

Re: can't run nvme-mp blktests

[Chaitanya Kulkarni]

On 4/14/22 22:20, Christoph Hellwig wrote:
> On Wed, Apr 13, 2022 at 03:46:16PM -0700, Bart Van Assche wrote:
>> I'm not sure whether the nvme-mp tests test any code that is not yet tested by
>> any nvme or srp test. How about removing these tests since these tests make
>> it harder than necessary to run blktests?
> 
> I haven't looked at the details recently, but if these tests still are
> basically a copy and paste of the srp mpath tests I'm all for removing
> them!

If it doesn't add a new test coverage in the blktest framework
then please remove.

-ck

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Fri, Apr 15, 2022 at 05:58:16AM +0000, Chaitanya Kulkarni wrote:
> On 4/14/22 22:20, Christoph Hellwig wrote:
> > On Wed, Apr 13, 2022 at 03:46:16PM -0700, Bart Van Assche wrote:
> >> I'm not sure whether the nvme-mp tests test any code that is not yet tested by
> >> any nvme or srp test. How about removing these tests since these tests make
> >> it harder than necessary to run blktests?
> > 
> > I haven't looked at the details recently, but if these tests still are
> > basically a copy and paste of the srp mpath tests I'm all for removing
> > them!
> 
> If it doesn't add a new test coverage in the blktest framework
> then please remove.

That's about 3 block folks not being sure whether or not it helps. And
the complexities of it, to test it, requiring a different kernel seems
just stupid. So I'm going to drop the tests from kdevops and not bother
with them.

With regards to srp tests there seems to be two modules to use. Do we
want to prioritize both just as much? If so then I'll just create two
guests for srp tests.

  Luis

Re: can't run nvme-mp blktests

[Bart Van Assche]

On 4/15/22 10:59, Luis Chamberlain wrote:
> That's about 3 block folks not being sure whether or not it helps. And
> the complexities of it, to test it, requiring a different kernel seems
> just stupid. So I'm going to drop the tests from kdevops and not bother
> with them.

How about modifying the nvmeof-mp tests such that these use NVMe
multipathing instead of the dm-multipath driver?

> With regards to srp tests there seems to be two modules to use. Do we
> want to prioritize both just as much? If so then I'll just create two
> guests for srp tests.

The rdma_rxe driver is older than the siw (Soft-iWARP) driver. That's why
the rdma_rxe driver is the default for the SRP tests. Ideally it shouldn't
matter whether the rdma_rxe or the siw driver is used for the SRP tests.
There has been a time when the rdma_rxe driver worked perfectly when running
the SRP tests. Unfortunately that is no longer the case ...

Thanks,

Bart.

Re: can't run nvme-mp blktests

[Hannes Reinecke]

On 4/15/22 20:09, Bart Van Assche wrote:
> On 4/15/22 10:59, Luis Chamberlain wrote:
>> That's about 3 block folks not being sure whether or not it helps. And
>> the complexities of it, to test it, requiring a different kernel seems
>> just stupid. So I'm going to drop the tests from kdevops and not bother
>> with them.
> 
> How about modifying the nvmeof-mp tests such that these use NVMe
> multipathing instead of the dm-multipath driver?
> 
I'd rather enable the nvme tests to work with rdma_rxe / siw. It should 
already work with TCP, so hooking in rdma_rxe and/or siw would be the 
logical step.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Fri, Apr 15, 2022 at 11:09:33AM -0700, Bart Van Assche wrote:
> On 4/15/22 10:59, Luis Chamberlain wrote:
> > That's about 3 block folks not being sure whether or not it helps. And
> > the complexities of it, to test it, requiring a different kernel seems
> > just stupid. So I'm going to drop the tests from kdevops and not bother
> > with them.
> 
> How about modifying the nvmeof-mp tests such that these use NVMe
> multipathing instead of the dm-multipath driver?

Patches welcomed, I'm just going to ignore nvme-mp for now.

  Luis

Re: can't run nvme-mp blktests

[Bart Van Assche]

On 4/14/22 22:58, Chaitanya Kulkarni wrote:
> If it doesn't add a new test coverage in the blktest framework
> then please remove.

cd blktests && git grep -nH dev/nvme-fabrics
tests/nvmeof-mp/rc:94:	[ -c /dev/nvme-fabrics ] &&
tests/nvmeof-mp/rc:101:					echo -n "$loginparams" > /dev/nvme-fabrics
tests/nvmeof-mp/rc:161:	if [ ! -c /dev/nvme-fabrics ]; then
tests/nvmeof-mp/rc:162:		echo "Error:  /dev/nvme-fabrics not available"

Does this mean that there are no other tests in the blktests framework that
test NVMeOF and hence that it is useful to keep these tests?

Thanks,

Bart.

Re: can't run nvme-mp blktests

[Hannes Reinecke]

On 4/15/22 20:06, Bart Van Assche wrote:
> On 4/14/22 22:58, Chaitanya Kulkarni wrote:
>> If it doesn't add a new test coverage in the blktest framework
>> then please remove.
> 
> cd blktests && git grep -nH dev/nvme-fabrics
> tests/nvmeof-mp/rc:94:    [ -c /dev/nvme-fabrics ] &&
> tests/nvmeof-mp/rc:101:                    echo -n "$loginparams" > 
> /dev/nvme-fabrics
> tests/nvmeof-mp/rc:161:    if [ ! -c /dev/nvme-fabrics ]; then
> tests/nvmeof-mp/rc:162:        echo "Error:  /dev/nvme-fabrics not 
> available"
> 
> Does this mean that there are no other tests in the blktests framework that
> test NVMeOF and hence that it is useful to keep these tests?
> 

# git grep _nvme_connect_subsys
tests/nvme/003: _nvme_connect_subsys "${nvme_trtype}" 
nqn.2014-08.org.nvmexpress.discovery
tests/nvme/004: _nvme_connect_subsys "${nvme_trtype}" blktests-subsystem-1
tests/nvme/005: _nvme_connect_subsys "${nvme_trtype}" blktests-subsystem-1
tests/nvme/008: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/009: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/010: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/011: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/012: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/013: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/014: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/015: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/018: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/019: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/020: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
tests/nvme/021: _nvme_connect_subsys "${nvme_trtype}" "${subsys_name}"
[ .. ]

So: no, it just means you've been using the wrong filter.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

Re: can't run nvme-mp blktests

[Bart Van Assche]

On 4/12/22 17:24, Luis Chamberlain wrote:
> I do have CONFIG_NVME_MULTIPATH=y but I also have:
> 
> cat /etc/modprobe.d/nvme.conf
> options nvme_core multipath=N
> 
> And yet I always end up booting with:
> 
> cat /sys/module/nvme_core/parameters/multipath
> Y

Is the nvme_core module loaded from the initial ramdisk or from 
/lib/modules? In the former case, does the initial ramdisk perhaps have 
to be regenerated?

Thanks,

Bart.

Re: can't run nvme-mp blktests

[Luis Chamberlain]

On Tue, Apr 12, 2022 at 10:53:40PM -0700, Bart Van Assche wrote:
> On 4/12/22 17:24, Luis Chamberlain wrote:
> > I do have CONFIG_NVME_MULTIPATH=y but I also have:
> > 
> > cat /etc/modprobe.d/nvme.conf
> > options nvme_core multipath=N
> > 
> > And yet I always end up booting with:
> > 
> > cat /sys/module/nvme_core/parameters/multipath
> > Y
> 
> Is the nvme_core module loaded from the initial ramdisk or from
> /lib/modules? In the former case, does the initial ramdisk perhaps have to
> be regenerated?

No, no ramdisk, but I just had to use the /etc/default/grub file to set
nvme_core=N as the /etc/modprobe.d/nvme.conf was not being picked up
for whatever reason.

cat /etc/modprobe.d/nvme.conf 
options nvme_core multipath=N

Either way, from a test automation perspective this requirement is a
pain, as I can't do a change dynamically at run time. So for this test
I have to manually change a kernel boot param. Is doing the required
changes dynamic not possible in the future?

I take it that it does not make sense at all to have multipath=N for all
blktests tests.

  Luis