* nvme native multipath question with two physical paths
@ 2024-02-19 19:03 Wen Xiong
2024-02-19 19:28 ` Keith Busch
0 siblings, 1 reply; 3+ messages in thread
From: Wen Xiong @ 2024-02-19 19:03 UTC (permalink / raw)
To: linux-nvme; +Cc: Wenxiong, Christoph Hellwig, Keith Busch
Hi All,
In our nvme native multipath support drawer, each nvme has two physical
paths/two controllers.
We created a shared namespace on it.
# nvme list-ns /dev/nvme2
[ 0]:0x1
# nvme list-ns /dev/nvme6
[ 0]:0x1
# ls -l /dev/nvme2*
crw-------. 1 root root 240, 2 Feb 18 18:05 /dev/nvme2
brw-rw----. 1 root disk 259, 4 Feb 18 18:14 /dev/nvme2n1
# ls -l /dev/nvme6*
crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6
nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:
\
+- nvme2 pcie 052f:80:00.0 live
+- nvme6 pcie 058f:80:00.0 live
#nvme list
/dev/nvme2n1 /dev/ng2n1
Run stress io test over nvme2n1, we hot unplug nvme2(052f:80:00),
/dev/nvme2 controller is gone.
/dev/nvme2n1 still existed and io still running over nvme2n1 without
error. Looks io test still
running over another path(nvme6)bbut we didn’t see /dev/nvme6n1 in /dev
dir.
# ls -l /dev/nvme2*
brw-rw----. 1 root disk 259, 4 Feb 18 18:22 /dev/nvme2n1
# ls -l /dev/nvme6*
crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6
With nvme native mpath enable,
- What multipath disk name we should use for ANA nvme multipath?
- Why IO still running over nvme2n1 without error even nvme2 controller
already unplug?
- Shared namespace is still existed since shared namespace has another
controller on it?
- How can let nvme6n1 showing up after nvme2 controller gone.
Thanks for your help!
Wendy
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: nvme native multipath question with two physical paths
2024-02-19 19:03 nvme native multipath question with two physical paths Wen Xiong
@ 2024-02-19 19:28 ` Keith Busch
2024-02-19 21:55 ` Wen Xiong
0 siblings, 1 reply; 3+ messages in thread
From: Keith Busch @ 2024-02-19 19:28 UTC (permalink / raw)
To: Wen Xiong; +Cc: linux-nvme, Wenxiong, Christoph Hellwig
On Mon, Feb 19, 2024 at 01:03:48PM -0600, Wen Xiong wrote:
> Hi All,
>
> In our nvme native multipath support drawer, each nvme has two physical
> paths/two controllers.
> We created a shared namespace on it.
>
> # nvme list-ns /dev/nvme2
> [ 0]:0x1
> # nvme list-ns /dev/nvme6
> [ 0]:0x1
>
> # ls -l /dev/nvme2*
> crw-------. 1 root root 240, 2 Feb 18 18:05 /dev/nvme2
> brw-rw----. 1 root disk 259, 4 Feb 18 18:14 /dev/nvme2n1
> # ls -l /dev/nvme6*
> crw-------. 1 root root 240, 6 Feb 18 18:05 /dev/nvme6
>
> nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:
> \
> +- nvme2 pcie 052f:80:00.0 live
> +- nvme6 pcie 058f:80:00.0 live
>
> #nvme list
> /dev/nvme2n1 /dev/ng2n1
>
>
> Run stress io test over nvme2n1, we hot unplug nvme2(052f:80:00), /dev/nvme2
> controller is gone.
> /dev/nvme2n1 still existed and io still running over nvme2n1 without error.
> Looks io test still
> running over another path(nvme6)bbut we didn´t see /dev/nvme6n1 in /dev dir.
Nor should you see a /dev/nvme6n1 in this scenario. The mulitpath device
doesn't change names. It assumes the identifier from the subsystem. In
your example, it uses subsystem '2' and will continue to use it no
matter which controller instances are attach to it. This feature is why
applications that have opened the /dev/nvme2n1 handle don't need to do
anything if the kernel experiences a path failover condition.
You should see the different paths in the hidden block hierarchy, which
you can find in the sysfs directory. There should be an entry in your
example at /sys/block/nvme2c6n1/. I am pretty sure programs like
'iostat' can show how IO is distributed among all the paths of an nvme
mulitpath device.
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: nvme native multipath question with two physical paths
2024-02-19 19:28 ` Keith Busch
@ 2024-02-19 21:55 ` Wen Xiong
0 siblings, 0 replies; 3+ messages in thread
From: Wen Xiong @ 2024-02-19 21:55 UTC (permalink / raw)
To: Keith Busch; +Cc: linux-nvme, Wenxiong, Christoph Hellwig
On 2024-02-19 13:28, Keith Busch wrote:
> Nor should you see a /dev/nvme6n1 in this scenario. The mulitpath
> device
> doesn't change names. It assumes the identifier from the subsystem. In
> your example, it uses subsystem '2' and will continue to use it no
> matter which controller instances are attach to it. This feature is why
> applications that have opened the /dev/nvme2n1 handle don't need to do
> anything if the kernel experiences a path failover condition.
>
> You should see the different paths in the hidden block hierarchy, which
> you can find in the sysfs directory. There should be an entry in your
> example at /sys/block/nvme2c6n1/. I am pretty sure programs like
> 'iostat' can show how IO is distributed among all the paths of an nvme
> mulitpath device.
Hi Ketith,
Thanks for your detail explanation!
Wendy
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2024-02-19 21:55 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-19 19:03 nvme native multipath question with two physical paths Wen Xiong
2024-02-19 19:28 ` Keith Busch
2024-02-19 21:55 ` Wen Xiong
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.