Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]

On Mon, 2019-01-14 at 17:06 +0200, Sasha Kotchubievsky wrote:
> Hi,
> 
> I ran some tests for nvmf target using various configurations in host 
> side: perf, fio (SPDK plugin), fio (libaio). It looks like, fio+SPDK 
> produces non-uniform load on cores in nvmf application.
> 
> SPDK's FIO plugin opens 3 QPs for each fio job. I suppose, they are: 
> admin QP for discovery, admin QP and IO QP. On the other side, nvmf 
> target, by default, use round-robin policy for QP assignment to cores.  
> As result, in some configurations, some cores are busy with IO and other 
> handle admin commands only.
> 
> In opposite, in tests with fio above libaio, a single admin QP is opened 
> at connect and all target's core are busy with IO.
> 
> For example, nvmf target rans on 3 cores and then I ran 2 fio jobs with 
> SPDK initiator. Core #14 handles IO from 2 jobs.
> 
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x10295e0 to core 12
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x102cfd0 to core 13
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x102d580 to core 14
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
> IBV_QPS_ERR
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x10295e0 to core 12
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x102daf0 to core 13
> nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
> 0x102e0a0 to core 14
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
> IBV_QPS_ERR
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#2 changed to: 
> IBV_QPS_ERR
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
> IBV_QPS_ERR
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#1 changed to: 
> IBV_QPS_ERR
> rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
> IBV_QPS_ERR
> 
> If the same test runs above libaio, 24 IO QPs and 1 admin QP is opened 
> at connect, and the test utilizes two cores in nvmf target.
> 
> Have I got that right? Is there any reason to open admin QPs in each fio 
> job ? (I understand,  from implementation perspective that's is simplest 
> way).
> 
> If you ask me, round-robin approach in target can be improved. I think, 
> admin and IO QPs should be handled independently. IO QPs should be 
> spread among cores regardless admin QPs. What do you think?

I think this is the best solution - we should make the distribution of the
qpairs smarter such that admin queues don't count when advancing the round-robin 
pointer.

> 
> Best regards
> 
> Sasha
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]



>      >
>      > On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" <
>      > spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
>      >
>      >     >
>      >     > If you ask me, round-robin approach in target can be improved. I think,
>      >     > admin and IO QPs should be handled independently. IO QPs should be
>      >     > spread among cores regardless admin QPs. What do you think?
>      >
>      >     I think this is the best solution - we should make the distribution of the
>      >     qpairs smarter such that admin queues don't count when advancing the
>      > round-robin
>      >     pointer.
>      >
>      > Do we know that a queue is admin or IO when we make the decision on advancing
>      > the round-robin pointer?
>      
>      Fortunately, it's one of the few things we do actually know up front. This is
>      what the target knows when a new connection is requested:
>      
>      struct spdk_nvmf_rdma_request_private_data {
>      	uint16_t	recfmt; /* record format */
>      	uint16_t	qid;	/* queue id */
>      	uint16_t	hrqsize;	/* host receive queue size */
>      	uint16_t	hsqsize;	/* host send queue size */
>      	uint16_t	cntlid;		/* controller id */
>      	uint8_t		reserved[22];
>      };
>      
>      So qid of 0 means they want to request an admin queue.
>
> Perfect.  Maybe we keep separate admin and IO round-robin pointers?  To avoid the case where some non-trivial number of admin queues are established between I/O queue connections.  It's not a likely scenario, and probably wouldn't make much of a performance difference if it did, but keeping separate pointers should be easy to implement.
>
> -Jim

Agree, for admin and IO QPs, separate round-robin pointers should be 
used. For example, fio + spdk plugin, as I mention, creates 2 admin QPs 
per job. Those QPs should be distributed between cores.

Sasha

Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1870 bytes --]



On 1/14/19, 2:18 PM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:

    On Mon, 2019-01-14 at 18:58 +0000, Harris, James R wrote:
    > 
    > On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" <
    > spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
    > 
    >     > 
    >     > If you ask me, round-robin approach in target can be improved. I think, 
    >     > admin and IO QPs should be handled independently. IO QPs should be 
    >     > spread among cores regardless admin QPs. What do you think?
    >     
    >     I think this is the best solution - we should make the distribution of the
    >     qpairs smarter such that admin queues don't count when advancing the
    > round-robin 
    >     pointer.
    >     
    > Do we know that a queue is admin or IO when we make the decision on advancing
    > the round-robin pointer?
    
    Fortunately, it's one of the few things we do actually know up front. This is
    what the target knows when a new connection is requested:
    
    struct spdk_nvmf_rdma_request_private_data {
    	uint16_t	recfmt; /* record format */
    	uint16_t	qid;	/* queue id */
    	uint16_t	hrqsize;	/* host receive queue size */
    	uint16_t	hsqsize;	/* host send queue size */
    	uint16_t	cntlid;		/* controller id */
    	uint8_t		reserved[22];
    };
    
    So qid of 0 means they want to request an admin queue.

Perfect.  Maybe we keep separate admin and IO round-robin pointers?  To avoid the case where some non-trivial number of admin queues are established between I/O queue connections.  It's not a likely scenario, and probably wouldn't make much of a performance difference if it did, but keeping separate pointers should be easy to implement.

-Jim

Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On Mon, 2019-01-14 at 18:58 +0000, Harris, James R wrote:
> 
> On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" <
> spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:
> 
>     > 
>     > If you ask me, round-robin approach in target can be improved. I think, 
>     > admin and IO QPs should be handled independently. IO QPs should be 
>     > spread among cores regardless admin QPs. What do you think?
>     
>     I think this is the best solution - we should make the distribution of the
>     qpairs smarter such that admin queues don't count when advancing the
> round-robin 
>     pointer.
>     
> Do we know that a queue is admin or IO when we make the decision on advancing
> the round-robin pointer?

Fortunately, it's one of the few things we do actually know up front. This is
what the target knows when a new connection is requested:

struct spdk_nvmf_rdma_request_private_data {
	uint16_t	recfmt; /* record format */
	uint16_t	qid;	/* queue id */
	uint16_t	hrqsize;	/* host receive queue size */
	uint16_t	hsqsize;	/* host send queue size */
	uint16_t	cntlid;		/* controller id */
	uint8_t		reserved[22];
};

So qid of 0 means they want to request an admin queue.


> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]



On 1/14/19, 11:55 AM, "SPDK on behalf of Walker, Benjamin" <spdk-bounces(a)lists.01.org on behalf of benjamin.walker(a)intel.com> wrote:

    > 
    > If you ask me, round-robin approach in target can be improved. I think, 
    > admin and IO QPs should be handled independently. IO QPs should be 
    > spread among cores regardless admin QPs. What do you think?
    
    I think this is the best solution - we should make the distribution of the
    qpairs smarter such that admin queues don't count when advancing the round-robin 
    pointer.
    
Do we know that a queue is admin or IO when we make the decision on advancing the round-robin pointer?

[SPDK] Number of QPs in NVME-OF testing with FIO (SPDK)

[Sasha Kotchubievsky]

[-- Attachment #1: Type: text/plain, Size: 2383 bytes --]

Hi,

I ran some tests for nvmf target using various configurations in host 
side: perf, fio (SPDK plugin), fio (libaio). It looks like, fio+SPDK 
produces non-uniform load on cores in nvmf application.

SPDK's FIO plugin opens 3 QPs for each fio job. I suppose, they are: 
admin QP for discovery, admin QP and IO QP. On the other side, nvmf 
target, by default, use round-robin policy for QP assignment to cores.  
As result, in some configurations, some cores are busy with IO and other 
handle admin commands only.

In opposite, in tests with fio above libaio, a single admin QP is opened 
at connect and all target's core are busy with IO.

For example, nvmf target rans on 3 cores and then I ran 2 fio jobs with 
SPDK initiator. Core #14 handles IO from 2 jobs.

nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x10295e0 to core 12
nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x102cfd0 to core 13
nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x102d580 to core 14
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
IBV_QPS_ERR
nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x10295e0 to core 12
nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x102daf0 to core 13
nvmf_tgt.c: 212:nvmf_tgt_get_qpair_core: *NOTICE*: Assigning QP 
0x102e0a0 to core 14
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
IBV_QPS_ERR
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#2 changed to: 
IBV_QPS_ERR
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
IBV_QPS_ERR
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#1 changed to: 
IBV_QPS_ERR
rdma.c: 526:spdk_nvmf_rdma_set_ibv_state: *NOTICE*: IBV QP#0 changed to: 
IBV_QPS_ERR

If the same test runs above libaio, 24 IO QPs and 1 admin QP is opened 
at connect, and the test utilizes two cores in nvmf target.

Have I got that right? Is there any reason to open admin QPs in each fio 
job ? (I understand,  from implementation perspective that's is simplest 
way).

If you ask me, round-robin approach in target can be improved. I think, 
admin and IO QPs should be handled independently. IO QPs should be 
spread among cores regardless admin QPs. What do you think?

Best regards

Sasha