RE: rdma_create_qp() and max_send_wr

RE: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

Le jeudi 21 avril 2011 à 11:53 -0700, cait-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org a écrit :
> An ENOMEM return does not mean that the subsystem *just* failed to
> allocate system memory.

> The memory that could not be allocated could be device memory.
>         

I'm also having some difficulties with system memory allocation.

In my test, a user is allowed to lock 4MBytes of memory, but not all
this memory is available to ibv_reg_mr() since ibv_create_cq() and
ibv_create_qp()/rdma_create_qp() lock memory respectively for CQ and QP.
The question is how much memory is needed for the CQ and QP queues ?

In my case, the maximum message of size is 4MBytes - 20KBytes, for a CQ
and QP (half duplex) queues length of 1.

Using message size of 128 bytes and less hit the QP WR limit of 16351
length.

When using messages of size 256 bytes, I'm only able to register 2609152
bytes, then CQ and QP (half duplex) queues are 10192 entries length. So
they seems to requires about 1585152 bytes. Taking in account a fixed
amount of reserved memory of 20KBytes, this give about 154 bytes per (CQ
+ QP (half duplex)) entry.

When doing the same math with size 512 and 1024, the size of (CQ + QP
(half duplex)) is going down.

msg  512 memory 3395584 length 6632
msg 1024 memory 3788800 length 3700
msg 2048 memory 3985408 length 1946

Note that the memory used for the message is allocated as an aligned big
chunk and registered as whole, and then sliced to be posted in WR. 

But the memory required for the CQ and QP elements (and other) is also
subject to alignment to a page size.

At least, I known that CQ / QP "overhead" is not going to hurt users, if
they are allocated "modern" memory limits, let's say 1GBytes ;)

Regards.

-- 
Yann Droneaud
OPTEYA



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma_create_qp() and max_send_wr

[Yann Droneaud]

And I forgot to mention:

Le vendredi 22 avril 2011 à 12:20 +0200, Yann Droneaud a écrit :
> I'm also having some difficulties with system memory allocation.

In this case of failure, strace shows the last write() syscall returning
ENOMEM.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Eli Cohen]

Hi Yan,
it appears that you're using quite an old firmware. Could you upgrade
the firmware to the latest version and check again the failure to
create a QP with the max depth. FW and burning tools can be downloaded
from www.mellanox.com.

Another possible reason for the failures you were seeing is the 4 MB
limit on locking memory from usersapce. Could you repeat the
experiment as root?

P.S: I checked on my setup and was able to create a QP with the max
size.

On Fri, Apr 22, 2011 at 12:20:45PM +0200, Yann Droneaud wrote:
> Hi,
> 
> Le jeudi 21 avril 2011 à 11:53 -0700, cait-DpaxOq6QOWMAvxtiuMwx3w@public.gmane.org a écrit :
> > An ENOMEM return does not mean that the subsystem *just* failed to
> > allocate system memory.
> 
> > The memory that could not be allocated could be device memory.
> >         
> 
> I'm also having some difficulties with system memory allocation.
> 
> In my test, a user is allowed to lock 4MBytes of memory, but not all
> this memory is available to ibv_reg_mr() since ibv_create_cq() and
> ibv_create_qp()/rdma_create_qp() lock memory respectively for CQ and QP.
> The question is how much memory is needed for the CQ and QP queues ?
> 
> In my case, the maximum message of size is 4MBytes - 20KBytes, for a CQ
> and QP (half duplex) queues length of 1.
> 
> Using message size of 128 bytes and less hit the QP WR limit of 16351
> length.
> 
> When using messages of size 256 bytes, I'm only able to register 2609152
> bytes, then CQ and QP (half duplex) queues are 10192 entries length. So
> they seems to requires about 1585152 bytes. Taking in account a fixed
> amount of reserved memory of 20KBytes, this give about 154 bytes per (CQ
> + QP (half duplex)) entry.
> 
> When doing the same math with size 512 and 1024, the size of (CQ + QP
> (half duplex)) is going down.
> 
> msg  512 memory 3395584 length 6632
> msg 1024 memory 3788800 length 3700
> msg 2048 memory 3985408 length 1946
> 
> Note that the memory used for the message is allocated as an aligned big
> chunk and registered as whole, and then sliced to be posted in WR. 
> 
> But the memory required for the CQ and QP elements (and other) is also
> subject to alignment to a page size.
> 
> At least, I known that CQ / QP "overhead" is not going to hurt users, if
> they are allocated "modern" memory limits, let's say 1GBytes ;)
> 
> Regards.
> 
> -- 
> Yann Droneaud
> OPTEYA
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

Le jeudi 19 mai 2011 à 09:07 +0300, Eli Cohen a écrit :
> Hi Yan,
> it appears that you're using quite an old firmware. Could you upgrade
> the firmware to the latest version and check again the failure to
> create a QP with the max depth. FW and burning tools can be downloaded
> from www.mellanox.com.
> 

Upgraded to firmware 2.8.0600, I have the same problem.

> Another possible reason for the failures you were seeing is the 4 MB
> limit on locking memory from usersapce. Could you repeat the
> experiment as root?
> 

The '16351' limit also hit root on my testbench.
I have also raised the memory limit to 1GBytes, and it doesn't work too.

> P.S: I checked on my setup and was able to create a QP with the max
> size.
> 

Have you some test code for me to test ?

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Eli Cohen]

On Thu, May 19, 2011 at 11:17:16AM +0200, Yann Droneaud wrote:
> 
> Have you some test code for me to test ?
> 

I used ibv_rc_pingpong which is part of libiberbs. the '-r' option
allows you to define the queue depth. Please try it and let me know.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Le jeudi 19 mai 2011 à 12:34 +0300, Eli Cohen a écrit :
> On Thu, May 19, 2011 at 11:17:16AM +0200, Yann Droneaud wrote:
> > 
> > Have you some test code for me to test ?
> > 
> 
> I used ibv_rc_pingpong which is part of libiberbs. the '-r' option
> allows you to define the queue depth. Please try it and let me know.

ibv_rc_pingpong works well with the max value eg 16384.
but ib_rdma_bw doesn't:

ib_rdma_bw -t 16384 doesn't work, but ib_rdma_bw -t 16351 does.

Creating QP through ibv_create_qp() seems to allow use 
of the maximum QP WR, but rmda_create_qp() limits to 16351.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma_create_qp() and max_send_wr

[Hefty, Sean]

> Creating QP through ibv_create_qp() seems to allow use
> of the maximum QP WR, but rmda_create_qp() limits to 16351.

rdma_create_qp() passes the QP creation call straight through to ibv_create_qp().  It will also allocate CQs if those don't already exist using the same limits as specified for the QP, which could be the reason for the failure.

- Sean

RE: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

Le jeudi 19 mai 2011 à 14:25 +0000, Hefty, Sean a écrit :
> > Creating QP through ibv_create_qp() seems to allow use
> > of the maximum QP WR, but rmda_create_qp() limits to 16351.
> 
> rdma_create_qp() passes the QP creation call straight through to ibv_create_qp().  

That's what I see when tracing the code in GDB.

> It will also allocate CQs if those don't already exist using the same 
> limits as specified for the QP, which could be the reason for the 
> failure.
> 

The CQs are already allocated, I've also checked that with the debugger.

So I'm a bit puzzled : why does it work in ibv_rc_pingpong but not in
rdma_bw ?

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

Le jeudi 19 mai 2011 à 16:45 +0200, Yann Droneaud a écrit :

> So I'm a bit puzzled : why does it work in ibv_rc_pingpong but not in
> rdma_bw ?
> 

Because ibv_rc_pingpong -r modify the max_recv_wr attributes 
and rdma_bw -t modify the max_send_rw instead.

After modifiying ibv_rc_pingpong to set max_recv_wr attributes, it fails
to create the QP when trying to use more than 16351 WR in the SQ.

To sum up:

- ibv_qp_init_attr.max_recv_wr can be set to ibv_device_attr.max_qp_wr,
16384 in my case,
 
- ibv_qp_init_attr.max_send_wr *cannot* be set to
ibv_device_attr.max_qp_wr, but it can be set to 16351.

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Eli Cohen]

On Thu, May 19, 2011 at 05:03:45PM +0200, Yann Droneaud wrote:
> 
> To sum up:
> 
> - ibv_qp_init_attr.max_recv_wr can be set to ibv_device_attr.max_qp_wr,
> 16384 in my case,
>  
> - ibv_qp_init_attr.max_send_wr *cannot* be set to
> ibv_device_attr.max_qp_wr, but it can be set to 16351.
> 
Thanks for investigating this. We'll check and send a fix. Meanwhile
you can workaround this in your code.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Le jeudi 19 mai 2011 à 18:46 +0300, Eli Cohen a écrit :
> On Thu, May 19, 2011 at 05:03:45PM +0200, Yann Droneaud wrote:
> > 
> > To sum up:
> > 
> > - ibv_qp_init_attr.max_recv_wr can be set to ibv_device_attr.max_qp_wr,
> > 16384 in my case,
> >  
> > - ibv_qp_init_attr.max_send_wr *cannot* be set to
> > ibv_device_attr.max_qp_wr, but it can be set to 16351.
> > 
> Thanks for investigating this. We'll check and send a fix. Meanwhile
> you can workaround this in your code.

I understand the problem (which is probably not really a problem):

In libmlx4/src/verbs.c:mlx4_create_qp()

        mlx4_calc_sq_wqe_size(&attr->cap, attr->qp_type, qp);

        /*
         * We need to leave 2 KB + 1 WQE of headroom in the SQ to
         * allow HW to prefetch.
         */
        qp->sq_spare_wqes = (2048 >> qp->sq.wqe_shift) + 1;
        qp->sq.wqe_cnt = align_queue_size(attr->cap.max_send_wr +
qp->sq_spare_wqes);
        qp->rq.wqe_cnt = align_queue_size(attr->cap.max_recv_wr);

For a requested SQ length of 1:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 1
 qp->sq_wge_cnt : 64

For a requested SQ length of 16351:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 16351
 qp->sq_wge_cnt : 16384

For a requested SQ length of 16384:

 qp->sq.wge_shift : 6
 qp->sq_spare_wges : 33
 attr->cap.max_send_wr : 16384
 qp->sq_wge_cnt : 32768

32768 is clearly above the ibv_device_attr.max_qp_wr limit, so the
creation of the QP failed later:

        ret = ibv_cmd_create_qp(pd, &qp->ibv_qp, attr, &cmd.ibv_cmd,
sizeof cmd,
                                &resp, sizeof resp);

ibv_cmd_create_qp() fails and returns 22.


If spare WQEs are taken in account here, it should be taken in account
in the data reported by ibv_query_device().

Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

Le jeudi 19 mai 2011 à 18:59 +0200, Yann Droneaud a écrit :
> Le jeudi 19 mai 2011 à 18:46 +0300, Eli Cohen a écrit :
> > On Thu, May 19, 2011 at 05:03:45PM +0200, Yann Droneaud wrote:
> > > 
> > > To sum up:
> > > 
> > > - ibv_qp_init_attr.max_recv_wr can be set to ibv_device_attr.max_qp_wr,
> > > 16384 in my case,
> > >  
> > > - ibv_qp_init_attr.max_send_wr *cannot* be set to
> > > ibv_device_attr.max_qp_wr, but it can be set to 16351.
> > > 
> > Thanks for investigating this. We'll check and send a fix. Meanwhile
> > you can workaround this in your code.
> 
> I understand the problem (which is probably not really a problem):
> 
> In libmlx4/src/verbs.c:mlx4_create_qp()
> 
>         mlx4_calc_sq_wqe_size(&attr->cap, attr->qp_type, qp);
> 
>         /*
>          * We need to leave 2 KB + 1 WQE of headroom in the SQ to
>          * allow HW to prefetch.
>          */

This was introduced by commit 561da8d1

Handle new FW requirement for send request prefetching

http://git.kernel.org/?p=libs/infiniband/libmlx4.git;a=commit;h=561da8d1

Regards.

-- 
Yann Droneaud
OPTEYA



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: rdma_create_qp() and max_send_wr

[Hefty, Sean]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 559 bytes --]

> If spare WQEs are taken in account here, it should be taken in account
> in the data reported by ibv_query_device().

max_qp_wr does not distinguish between max send or receive or indicate if those values should be the same.  IMO, Setting

max_qp_wr = max(send wr, recv wr)

makes more sense than

max_qp_wr = min(send wr, recv wr)

The maximum values reported are not meant to be guaranteed minimums.

- Sean
N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: rdma_create_qp() and max_send_wr

[Eli Cohen]

On Thu, May 19, 2011 at 06:06:03PM +0000, Hefty, Sean wrote:
> 
> max_qp_wr does not distinguish between max send or receive or indicate if those values should be the same.  IMO, Setting
> 
> max_qp_wr = max(send wr, recv wr)
> 
> makes more sense than
> 
> max_qp_wr = min(send wr, recv wr)
> 
> The maximum values reported are not meant to be guaranteed minimums.
> 

Here's what the spec says:
The maximum number of outstanding work requests on any
Work Queue supported by this HCA.

I think the text should be interpreted as you said:
max_qp_wr = max(send wr, recv wr)

But I also don't find much use for the value returned by query HCA.



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

rdma_create_qp() and max_send_wr

[Yann Droneaud]

Hi,

I have a problem with rdma_create_qp() when I set
qp_init_attr.cap.max_send_wr to something higher than 16351: 
it returns -1 and errno is set to ENOMEM "Cannot allocate memory".

strace doesn't show anything related to memory, but the last write()
syscall returns EINVAL "Invalid Argument".

I'm using a Mellanox ConnectX MT26428 (v2.6.100 / a0) HCA, under Debian
6.0 (kernel 2.6.32-5, librdmacm 1.0.10-1, libibverbs 1.1.3-2, libmlx4
1.0-1).

According to ibv_query_device() informations, 
ibv_device_attr.max_qp_wr is 16384 (and ibv_device_attr.max_qp_wr is
16383). So one might think that 16383 outstanding WR should be OK.

I've also tried to increase the length of the associated CQ, but it
doesn't change anything.

So what's the limit of WR in the QP's SQ ? 
Is it ibv_device_attr.max_qp_wr - 32 ?


Regards.

-- 
Yann Droneaud
OPTEYA


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Roland Dreier]

On Thu, Apr 21, 2011 at 9:44 AM, Yann Droneaud <ydroneaud-RlY5vtjFyJ3QT0dZR+AlfA@public.gmane.org> wrote:
> I have a problem with rdma_create_qp() when I set
> qp_init_attr.cap.max_send_wr to something higher than 16351:
> it returns -1 and errno is set to ENOMEM "Cannot allocate memory".
>
> strace doesn't show anything related to memory, but the last write()
> syscall returns EINVAL "Invalid Argument".
>
> I'm using a Mellanox ConnectX MT26428 (v2.6.100 / a0) HCA, under Debian
> 6.0 (kernel 2.6.32-5, librdmacm 1.0.10-1, libibverbs 1.1.3-2, libmlx4
> 1.0-1).
>
> According to ibv_query_device() informations,
> ibv_device_attr.max_qp_wr is 16384 (and ibv_device_attr.max_qp_wr is
> 16383). So one might think that 16383 outstanding WR should be OK.

Getting exactly the right value for max_qp_wr is kind of tricky because
of complicated allocation rules.  I guess this is just a mlx4 bug in
reporting not quite the right value from ibv_query_device().
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Or Gerlitz]

Roland Dreier wrote:
> Getting exactly the right value for max_qp_wr is kind of tricky because
> of complicated allocation rules.  I guess this is just a mlx4 bug in
> reporting not quite the right value from ibv_query_device().

Maybe the correct way to go for mlx4 is to go min/max that is, report 
the --minimal-- of max(recv, send) value that would work for an app 
setting of either of their send or the recv WR numbers.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rdma_create_qp() and max_send_wr

[Eli Cohen]

On Sun, May 22, 2011 at 09:46:21AM +0300, Or Gerlitz wrote:
> 
> Maybe the correct way to go for mlx4 is to go min/max that is,
> report the --minimal-- of max(recv, send) value that would work for
> an app setting of either of their send or the recv WR numbers.
> 

I see that OFED already contains a fix for this that Jack pushed. So I
guess Jack will send it for upstream submission.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html