* Questions about masked atomic
@ 2020-05-11 13:54 liweihang
2020-05-12 11:35 ` Leon Romanovsky
0 siblings, 1 reply; 6+ messages in thread
From: liweihang @ 2020-05-11 13:54 UTC (permalink / raw)
To: jgg, dledford; +Cc: linux-rdma, Linuxarm
Hi All,
I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
1. The kernel now supports masked atomic, but the it does not support atomic
operation. Is the masked atomic valid in kernel currently?
2. In the userspace, ofed does not have the corresponding opcode for the masked
atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
and ibv_send_wr also has no related data segment for it. How to support it in
userspace?
Thanks
Weihang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Questions about masked atomic
2020-05-11 13:54 Questions about masked atomic liweihang
@ 2020-05-12 11:35 ` Leon Romanovsky
2020-05-15 9:40 ` liweihang
0 siblings, 1 reply; 6+ messages in thread
From: Leon Romanovsky @ 2020-05-12 11:35 UTC (permalink / raw)
To: liweihang; +Cc: jgg, dledford, linux-rdma, Linuxarm
On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote:
> Hi All,
>
> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
>
> 1. The kernel now supports masked atomic, but the it does not support atomic
> operation. Is the masked atomic valid in kernel currently?
Yes, it is valid, but probably has a very little real value for the kernel ULPs.
I see code in the RDS that uses atomics, but it says nothing to me, because
upstream RDS and version in-real-use are completely different.
> 2. In the userspace, ofed does not have the corresponding opcode for the masked
> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
> and ibv_send_wr also has no related data segment for it. How to support it in
> userspace?
ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1]
with specific and new post builders.
Thanks
[1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md
>
> Thanks
> Weihang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Questions about masked atomic
2020-05-12 11:35 ` Leon Romanovsky
@ 2020-05-15 9:40 ` liweihang
2020-05-17 13:14 ` Leon Romanovsky
0 siblings, 1 reply; 6+ messages in thread
From: liweihang @ 2020-05-15 9:40 UTC (permalink / raw)
To: Leon Romanovsky; +Cc: jgg, dledford, linux-rdma, Linuxarm
On 2020/5/12 19:35, Leon Romanovsky wrote:
> On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote:
>> Hi All,
>>
>> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
>>
>> 1. The kernel now supports masked atomic, but the it does not support atomic
>> operation. Is the masked atomic valid in kernel currently?
>
> Yes, it is valid, but probably has a very little real value for the kernel ULPs.
> I see code in the RDS that uses atomics, but it says nothing to me, because
> upstream RDS and version in-real-use are completely different.
>
>> 2. In the userspace, ofed does not have the corresponding opcode for the masked
>> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
>> and ibv_send_wr also has no related data segment for it. How to support it in
>> userspace?
>
> ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1]
> with specific and new post builders.
>
> Thanks
>
> [1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md
>
Hi Leon,
Thanks for your response. May I ask another question:
Why it's not encouraged to use atomic/extended atomic/masked atomic operations in kernel?
Jason said that there seems no kernel users of extended atomic, is there any other reasons?
Weihang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Questions about masked atomic
2020-05-15 9:40 ` liweihang
@ 2020-05-17 13:14 ` Leon Romanovsky
2020-05-17 18:58 ` Tom Talpey
0 siblings, 1 reply; 6+ messages in thread
From: Leon Romanovsky @ 2020-05-17 13:14 UTC (permalink / raw)
To: liweihang; +Cc: jgg, dledford, linux-rdma, Linuxarm
On Fri, May 15, 2020 at 09:40:26AM +0000, liweihang wrote:
> On 2020/5/12 19:35, Leon Romanovsky wrote:
> > On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote:
> >> Hi All,
> >>
> >> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
> >>
> >> 1. The kernel now supports masked atomic, but the it does not support atomic
> >> operation. Is the masked atomic valid in kernel currently?
> >
> > Yes, it is valid, but probably has a very little real value for the kernel ULPs.
> > I see code in the RDS that uses atomics, but it says nothing to me, because
> > upstream RDS and version in-real-use are completely different.
> >
> >> 2. In the userspace, ofed does not have the corresponding opcode for the masked
> >> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
> >> and ibv_send_wr also has no related data segment for it. How to support it in
> >> userspace?
> >
> > ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1]
> > with specific and new post builders.
> >
> > Thanks
> >
> > [1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md
> >
>
> Hi Leon,
>
> Thanks for your response. May I ask another question:
>
> Why it's not encouraged to use atomic/extended atomic/masked atomic operations in kernel?
> Jason said that there seems no kernel users of extended atomic, is there any other reasons?
I don't think that "it is not encouraged", the more accurate will be
"the IBTA atomics will give nothing to the kernel ULPs".
The atomic data is not necessary stored in the host memory, while ULPs
need it in the memory. It means that they anyway will need to do some
synchronization in the host and "cancel" any advantage of atomics if
they exist.
Thanks
>
> Weihang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Questions about masked atomic
2020-05-17 13:14 ` Leon Romanovsky
@ 2020-05-17 18:58 ` Tom Talpey
2020-05-19 1:27 ` liweihang
0 siblings, 1 reply; 6+ messages in thread
From: Tom Talpey @ 2020-05-17 18:58 UTC (permalink / raw)
To: Leon Romanovsky, liweihang; +Cc: jgg, dledford, linux-rdma, Linuxarm
On 5/17/2020 9:14 AM, Leon Romanovsky wrote:
> On Fri, May 15, 2020 at 09:40:26AM +0000, liweihang wrote:
>> On 2020/5/12 19:35, Leon Romanovsky wrote:
>>> On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote:
>>>> Hi All,
>>>>
>>>> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
>>>>
>>>> 1. The kernel now supports masked atomic, but the it does not support atomic
>>>> operation. Is the masked atomic valid in kernel currently?
>>>
>>> Yes, it is valid, but probably has a very little real value for the kernel ULPs.
>>> I see code in the RDS that uses atomics, but it says nothing to me, because
>>> upstream RDS and version in-real-use are completely different.
>>>
>>>> 2. In the userspace, ofed does not have the corresponding opcode for the masked
>>>> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
>>>> and ibv_send_wr also has no related data segment for it. How to support it in
>>>> userspace?
>>>
>>> ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1]
>>> with specific and new post builders.
>>>
>>> Thanks
>>>
>>> [1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md
>>>
>>
>> Hi Leon,
>>
>> Thanks for your response. May I ask another question:
>>
>> Why it's not encouraged to use atomic/extended atomic/masked atomic operations in kernel?
>> Jason said that there seems no kernel users of extended atomic, is there any other reasons?
>
> I don't think that "it is not encouraged", the more accurate will be
> "the IBTA atomics will give nothing to the kernel ULPs".
>
> The atomic data is not necessary stored in the host memory, while ULPs
> need it in the memory. It means that they anyway will need to do some
> synchronization in the host and "cancel" any advantage of atomics if
> they exist.
Indeed, it is a common misconception by upper layer implementers that
the atomicity is available to the responder CPU. In fact, atomics work
only from the HCA that executes them, and the result is flushed to
memory, non-atomically, at some later time. These limitations greatly
reduce the motivation to use them at all, much less the exotic masked
ones.
I believe another reason they're not surfaced for kernel consumers is
that there aren't any. Primarily, the kernel consumers are storage, and
storage protocols stay far away from atomics.
Tom.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Questions about masked atomic
2020-05-17 18:58 ` Tom Talpey
@ 2020-05-19 1:27 ` liweihang
0 siblings, 0 replies; 6+ messages in thread
From: liweihang @ 2020-05-19 1:27 UTC (permalink / raw)
To: Tom Talpey, Leon Romanovsky; +Cc: jgg, dledford, linux-rdma, Linuxarm
On 2020/5/18 2:58, Tom Talpey wrote:
> On 5/17/2020 9:14 AM, Leon Romanovsky wrote:
>> On Fri, May 15, 2020 at 09:40:26AM +0000, liweihang wrote:
>>> On 2020/5/12 19:35, Leon Romanovsky wrote:
>>>> On Mon, May 11, 2020 at 01:54:48PM +0000, liweihang wrote:
>>>>> Hi All,
>>>>>
>>>>> I have two questions about masked atomic (Masked Compare and Swap & MFetchAdd):
>>>>>
>>>>> 1. The kernel now supports masked atomic, but the it does not support atomic
>>>>> operation. Is the masked atomic valid in kernel currently?
>>>>
>>>> Yes, it is valid, but probably has a very little real value for the kernel ULPs.
>>>> I see code in the RDS that uses atomics, but it says nothing to me, because
>>>> upstream RDS and version in-real-use are completely different.
>>>>
>>>>> 2. In the userspace, ofed does not have the corresponding opcode for the masked
>>>>> atomic (IB_WR_MASKED_ATOMIC_CMP_AND_SWP, IB_WR_MASKED_ATOMIC_FETCH_AND_ADD),
>>>>> and ibv_send_wr also has no related data segment for it. How to support it in
>>>>> userspace?
>>>>
>>>> ibv_send_wr is not extensible, so the real solution will need to extend ibv_wr_post() [1]
>>>> with specific and new post builders.
>>>>
>>>> Thanks
>>>>
>>>> [1] https://github.com/linux-rdma/rdma-core/blob/master/libibverbs/man/ibv_wr_post.3.md
>>>>
>>>
>>> Hi Leon,
>>>
>>> Thanks for your response. May I ask another question:
>>>
>>> Why it's not encouraged to use atomic/extended atomic/masked atomic operations in kernel?
>>> Jason said that there seems no kernel users of extended atomic, is there any other reasons?
>>
>> I don't think that "it is not encouraged", the more accurate will be
>> "the IBTA atomics will give nothing to the kernel ULPs".
>>
>> The atomic data is not necessary stored in the host memory, while ULPs
>> need it in the memory. It means that they anyway will need to do some
>> synchronization in the host and "cancel" any advantage of atomics if
>> they exist.
>
> Indeed, it is a common misconception by upper layer implementers that
> the atomicity is available to the responder CPU. In fact, atomics work
> only from the HCA that executes them, and the result is flushed to
> memory, non-atomically, at some later time. These limitations greatly
> reduce the motivation to use them at all, much less the exotic masked
> ones.
>
> I believe another reason they're not surfaced for kernel consumers is
> that there aren't any. Primarily, the kernel consumers are storage, and
> storage protocols stay far away from atomics.
>
> Tom.
>
Hi Tom and Leon,
Thank you for the explanation, it helps me lot.
Weihang
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-05-19 1:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-11 13:54 Questions about masked atomic liweihang
2020-05-12 11:35 ` Leon Romanovsky
2020-05-15 9:40 ` liweihang
2020-05-17 13:14 ` Leon Romanovsky
2020-05-17 18:58 ` Tom Talpey
2020-05-19 1:27 ` liweihang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).