Re: [PATCH for-next] RDMA/hns: Add support for extended atomic in userspace

[Weihang Li <liweihang@huawei.com>]

On 2020/1/22 22:08, Tom Talpey wrote:
> On 1/22/2020 3:54 AM, Weihang Li wrote:
>>
>>
>> On 2020/1/17 3:51, Jason Gunthorpe wrote:
>>>>> What happens to your userspace if it runs on an old kernel and tries
>>>>> to use extended atomic?
>>>>>
>>>>> Jason
>>>>>
>>>> Hi Jason,
>>>>
>>>> If the hns userspace runs with old kernel, the hardware will report a asynchronous
>>>> event for the extended atomic operation and modify the qp to error state because
>>>> the enable bit in this qp's context hasn't been set.
>>>>
>>>> The driver will print like this:
>>>>
>>>> [ 1252.240921] hns3 0000:7d:00.0: Invalid request local work queue 0x9 error.
>>>> [ 1252.247772] hns3 0000:7d:00.0: no hr_qp can be found!
>>> Ideally the provider will not set
>>> IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP and related without kernel
>>> support..
>>>
>>> I've applied this patch, but I feel like you may need a followup to
>>> fix the capability reporting?
>>>
>>> Jason
>>
>> Hi Jason,
>>
>> Thank for your suggestions.
>>
>> But I'm confuse about the relationship between "PCI ATOMIC" in this macro
>> and atomic operations in RDMA.
> 
> PCI Atomics are optonal and are a much more recent facility.
> 
> RDMA Atomics do not require PCI Atomics, because they have
> different semantics with respect to memory atomicity. Read
> carefully and you'll see that they operate atomically only
> within the adapter, and are not atomic all the way to the
> underlying memory. It's a long and somewhat historical story.
> 
> Now that PCIe Atomics are becoming more widely supported by
> processor complexes, there is the possibility these may be
> more tightly embraced by RDMA implementations. There is
> discussion in IBTA and IETF around this currently, in fact,
> for the new RDMA Atomic Write operation.
> 
> Be aware that PCI Atomics are relatively expensive operations.
> The existing ones perform a read-modify-write cycle on both
> PCI and memory busses. This overhead is not to be taken lightly.
> 
> Tom.
> 

Hi Tom,

Thanks for your patience and detailed explanation :)
I know more about their relationship now.

Weihang


>> I found the related series on patchwork:
>> https://patchwork.kernel.org/cover/10782873/
>>
>> And I found the description about atomic operations in PCIe specification
>> v4.0:
>>
>> "An Atomic Operation (AtomicOp) is a single PCI Express transaction that
>> targets a location in Memory Space, reads the location’s value, potentially
>> writes a new value back to the location, and returns the original value. This
>> "read-modify-write" sequence to the location is performed atomically."
>>
>> It seems that the atomic for PCIe and RDMA is different concepts, and the macro
>> IBV_PCI_ATOMIC_OPERATION_4_BYTE_SIZE_SUP is for PCIe atomic.
>>
>> Could you please give me more suggestions about them?
>>
>> Thanks
>> Weihang
>>
>>
>>
> 
> .