From: Tariq Toukan <tariqt@mellanox.com>
To: Eric Dumazet <eric.dumazet@gmail.com>,
"jianchao.wang" <jianchao.w.wang@oracle.com>,
Tariq Toukan <tariqt@mellanox.com>,
Jason Gunthorpe <jgg@ziepe.ca>
Cc: junxiao.bi@oracle.com, netdev@vger.kernel.org,
linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org,
Saeed Mahameed <saeedm@mellanox.com>
Subject: Re: [PATCH] net/mlx4_en: ensure rx_desc updating reaches HW before prod db updating
Date: Sun, 21 Jan 2018 11:31:30 +0200 [thread overview]
Message-ID: <dfc02a48-7d2a-56da-dc4e-d90a9fcc559c@mellanox.com> (raw)
In-Reply-To: <1516376999.3606.39.camel@gmail.com>
On 19/01/2018 5:49 PM, Eric Dumazet wrote:
> On Fri, 2018-01-19 at 23:16 +0800, jianchao.wang wrote:
>> Hi Tariq
>>
>> Very sad that the crash was reproduced again after applied the patch.
>>
>> --- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c
>> +++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
>> @@ -252,6 +252,7 @@ static inline bool mlx4_en_is_ring_empty(struct mlx4_en_rx_ring *ring)
>>
>> static inline void mlx4_en_update_rx_prod_db(struct mlx4_en_rx_ring *ring)
>> {
>> + dma_wmb();
>
> So... is wmb() here fixing the issue ?
>
>> *ring->wqres.db.db = cpu_to_be32(ring->prod & 0xffff);
>> }
>>
>> I analyzed the kdump, it should be a memory corruption.
>>
>> Thanks
>> Jianchao
Hmm, this is actually consistent with the example below [1].
AIU from the example, it seems that the dma_wmb/dma_rmb barriers are
good for synchronizing cpu/device accesses to the "Streaming DMA mapped"
buffers (the descriptors, went through the dma_map_page() API), but not
for the doorbell (a coherent memory, typically allocated via
dma_alloc_coherent) that requires using the stronger wmb() barrier.
[1] Documentation/memory-barriers.txt
(*) dma_wmb();
(*) dma_rmb();
These are for use with consistent memory to guarantee the ordering
of writes or reads of shared memory accessible to both the CPU and a
DMA capable device.
For example, consider a device driver that shares memory with a device
and uses a descriptor status value to indicate if the descriptor
belongs
to the device or the CPU, and a doorbell to notify it when new
descriptors are available:
if (desc->status != DEVICE_OWN) {
/* do not read data until we own descriptor */
dma_rmb();
/* read/modify data */
read_data = desc->data;
desc->data = write_data;
/* flush modifications before status update */
dma_wmb();
/* assign ownership */
desc->status = DEVICE_OWN;
/* force memory to sync before notifying device via MMIO */
wmb();
/* notify device of new descriptors */
writel(DESC_NOTIFY, doorbell);
}
The dma_rmb() allows us guarantee the device has released ownership
before we read the data from the descriptor, and the dma_wmb() allows
us to guarantee the data is written to the descriptor before the
device
can see it now has ownership. The wmb() is needed to guarantee
that the
cache coherent memory writes have completed before attempting a
write to
the cache incoherent MMIO region.
See Documentation/DMA-API.txt for more information on consistent
memory.
>> On 01/15/2018 01:50 PM, jianchao.wang wrote:
>>> Hi Tariq
>>>
>>> Thanks for your kindly response.
>>>
>>> On 01/14/2018 05:47 PM, Tariq Toukan wrote:
>>>> Thanks Jianchao for your patch.
>>>>
>>>> And Thank you guys for your reviews, much appreciated.
>>>> I was off-work on Friday and Saturday.
>>>>
>>>> On 14/01/2018 4:40 AM, jianchao.wang wrote:
>>>>> Dear all
>>>>>
>>>>> Thanks for the kindly response and reviewing. That's really appreciated.
>>>>>
>>>>> On 01/13/2018 12:46 AM, Eric Dumazet wrote:
>>>>>>> Does this need to be dma_wmb(), and should it be in
>>>>>>> mlx4_en_update_rx_prod_db ?
>>>>>>>
>>>>>>
>>>>>> +1 on dma_wmb()
>>>>>>
>>>>>> On what architecture bug was observed ?
>>>>>
>>>>> This issue was observed on x86-64.
>>>>> And I will send a new patch, in which replace wmb() with dma_wmb(), to customer
>>>>> to confirm.
>>>>
>>>> +1 on dma_wmb, let us know once customer confirms.
>>>> Please place it within mlx4_en_update_rx_prod_db as suggested.
>>>
>>> Yes, I have recommended it to customer.
>>> Once I get the result, I will share it here.
>>>> All other calls to mlx4_en_update_rx_prod_db are in control/slow path so I prefer being on the safe side, and care less about bulking the barrier.
>>>>
>>>> Thanks,
>>>> Tariq
>>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2018-01-21 9:31 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-12 3:42 [PATCH] net/mlx4_en: ensure rx_desc updating reaches HW before prod db updating Jianchao Wang
2018-01-12 3:42 ` Jianchao Wang
2018-01-12 16:32 ` Jason Gunthorpe
[not found] ` <20180112163247.GB15974-uk2M96/98Pc@public.gmane.org>
2018-01-12 16:46 ` Eric Dumazet
2018-01-12 16:46 ` Eric Dumazet
2018-01-12 19:53 ` Saeed Mahameed
[not found] ` <85116e56-52b1-944d-6ee2-916ccfc3a7a6-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2018-01-12 20:16 ` Eric Dumazet
2018-01-12 20:16 ` Eric Dumazet
2018-01-12 21:01 ` Saeed Mahameed
2018-01-12 21:21 ` Eric Dumazet
[not found] ` <e902138a-3508-3504-51e5-46152cc2fb31-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2018-01-13 19:15 ` Jason Gunthorpe
2018-01-13 19:15 ` Jason Gunthorpe
2018-01-14 2:40 ` jianchao.wang
[not found] ` <a40e44f4-106b-1075-8f92-f7741508372c-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-14 9:47 ` Tariq Toukan
2018-01-14 9:47 ` Tariq Toukan
2018-01-15 5:50 ` jianchao.wang
[not found] ` <fea0aa1c-b68e-9485-3826-2dfad7824911-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-19 15:16 ` jianchao.wang
2018-01-19 15:16 ` jianchao.wang
[not found] ` <53b1ac4d-a294-eb98-149e-65d7954243da-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-19 15:49 ` Eric Dumazet
2018-01-19 15:49 ` Eric Dumazet
2018-01-21 9:31 ` Tariq Toukan [this message]
[not found] ` <dfc02a48-7d2a-56da-dc4e-d90a9fcc559c-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2018-01-21 16:24 ` Tariq Toukan
2018-01-21 16:24 ` Tariq Toukan
2018-01-21 16:43 ` Eric Dumazet
[not found] ` <1516552998.3478.5.camel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-22 2:40 ` jianchao.wang
2018-01-22 2:40 ` jianchao.wang
2018-01-22 15:47 ` Jason Gunthorpe
2018-01-23 3:25 ` jianchao.wang
2018-01-22 2:12 ` jianchao.wang
[not found] ` <c8b0955b-a3fc-afe2-2c67-e655ca2ee6f6-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-25 3:27 ` jianchao.wang
2018-01-25 3:27 ` jianchao.wang
[not found] ` <532b4d71-e2eb-35f3-894e-1c3288e7bc3f-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-25 3:55 ` Eric Dumazet
2018-01-25 3:55 ` Eric Dumazet
[not found] ` <1516852543.3715.43.camel-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-25 6:25 ` jianchao.wang
2018-01-25 6:25 ` jianchao.wang
[not found] ` <89066a75-43db-0f62-f171-70b0abaa8ea0-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-01-25 9:54 ` Tariq Toukan
2018-01-25 9:54 ` Tariq Toukan
[not found] ` <918db4ec-8c3c-aafa-4be6-0e00a99632e2-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2018-01-27 12:41 ` jianchao.wang
2018-01-27 12:41 ` jianchao.wang
[not found] ` <d9883261-e93e-400a-757c-3a81d8b6aca1@mellanox.com>
2019-01-02 1:43 ` jianchao.wang
2018-01-21 20:40 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dfc02a48-7d2a-56da-dc4e-d90a9fcc559c@mellanox.com \
--to=tariqt@mellanox.com \
--cc=eric.dumazet@gmail.com \
--cc=jgg@ziepe.ca \
--cc=jianchao.w.wang@oracle.com \
--cc=junxiao.bi@oracle.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.