From: Liran Alon <liran.alon@oracle.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: saeedm@mellanox.com, leon@kernel.org, netdev@vger.kernel.org,
linux-rdma@vger.kernel.org, eli@mellanox.com,
tariqt@mellanox.com, danielm@mellanox.com,
"Håkon Bugge" <haakon.bugge@oracle.com>
Subject: Re: [PATCH] net: mlx5: Use writeX() to ring doorbell and remove reduntant wmb()
Date: Thu, 2 Jan 2020 21:45:52 +0200 [thread overview]
Message-ID: <6524AE07-2ED7-41B5-B761-9F6BE8D2049B@oracle.com> (raw)
In-Reply-To: <20200102192934.GH9282@ziepe.ca>
> On 2 Jan 2020, at 21:29, Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Thu, Jan 02, 2020 at 07:44:36PM +0200, Liran Alon wrote:
>> Currently, mlx5e_notify_hw() executes wmb() to complete writes to cache-coherent
>> memory before ringing doorbell. Doorbell is written to by mlx5_write64()
>> which use __raw_writeX().
>>
>> This is semantically correct but executes reduntant wmb() in some architectures.
>> For example, in x86, a write to UC memory guarantees that any previous write to
>> WB memory will be globally visible before the write to UC memory. Therefore, there
>> is no need to also execute wmb() before write to doorbell which is mapped as UC memory.
>>
>> The consideration regarding this between different architectures is handled
>> properly by the writeX() macro. Which is defined differently for different
>> architectures. E.g. On x86, it is just a memory write. However, on ARM, it
>> is defined as __iowmb() folowed by a memory write. __iowmb() is defined
>> as wmb().
>
> This reasoning seems correct, though I would recommend directly
> refering to locking/memory-barriers.txt which explains this.
I find memory-barriers.txt not explicit enough on the semantics of writeX().
(For example: Should it flush write-combined buffers before writing to the UC memory?)
That’s why I preferred to explicitly state here how I perceive it.
But I don’t mind of course adding a pointer to the memory-barriers.txt file.
>
>> Therefore, change mlx5_write64() to use writeX() and remove wmb() from
>> it's callers.
>
> Yes, wmb(); writel(); is always redundant
Well, unfortunately not…
See: https://marc.info/?l=linux-netdev&m=157798859215697&w=2
(See my suggestion to add flush_wc_writeX())
>
>> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
>> index 40748fc1b11b..28744a725e64 100644
>> +++ b/include/linux/mlx5/cq.h
>> @@ -162,11 +162,6 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd,
>>
>> *cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
>>
>> - /* Make sure that the doorbell record in host memory is
>> - * written before ringing the doorbell via PCI MMIO.
>> - */
>> - wmb();
>> -
>
> Why did this one change? The doorbell memory here is not a writel():
Well, it’s not seen in the diff but actually the full code is:
/* Make sure that the doorbell record in host memory is
* written before ringing the doorbell via PCI MMIO.
*/
wmb();
doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
doorbell[1] = cpu_to_be32(cq->cqn);
mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL);
i.e. doorbell is not the write to the doorbell itself. It’s still done via mlx5_write64().
>
>> doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
>> doorbell[1] = cpu_to_be32(cq->cqn);
>
>> static inline void mlx5_write64(__be32 val[2], void __iomem *dest)
>> {
>> #if BITS_PER_LONG == 64
>> - __raw_writeq(*(u64 *)val, dest);
>> + writeq(*(u64 *)val, dest);
>
> I want to say this might cause problems with endian swapping as writeq
> also does some swaps that __raw does not? Is this true?
Hmm... Looking at ARM64 version, writeq() indeed calls cpu_to_le64() on parameter before passing it to __raw_writeq().
Quite surprising from API perspective to be honest.
So should I change this instead to iowrite64be(*(u64 *)val, dest)?
-Liran
>
> ie writeq does not accept a be32
>
> Some time ago I reworked this similar code in userspace to use a u64
> and remove the swapping from the caller.
>
> Jason
next prev parent reply other threads:[~2020-01-02 19:46 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-02 17:44 [PATCH] net: mlx5: Use writeX() to ring doorbell and remove reduntant wmb() Liran Alon
2020-01-02 19:29 ` Jason Gunthorpe
2020-01-02 19:45 ` Liran Alon [this message]
2020-01-02 20:58 ` Jason Gunthorpe
2020-01-02 22:21 ` Liran Alon
2020-01-03 13:37 ` Jason Gunthorpe
2020-01-03 16:31 ` Liran Alon
2020-01-03 16:36 ` Jason Gunthorpe
2020-01-03 16:42 ` Liran Alon
2020-01-03 18:38 ` Liran Alon
2020-01-03 19:06 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6524AE07-2ED7-41B5-B761-9F6BE8D2049B@oracle.com \
--to=liran.alon@oracle.com \
--cc=danielm@mellanox.com \
--cc=eli@mellanox.com \
--cc=haakon.bugge@oracle.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
--cc=tariqt@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).