From: Liran Alon <liran.alon@oracle.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: saeedm@mellanox.com, leon@kernel.org, netdev@vger.kernel.org,
linux-rdma@vger.kernel.org, eli@mellanox.com,
tariqt@mellanox.com, danielm@mellanox.com,
"Håkon Bugge" <haakon.bugge@oracle.com>
Subject: Re: [PATCH v2] net: mlx5: Use iowriteXbe() to ring doorbell and remove reduntant wmb()
Date: Sat, 4 Jan 2020 00:28:36 +0200 [thread overview]
Message-ID: <11EBB1F4-6D5A-4047-99A1-D339B7C8A697@oracle.com> (raw)
In-Reply-To: <20200103191749.GE9706@ziepe.ca>
> On 3 Jan 2020, at 21:17, Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Fri, Jan 03, 2020 at 07:52:07PM +0200, Liran Alon wrote:
>> diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
>> index 40748fc1b11b..4631ad35da53 100644
>> +++ b/include/linux/mlx5/cq.h
>> @@ -162,13 +162,8 @@ static inline void mlx5_cq_arm(struct mlx5_core_cq *cq, u32 cmd,
>>
>> *cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
>>
>> - /* Make sure that the doorbell record in host memory is
>> - * written before ringing the doorbell via PCI MMIO.
>> - */
>> - wmb();
>> -
>> - doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
>> - doorbell[1] = cpu_to_be32(cq->cqn);
>> + doorbell[0] = sn << 28 | cmd | ci;
>> + doorbell[1] = cq->cqn;
>
> This does actually have to change to a u64 otherwise it is not the
> same.
>
> On x86 LE, it was
> db[0] = swab(a)
> db[1] = swab(b)
> __raw_writel(db)
>
> Now it is
> db[0] = a
> db[1] = b
> __raw_writel(swab(db))
>
> Putting the swab around the u64 swaps the order of a/b in the TLP.
>
> It might be tempting to swap db[0]/db[1] but IIRC this messed it up on
> BE.
Oops. You are right...
>
> The sanest, simplest solution is to use a u64 natively, as the example
> I gave did.
I agree.
>
> There is also the issue of casting a u32 to a u64 and possibly
> triggering a unaligned kernel access, presumably this doesn't happen
> today only by some lucky chance..
>
>> mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL);
>> }
>> diff --git a/include/linux/mlx5/doorbell.h b/include/linux/mlx5/doorbell.h
>> index 5c267707e1df..9c1d35777323 100644
>> +++ b/include/linux/mlx5/doorbell.h
>> @@ -43,17 +43,15 @@
>> * Note that the write is not atomic on 32-bit systems! In contrast to 64-bit
>> * ones, it requires proper locking. mlx5_write64 doesn't do any locking, so use
>> * it at your own discretion, protected by some kind of lock on 32 bits.
>> - *
>> - * TODO: use write{q,l}_relaxed()
>> */
>>
>> -static inline void mlx5_write64(__be32 val[2], void __iomem *dest)
>> +static inline void mlx5_write64(u32 val[2], void __iomem *dest)
>> {
>
> So this should accept a straight u64, the goofy arrays have to go away
I agree.
>
>> #if BITS_PER_LONG == 64
>> - __raw_writeq(*(u64 *)val, dest);
>> + iowrite64be(*(u64 *)val, dest);
>> #else
>> - __raw_writel((__force u32) val[0], dest);
>> - __raw_writel((__force u32) val[1], dest + 4);
>> + iowrite32be(val[0], dest);
>> + iowrite32be(val[1], dest + 4);
>
> With a u64 input this fallback is written as
>
> iowrite32be(val >> 32, dest)
> iowrite32be((u32)val, dest + 4)
>
> Which matches the definition for how write64 must construct a TLP.
>
> And arguably the first one should be _relaxed (but nobody cares about
> this code path)
I agree with everything. Will fix on v3.
Thanks!
-Liran
>
> Jason
next prev parent reply other threads:[~2020-01-03 22:28 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-03 17:52 [PATCH v2] net: mlx5: Use iowriteXbe() to ring doorbell and remove reduntant wmb() Liran Alon
2020-01-03 19:17 ` Jason Gunthorpe
2020-01-03 22:28 ` Liran Alon [this message]
2020-01-06 22:05 ` Saeed Mahameed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=11EBB1F4-6D5A-4047-99A1-D339B7C8A697@oracle.com \
--to=liran.alon@oracle.com \
--cc=danielm@mellanox.com \
--cc=eli@mellanox.com \
--cc=haakon.bugge@oracle.com \
--cc=jgg@ziepe.ca \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=saeedm@mellanox.com \
--cc=tariqt@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).