From: Robin Murphy <robin.murphy@arm.com>
To: Will Deacon <will.deacon@arm.com>,
Zhangshaokun <zhangshaokun@hisilicon.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>,
steve.capper@arm.com, netdev@vger.kernel.org,
ilias.apalodimas@linaro.org,
"huanglingyan \(A\)" <huanglingyan2@huawei.com>,
linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH] arm64: do_csum: implement accelerated scalar version
Date: Mon, 15 Apr 2019 19:18:22 +0100 [thread overview]
Message-ID: <41b30c72-c1c5-14b2-b2e1-3507d552830d@arm.com> (raw)
In-Reply-To: <20190412095243.GA27193@fuggles.cambridge.arm.com>
On 12/04/2019 10:52, Will Deacon wrote:
> On Fri, Apr 12, 2019 at 10:31:16AM +0800, Zhangshaokun wrote:
>> On 2019/2/19 7:08, Ard Biesheuvel wrote:
>>> It turns out that the IP checksumming code is still exercised often,
>>> even though one might expect that modern NICs with checksum offload
>>> have no use for it. However, as Lingyan points out, there are
>>> combinations of features where the network stack may still fall back
>>> to software checksumming, and so it makes sense to provide an
>>> optimized implementation in software as well.
>>>
>>> So provide an implementation of do_csum() in scalar assembler, which,
>>> unlike C, gives direct access to the carry flag, making the code run
>>> substantially faster. The routine uses overlapping 64 byte loads for
>>> all input size > 64 bytes, in order to reduce the number of branches
>>> and improve performance on cores with deep pipelines.
>>>
>>> On Cortex-A57, this implementation is on par with Lingyan's NEON
>>> implementation, and roughly 7x as fast as the generic C code.
>>>
>>> Cc: "huanglingyan (A)" <huanglingyan2@huawei.com>
>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> ---
>>> Test code after the patch.
>>
>> Hi maintainers and Ard,
>>
>> Any update on it?
>
> I'm waiting for Robin to come back with numbers for a C implementation.
>
> Robin -- did you get anywhere with that?
Still not what I would call finished, but where I've got so far (besides
an increasingly elaborate test rig) is as below - it still wants some
unrolling in the middle to really fly (and actual testing on BE), but
the worst-case performance already equals or just beats this asm version
on Cortex-A53 with GCC 7 (by virtue of being alignment-insensitive and
branchless except for the loop). Unfortunately, the advantage of C code
being instrumentable does also come around to bite me...
Robin.
----->8-----
/* Looks dumb, but generates nice-ish code */
static u64 accumulate(u64 sum, u64 data)
{
__uint128_t tmp = (__uint128_t)sum + data;
return tmp + (tmp >> 64);
}
unsigned int do_csum_c(const unsigned char *buff, int len)
{
unsigned int offset, shift, sum, count;
u64 data, *ptr;
u64 sum64 = 0;
offset = (unsigned long)buff & 0x7;
/*
* This is to all intents and purposes safe, since rounding down cannot
* result in a different page or cache line being accessed, and @buff
* should absolutely not be pointing to anything read-sensitive.
* It does, however, piss off KASAN...
*/
ptr = (u64 *)(buff - offset);
shift = offset * 8;
/*
* Head: zero out any excess leading bytes. Shifting back by the same
* amount should be at least as fast as any other way of handling the
* odd/even alignment, and means we can ignore it until the very end.
*/
data = *ptr++;
#ifdef __LITTLE_ENDIAN
data = (data >> shift) << shift;
#else
data = (data << shift) >> shift;
#endif
count = 8 - offset;
/* Body: straightforward aligned loads from here on... */
//TODO: fancy stuff with larger strides and uint128s?
while(len > count) {
sum64 = accumulate(sum64, data);
data = *ptr++;
count += 8;
}
/*
* Tail: zero any over-read bytes similarly to the head, again
* preserving odd/even alignment.
*/
shift = (count - len) * 8;
#ifdef __LITTLE_ENDIAN
data = (data << shift) >> shift;
#else
data = (data >> shift) << shift;
#endif
sum64 = accumulate(sum64, data);
/* Finally, folding */
sum64 += (sum64 >> 32) | (sum64 << 32);
sum = sum64 >> 32;
sum += (sum >> 16) | (sum << 16);
if (offset & 1)
return (u16)swab32(sum);
return sum >> 16;
}
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2019-04-15 18:18 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-18 23:08 [PATCH] arm64: do_csum: implement accelerated scalar version Ard Biesheuvel
2019-02-19 15:08 ` Ilias Apalodimas
2019-02-28 14:16 ` Ard Biesheuvel
2019-02-28 15:13 ` Robin Murphy
2019-02-28 15:28 ` Ard Biesheuvel
2019-04-12 2:31 ` Zhangshaokun
2019-04-12 9:52 ` Will Deacon
2019-04-15 18:18 ` Robin Murphy [this message]
2019-05-15 9:47 ` Will Deacon
2019-05-15 10:15 ` David Laight
2019-05-15 10:57 ` Robin Murphy
2019-05-15 11:13 ` David Laight
2019-05-15 12:39 ` Robin Murphy
2019-05-15 13:54 ` David Laight
2019-05-15 11:02 ` Robin Murphy
2019-05-16 3:14 ` Zhangshaokun
2019-08-15 16:46 ` Will Deacon
2019-08-16 8:15 ` Shaokun Zhang
2019-08-16 14:55 ` Robin Murphy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=41b30c72-c1c5-14b2-b2e1-3507d552830d@arm.com \
--to=robin.murphy@arm.com \
--cc=ard.biesheuvel@linaro.org \
--cc=huanglingyan2@huawei.com \
--cc=ilias.apalodimas@linaro.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=netdev@vger.kernel.org \
--cc=steve.capper@arm.com \
--cc=will.deacon@arm.com \
--cc=zhangshaokun@hisilicon.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).