Re: Advertise maximum number of sg supported by driver in single request

Re: Advertise maximum number of sg supported by driver in single request

[Herbert Xu]

On Tue, Jan 14, 2020 at 03:23:30PM +0530, Ayush Sawal wrote:
> Hi all,
> 
> The hardware crypto drivers have a limit on max number of sgs they can
> handle per crypto request.If data size in one crypto request is
> huge,hardware crypto driver may not be able to send the request in single
> shot to hardware and end up using fallback to software.
> 
> Does it make sense to have a new API for crypto drivers using that drivers
> can advertise the max number of sg it can handle in one crypto request?
> 
> and then  crypto framework may also have to include the similar API which
> crypto framework user can use while forming the crypto request .
> 
> Does this implementation make sense?

What is the actual limit? Are you running into this limit with
real-life requests?

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Advertise maximum number of sg supported by driver in single request

[Ayush Sawal]

Hi Herbert,

Sorry for the late reply

On 15/01/2020 14:02:34 +0800, Herbert Xu wrote:

> On Tue, Jan 14, 2020 at 03:23:30PM +0530, Ayush Sawal wrote
>> Hi all,
>>
>> The hardware crypto drivers have a limit on max number of sgs they can
>> handle per crypto request.If data size in one crypto request is
>> huge,hardware crypto driver may not be able to send the request in single
>> shot to hardware and end up using fallback to software.
>>
>> Does it make sense to have a new API for crypto drivers using that 
>> drivers
>> can advertise the max number of sg it can handle in one crypto request?
>>
>> and then  crypto framework may also have to include the similar API which
>> crypto framework user can use while forming the crypto request .
>>
>> Does this implementation make sense?
>
> What is the actual limit? Are you running into this limit with
> real-life requests?

The max data limit is 15 sgs where each sg contains data of mtu size .
we are running a netperf udp stream test over ipsec tunnel .The ipsec 
tunnel is established between two hosts which are directly connected

Thanks,

Ayush

Re: Advertise maximum number of sg supported by driver in single request

[Herbert Xu]

On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>
> The max data limit is 15 sgs where each sg contains data of mtu size .
> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> is established between two hosts which are directly connected

Are you actually getting 15-element SG lists from IPsec? What is
generating an skb with 15-element SG lists?

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Advertise maximum number of sg supported by driver in single request

[Ayush Sawal]

Hi Herbert,

On 1/17/2020 11:53 AM, Herbert Xu wrote:
> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>> The max data limit is 15 sgs where each sg contains data of mtu size .
>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>> is established between two hosts which are directly connected
> Are you actually getting 15-element SG lists from IPsec? What is
> generating an skb with 15-element SG lists?
we have established the ipsec tunnel in transport mode using ip xfrm.
and running traffic using netserver and netperf.

In server side we are running
netserver -4
In client side we are running
"netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
where the packet size is 21k ,which is then fragmented into 15 ip 
fragments each of mtu size.
The mtu size currently is 1500bytes.

Re: Advertise maximum number of sg supported by driver in single request

[Steffen Klassert]

On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> Hi Herbert,
> 
> On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > is established between two hosts which are directly connected
> > Are you actually getting 15-element SG lists from IPsec? What is
> > generating an skb with 15-element SG lists?
> we have established the ipsec tunnel in transport mode using ip xfrm.
> and running traffic using netserver and netperf.
> 
> In server side we are running
> netserver -4
> In client side we are running
> "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
> where the packet size is 21k ,which is then fragmented into 15 ip fragments
> each of mtu size.

I'm lacking a bit of context here, but this should generate 15 IP
packets that are encrypted one by one.

Re: Advertise maximum number of sg supported by driver in single request

[Ayush Sawal]

Hi steffen,

On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>> Hi Herbert,
>>
>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>> is established between two hosts which are directly connected
>>> Are you actually getting 15-element SG lists from IPsec? What is
>>> generating an skb with 15-element SG lists?
>> we have established the ipsec tunnel in transport mode using ip xfrm.
>> and running traffic using netserver and netperf.
>>
>> In server side we are running
>> netserver -4
>> In client side we are running
>> "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>> each of mtu size.
> I'm lacking a bit of context here, but this should generate 15 IP
> packets that are encrypted one by one.
This is what i observed ,please correct me if i am wrong.
The packet when reaches esp_output(),is in socket buffer and based on 
the number of frags ,sg is initialized  using
sg_init_table(sg,frags),where frags are 15 in our case.

The socket buffer data is then copied to this sg and then struct 
aead_request members are filled.
After this crypto aead request which contains all data in its sg list 
goes to hw crypto driver for encryption in a single request.

In the crypto driver we are receiving a single aead-request with all 15 
sgs in that request.

Thanks,

Ayush

Re: Advertise maximum number of sg supported by driver in single request

[Steffen Klassert]

On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
> Hi steffen,
> 
> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> > On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> > > Hi Herbert,
> > > 
> > > On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > > > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > > > is established between two hosts which are directly connected
> > > > Are you actually getting 15-element SG lists from IPsec? What is
> > > > generating an skb with 15-element SG lists?
> > > we have established the ipsec tunnel in transport mode using ip xfrm.
> > > and running traffic using netserver and netperf.
> > > 
> > > In server side we are running
> > > netserver -4
> > > In client side we are running
> > > "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
> > > where the packet size is 21k ,which is then fragmented into 15 ip fragments
> > > each of mtu size.
> > I'm lacking a bit of context here, but this should generate 15 IP
> > packets that are encrypted one by one.
> This is what i observed ,please correct me if i am wrong.
> The packet when reaches esp_output(),is in socket buffer and based on the
> number of frags ,sg is initialized  using
> sg_init_table(sg,frags),where frags are 15 in our case.

The packet should be IP fragmented before it enters esp_output()
unless this is a UDP GSO packet. What kind of device do you use
here? Is it a crypto accelerator or a NIC that can do ESP offloads?

Re: Advertise maximum number of sg supported by driver in single request

[Ayush Sawal]

Hi steffen,

On 1/17/2020 5:47 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
>> Hi steffen,
>>
>> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
>>> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>>>> Hi Herbert,
>>>>
>>>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>>>> is established between two hosts which are directly connected
>>>>> Are you actually getting 15-element SG lists from IPsec? What is
>>>>> generating an skb with 15-element SG lists?
>>>> we have established the ipsec tunnel in transport mode using ip xfrm.
>>>> and running traffic using netserver and netperf.
>>>>
>>>> In server side we are running
>>>> netserver -4
>>>> In client side we are running
>>>> "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
>>>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>>>> each of mtu size.
>>> I'm lacking a bit of context here, but this should generate 15 IP
>>> packets that are encrypted one by one.
>> This is what i observed ,please correct me if i am wrong.
>> The packet when reaches esp_output(),is in socket buffer and based on the
>> number of frags ,sg is initialized  using
>> sg_init_table(sg,frags),where frags are 15 in our case.
> The packet should be IP fragmented before it enters esp_output()
> unless this is a UDP GSO packet. What kind of device do you use
> here? Is it a crypto accelerator or a NIC that can do ESP offloads?

We have device which works as a crypto accelerator . It just encrypts 
the packets and send it back to kernel.

Re: Advertise maximum number of sg supported by driver in single request

[Steffen Klassert]

On Fri, Jan 17, 2020 at 07:08:05PM +0530, Ayush Sawal wrote:
> Hi steffen,
> 
> On 1/17/2020 5:47 PM, Steffen Klassert wrote:
> > On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
> > > Hi steffen,
> > > 
> > > On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> > > > On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> > > > > Hi Herbert,
> > > > > 
> > > > > On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > > > > > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > > > > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > > > > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > > > > > is established between two hosts which are directly connected
> > > > > > Are you actually getting 15-element SG lists from IPsec? What is
> > > > > > generating an skb with 15-element SG lists?
> > > > > we have established the ipsec tunnel in transport mode using ip xfrm.
> > > > > and running traffic using netserver and netperf.
> > > > > 
> > > > > In server side we are running
> > > > > netserver -4
> > > > > In client side we are running
> > > > > "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
> > > > > where the packet size is 21k ,which is then fragmented into 15 ip fragments
> > > > > each of mtu size.
> > > > I'm lacking a bit of context here, but this should generate 15 IP
> > > > packets that are encrypted one by one.
> > > This is what i observed ,please correct me if i am wrong.
> > > The packet when reaches esp_output(),is in socket buffer and based on the
> > > number of frags ,sg is initialized  using
> > > sg_init_table(sg,frags),where frags are 15 in our case.
> > The packet should be IP fragmented before it enters esp_output()
> > unless this is a UDP GSO packet. What kind of device do you use
> > here? Is it a crypto accelerator or a NIC that can do ESP offloads?
> 
> We have device which works as a crypto accelerator . It just encrypts the
> packets and send it back to kernel.

I just did a test and I see the same behaviour. Seems like I was
mistaken, we actually fragment the ESP packets. The only case
where we do pre-encap fragmentation is IPv6 tunnel mode. But I
wonder if it would make sense to avoid to have ESP fragments on
the wire.

RE: Advertise maximum number of sg supported by driver in single request

[Van Leeuwen, Pascal]

Steffen,

> -----Original Message-----
> From: linux-crypto-owner@vger.kernel.org <linux-crypto-owner@vger.kernel.org> On Behalf Of Steffen Klassert
> Sent: Monday, January 20, 2020 10:37 AM
> To: Ayush Sawal <ayush.sawal@asicdesigners.com>
> Cc: Herbert Xu <herbert@gondor.apana.org.au>; linux-crypto@vger.kernel.org; manojmalviya@chelsio.com; Ayush Sawal
> <ayush.sawal@chelsio.com>; netdev@vger.kernel.org
> Subject: Re: Advertise maximum number of sg supported by driver in single request
>
> <<< External Email >>>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the
> sender/sender address and know the content is safe.
>
>
> On Fri, Jan 17, 2020 at 07:08:05PM +0530, Ayush Sawal wrote:
> > Hi steffen,
> >
> > On 1/17/2020 5:47 PM, Steffen Klassert wrote:
> > > On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
> > > > Hi steffen,
> > > >
> > > > On 1/17/2020 12:34 PM, Steffen Klassert wrote:
> > > > > On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
> > > > > > Hi Herbert,
> > > > > >
> > > > > > On 1/17/2020 11:53 AM, Herbert Xu wrote:
> > > > > > > On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
> > > > > > > > The max data limit is 15 sgs where each sg contains data of mtu size .
> > > > > > > > we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
> > > > > > > > is established between two hosts which are directly connected
> > > > > > > Are you actually getting 15-element SG lists from IPsec? What is
> > > > > > > generating an skb with 15-element SG lists?
> > > > > > we have established the ipsec tunnel in transport mode using ip xfrm.
> > > > > > and running traffic using netserver and netperf.
> > > > > >
> > > > > > In server side we are running
> > > > > > netserver -4
> > > > > > In client side we are running
> > > > > > "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
> > > > > > where the packet size is 21k ,which is then fragmented into 15 ip fragments
> > > > > > each of mtu size.
> > > > > I'm lacking a bit of context here, but this should generate 15 IP
> > > > > packets that are encrypted one by one.
> > > > This is what i observed ,please correct me if i am wrong.
> > > > The packet when reaches esp_output(),is in socket buffer and based on the
> > > > number of frags ,sg is initialized  using
> > > > sg_init_table(sg,frags),where frags are 15 in our case.
> > > The packet should be IP fragmented before it enters esp_output()
> > > unless this is a UDP GSO packet. What kind of device do you use
> > > here? Is it a crypto accelerator or a NIC that can do ESP offloads?
> >
> > We have device which works as a crypto accelerator . It just encrypts the
> > packets and send it back to kernel.
>
> I just did a test and I see the same behaviour. Seems like I was
> mistaken, we actually fragment the ESP packets. The only case
> where we do pre-encap fragmentation is IPv6 tunnel mode. But I
> wonder if it would make sense to avoid to have ESP fragments on
> the wire.
>
Well, for one thing, I don't know of any HW IPsec accelerator that can
handle fragmented IPsec packets directly. None of our hardware, that we've
been developing for over 2 decades now, can do that. All fragments would be
deferred to the slowpath for reassembly, killing performance.
So from that perspective you'd want to avoid systematic post-encapsulation
fragmentation whenever possible.
Proper path MTU discovery, accounting for the added IPsec headers, should
normally prevent this from being necessary.

Having said all that, it's not possible to encapsulate IPv4 fragments in transport
mode. So if PMTU discovery does not properly avoid that situation, then you
have no choice but to fragment _after_ ESP. But _only_ for that specific case.

Regards,
Pascal van Leeuwen
Silicon IP Architect Multi-Protocol Engines, Rambus Security
Rambus ROTW Holding BV
+31-73 6581953

Note: The Inside Secure/Verimatrix Silicon IP team was recently acquired by Rambus.
Please be so kind to update your e-mail address book with my new e-mail address.


** This message and any attachments are for the sole use of the intended recipient(s). It may contain information that is confidential and privileged. If you are not the intended recipient of this message, you are prohibited from printing, copying, forwarding or saving it. Please delete the message and attachments and notify the sender immediately. **

Rambus Inc.<http://www.rambus.com>

Re: Advertise maximum number of sg supported by driver in single request

[Ayush Sawal]

Hi Steffen,

On 1/20/2020 3:07 PM, Steffen Klassert wrote:
> On Fri, Jan 17, 2020 at 07:08:05PM +0530, Ayush Sawal wrote:
>> Hi steffen,
>>
>> On 1/17/2020 5:47 PM, Steffen Klassert wrote:
>>> On Fri, Jan 17, 2020 at 04:28:54PM +0530, Ayush Sawal wrote:
>>>> Hi steffen,
>>>>
>>>> On 1/17/2020 12:34 PM, Steffen Klassert wrote:
>>>>> On Fri, Jan 17, 2020 at 12:13:07PM +0530, Ayush Sawal wrote:
>>>>>> Hi Herbert,
>>>>>>
>>>>>> On 1/17/2020 11:53 AM, Herbert Xu wrote:
>>>>>>> On Thu, Jan 16, 2020 at 01:27:24PM +0530, Ayush Sawal wrote:
>>>>>>>> The max data limit is 15 sgs where each sg contains data of mtu size .
>>>>>>>> we are running a netperf udp stream test over ipsec tunnel .The ipsec tunnel
>>>>>>>> is established between two hosts which are directly connected
>>>>>>> Are you actually getting 15-element SG lists from IPsec? What is
>>>>>>> generating an skb with 15-element SG lists?
>>>>>> we have established the ipsec tunnel in transport mode using ip xfrm.
>>>>>> and running traffic using netserver and netperf.
>>>>>>
>>>>>> In server side we are running
>>>>>> netserver -4
>>>>>> In client side we are running
>>>>>> "netperf -H <serverip> -p <port> -t UDP_STREAM  -Cc -- -m 21k"
>>>>>> where the packet size is 21k ,which is then fragmented into 15 ip fragments
>>>>>> each of mtu size.
>>>>> I'm lacking a bit of context here, but this should generate 15 IP
>>>>> packets that are encrypted one by one.
>>>> This is what i observed ,please correct me if i am wrong.
>>>> The packet when reaches esp_output(),is in socket buffer and based on the
>>>> number of frags ,sg is initialized  using
>>>> sg_init_table(sg,frags),where frags are 15 in our case.
>>> The packet should be IP fragmented before it enters esp_output()
>>> unless this is a UDP GSO packet. What kind of device do you use
>>> here? Is it a crypto accelerator or a NIC that can do ESP offloads?
>> We have device which works as a crypto accelerator . It just encrypts the
>> packets and send it back to kernel.
> I just did a test and I see the same behaviour. Seems like I was
> mistaken, we actually fragment the ESP packets. The only case
> where we do pre-encap fragmentation is IPv6 tunnel mode. But I
> wonder if it would make sense to avoid to have ESP fragments on
> the wire.


As we have a crypto accelarator as device when the request is send to 
the crypto driver from esp_output ,
the aead_request has all the fragments in its src sg and the problem 
which we are facing is when this
src sg nents becomes greater than 15 ,15 is our crypto driver's max sg 
limit to handle the request in one shot.

Does it make sense for a crypto driver to advertise the maximum amount 
of sg it can handle for a single
request and then handling this in crypto framework while forming the 
crypto request?

Thanks,
Ayush

Re: Advertise maximum number of sg supported by driver in single request

[Gilad Ben-Yossef]

On Mon, Jan 20, 2020 at 2:35 PM Ayush Sawal
<ayush.sawal@asicdesigners.com> wrote:

> As we have a crypto accelarator as device when the request is send to
> the crypto driver from esp_output ,
> the aead_request has all the fragments in its src sg and the problem
> which we are facing is when this
> src sg nents becomes greater than 15 ,15 is our crypto driver's max sg
> limit to handle the request in one shot.
>
> Does it make sense for a crypto driver to advertise the maximum amount
> of sg it can handle for a single
> request and then handling this in crypto framework while forming the
> crypto request?
>

As I maintain the driver of another crypto accelerator I sympathize
with the need but I question the proposed solution.
Consider: your specific driver is limited by the number of
scattergather entries. Another implementation might be limited
by something else such as the total overall size of the request buffer
and probably half a dozen other considerations.
Should we now be passing all this capability information to the crypto
API core? and what happens if a new driver
has a limitation in a different quality?

So no, the solution to advertise the specific capability limitation of
each implementation does not seem to be a good one.
We already have a solution to the problem - initiate a fallback TFM
request and use it if you cannot fulfill the request on your own.

I do agree however that having each implementation registering and
keeping their own fallback tfm request just for these cases has some
overhead and a redundancy.

Maybe a better solution would be to allow implementation to return to
the Crypto API core a special return value (maybe -EAGAIN?) that tells
it that although the request is a valid one, this specific
implementation cannot fulfil it and let the crypto API core do the
fallback?

It sounds like it can be simpler to the implementation providers AND
save some redundant code...

-- 
Gilad Ben-Yossef
Chief Coffee Drinker

values of β will give rise to dom!