Performance issue with redfish TLS handshake

Performance issue with redfish TLS handshake

[sharad yadav]

[-- Attachment #1: Type: text/plain, Size: 1286 bytes --]

Hi All,

We have tried to measure redfish APIs performance benchmarking on AST2600.
On redfish GET request there is a penalty added for ~100ms on TLS handshake
at
https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L297

On trying below all methods, each request calls `async_handshake` which
adds 100ms delay
before the actual redfish handler code gets called.
*Method 1:*
curl --insecure -X POST -D headers.txt
https://${bmc}/redfish/v1/SessionService/Sessions
-d    '{"UserName":"root", "Password":"0penBmc"}'
export token=<Read X-Auth-Token from the headers.txt>
curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X
GET https://${bmc}/redfish/v1/Systems/system

*Method 2:*
export token=`curl -k -H "Content-Type: application/json" -X POST
https://${bmc}/login
-d '{"username" : "root", "password" : "0penBmc"}' | grep token | awk
'{print $2;}' | tr -d '"'`
curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X
GET https://${bmc}/redfish/v1/Systems/system

*Method 3:*
curl https://${bmc}/redfish/v1/Systems/system --insecure -u root:0penBmc -L

We want to avoid this ~100ms delay for better performance.
Please suggest if there is a way to skip the `async_handshake` call by
modifying the requests method?

Thanks,
Sharad

[-- Attachment #2: Type: text/html, Size: 2871 bytes --]

Re: Performance issue with redfish TLS handshake

[John Broadbent]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

On Tue, Oct 5, 2021 at 1:42 AM sharad yadav <sharad.openbmc@gmail.com>
wrote:

> Hi All,
>
> We have tried to measure redfish APIs performance benchmarking on AST2600.
> On redfish GET request there is a penalty added for ~100ms on TLS
> handshake at
> https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L297
>
> On trying below all methods, each request calls `async_handshake` which
> adds 100ms delay
> before the actual redfish handler code gets called.
> *Method 1:*
> curl --insecure -X POST -D headers.txt https://${bmc}/redfish/v1/SessionService/Sessions
> -d    '{"UserName":"root", "Password":"0penBmc"}'
> export token=<Read X-Auth-Token from the headers.txt>
> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X
> GET https://${bmc}/redfish/v1/Systems/system
>
> *Method 2:*
> export token=`curl -k -H "Content-Type: application/json" -X POST https://${bmc}/login
> -d '{"username" : "root", "password" : "0penBmc"}' | grep token | awk
> '{print $2;}' | tr -d '"'`
> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X
> GET https://${bmc}/redfish/v1/Systems/system
>
> *Method 3:*
> curl https://${bmc}/redfish/v1/Systems/system --insecure -u root:0penBmc
> -L
>
> We want to avoid this ~100ms delay for better performance.
> Please suggest if there is a way to skip the `async_handshake` call by
> modifying the requests method?
>
> Thanks,
> Sharad
>



There is logic in the crow::connection object that should allow you to use
tcp keep-alive and avoid the handshake in start.
https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L694

I have looked at the connection class in bmcweb before, and found it
difficult to understand.
However, this is a simplified version of the states within the connection
class:

start->doReadHeaders->doRead->handle->completeRequest->doWrite[if keep
alive]->doReadHeaders

The async_handshake is in the start, so if you are able to use the same
connection, you should only pay for the handshake once.
Ed Tanous and Gunnar Mills are the definitive experts.

Let us know what you find.
Thank you

[-- Attachment #2: Type: text/html, Size: 4202 bytes --]

Re: Performance issue with redfish TLS handshake

[Ed Tanous]

On Tue, Oct 5, 2021 at 11:48 AM John Broadbent <jebr@google.com> wrote:
>
>
>
> On Tue, Oct 5, 2021 at 1:42 AM sharad yadav <sharad.openbmc@gmail.com> wrote:
>>
>> Hi All,
>>
>> We have tried to measure redfish APIs performance benchmarking on AST2600.
>> On redfish GET request there is a penalty added for ~100ms on TLS handshake at

This is a little higher than I would've expected, but not outside the
realm of reasonable.  Can you triage what cipher suite you're
negotiating between the client and server?  Are you using a DH+EC key
cipher?  That should be faster than RSA.

>> https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L297
>>
>> On trying below all methods, each request calls `async_handshake` which adds 100ms delay
>> before the actual redfish handler code gets called.
>> Method 1:
>> curl --insecure -X POST -D headers.txt https://${bmc}/redfish/v1/SessionService/Sessions -d    '{"UserName":"root", "Password":"0penBmc"}'
>> export token=<Read X-Auth-Token from the headers.txt>
>> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X GET https://${bmc}/redfish/v1/Systems/system
>>
>> Method 2:
>> export token=`curl -k -H "Content-Type: application/json" -X POST https://${bmc}/login -d '{"username" : "root", "password" : "0penBmc"}' | grep token | awk '{print $2;}' | tr -d '"'`
>> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X GET https://${bmc}/redfish/v1/Systems/system
>>
>> Method 3:
>> curl https://${bmc}/redfish/v1/Systems/system --insecure -u root:0penBmc -L
>>
>> We want to avoid this ~100ms delay for better performance.
>> Please suggest if there is a way to skip the `async_handshake` call by modifying the requests method?
>>
>> Thanks,
>> Sharad
>
>
>
>
> There is logic in the crow::connection object that should allow you to use tcp keep-alive and avoid the handshake in start.
> https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L694
>
> I have looked at the connection class in bmcweb before, and found it difficult to understand.
> However, this is a simplified version of the states within the connection class:
>
> start->doReadHeaders->doRead->handle->completeRequest->doWrite[if keep alive]->doReadHeaders
>
> The async_handshake is in the start, so if you are able to use the same connection, you should only pay for the handshake once.
> Ed Tanous and Gunnar Mills are the definitive experts.

Yep, John got this exactly right.  Make sure whatever client you're
using is taking advantage of keepalive, and you will only take this
TLS performance hit for the first request.

>
>
> Let us know what you find.
> Thank you

Re: Performance issue with redfish TLS handshake

[sharad yadav]

[-- Attachment #1: Type: text/plain, Size: 3257 bytes --]

Thanks for the help. It worked out with the client setting keepalive=true.
Though curl is pre-enabled with keepalive=true. Below curl command worked
out to avoid TLS handshake in subsequent calls.
curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json" -X
GET https://${bmc}/redfish/v1/Systems/system https://${bmc}/redfish/v1/Syst
ems/system

Thanks,
Sharad

On Wed, 6 Oct 2021 at 00:23, Ed Tanous <edtanous@google.com> wrote:

> On Tue, Oct 5, 2021 at 11:48 AM John Broadbent <jebr@google.com> wrote:
> >
> >
> >
> > On Tue, Oct 5, 2021 at 1:42 AM sharad yadav <sharad.openbmc@gmail.com>
> wrote:
> >>
> >> Hi All,
> >>
> >> We have tried to measure redfish APIs performance benchmarking on
> AST2600.
> >> On redfish GET request there is a penalty added for ~100ms on TLS
> handshake at
>
> This is a little higher than I would've expected, but not outside the
> realm of reasonable.  Can you triage what cipher suite you're
> negotiating between the client and server?  Are you using a DH+EC key
> cipher?  That should be faster than RSA.
>
> >>
> https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L297
> >>
> >> On trying below all methods, each request calls `async_handshake` which
> adds 100ms delay
> >> before the actual redfish handler code gets called.
> >> Method 1:
> >> curl --insecure -X POST -D headers.txt https://${bmc}/redfish/v1/SessionService/Sessions
> -d    '{"UserName":"root", "Password":"0penBmc"}'
> >> export token=<Read X-Auth-Token from the headers.txt>
> >> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json"
> -X GET https://${bmc}/redfish/v1/Systems/system
> >>
> >> Method 2:
> >> export token=`curl -k -H "Content-Type: application/json" -X POST
> https://${bmc}/login -d '{"username" : "root", "password" : "0penBmc"}' |
> grep token | awk '{print $2;}' | tr -d '"'`
> >> curl -k -H "X-Auth-Token: $token" -H "Content-Type: application/json"
> -X GET https://${bmc}/redfish/v1/Systems/system
> >>
> >> Method 3:
> >> curl https://${bmc}/redfish/v1/Systems/system --insecure -u
> root:0penBmc -L
> >>
> >> We want to avoid this ~100ms delay for better performance.
> >> Please suggest if there is a way to skip the `async_handshake` call by
> modifying the requests method?
> >>
> >> Thanks,
> >> Sharad
> >
> >
> >
> >
> > There is logic in the crow::connection object that should allow you to
> use tcp keep-alive and avoid the handshake in start.
> >
> https://github.com/openbmc/bmcweb/blob/master/http/http_connection.hpp#L694
> >
> > I have looked at the connection class in bmcweb before, and found it
> difficult to understand.
> > However, this is a simplified version of the states within the
> connection class:
> >
> > start->doReadHeaders->doRead->handle->completeRequest->doWrite[if keep
> alive]->doReadHeaders
> >
> > The async_handshake is in the start, so if you are able to use the same
> connection, you should only pay for the handshake once.
> > Ed Tanous and Gunnar Mills are the definitive experts.
>
> Yep, John got this exactly right.  Make sure whatever client you're
> using is taking advantage of keepalive, and you will only take this
> TLS performance hit for the first request.
>
> >
> >
> > Let us know what you find.
> > Thank you
>

[-- Attachment #2: Type: text/html, Size: 4726 bytes --]