* Re: [BUG] Layerscape CAAM+kTLS+io_uring
[not found] <878rrqrgaj.fsf@pengutronix.de>
@ 2022-05-05 19:20 ` Marc Kleine-Budde
2022-05-06 0:10 ` Jakub Kicinski
0 siblings, 1 reply; 4+ messages in thread
From: Marc Kleine-Budde @ 2022-05-05 19:20 UTC (permalink / raw)
To: Steffen Trumtrar
Cc: linux-crypto, io-uring, kernel, Horia Geantă,
Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
Boris Pismenny, John Fastabend, Daniel Borkmann, Jakub Kicinski,
netdev
[-- Attachment #1: Type: text/plain, Size: 12690 bytes --]
Hello,
no one seems to care about this problem. :/
Maybe too many components are involved, I'm the respective maintainers
on Cc.
Cc += the CAAM maintainers
Cc += the io_uring maintainers
Cc += the kTLS maintainers
On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> Hi all,
>
> I have a Layerscape-1046a based board where I'm trying to use a
> combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> v5.17). The problem I see is that on shutdown the last bytes are
> missing. It looks like io_uring is not waiting for all completions
> from the CAAM driver.
>
> With ARM-ASM instead of the CAAM, the setup works fine.
What's the difference between the CAAM and ARM-ASM crypto? Without
looking into the code I think the CAAM is asynchron while ARM-ASM is
synchron. Is this worth investigating?
> I tried to debug with ftrace and see where it goes wrong. Here is what
> seems to be (at least to me) some of the last bytes:
>
> webserver-612 [002] ..... 135.300350: io_uring_file_get: ring 0000000078f4a859, fd 6 00078f4a859, fd 7
> webserver-612 [002] ..... 135.300352: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fbb9b849, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300353: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ff858bdf, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300353: io_uring_link: ring 0000000078f4a859, request 00000000ff858bdf linked after 00000000fbb9b849
> webserver-612 [002] ..... 135.300354: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000fbb9b849, flags 798724, normal queue, work 0000000060cd323f
> webserver-612 [002] ..... 135.300358: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300375: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300379: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300388: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300389: io_uring_file_get: ring 0000000078f4a859, fd 7 0000078f4a859, req 000000008c2bf2be, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300390: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fe4e50d1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300391: io_uring_link: ring 0000000078f4a859, request 00000000fe4e50d1 linked after 000000008c2bf2be
> webserver-612 [002] ..... 135.300392: io_uring_queue_async_work: ring 0000000078f4a859, request 000000008c2bf2be, flags 798724, normal queue, work 00000000f2b434fc
> webserver-612 [002] ..... 135.300396: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300410: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300414: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300423: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300424: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300424: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e48f3098, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300426: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ec67d53c, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300426: io_uring_link: ring 0000000078f4a859, request 00000000ec67d53c linked after 00000000e48f3098
> webserver-612 [002] ..... 135.300427: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000e48f3098, flags 798724, normal queue, work 000000009e3701da
> webserver-612 [002] ..... 135.300431: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300447: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300452: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300461: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300462: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300462: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f658f96f, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0, sq_thread 0
> webserver-612 [002] ..... 135.300464: io_uring_link: ring 0000000078f4a859, request 000000006c29e721 linked after 00000000f658f96f
> webserver-612 [002] ..... 135.300465: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000f658f96f, flags 798724, normal queue, work 000000007434c68b
> webserver-612 [002] ..... 135.300469: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300479: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300483: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300492: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300493: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300494: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000757ef148, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300495: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e5c82137, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 8f4a859, request 00000000e5c82137 linked after 00000000757ef148
> webserver-612 [002] ..... 135.300496: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000757ef148, flags 798724, normal queue, work 00000000b55630dd
> webserver-612 [002] ..... 135.300500: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300516: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300520: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300529: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300530: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300531: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000085e5cac4, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300532: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000008e6a863, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300532: io_uring_link: ring 0000000078f4a859, request 0000000008e6a863 linked after 0000000085e5cac4 ng 0000000078f4a859, request 0000000085e5cac4, flags 798724, normal queue, work 0000000036c4ff52
> webserver-612 [002] ..... 135.300537: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300553: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300557: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300566: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300567: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300567: io_uring_submit_sqe: ring 0000000078f4a859, req 000000000f7fdd39, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300568: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000741c64e1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300569: io_uring_link: ring 0000000078f4a859, request 00000000741c64e1 linked after 000000000f7fdd39
> webserver-612 [002] ..... 135.300570: io_uring_queue_async_work: ring 0000000078f4a859, request 000000000f7fdd39, flags 798724, normal queue, work 00000000fc4accf1
> webserver-612 [002] ..... 135.300574: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300594: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300598: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.300607: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.300608: io_uring_file_get: ring 0000000078f4a859, fd 7
> webserver-612 [002] ..... 135.300608: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000052b47765, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300610: io_uring_submit_sqe: ring 0000000078f4a859, req 000000003904ded9, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.300611: io_uring_link: ring 0000000078f4a859, request 000000003904ded9 linked after 0000000052b47765
> webserver-612 [002] ..... 135.300612: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000052b47765, flags 798724, normal queue, work 00000000e11c8599
> webserver-612 [002] ..... 135.300615: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-647 [003] ...1. 135.300631: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 768, cflags 0
> iou-wrk-612-647 [003] d..2. 135.300634: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
> webserver-612 [002] ..... 135.301668: io_uring_file_get: ring 0000000078f4a859, fd 6
> webserver-612 [002] ..... 135.301669: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000012863980, op 34, data 0x0, flags 8, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.301670: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f4b07ff9, op 19, data 0x0, flags 0, non block 1, sq_thread 0
> webserver-612 [002] ..... 135.301671: io_uring_link: ring 0000000078f4a859, request 00000000f4b07ff9 linked after 0000000012863980
> webserver-612 [002] ..... 135.301672: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000012863980, flags 262152, normal queue, work 00000000102270ed
> webserver-612 [002] ..... 135.301740: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> iou-wrk-612-648 [000] ...1. 135.301757: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
> iou-wrk-612-648 [000] ...1. 135.301767: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
> webserver-612 [002] ..... 135.301769: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>
>
> Userspace said that 768 bytes where missing.
>
> Any ideas for how to debug this or why the async work queue doesn't work with
> the CAMM but does with ARM-ASM?
> If I can provide more info that might help, I'll try to produce and
> provide it. As there are multiple components involved, I'm not sure
> where to start or what information is useful. Currently looks like CAAM
> is the culprit.
Can you provide test code or at least illustrate with code how you plug
the components together?
Marc
--
Pengutronix e.K. | Marc Kleine-Budde |
Embedded Linux | https://www.pengutronix.de |
Vertretung West/Dortmund | Phone: +49-231-2826-924 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring
2022-05-05 19:20 ` [BUG] Layerscape CAAM+kTLS+io_uring Marc Kleine-Budde
@ 2022-05-06 0:10 ` Jakub Kicinski
2022-05-16 10:06 ` Steffen Trumtrar
0 siblings, 1 reply; 4+ messages in thread
From: Jakub Kicinski @ 2022-05-06 0:10 UTC (permalink / raw)
To: Marc Kleine-Budde
Cc: Steffen Trumtrar, linux-crypto, io-uring, kernel,
Horia Geantă,
Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
Boris Pismenny, John Fastabend, Daniel Borkmann, netdev
On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
> Hello,
>
> no one seems to care about this problem. :/
>
> Maybe too many components are involved, I'm the respective maintainers
> on Cc.
>
> Cc += the CAAM maintainers
> Cc += the io_uring maintainers
> Cc += the kTLS maintainers
>
> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> > Hi all,
> >
> > I have a Layerscape-1046a based board where I'm trying to use a
> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> > v5.17). The problem I see is that on shutdown the last bytes are
> > missing. It looks like io_uring is not waiting for all completions
> > from the CAAM driver.
> >
> > With ARM-ASM instead of the CAAM, the setup works fine.
>
> What's the difference between the CAAM and ARM-ASM crypto? Without
> looking into the code I think the CAAM is asynchron while ARM-ASM is
> synchron. Is this worth investigating?
Sounds like
20ffc7adf53a ("net/tls: missing received data after fast remote close")
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring
2022-05-06 0:10 ` Jakub Kicinski
@ 2022-05-16 10:06 ` Steffen Trumtrar
2022-05-17 6:30 ` [EXT] " Gaurav Jain
0 siblings, 1 reply; 4+ messages in thread
From: Steffen Trumtrar @ 2022-05-16 10:06 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Marc Kleine-Budde, linux-crypto, io-uring, kernel,
Horia Geantă,
Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
Boris Pismenny, John Fastabend, Daniel Borkmann, netdev
Jakub Kicinski <kuba@kernel.org> writes:
> On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
>> Hello,
>>
>> no one seems to care about this problem. :/
>>
>> Maybe too many components are involved, I'm the respective maintainers
>> on Cc.
>>
>> Cc += the CAAM maintainers
>> Cc += the io_uring maintainers
>> Cc += the kTLS maintainers
>>
>> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
>> > Hi all,
>> >
>> > I have a Layerscape-1046a based board where I'm trying to use a
>> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
>> > v5.17). The problem I see is that on shutdown the last bytes are
>> > missing. It looks like io_uring is not waiting for all completions
>> > from the CAAM driver.
>> >
>> > With ARM-ASM instead of the CAAM, the setup works fine.
>>
>> What's the difference between the CAAM and ARM-ASM crypto? Without
>> looking into the code I think the CAAM is asynchron while ARM-ASM is
>> synchron. Is this worth investigating?
>
> Sounds like
> 20ffc7adf53a ("net/tls: missing received data after fast remote close")
That fixes something in tls_sw. I have a kernel that includes this
patch. So this sounds right, but can't be it, right?
Best regards,
Steffen
--
Pengutronix e.K. | Dipl.-Inform. Steffen Trumtrar |
Steuerwalder Str. 21 | https://www.pengutronix.de/ |
31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686| Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring
2022-05-16 10:06 ` Steffen Trumtrar
@ 2022-05-17 6:30 ` Gaurav Jain
0 siblings, 0 replies; 4+ messages in thread
From: Gaurav Jain @ 2022-05-17 6:30 UTC (permalink / raw)
To: Steffen Trumtrar, Jakub Kicinski
Cc: Marc Kleine-Budde, linux-crypto, io-uring, kernel, Horia Geanta,
Pankaj Gupta, Jens Axboe, Pavel Begunkov, Boris Pismenny,
John Fastabend, Daniel Borkmann, netdev
Hello Steffen
> -----Original Message-----
> From: Steffen Trumtrar <s.trumtrar@pengutronix.de>
> Sent: Monday, May 16, 2022 3:36 PM
> To: Jakub Kicinski <kuba@kernel.org>
> Cc: Marc Kleine-Budde <mkl@pengutronix.de>; linux-crypto@vger.kernel.org;
> io-uring@vger.kernel.org; kernel@pengutronix.de; Horia Geanta
> <horia.geanta@nxp.com>; Pankaj Gupta <pankaj.gupta@nxp.com>; Gaurav Jain
> <gaurav.jain@nxp.com>; Jens Axboe <axboe@kernel.dk>; Pavel Begunkov
> <asml.silence@gmail.com>; Boris Pismenny <borisp@nvidia.com>; John
> Fastabend <john.fastabend@gmail.com>; Daniel Borkmann
> <daniel@iogearbox.net>; netdev@vger.kernel.org
> Subject: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring
>
> Caution: EXT Email
>
> Jakub Kicinski <kuba@kernel.org> writes:
>
> > On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
> >> Hello,
> >>
> >> no one seems to care about this problem. :/
> >>
> >> Maybe too many components are involved, I'm the respective
> >> maintainers on Cc.
> >>
> >> Cc += the CAAM maintainers
> >> Cc += the io_uring maintainers
> >> Cc += the kTLS maintainers
> >>
> >> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> >> > Hi all,
> >> >
> >> > I have a Layerscape-1046a based board where I'm trying to use a
> >> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> >> > v5.17). The problem I see is that on shutdown the last bytes are
> >> > missing. It looks like io_uring is not waiting for all completions
> >> > from the CAAM driver.
> >> >
> >> > With ARM-ASM instead of the CAAM, the setup works fine.
> >>
> >> What's the difference between the CAAM and ARM-ASM crypto? Without
> >> looking into the code I think the CAAM is asynchron while ARM-ASM is
> >> synchron. Is this worth investigating?
> >
> > Sounds like
> > 20ffc7adf53a ("net/tls: missing received data after fast remote
> > close")
>
> That fixes something in tls_sw. I have a kernel that includes this patch. So this
> sounds right, but can't be it, right?
>
To understand more on problem, can you share how you are offloading ktls to CAAM?
Gaurav
>
> Best regards,
> Steffen
>
> --
> Pengutronix e.K. | Dipl.-Inform. Steffen Trumtrar |
> Steuerwalder Str. 21 |
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pe
> ngutronix.de%2F&data=05%7C01%7Cgaurav.jain%40nxp.com%7Cc225fa4
> 3da7f4a43f16b08da3723dc4a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%
> 7C0%7C637882924400875890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL
> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&sdata=fI1BHNZIT5RlpjEdEMXVkv1WaHcvgTlZOcaC0LRojac%3D&
> reserved=0 |
> 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
> Amtsgericht Hildesheim, HRA 2686| Fax: +49-5121-206917-5555 |
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-05-17 6:30 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <878rrqrgaj.fsf@pengutronix.de>
2022-05-05 19:20 ` [BUG] Layerscape CAAM+kTLS+io_uring Marc Kleine-Budde
2022-05-06 0:10 ` Jakub Kicinski
2022-05-16 10:06 ` Steffen Trumtrar
2022-05-17 6:30 ` [EXT] " Gaurav Jain
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).