All of lore.kernel.org
 help / color / mirror / Atom feed
* [BUG] Layerscape CAAM+kTLS+io_uring
@ 2022-04-27  8:20 Steffen Trumtrar
  2022-05-05 19:20 ` Marc Kleine-Budde
  0 siblings, 1 reply; 5+ messages in thread
From: Steffen Trumtrar @ 2022-04-27  8:20 UTC (permalink / raw)
  To: linux-crypto; +Cc: io-uring


Hi all,

I have a Layerscape-1046a based board where I'm trying to use a
combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
v5.17). The problem I see is that on shutdown the last bytes are
missing. It looks like io_uring is not waiting for all completions
from the CAAM driver.

With ARM-ASM instead of the CAAM, the setup works fine.

I tried to debug with ftrace and see where it goes wrong. Here is what
seems to be (at least to me) some of the last bytes:

 webserver-612     [002] .....   135.300350: io_uring_file_get: ring 0000000078f4a859, fd 6 00078f4a859, fd 7
       webserver-612     [002] .....   135.300352: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fbb9b849, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300353: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ff858bdf, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300353: io_uring_link: ring 0000000078f4a859, request 00000000ff858bdf linked after 00000000fbb9b849
       webserver-612     [002] .....   135.300354: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000fbb9b849, flags 798724, normal queue, work 0000000060cd323f
       webserver-612     [002] .....   135.300358: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300375: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300379: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300388: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300389: io_uring_file_get: ring 0000000078f4a859, fd 7 0000078f4a859, req 000000008c2bf2be, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300390: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fe4e50d1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300391: io_uring_link: ring 0000000078f4a859, request 00000000fe4e50d1 linked after 000000008c2bf2be
       webserver-612     [002] .....   135.300392: io_uring_queue_async_work: ring 0000000078f4a859, request 000000008c2bf2be, flags 798724, normal queue, work 00000000f2b434fc
       webserver-612     [002] .....   135.300396: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300410: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300414: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300423: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300424: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300424: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e48f3098, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300426: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ec67d53c, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300426: io_uring_link: ring 0000000078f4a859, request 00000000ec67d53c linked after 00000000e48f3098
       webserver-612     [002] .....   135.300427: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000e48f3098, flags 798724, normal queue, work 000000009e3701da
       webserver-612     [002] .....   135.300431: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300447: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300452: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300461: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300462: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300462: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f658f96f, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0, sq_thread 0
       webserver-612     [002] .....   135.300464: io_uring_link: ring 0000000078f4a859, request 000000006c29e721 linked after 00000000f658f96f
       webserver-612     [002] .....   135.300465: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000f658f96f, flags 798724, normal queue, work 000000007434c68b
       webserver-612     [002] .....   135.300469: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300479: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300483: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300492: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300493: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300494: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000757ef148, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300495: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e5c82137, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 8f4a859, request 00000000e5c82137 linked after 00000000757ef148
       webserver-612     [002] .....   135.300496: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000757ef148, flags 798724, normal queue, work 00000000b55630dd
       webserver-612     [002] .....   135.300500: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300516: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300520: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300529: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300530: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300531: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000085e5cac4, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300532: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000008e6a863, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300532: io_uring_link: ring 0000000078f4a859, request 0000000008e6a863 linked after 0000000085e5cac4 ng 0000000078f4a859, request 0000000085e5cac4, flags 798724, normal queue, work 0000000036c4ff52
       webserver-612     [002] .....   135.300537: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300553: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300557: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300566: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300567: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300567: io_uring_submit_sqe: ring 0000000078f4a859, req 000000000f7fdd39, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300568: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000741c64e1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300569: io_uring_link: ring 0000000078f4a859, request 00000000741c64e1 linked after 000000000f7fdd39
       webserver-612     [002] .....   135.300570: io_uring_queue_async_work: ring 0000000078f4a859, request 000000000f7fdd39, flags 798724, normal queue, work 00000000fc4accf1
       webserver-612     [002] .....   135.300574: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300594: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300598: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.300607: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.300608: io_uring_file_get: ring 0000000078f4a859, fd 7
       webserver-612     [002] .....   135.300608: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000052b47765, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300610: io_uring_submit_sqe: ring 0000000078f4a859, req 000000003904ded9, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
       webserver-612     [002] .....   135.300611: io_uring_link: ring 0000000078f4a859, request 000000003904ded9 linked after 0000000052b47765
       webserver-612     [002] .....   135.300612: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000052b47765, flags 798724, normal queue, work 00000000e11c8599
       webserver-612     [002] .....   135.300615: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-647     [003] ...1.   135.300631: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 768, cflags 0
     iou-wrk-612-647     [003] d..2.   135.300634: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
       webserver-612     [002] .....   135.301668: io_uring_file_get: ring 0000000078f4a859, fd 6
       webserver-612     [002] .....   135.301669: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000012863980, op 34, data 0x0, flags 8, non block 1, sq_thread 0
       webserver-612     [002] .....   135.301670: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f4b07ff9, op 19, data 0x0, flags 0, non block 1, sq_thread 0
       webserver-612     [002] .....   135.301671: io_uring_link: ring 0000000078f4a859, request 00000000f4b07ff9 linked after 0000000012863980
       webserver-612     [002] .....   135.301672: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000012863980, flags 262152, normal queue, work 00000000102270ed
       webserver-612     [002] .....   135.301740: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
     iou-wrk-612-648     [000] ...1.   135.301757: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
     iou-wrk-612-648     [000] ...1.   135.301767: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
       webserver-612     [002] .....   135.301769: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1


Userspace said that 768 bytes where missing.

Any ideas for how to debug this or why the async work queue doesn't work with
the CAMM but does with ARM-ASM?
If I can provide more info that might help, I'll try to produce and
provide it. As there are multiple components involved, I'm not sure
where to start or what information is useful. Currently looks like CAAM
is the culprit.


Thanks,
Steffen Trumtrar

--
Pengutronix e.K.                | Dipl.-Inform. Steffen Trumtrar |
Steuerwalder Str. 21            | https://www.pengutronix.de/    |
31137 Hildesheim, Germany       | Phone: +49-5121-206917-0       |
Amtsgericht Hildesheim, HRA 2686| Fax:   +49-5121-206917-5555    |

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring
  2022-04-27  8:20 [BUG] Layerscape CAAM+kTLS+io_uring Steffen Trumtrar
@ 2022-05-05 19:20 ` Marc Kleine-Budde
  2022-05-06  0:10   ` Jakub Kicinski
  0 siblings, 1 reply; 5+ messages in thread
From: Marc Kleine-Budde @ 2022-05-05 19:20 UTC (permalink / raw)
  To: Steffen Trumtrar
  Cc: linux-crypto, io-uring, kernel, Horia Geantă,
	Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
	Boris Pismenny, John Fastabend, Daniel Borkmann, Jakub Kicinski,
	netdev

[-- Attachment #1: Type: text/plain, Size: 12690 bytes --]

Hello,

no one seems to care about this problem. :/

Maybe too many components are involved, I'm the respective maintainers
on Cc.

Cc += the CAAM maintainers
Cc += the io_uring maintainers
Cc += the kTLS maintainers

On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> Hi all,
> 
> I have a Layerscape-1046a based board where I'm trying to use a
> combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> v5.17). The problem I see is that on shutdown the last bytes are
> missing. It looks like io_uring is not waiting for all completions
> from the CAAM driver.
> 
> With ARM-ASM instead of the CAAM, the setup works fine.

What's the difference between the CAAM and ARM-ASM crypto? Without
looking into the code I think the CAAM is asynchron while ARM-ASM is
synchron. Is this worth investigating?

> I tried to debug with ftrace and see where it goes wrong. Here is what
> seems to be (at least to me) some of the last bytes:
> 
>  webserver-612     [002] .....   135.300350: io_uring_file_get: ring 0000000078f4a859, fd 6 00078f4a859, fd 7
>        webserver-612     [002] .....   135.300352: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fbb9b849, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300353: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ff858bdf, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300353: io_uring_link: ring 0000000078f4a859, request 00000000ff858bdf linked after 00000000fbb9b849
>        webserver-612     [002] .....   135.300354: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000fbb9b849, flags 798724, normal queue, work 0000000060cd323f
>        webserver-612     [002] .....   135.300358: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300375: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300379: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300388: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300389: io_uring_file_get: ring 0000000078f4a859, fd 7 0000078f4a859, req 000000008c2bf2be, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300390: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000fe4e50d1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300391: io_uring_link: ring 0000000078f4a859, request 00000000fe4e50d1 linked after 000000008c2bf2be
>        webserver-612     [002] .....   135.300392: io_uring_queue_async_work: ring 0000000078f4a859, request 000000008c2bf2be, flags 798724, normal queue, work 00000000f2b434fc
>        webserver-612     [002] .....   135.300396: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300410: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300414: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300423: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300424: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300424: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e48f3098, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300426: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000ec67d53c, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300426: io_uring_link: ring 0000000078f4a859, request 00000000ec67d53c linked after 00000000e48f3098
>        webserver-612     [002] .....   135.300427: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000e48f3098, flags 798724, normal queue, work 000000009e3701da
>        webserver-612     [002] .....   135.300431: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300447: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300452: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300461: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300462: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300462: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f658f96f, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0, sq_thread 0
>        webserver-612     [002] .....   135.300464: io_uring_link: ring 0000000078f4a859, request 000000006c29e721 linked after 00000000f658f96f
>        webserver-612     [002] .....   135.300465: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000f658f96f, flags 798724, normal queue, work 000000007434c68b
>        webserver-612     [002] .....   135.300469: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300479: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300483: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300492: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300493: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300494: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000757ef148, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300495: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000e5c82137, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0 8f4a859, request 00000000e5c82137 linked after 00000000757ef148
>        webserver-612     [002] .....   135.300496: io_uring_queue_async_work: ring 0000000078f4a859, request 00000000757ef148, flags 798724, normal queue, work 00000000b55630dd
>        webserver-612     [002] .....   135.300500: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300516: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300520: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300529: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300530: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300531: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000085e5cac4, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300532: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000008e6a863, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300532: io_uring_link: ring 0000000078f4a859, request 0000000008e6a863 linked after 0000000085e5cac4 ng 0000000078f4a859, request 0000000085e5cac4, flags 798724, normal queue, work 0000000036c4ff52
>        webserver-612     [002] .....   135.300537: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300553: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4608, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300557: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300566: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300567: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300567: io_uring_submit_sqe: ring 0000000078f4a859, req 000000000f7fdd39, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300568: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000741c64e1, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300569: io_uring_link: ring 0000000078f4a859, request 00000000741c64e1 linked after 000000000f7fdd39
>        webserver-612     [002] .....   135.300570: io_uring_queue_async_work: ring 0000000078f4a859, request 000000000f7fdd39, flags 798724, normal queue, work 00000000fc4accf1
>        webserver-612     [002] .....   135.300574: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300594: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 4352, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300598: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.300607: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.300608: io_uring_file_get: ring 0000000078f4a859, fd 7
>        webserver-612     [002] .....   135.300608: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000052b47765, op 30, data 0xf3e096f0, flags 8196, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300610: io_uring_submit_sqe: ring 0000000078f4a859, req 000000003904ded9, op 15, data 0x0, flags 2097152, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.300611: io_uring_link: ring 0000000078f4a859, request 000000003904ded9 linked after 0000000052b47765
>        webserver-612     [002] .....   135.300612: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000052b47765, flags 798724, normal queue, work 00000000e11c8599
>        webserver-612     [002] .....   135.300615: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-647     [003] ...1.   135.300631: io_uring_complete: ring 0000000078f4a859, user_data 0xf3e096f0, result 768, cflags 0
>      iou-wrk-612-647     [003] d..2.   135.300634: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result -125, cflags 0
>        webserver-612     [002] .....   135.301668: io_uring_file_get: ring 0000000078f4a859, fd 6
>        webserver-612     [002] .....   135.301669: io_uring_submit_sqe: ring 0000000078f4a859, req 0000000012863980, op 34, data 0x0, flags 8, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.301670: io_uring_submit_sqe: ring 0000000078f4a859, req 00000000f4b07ff9, op 19, data 0x0, flags 0, non block 1, sq_thread 0
>        webserver-612     [002] .....   135.301671: io_uring_link: ring 0000000078f4a859, request 00000000f4b07ff9 linked after 0000000012863980
>        webserver-612     [002] .....   135.301672: io_uring_queue_async_work: ring 0000000078f4a859, request 0000000012863980, flags 262152, normal queue, work 00000000102270ed
>        webserver-612     [002] .....   135.301740: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
>      iou-wrk-612-648     [000] ...1.   135.301757: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
>      iou-wrk-612-648     [000] ...1.   135.301767: io_uring_complete: ring 0000000078f4a859, user_data 0x0, result 0, cflags 0
>        webserver-612     [002] .....   135.301769: io_uring_cqring_wait: ring 0000000078f4a859, min_events 1
> 
> 
> Userspace said that 768 bytes where missing.
> 
> Any ideas for how to debug this or why the async work queue doesn't work with
> the CAMM but does with ARM-ASM?
> If I can provide more info that might help, I'll try to produce and
> provide it. As there are multiple components involved, I'm not sure
> where to start or what information is useful. Currently looks like CAAM
> is the culprit.

Can you provide test code or at least illustrate with code how you plug
the components together?

Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring
  2022-05-05 19:20 ` Marc Kleine-Budde
@ 2022-05-06  0:10   ` Jakub Kicinski
  2022-05-16 10:06     ` Steffen Trumtrar
  0 siblings, 1 reply; 5+ messages in thread
From: Jakub Kicinski @ 2022-05-06  0:10 UTC (permalink / raw)
  To: Marc Kleine-Budde
  Cc: Steffen Trumtrar, linux-crypto, io-uring, kernel,
	Horia Geantă,
	Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
	Boris Pismenny, John Fastabend, Daniel Borkmann, netdev

On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
> Hello,
> 
> no one seems to care about this problem. :/
> 
> Maybe too many components are involved, I'm the respective maintainers
> on Cc.
> 
> Cc += the CAAM maintainers
> Cc += the io_uring maintainers
> Cc += the kTLS maintainers
> 
> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> > Hi all,
> > 
> > I have a Layerscape-1046a based board where I'm trying to use a
> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> > v5.17). The problem I see is that on shutdown the last bytes are
> > missing. It looks like io_uring is not waiting for all completions
> > from the CAAM driver.
> > 
> > With ARM-ASM instead of the CAAM, the setup works fine.  
> 
> What's the difference between the CAAM and ARM-ASM crypto? Without
> looking into the code I think the CAAM is asynchron while ARM-ASM is
> synchron. Is this worth investigating?

Sounds like
20ffc7adf53a ("net/tls: missing received data after fast remote close")

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [BUG] Layerscape CAAM+kTLS+io_uring
  2022-05-06  0:10   ` Jakub Kicinski
@ 2022-05-16 10:06     ` Steffen Trumtrar
  2022-05-17  6:30       ` [EXT] " Gaurav Jain
  0 siblings, 1 reply; 5+ messages in thread
From: Steffen Trumtrar @ 2022-05-16 10:06 UTC (permalink / raw)
  To: Jakub Kicinski
  Cc: Marc Kleine-Budde, linux-crypto, io-uring, kernel,
	Horia Geantă,
	Pankaj Gupta, Gaurav Jain, Jens Axboe, Pavel Begunkov,
	Boris Pismenny, John Fastabend, Daniel Borkmann, netdev


Jakub Kicinski <kuba@kernel.org> writes:

> On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
>> Hello,
>>
>> no one seems to care about this problem. :/
>>
>> Maybe too many components are involved, I'm the respective maintainers
>> on Cc.
>>
>> Cc += the CAAM maintainers
>> Cc += the io_uring maintainers
>> Cc += the kTLS maintainers
>>
>> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
>> > Hi all,
>> >
>> > I have a Layerscape-1046a based board where I'm trying to use a
>> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
>> > v5.17). The problem I see is that on shutdown the last bytes are
>> > missing. It looks like io_uring is not waiting for all completions
>> > from the CAAM driver.
>> >
>> > With ARM-ASM instead of the CAAM, the setup works fine.
>>
>> What's the difference between the CAAM and ARM-ASM crypto? Without
>> looking into the code I think the CAAM is asynchron while ARM-ASM is
>> synchron. Is this worth investigating?
>
> Sounds like
> 20ffc7adf53a ("net/tls: missing received data after fast remote close")

That fixes something in tls_sw. I have a kernel that includes this
patch. So this sounds right, but can't be it, right?


Best regards,
Steffen

--
Pengutronix e.K.                | Dipl.-Inform. Steffen Trumtrar |
Steuerwalder Str. 21            | https://www.pengutronix.de/    |
31137 Hildesheim, Germany       | Phone: +49-5121-206917-0       |
Amtsgericht Hildesheim, HRA 2686| Fax:   +49-5121-206917-5555    |

^ permalink raw reply	[flat|nested] 5+ messages in thread
* RE: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring
  2022-05-16 10:06     ` Steffen Trumtrar
@ 2022-05-17  6:30       ` Gaurav Jain
  0 siblings, 0 replies; 5+ messages in thread
From: Gaurav Jain @ 2022-05-17  6:30 UTC (permalink / raw)
  To: Steffen Trumtrar, Jakub Kicinski
  Cc: Marc Kleine-Budde, linux-crypto, io-uring, kernel, Horia Geanta,
	Pankaj Gupta, Jens Axboe, Pavel Begunkov, Boris Pismenny,
	John Fastabend, Daniel Borkmann, netdev

Hello Steffen

> -----Original Message-----
> From: Steffen Trumtrar <s.trumtrar@pengutronix.de>
> Sent: Monday, May 16, 2022 3:36 PM
> To: Jakub Kicinski <kuba@kernel.org>
> Cc: Marc Kleine-Budde <mkl@pengutronix.de>; linux-crypto@vger.kernel.org;
> io-uring@vger.kernel.org; kernel@pengutronix.de; Horia Geanta
> <horia.geanta@nxp.com>; Pankaj Gupta <pankaj.gupta@nxp.com>; Gaurav Jain
> <gaurav.jain@nxp.com>; Jens Axboe <axboe@kernel.dk>; Pavel Begunkov
> <asml.silence@gmail.com>; Boris Pismenny <borisp@nvidia.com>; John
> Fastabend <john.fastabend@gmail.com>; Daniel Borkmann
> <daniel@iogearbox.net>; netdev@vger.kernel.org
> Subject: [EXT] Re: [BUG] Layerscape CAAM+kTLS+io_uring
> 
> Caution: EXT Email
> 
> Jakub Kicinski <kuba@kernel.org> writes:
> 
> > On Thu, 5 May 2022 21:20:46 +0200 Marc Kleine-Budde wrote:
> >> Hello,
> >>
> >> no one seems to care about this problem. :/
> >>
> >> Maybe too many components are involved, I'm the respective
> >> maintainers on Cc.
> >>
> >> Cc += the CAAM maintainers
> >> Cc += the io_uring maintainers
> >> Cc += the kTLS maintainers
> >>
> >> On 27.04.2022 10:20:40, Steffen Trumtrar wrote:
> >> > Hi all,
> >> >
> >> > I have a Layerscape-1046a based board where I'm trying to use a
> >> > combination of liburing (v2.0) with splice, kTLS and CAAM (kernel
> >> > v5.17). The problem I see is that on shutdown the last bytes are
> >> > missing. It looks like io_uring is not waiting for all completions
> >> > from the CAAM driver.
> >> >
> >> > With ARM-ASM instead of the CAAM, the setup works fine.
> >>
> >> What's the difference between the CAAM and ARM-ASM crypto? Without
> >> looking into the code I think the CAAM is asynchron while ARM-ASM is
> >> synchron. Is this worth investigating?
> >
> > Sounds like
> > 20ffc7adf53a ("net/tls: missing received data after fast remote
> > close")
> 
> That fixes something in tls_sw. I have a kernel that includes this patch. So this
> sounds right, but can't be it, right?
> 
To understand more on problem, can you share how you are offloading ktls to CAAM?

Gaurav
> 
> Best regards,
> Steffen
> 
> --
> Pengutronix e.K.                | Dipl.-Inform. Steffen Trumtrar |
> Steuerwalder Str. 21            |
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pe
> ngutronix.de%2F&amp;data=05%7C01%7Cgaurav.jain%40nxp.com%7Cc225fa4
> 3da7f4a43f16b08da3723dc4a%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%
> 7C0%7C637882924400875890%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL
> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C
> %7C&amp;sdata=fI1BHNZIT5RlpjEdEMXVkv1WaHcvgTlZOcaC0LRojac%3D&amp;
> reserved=0    |
> 31137 Hildesheim, Germany       | Phone: +49-5121-206917-0       |
> Amtsgericht Hildesheim, HRA 2686| Fax:   +49-5121-206917-5555    |

^ permalink raw reply	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-05-17  6:30 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-27  8:20 [BUG] Layerscape CAAM+kTLS+io_uring Steffen Trumtrar
2022-05-05 19:20 ` Marc Kleine-Budde
2022-05-06  0:10   ` Jakub Kicinski
2022-05-16 10:06     ` Steffen Trumtrar
2022-05-17  6:30       ` [EXT] " Gaurav Jain

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.